Which Parts of an Ontology Matter, and Whether an LLM Can Write Them

The Question

The ontology in the first post raised correct answers from 60% to 85% when the model was allowed to decline. Claude Code wrote it ahead of time, and it also wrote the questions. Two questions follow from that. Which part of the ontology did the work? And could the model have written that part itself?

The ontology has four sections:

The Test

The setup is the same as the first post: the Chinook sample database, qwen2.5:7b-instruct through Ollama, text-to-SQL, 3 runs per question at temperature 0.2, and grading by program. The model could reply CANNOT_ANSWER. Claude Code designed and ran the test, and the versions and a written expectation were fixed before the run.

The questions are the 48 from Round 4 of the first post: 32 the database can answer and 16 it can’t. The test has two parts.

Part 1: Remove One Section at a Time

Wrongly declined is out of 96 runs on questions that can be answered.

Relations did the most for answering. Without them, correct answers fell from 85% to 71%, wrong declines doubled from 8 to 16, and multi-hop questions fell from 7.0 to 5.0 out of 8. Seven questions got worse and none got better. “How many tracks are on albums by Iron Maiden?” went from 3 of 3 runs to 0. The schema already lists its 11 foreign keys, and these lines still mattered.

Classes and terms helped the model notice what isn’t there. Without the class descriptions, 6 runs on unanswerable questions weren’t declined. All of them were SQL errors from made-up columns like ReleaseDate. Without the terms, 5 weren’t declined, and 3 of those were made-up answers: asked which record label released ‘Let There Be Rock’, the model returned the artist, AC/DC.

Removing the terms helped some answers. It was better on 6 questions and worse on 1, so the total went up to 94%. The one it lost was “What is the revenue from German clients?”, which went from 3 of 3 runs to 0. “Revenue” is defined only in the terms.

The three rules made little difference. Without them, 3 questions were better and 1 was worse.

The expectation written before the run said removing the terms would cost the most. For answering, the opposite happened. It said removing the relations would cost the most on multi-hop questions, which happened.

Part 2: Let the Model Write Its Own

A 2023 study, LLMs4OL, tested models on “term typing, taxonomy discovery, and extraction of non-taxonomic relations” and found that “foundational LLMs are not sufficiently suitable for ontology construction that entails a high degree of reasoning skills and domain expertise”. The task here is much narrower: describing a database that already exists.

The same small model was asked for a short ontology with the same four sections. It wrote two.

Each was generated once, and the first reply was used as it was. Both were fixed before the questions ran. Part of the version with sample rows:

### Relations
- **Album Artist**: Album.ArtistId = Artist.ArtistId
- **Customer Invoices**: Customer.CustomerId = Invoice.CustomerId

### Terms
- **Total Sales**: `SELECT SUM(Total) FROM Invoice`

Both had mistakes. The one from the schema has no line for a track’s album or genre, and its query for the top-selling artist joins albums to invoice lines on the wrong key. The one with sample rows has no line for a customer’s support rep or for which track an invoice line sells, and it describes the manager link backwards: “ReportsTo: The manager who reports to this employee.”

The model’s own ontologies did about as well as the one written ahead of time. That held even with their mistakes. Compared with it, the one from the schema was better on 3 questions and worse on 3, and the one with sample rows was better on 4 and worse on 2.

Sample rows helped a little. Correct answers went from 84% to 91%, and multi-hop questions from 7.0 to 8.0 out of 8.

Neither one knew the store’s own words. “What is the revenue from German clients?” was declined in every run with both of the model’s ontologies. They defined “Total Sales”, but not “revenue” or “client”.

The expectation written before the run said both would land between the schema alone and the ontology written ahead of time. The one from the schema landed about even with it, and the one with sample rows landed above it.

What the Results Show

The joins did most of the work. Removing them cost more than removing anything else.

The model could write that part. Its own ontologies had most of the joins, missed a few, and still did about as well as the one written ahead of time.

The meanings were the part it couldn’t write. The one question that needed the word “revenue” failed whenever that definition was missing, whether it was removed or never written. The previous post found the same with familiar words like “long track”.

Lines about meaning also helped the model notice what isn’t there. Without them, it was more likely to invent a column or an answer.

More text was not always better. The terms helped with the one word they defined and got in the way of a few other questions.

What I Take From It

If I write an ontology for an LLM, I’d start with how the tables connect, and the model can draft that part from the database itself. The meanings only we know have to come from people. I’d add them where the model would otherwise guess, and check that each one doesn’t make other answers worse.

What This Can’t Say

• The differences are small. A few questions decide each comparison, and runs of the same question are not independent.
• The sections differ in size. Removing 10 lines changes the prompt more than removing 3.
• The ontology written ahead of time came from Claude Code. It was not written by someone who knows the business, and another ontology could split the work differently.
• Each of the model’s ontologies was generated once. Another run could write a better or worse one, and the same model that wrote it also used it.
• Chinook is well known. The model may have seen it in training, which would make writing about it easier.
• This only covers a first version. Keeping an ontology up to date is a different question.
• The setup is narrow. This is one model, text-to-SQL only, with declining offered in the prompt.

---

Further Reading:

• LLMs4OL: Large Language Models for Ontology Learning, Hamed Babaei Giglou, Jennifer D’Souza and Sören Auer, 2023
• Chinook database
• qwen2.5, Ollama