Testing Whether an Ontology Helps an LLM Write Its Answers

The Question

An ontology writes down what things in a domain are and how they relate. One long-standing use is reasoning. SNOMED CT, for example, is written so that “a logic reasoner (also called a classifier)” can “compute the appropriate hierarchical relationships.” No language model is involved in that.

I was curious about a different use. A trained model is already finished. Is it worth handing it an ontology I wrote when it writes an answer, especially a frontier model? I also wanted to know two related things: whether it would then answer only from facts, and whether it would say so when it can’t answer.

Before running anything, I assumed the ontology would always make the answers better, because the model gets more meaning at the moment it writes the answer.

This test only uses a small local model, so it doesn’t settle the frontier question. The closest published result is Sequeda et al. from 2023. With GPT-4 on an insurance database, accuracy was 16.7% over SQL and 54.2% over “a Knowledge Graph representation of the SQL database”. That is 43 questions on a 199-table schema, and all three authors work at data.world.

Where an Ontology Can Enter

The word “inference” gets used for two of these. For an LLM it means producing output. For an ontology it means a reasoner deriving new facts. This post is only about the second row: the weights stay the same, and the ontology is text the model reads for that one request.

The Test

Who did what. I asked the question and said yes to the follow-up runs. Claude Code designed the test, wrote the questions, the ontology and the code, ran everything, and reported the results. Claude itself was not one of the models tested.

In every condition except documents, the model’s job is text-to-SQL. The prompt gives it the schema, meaning the table definitions, and the model writes one SQL query. The database runs that query, so the answer comes from the data. In the documents condition, the model writes the answer itself from five retrieved passages.

The questions and the ontology were fixed before each round. Rounds 3 and 4 ran on a newer version of Ollama than Rounds 1 and 2. Running the Round 1 schema and ontology conditions again on the newer version gave 82 of 96 (same as before) and 90 of 96 (89 before).

The ontology is about 40 lines of plain text: classes, relations with their join keys, term definitions and three rules. It isn’t OWL, and no reasoner ran. It is closer to careful notes about a schema than to a formal ontology. Some of it looks like this:

Classes:
- Customer: a client, buyer or shopper.

Relations:
- Album releasedBy Artist: Album.ArtistId = Artist.ArtistId

Terms:
- Revenue, sales or amount spent: SUM(Invoice.Total), or SUM(InvoiceLine.UnitPrice * InvoiceLine.Quantity).

Rules:
- Every Album has exactly one Artist.

Round 1: A Readable Database

Correct answers out of 8 per type, averaged over the three seeds:

The ontology was better on 6 questions, worse on 1, and the same on 25. Most of the gain was on multi-hop questions, and aggregate questions went down a little.

The bigger jump was from documents to tables. Five passages can hold the answer to “which artist released this album?” They usually don’t hold “how many tracks are there in total?”

Those tables also have names like Track and Customer, which already carry meaning. So the next round took the meaning out of the names.

Round 2: Hide the Names

Every table and column was renamed to something like t_04 and c_08, with the data unchanged. That is a deliberately extreme version of a schema whose names don’t explain themselves.

The 3% is one question, answered correctly in all three runs. With the ontology, all 77 wrong answers were SQL errors. A typical one picked the right table and then used the ontology’s attribute names:

SELECT Email FROM t_04 WHERE FirstName = 'Helena' AND LastName = 'Holý'

no such column: Email

Round 3: Don’t Make It Write the Names

The model had the right table and the wrong column names, so in this round it didn’t have to write the hidden names at all. It saw only the ontology and the class and attribute names, and wrote SQL in those names. Views on the database turned them into the real ones:

CREATE VIEW Customer AS
SELECT c_08 AS CustomerId, c_03 AS FirstName, c_06 AS LastName, c_09 AS City
FROM t_04;

That is 87 of 96, close to the 85% of the readable schema in Round 1. The first version of this round used a regex translator to swap the names. It had bugs, so this version uses views and was run again from the start.

Round 4: Can It Say “I Can’t Answer”?

In the rounds above, the model had to answer. Every question had an answer, the prompts said “Give your best answer” or “Reply with only the SQL”, and no reply declined.

This round added 16 questions the database can’t answer, next to the 32 it can. Eight ask for something that is nowhere in the data, like “What year was the album ‘Let There Be Rock’ released?” Eight ask for something that exists only on another table, like “What is the birth date of the customer Helena Holý?”, where only employees have a birth date. The SQL prompts gained one line: “If this database does not hold the information needed, reply with only CANNOT_ANSWER.” The document prompt got the same option.

It can decline. With the option in the prompt, the model declined almost every question it couldn’t answer.

Declining has a cost. It also declined questions it could answer, so correct answers fell in every condition. Compared with Round 1, documents went from 42% to 34%, the schema from 85% to 60%, and the schema with the ontology from 93% to 85%.

This is where the ontology made the largest difference. The gap between schema and ontology was 8 points in Round 1 and 25 points here. With only the schema, the model declined “What is the revenue from German clients?” in all three runs. With the ontology it answered correctly in all three. The words “revenue” and “client” are not in the schema, and the ontology defines them. Out of 8, term questions went from 3.0 to 6.0 and multi-hop questions from 3.0 to 7.0.

Nothing from training showed up. No reply answered an unanswerable question from outside knowledge, like the year of that album. The one made-up answer was in the document condition: “Who is the support rep of the artist Aerosmith?” got “Jane Peacock” in all three runs. That name is a real employee in the data, and artists don’t have support reps.

The ontology did not make every answer right. It failed on 14 of the 96 answerable runs: 8 wrongly declined, 3 ran and returned the wrong name, and 3 returned nothing.

What the Rounds Show

Tables and readable names made the biggest difference. Going from documents to a schema raised the score from 42% to 85%. Hiding the names dropped it to 3%. Adding the ontology to a readable schema raised it from 85% to 93%.

Knowing the meaning was not the same as writing the query. With hidden names, the ontology raised the score only to 20%, and every failure was a SQL error. When views did the renaming, it reached 91%.

The ontology mattered most when the model could decline. Over the schema alone, it added 8 points when the model had to answer, and 25 points when it could decline. Wrong declines fell from 21 runs to 8, and wrong or failed queries from 17 to 6.

What I Take From It

I assumed an ontology would always make the answers better, and it didn’t. On a readable schema, most questions came out the same.

Looking across the four rounds, the ontology did more to help the model judge what the data has and doesn’t have than to make it get more answers right.

Giving one to a trained model can still help. Meanings and information that aren’t common may not be in a pre-trained model at all, so handing them over seems useful. This test didn’t measure that directly, because Chinook is well known and its terms are everyday words.

Meaning alone may not be enough, though. An LLM can still put something it learned in training into an answer. That didn’t happen here, but the questions were easy and the prompt told it to decline, so I don’t treat that concern as settled.

So an ontology can help make the answers more trustworthy, but it doesn’t guarantee them. The ontology is text the model reads, and nothing forces the model to follow it.

The questions I want to look at next are what it takes to keep training knowledge out of an answer, and how an ontology should be stored for that.

What This Can’t Say

• The scale is small. This is one model, one dataset and 32 questions, and runs of the same question are not independent. A difference of 6 questions against 1 is weak evidence.
• No frontier model was tested. A larger model might copy the mapping correctly or decline differently, and this test can’t say.
• The data is clean and well known. Chinook is a sample database, and the model may have seen it in training. The Sequeda benchmark has 199 tables.
• Claude Code wrote the questions and the ontology. They were fixed before the runs, but the ontology can still favor the questions.
• The ontology is plain text. A formal ontology, a reasoner, or a graph lookup could behave differently.
• The answers are short. Most of them are SQL. That is what makes grading by program possible, and it says nothing about long free-form answers.
• The unanswerable questions are easy. There are only 16, the missing information is clearly missing, and the prompt says what to do. The 100% doesn’t mean the problem is solved.

The next post asks what happens with words only the store knows: How an LLM Handles Words Only Your Data Knows

---

Further Reading:

• A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model’s Accuracy for Question Answering on Enterprise SQL Databases, Juan Sequeda, Dean Allemang and Bryon Jacob, 2023
• SNOMED CT logic model, SNOMED International
• Chinook database
• qwen2.5, Ollama