The Question
In the previous post, the ontology mostly explained everyday words, like “client” for Customer and “song” for Track. A trained model already knows those words. The more useful case is a word whose meaning only your data has. When the model meets one, does it use the meaning you give it, or the one it learned in training?
There are two kinds of these words.
| Kind | Example | What training gives the model |
|---|---|---|
| Invented | “Kestrel invoice”: Total of at least 10 | Nothing |
| Familiar, with a new meaning | “Long track”: longer than 15 minutes | A different, everyday meaning |
The Test
The setup is the same as the previous post: the Chinook sample database, qwen2.5:7b-instruct through Ollama, text-to-SQL, 3 runs per question at temperature 0.2, and grading by program. The model could reply CANNOT_ANSWER, as in Round 4 there. Claude Code designed and ran the test, and the questions, the definitions and a written expectation were fixed before the run.
There are 16 questions, 8 for each kind of word. Each question ran in two conditions: with the ontology from the previous post, and with the same ontology plus one definition line per word.
| Word | Definition given | Question |
|---|---|---|
| Kestrel invoice | an Invoice whose Total is at least 10 | How many kestrel invoices are there? |
| Gannet track | a Track that appears on no InvoiceLine | How many gannet tracks are there? |
| Long track | a Track longer than 15 minutes | How many long tracks are there? |
| Recent invoice | an Invoice whose InvoiceDate is in 2025 | How many recent invoices are there? |
Results
Runs out of 24 for each kind of word:
| Outcome | Invented | Familiar |
|---|---|---|
| No definitions: declined | 24 | 12 |
| No definitions: answered with its own meaning | 0 | 12 |
| With definitions: correct | 20 | 21 |
With an invented word, the model knew it didn’t know. It declined all 24 runs.
With a familiar word, half the time it used its own meaning. None of these were errors. Each query ran and returned an answer that looked reasonable.
| Word | Store meaning | What the model used | Answer |
|---|---|---|---|
| Long track | over 15 minutes | over 5 minutes | 1,069, not 218 |
| Recent invoice | dated in 2025 | the last month from today | 0, not 80 |
| Top customer | spent more than 47 | the 10 who spent the most | 10 names, not 2 |
| Popular genre | more than 500 tracks | most units sold | genres sorted by sales, not Rock and Latin |
The other four familiar words, big order, short album, hit track and senior employee, were declined in all runs.
Definitions fixed most of it, but not all. With definitions, 41 of 48 runs were correct. Of the 7 failures, “gannet track” was answered with the long-track rule (Milliseconds over 900000) in all three runs. “Hit track” got the meaning right and the SQL wrong, counting each track separately in all three runs. “Mulberry country” counted invoices instead of customers in one run.
The expectation written before the run said the model would answer many familiar-word questions with its own meaning, which happened. It also said familiar words would do worse than invented ones once the definitions were there, which didn’t happen (21 against 20).
What the Results Show
A word the model had never seen was safe. It declined every time.
A word it already knew was not. Half the time it brought in its own meaning, and nothing in the answer showed that anything was wrong. Being allowed to decline didn’t help, because the model didn’t notice that it didn’t know.
Definitions worked, and they could also mix. With 16 new definitions in one list, one word picked up another word’s rule.
What I Take From It
In the previous post I thought an ontology would help most with meanings that aren’t common, because a pre-trained model may not have them. This test agrees, but the trouble wasn’t the word the model had never seen. It was the familiar word with a store meaning, where the model’s own meaning from training came into the answer. That is the concern I had about training knowledge leaking into answers, showing up as a meaning instead of a fact.
So the words worth defining first are the ones that sound ordinary.
What This Can’t Say
- The scale is small. This is 16 questions, one model and 3 runs each, and runs of the same question are not independent.
- Claude Code chose the words. The thresholds and wording may be easier or harder than real business terms.
- The definitions were short and clear. Real definitions are often longer, and some disagree with each other.
- The answers are SQL. A model writing free-form text might behave differently.
- Declining was offered in the prompt. Without that line, the invented words might have been guessed too.
The next post asks which part of the ontology did the work, and whether the model can write that part itself: Which Parts of an Ontology Matter, and Whether an LLM Can Write Them
Further Reading:
- Chinook database
- qwen2.5, Ollama