The grader grew teeth: grading the intelligence engine on a real key

The DCS intelligence engine has one job that matters more than any other: read what a user tells it and extract durable facts to remember. If that extractor is wrong, everything built on top of it is confidently wrong. So this week we did the unglamorous thing and built a harness to grade it — and then let the harness say uncomfortable things.

It said several. Every number below is ours, author-reported and internally re-run, not a third-party benchmark — but the point of the week isn't the scores. It's that the measurements were built to catch us, and they did.

It had been learning nothing

The first finding is the one that stings. Since a recent provider switch, the engine's live learning path had made 148 model calls, thrown zero errors, and stored zero facts. Nothing was broken in the way anything alarms on — the job succeeded, health stayed green, memory simply never grew. The cause was mundane: the extraction call pinned a JSON response mode that cannot return a top-level array, while the prompt asked for exactly that. The model complied every time, and every answer was discarded.

It was found by grading the thing against a real key. That is the entire argument for an eval harness in one sentence: a job that succeeds and produces nothing looks identical to a healthy one until something actually measures the output.

The held-out corpus caught the memorization

The extractor's prompt had been tuned four times against the same two evaluation sets, and it scored well on them — around the mid-80s to mid-90s. So we built a third corpus, written by someone forbidden to read the extractor's code, and held it out. On that set the score collapsed and a handful of "must-not" safety cases fired. The roughly thirty-point gap between the tuned sets and the held-out one is the memorization — the prompt had learned the tests, not the task. A benchmark you've optimized against four times is no longer measuring you.

The grader that turned on its own work

Two more findings were about the harness catching itself, which is exactly what you want a harness to do:

• It was about to write into production. The grader's store selection followed a connection string that pointed at the real database. A full grading run would have written its invented test facts into real user memory. Caught before it ran.
• Almost half a run silently didn't execute. On one pass, a large fraction of cases came back rate-limited by the provider and were scored as "extracted nothing" — dragging recall down and printing a confident verdict over a run that never happened. A benchmark that grades its own failures as your failures is worse than no benchmark.

The most valuable output of the week wasn't a feature. It was a set of measurements that refused to flatter the codebase — including the measurements themselves.

The through-line

Behind all of it sits one structural hole worth naming: the test suite runs with authentication turned off, so the entire authenticated request path was never exercised. That single gap is what let the extraction bug, and the auth bugs in this week's companion post, live in a green codebase. Tests that skip the hard path are tests that agree with you.

— Deepak