Evals before scale: the cheapest insurance in AI engineering

The highest-leverage artifact in AI engineering is a file of thirty real queries with expected answers. Not a bigger model, not a clever prompt — a small eval set. LLM evals have a reputation as a research discipline with leaderboards and statistical machinery, and that reputation keeps teams from building the twenty-line version that would actually protect them. You don't need the machinery. You need enough signal to answer one question: did this change make my system worse?

Because here's the position you're otherwise in. Every cost decision you want to make — a cheaper model, a cheaper provider, a shorter prompt, a free rung before a paid one — is a change that might degrade quality. If you can't measure quality, every one of those changes is a leap of faith, so you don't make them, so you overpay indefinitely. Evals aren't a quality tool that happens to touch cost. They're the thing that makes cost optimization possible at all.

Vibes don't survive a model swap

The default evaluation method is vibes: run five prompts, squint at the output, ship it. Vibes have two failure modes. They're anchored to the prompts you happen to remember — the easy, representative ones, never the ugly tail. And they're not comparable across time. You can't diff a squint from March against a squint from June.

This bites hardest exactly when the stakes are highest: swapping models or providers. The pull is always toward cheaper — smaller models first is the right default — but "the small model seemed fine" is not a load-bearing sentence. I've watched teams stay pinned to an expensive model for months, not because it measured better, but because nobody could prove the cheap one wasn't worse. The absence of an eval set has a price, and it compounds monthly.

20–50 real queries beat a thousand synthetic ones

The counterintuitive part: the eval set should be small and it should be ugly.

Small, because you'll actually run it. A set of 30 queries runs in a couple of minutes and costs pennies, which means it runs on every change instead of quarterly. A 5,000-case benchmark that requires a scheduled job gets consulted twice a year and trusted never.

Ugly, because the value is in the tail. Pull the queries from real traffic — your logs, your support threads, your own usage — and deliberately keep the awkward ones: the ambiguous phrasing, the query that returns nothing, the input in the wrong language, the one that broke production in April. Synthetic test cases are written by the same brain that wrote the prompt, so they share its blind spots. Real traffic is generated by people who never read your prompt and don't care about your assumptions. Twenty of those are worth a thousand cases invented to pass.

For each query, write down what a good answer looks like, at the crudest fidelity that catches real failures: a string the output must contain, a fact it must state, a source it must cite, a JSON schema it must satisfy. Exact checks where you can. A model-as-judge only where you must — judges add cost and their own noise, so they're a last resort, not a default.

Gate every swap on a regression run

Once the set exists, the process is almost insultingly simple: run it on the current setup, record the score; run it on the proposed setup, diff. If the new score holds within a threshold you chose in advance, the swap proceeds. If it drops, the swap stops, and you now know something specific — which queries broke — instead of something vague.

The threshold matters more than it looks. Decide before you run what regression you'll accept for what savings — maybe two points of pass rate is fine for a 10× cost cut, maybe it isn't for your product. Deciding after you've seen the numbers is how every regression becomes retroactively acceptable.

Do the same arithmetic on the insurance itself: a 40-query run against a candidate model costs cents and minutes. An unmeasured regression costs you in the currency you care most about — user trust — and you pay it slowly, invisibly, before anyone connects the complaints to the swap. I gate everything behind this: model changes, provider changes, prompt rewrites, temperature nudges. Anything that touches output goes through the gate. It's the cheapest gate you will ever operate, and it belongs on your launch checklist before the feature ships, not after the first incident.

You can't route what you can't score

Here's where evals stop being defensive and start paying rent. Routing — sending each request to the cheapest option that's good enough — depends entirely on knowing what "good enough" means. That knowledge is an eval score. No score, no ladder; just guesses about which rung deserves the traffic.

Run your set against every candidate rung and you get a capability map: the small model passes 90% overall but fails the multi-step cases; the free search rung nails documentation queries and whiffs on news. Now the routing rule writes itself — route the shapes each rung passes, escalate the shapes it fails. The same map answers which model a new agent should start on, and it's the difference between a price ladder built on evidence and one built on brand reputation. Teams skip routing "because quality," but quality-as-a-blocker is usually quality-as-unmeasured. Score it and the blocker becomes a config file.

A starter recipe

The whole thing, this week, in an afternoon:

1. Pull 30 queries from the last month of real traffic. Skim for variety; keep the weird ones on purpose.
2. For each, write the expected outcome — a must-contain string, a must-be-true fact, or pass/fail criteria a colleague could apply without you in the room. A judge prompt only for the genuinely fuzzy few.
3. Write a runner: loop queries through your system, append input, output, and pass/fail to a JSONL file. Print the pass rate. Fifty lines of code, give or take.
4. Run it now to get a baseline. Then run it on every change to model, provider, or prompt, and diff before shipping.
5. Feed it. Every time production surprises you, that query joins the set. The set becomes the fossil record of everything that ever went wrong — which is exactly what you want standing between you and the next change.

Start before you scale, because afterward you're doing archaeology on incidents instead of curation on logs. Thirty queries, one afternoon, and every future swap becomes a measurement instead of a bet. There's more in this vein under building with AI, but nothing there matters more than this file.