How do you know your agent works?

Published · 32 min read

Selective manual focus

Selective manual focus

The most sought-after skill in the field is proving that your LLM works, and shipping it to production is the easy part.

Every conversation about agents in production that I follow, on teams, in communities, in LinkedIn threads, ALWAYS ends up in the same place in the same way. Someone describes the system: RAG, tool calling, fallback, retry, a pretty diagram. Then someone asks how they know it works, and the answer is some variation of “we tested it a lot and it seemed fine”, sometimes backed by a public benchmark that has nothing to do with the product.

This article was born out of being annoyed by that scene. The market spent years teaching everyone how to ship systems with LLMs. Frameworks, templates, weekend courses. Shipping is easy. Measuring is still hard, and measuring is what decides the fights that matter: whether to switch models, whether to automate the next step, whether to defend the project when its cost shows up in someone’s spreadsheet.

I wanted a text I could send to someone who just shipped their first agent: why public benchmarks don’t solve it, which stage the team is at, and the whole process of building the first eval harness, with the numbers and the decisions at each step. I couldn’t find one in Portuguese, so I decided to write it.

Everyone learned to ship, almost nobody learned to measure

There is a number that opens this discussion well.

The AI Evals for Engineers & PMs course, by Hamel Husain and Shreya Shankar, costs US$ 4,200 and has already been taken by more than 4,500 engineers and PMs, from teams at OpenAI, Google, Meta, Amazon and Microsoft. Lenny Rachitsky, who is probably the person who interviews the most product builders in the world today, describes the two of them as teaching “the most popular course in the world on AI evals” and calls it the hottest skill for anyone building AI products. O’Reilly has their book, Evals for AI Engineers, slated for October 2026.

Four thousand five hundred people paying four thousand dollars to learn how to measure. Feed hype does not move budgets on teams that size.

On the other side, “AI Engineer” was the fastest-growing job title in the United States on LinkedIn’s Jobs on the Rise list in 2025 and again in 2026, according to Axios and Forbes. The line of people learning to ship agents is enormous. The line of people learning to measure agents is much shorter.

The economic argument, which is what convinces whoever is spending the money

The part that matters to the business fits in two sentences.

The cost of being wrong grows with the degree of automation. While there is a human reviewing every output, a model error is an annoyance. When you take the human out of the loop, the model error becomes the company’s output.

And the market is already paying that bill in public.

In February 2024, the British Columbia Civil Resolution Tribunal ruled against Air Canada because the chatbot on its website made up a bereavement fare policy. The company argued it could not be held responsible for what the chatbot said. The tribunal replied, in paragraph 27, that Air Canada was suggesting “the chatbot is a separate legal entity that is responsible for its own actions”, and called that “a remarkable submission”. The total came to CAD 812.02. The amount is irrelevant. The precedent is not.

New York City kept a chatbot online for more than two years that, as The Markup documented, told business owners they could refuse tenants with housing vouchers and keep part of their employees’ tips. Both are illegal there. It was only ordered shut down in February 2026, after costing an estimated half a million dollars a year.

And the easiest case to measure: researcher Damien Charlotin maintains a database of court decisions in which the court explicitly recognized that someone used hallucinated content. As of August 16, 2026, there are 1,922 cases, forty-one of them in Brazil.

None of these teams was incompetent. All of them shipped something that passed someone’s eyeball test.

Gartner projects that more than 40% of agentic AI projects will be canceled by the end of 2027, due to cost, unclear business value or inadequate risk controls. It is a forecast, and it should be read as one. But the three reasons they list share the same root: nobody could show, with a number, that the thing works well enough to justify the next step.

Evals are the confidence gate. Without evals, you have no argument for automating further, and without automating further the project never pays for itself.

Nobody deploys without tests in traditional software. In AI the test exists too, except the answer key is probabilistic and you have to build the answer key first. It is extra work, and that is why almost nobody does it.

“But Claude Code doesn’t use evals”

This objection deserves its own answer, because it took over an entire week of timeline in September 2025.

The trigger was real. Boris Cherny, from Claude Code, said on a podcast that central product decisions were made “mostly on vibes”, with dogfooding in place of benchmarks. swyx packaged it into a list of famous products “without evals”, one agent startup CEO wrote that “evals are a scam”, another was less polite, and for a week it looked like the field had died before it was born.

Two details take the objection apart.

The first is context. In Claude Code, the developer is the domain expert and uses the product all day long. Dogfooding in that configuration is continuous error analysis with the best possible annotator. That was Shreya Shankar’s answer on Lenny’s episode: whoever says they don’t do evals is almost always doing them, just without calling them that.

The second is the epilogue. In July 2026, the Claude Code team publicly stated it has “thousands of evals”, and Cherny himself described the cycle: you use the product, you see where it stumbles, and that is where the eval set comes from, which then saturates, gets thrown away and rebuilt. That is, almost word for word, the process I describe two sections from now.

What survived the debate was the mature version of the critique, which can be summed up as “a badly done eval is theater”. An unvalidated judge, a golden set that doesn’t come from production, a corporate score used as a marketing piece. The rest of this article is about how to avoid doing that.

The three stages

I have been using this typology in conversations for a while and it has held up well. Every team I know is at one of the three. Including mine, at different moments.

Stage 1, vibe check

You open the playground, run five, ten, who knows how many examples that came to mind, read the outputs, think it looks good, and ship.

Described as a measurement, that is n=5, convenience sampled, evaluated by the same person who has a stake in the result. Three biases in one move.

There is nothing wrong with starting here. Everyone starts here. The problem is staying.

The symptoms are easy to recognize, and every one of them is behavioral, with nothing technical about it:

Technical knowledge is almost never what keeps a team from moving up a stage. The blocker is the belief that evals are something you do later, once the product matures. Anthropic put it very well: “evals get harder to build the longer you wait. Early on, product requirements naturally become test cases. Wait too long and you’ll be reverse-engineering success criteria from a live system.”

Stage 2, generic benchmark

Here the team understood it needs a number. So it went looking for a number where there was one ready: MMLU, accuracy on a public dataset, leaderboard position, model card score.

This stage looks mature and is, in practice, more dangerous than stage 1. Because now you have a number, and numbers give confidence. Except the number is about something else. So, in three steps, here we go.

First, the benchmark may be inside the training data. Scale AI built GSM1k, a thousand new problems mirroring the GSM8K distribution, and measured the drop. It was up to 13 percentage points, concentrated in model families optimized for math. Math-Shepherd-Mistral-7B-RL scored 74.5% on GSM8K and 61.1% on GSM1k. And don’t rely on n-gram decontamination either. A 2023 paper showed that simple paraphrase and translation get around the standard methods, and that 8% to 18% of HumanEval shows up overlapped in public pre-training sets.

Second, the benchmark may simply be wrong. The paper “Are We Done with MMLU?” re-annotated 5,700 MMLU questions across the 57 subjects and found 6.49% of questions with errors in the whole dataset. In the Virology subset, 57% of the questions analyzed had a problem, between wrong answer keys and ambiguous questions. Fixing that changed model rankings. In Virology, Llama 3.1 405B was ranked sixteenth considering all instances, and became first considering only the correct ones.

Third, the leaderboard has people gaming it. “The Leaderboard Illusion” documented that raising the share of arena data in training from 0% to 70% more than doubles the win rate on ArenaHard, from 23.5% to 49.9%. And that Meta tested 27 private variants in the run-up to Llama 4, with the option to choose which score to publish. Before that, LMSYS itself showed that a good part of human preference in the arena is explained by length and markdown formatting. Controlling for style, one model dropped from 6th to 18th place. The community even has a verb for the practice, benchmaxxing, optimizing for what the leaderboard measures instead of the capability it was supposed to measure.

If you want a single number to keep from this section, use this one. Epoch AI ran GPQA Diamond sixteen times independently on a model whose reported score was 65%, and got a mean of 55%, with a 3 point standard deviation. Ten points of difference, with no identifiable cause, because whoever publishes a score rarely reports prompt and temperature. A benchmark that is not reproducible across evaluators does not work as a product acceptance criterion.

The most structural critique came in 2025, when a team at the Oxford Internet Institute reviewed 445 LLM benchmarks published at the main conferences. Only 16% used any statistical test to compare models and fewer than 10% used complete real-world tasks. The paper’s title is Measuring what Matters, and the argument is exactly that.

Put the three together and only one conclusion remains: generic benchmarks are poor predictors of performance on your task, and none of them was built to predict it.

Clinical symptom of stage 2: the team debates which model to use by reading model cards instead of running both on its own use case.

Stage 3, product eval

Here the answer key is yours, and that is sort of the whole problem, isn’t it?

The team looked at real traffic, catalogued the failure modes that actually happen in the product, built a set of cases with human verdicts, built an automated judge calibrated against those verdicts, and tied it all into a gate that runs on every change.

What changes day to day is a single thing, and it is enormous: the team gets its speed back. You can swap the prompt, swap the model, change the chunking, touch the retriever, because there is a number that tells you if things got worse. The fear of touching things goes away.

Tooling is also not what keeps teams from getting here, and there is no technology gap either. The next section is about the actual blocker: the boring manual work that nobody wants to do, and that almost everyone skips.

The cookbook

Before the steps, the two observations that explain their shape. The recipe only makes sense if you accept both premises.

First: systems with LLMs fail silently. Traditional code breaks with a stack trace. An LLM does not break. It answers, fluently, well formatted, with exactly the same confidence when it is right and when it is wrong. No exception will sound an alarm for you, and that is why the recipe starts with reading data instead of with infrastructure.

Second: quality is not a property of the model. It comes out of the meeting between the model, your prompt, your data and your traffic. It changes when any of the four changes. That is why it cannot be outsourced to a benchmark, and that is why every step below works with your own material.

Accept both, and the rest is just getting started. Seven steps. No platform needed, no purchase approval, no team. You need access to the traces and the willingness to read bad text for two days. The first six steps fit in one week of one person’s time. The seventh runs forever, and it is what keeps the other six from rotting.

And there is no need to play hero. Run the standard recipe once, as is, before adapting it to your case. Premature adaptation is like premature optimization, only harder to reverse, because you will never know what the result of the process you didn’t run would have been.

1. Collect real traces

Gather somewhere between 50 and 100 production traces to start. The whole trace: user input, what the system retrieved, which tools it called, what it returned. If you have no traffic yet, generate synthetic data, and I come back to that in step 4.

2. Read everything, one by one

This part really is unpleasant, sorry. Read all of them, one by one, and write in free text what is wrong with each one. No predefined categories, no rubric, no score. Just a text field next to each trace, where you write “answered correctly but ignored that the user had already said they were vegetarian” or “called the tool twice with the same argument”.

Every time I do this I find something I wasn’t looking for. A tool called twice in a loop, context truncated mid-sentence, a router sending to the wrong agent, a user repeating the same question with growing irritation over three turns. None of that shows up on a dashboard, because a dashboard shows what you already knew you should look at.

This has a name. It comes from qualitative research and is called open coding. Hamel Husain and Shreya Shankar adapted it to LLM traces and it is the heart of their method. Three rules from their FAQ:

  1. Annotate only the first failure in each trace at the start, because an error upstream causes errors downstream and you end up counting the same thing several times.
  2. Whoever annotates has to understand the domain. They call this person the benevolent dictator: a psychologist for the mental health chatbot, a lawyer for legal document analysis, the head of customer service for support automation. One person. Five is a committee. And on delegating, they are blunt: “read the raw traces yourself at the start. That is how you discover new failure types. Never skip this or delegate it.”
  3. Stop when new traces stop revealing new categories, which also has a name borrowed from qualitative research, theoretical saturation. Their heuristic: if some 20 traces in a row bring no new category, you can stop, but review at least 100 in the first round.

If this sounds like a lot of work for an uncertain result, the number they give is: “we spend 60 to 80% of development time on error analysis and evaluation”. This is the work. Calling it preparation is the mistake.

3. Group and count

Now take all the free-text notes and group them into categories. This is axial coding, and it is the step Hamel says is the most important of the whole process.

Here an LLM genuinely helps. Throw it your hundred notes and ask it to propose a grouping. You correct it, merge categories that are the same thing under different names, split the ones that turned into a catch-all. The criterion is being able to count without hesitating.

Then count the frequency per category.

That count is the day’s deliverable, and it can be uncomfortable, because the distribution is almost never the one the team imagined. The public example Hamel uses is a product where three problems accounted for more than 60% of failures, and the worst of them was relative date interpretation. When the user said “two weeks from now”, the system got it wrong 66% of the time. After it was addressed, the success rate went from 33% to 95%. Nobody had guessed that was the main problem.

Prioritize by pain times frequency. The top of the list is usually obvious once the count is in front of you.

4. Golden dataset v0

Now you build the set of cases with human verdicts.

Size: start small. Anthropic’s recommendation is very specific and kills the most common excuse: “teams delay building evals because they think they need hundreds of tasks. In practice, 20 to 50 simple tasks drawn from real failures is already a great start.”

Rules I would follow at the beginning:

  1. Real cases beat synthetic cases. You just spent two days reading production. The best cases are already annotated.
  2. Include the hard ones and the ambiguous ones. A golden set with only easy cases gives you a 98% pass rate and zero information. Hamel has a good line on this: “if you’re passing 100% of your evals, you’re probably not challenging the system enough. A 70% pass rate might indicate a more meaningful evaluation.”
  3. If you have no traffic, generate synthetic data with structure. Asking the model to “give me test queries” produces generic, repetitive garbage. What works is defining dimensions and writing tuples. In a recipe app, the dimensions would be dietary restriction such as vegan, gluten-free, none, cuisine type such as Italian, Asian, comfort food and complexity simple, multi-step. Write 20 tuples by hand first, because that forces you to understand the problem space. Then scale, and convert each tuple into natural language in a separate prompt, otherwise every query comes out looking the same.
  4. Expect the criteria to change while you label. Shreya Shankar gave this a name in a 2024 paper, criteria drift. You need criteria to judge outputs, but it is by judging outputs that you discover the criteria. Don’t treat the v0 rubric as a contract, treat it as a draft.
  5. If more than one person labels, measure agreement before demanding it from the machine. And calibrate your expectations: in a December 2025 hate speech detection study, trained human annotators landed at a Cohen’s kappa between 0.47 and 0.48 on two of the three datasets. That is “moderate”. If your humans agree at 0.47, demanding 0.8 from your automated judge is incoherent.

A caveat on kappa, since we’re here. With imbalanced classes, which is exactly the case in evals, because most outputs pass, it can come out low even with very high observed agreement. If your distribution is very skewed, it is worth looking at Gwet’s AC1, and report the confidence interval along with the point estimate.

5. Binary judge, calibrated before use

Now we automate. Two decisions determine whether this will work.

Decision 1: the judge rules binary, one criterion at a time.

Don’t ask for a score from 1 to 10. Hamel’s justification is the best I know: “the difference between adjacent points, like 3 and 4, is subjective and inconsistent across annotators, detecting a statistical difference requires a larger sample, and annotators tend to retreat to the middle value to avoid making a hard decision”. Binary forces the decision instead of hiding the uncertainty in the middle of the scale.

One criterion per call. “Is the answer factually consistent with the retrieved context?” is one eval. “Did the answer respect the declared dietary restriction?” is another. Don’t combine them, because when the number drops you will want to know what dropped.

To make it concrete, this is the skeleton of a binary judge, using the recipe app example:

You evaluate a recipe assistant.

Criterion (just one): does the answer respect the dietary restrictions
the user declared at any point in the conversation?

<conversation>
{{conversation}}
</conversation>

<answer>
{{answer}}
</answer>

Steps:
1. List the restrictions declared by the user.
2. List every ingredient or instruction in the answer that conflicts
   with any of them. Valid substitutions are not a conflict.
3. Any conflict = FAIL. None = PASS. When in doubt, FAIL.

Reply in JSON: {"analysis": "...", "verdict": "PASS" | "FAIL"}

Three deliberate choices there. The analysis comes before the verdict, because forcing the model to list the evidence first improves the decision and gives you debugging material when it gets it wrong. “When in doubt, FAIL” settles the tie for you, instead of letting the model settle it a different way on every call. And the criterion fits in one sentence, which is the test that it is one criterion rather than three. When the judge gets cases from your golden set wrong, those cases become few-shot examples inside the prompt itself.

One caveat: this is no universal law. In tasks with a genuinely ordinal structure, like document relevance in information retrieval, there is 2025 work suggesting larger scales help. The strong case for binary is about human labeling and error analysis, which is where you are right now.

Decision 2: measure the judge’s alignment with humans BEFORE using the judge.

Set aside part of your labeled cases as a holdout, run the judge and look at true positive rate and true negative rate separately. Don’t look only at raw agreement, because with imbalanced classes raw agreement is misleading.

If alignment is poor, switch models, rewrite the criterion, or accept that this criterion is not automatable yet.

And there is a detail I only recently saw handled properly, in a November 2025 paper on how to report LLM-as-a-judge evaluations. If you know your judge’s sensitivity and specificity, its raw score is a biased estimator, and you can correct it with a closed-form formula. The measurement and the correction together are ten lines:

import numpy as np

humano = np.array([1, 1, 0, 1, 0, ...])  # 1 = pass, 0 = fail
juiz   = np.array([1, 1, 0, 0, 0, ...])

tpr = (juiz[humano == 1] == 1).mean()  # agrees on the ones that pass
tnr = (juiz[humano == 0] == 0).mean()  # agrees on the ones that fail

# true pass rate, corrected (Lee et al., 2025)
p_bruto = juiz_em_producao.mean()
p_real = (p_bruto + tnr - 1) / (tpr + tnr - 1)

What got me in that paper was the direction of the bias: it flips. A bad system looks better than it is, a good system looks worse than it is. Meaning, when your product starts getting genuinely good, the uncalibrated judge starts underestimating you, and the team reads that as diminishing returns on the work. It is a wrong diagnosis caused by the measuring instrument, while the measurements themselves are fine.

6. Baseline and gate

Run the current system, without changing anything, against the whole golden set. The number that comes out is your starting point. It will probably be worse than the team’s intuition. That is the value of the thing.

Then tie it into CI: a new version only ships if it does not regress on the golden set. The split Anthropic uses is helpful here. A capability eval answers “can the agent do this?” and should start with a low pass rate. A regression eval answers “does the agent still do what it already did?” and should stay close to 100%. When a capability eval saturates, it retires and becomes a regression eval.

Prefer deterministic checks over an LLM judge in CI whenever you can, because CI runs all the time and judges cost money. And since LLMs are not deterministic, handle that in configuration instead of fighting it. The promptfoo GitHub Action has repeat and repeat-min-pass, which runs each test three times and requires it to pass at least two. Consistent failure breaks the build, judge variance does not.

Anthropic also has a back-of-the-napkin calculation on consistency worth keeping: if the agent gets it right 75% of the time and you run the same task three times, the chance of passing all three is 0.75³, that is, 42%. Averages mislead badly when the user expects reliable behavior every time.

7. Close the loop

A hundred fresh traces per review cycle, with a cycle of two to four weeks. A new failure in production becomes a new case in the dataset. Without that the golden set ages and stops reflecting the product, and your gate starts protecting a version of the world that no longer exists.

A few things to watch for

Reaching stage 3 does not end the problem. It just trades the problem for a more interesting one.

Trap 1, the overconfident judge

The LLM judge agrees too much and is too sure of itself.

The paper Overconfidence in LLM-as-a-Judge, from August 2025, tested 14 judges on a benchmark of answer pairs where one is subtly wrong. The expected calibration error of the best model was 11.78%. The worst came in at 74.22%. And several models concentrate their predictions in the 90 to 100% confidence range, with accuracy well below the ideal line.

There is something worse, and it is the most useful thing to know. Work from Kensho and MIT, No Free Labels, showed that the judge cannot evaluate what it cannot solve. On questions GPT-4o itself cannot answer, its kappa against the human label collapses from 0.86 to 0.16, which is practically chance. And a small judge with a reference beats a much larger judge with no reference at all, and what matters about the reference is that it is correct, whether or not a human wrote it.

And there is sycophancy (yes, that is a real word, I checked), which is the most uncomfortable one. A September 2025 study measured how much the judge changes its mind when the user disagrees. A formal, well-grounded counterargument convinced it in 56% of cases. A casual, assertive counterargument with no justification at all convinced it in 84.5%.

Add to that the classic biases documented since MT-Bench: position, verbosity, preference for its own output. And pay attention to one reading detail, because the most cited number in the field comes from there. That 85% agreement between GPT-4 and humans excludes ties and inconsistent votes. Including everything, it drops to 66%. Whoever cites “85%” is citing the best case.

How do you solve it? Human labels are the anchor, always. The judge is a cheap approximation of an expensive label, and an approximation has to be measured against what it approximates. If the judge changed prompt or model, it is a new instrument and needs recalibration.

Trap 2, overfitting the golden set

You have a number. The number becomes a target. The target becomes optimization. And then the team tweaks the prompt until the number goes up, without anything improving for the user. The worst part is that the failure is silent again. The dashboard stays green, the demo still looks good, and the only evidence that something broke is that the improvement does not show up in any metric the team does not control.

It is pure Goodhart, and it is no hypothesis. It is what happened to the public benchmarks in the stages section, and there is nothing structural preventing it from happening to your private benchmark.

Actually, there is a finding that closes this escape hatch. A 2026 study on benchmark saturation analyzed 60 benchmarks and found 48.3% with high saturation. And it compared public against private: keeping the test set private did not protect against saturation. What protected was expert curation.

The fix: a holdout you don’t look at day to day, a monthly refresh with new production cases, and suspicion of any big improvement without a hypothesis. If the number went up eight points and nobody can explain why, the dataset probably leaked into the prompt.

Trap 3, proxy metrics

This is the quietest of the three.

Evals measure what is easy to measure. Correct format, absence of hallucination, adherence to the retrieved context. All of it is measurable and none of it is the business objective in the vast majority of cases.

You can have a system at 95% on the golden set that is destroying value, because the thing it does well is not the thing that needed doing. And the eval won’t warn you, because the eval is measuring exactly what you told it to measure.

I won’t develop this now, but it might become another article.

Tooling is the least of your problems

A short section on purpose.

The map changed quite a bit in 2026, so an update is worthwhile. Langfuse was acquired by ClickHouse in January and promptfoo by OpenAI in March, both staying open. Beyond those, Braintrust and LangSmith always show up on the commercial side, Arize Phoenix as source-available, DeepEval and Inspect AI, from the UK AI Security Institute with Meridian Labs, as open source. openai/evals today points to the Evals API in OpenAI’s own dashboard.

Pick one and move on with your life. Hamel, who has consulted for dozens of teams on this, wrote that he has no favorite vendor and that the criterion that weighs most in the choice is who gives the best support, rather than who has the most features.

His recommendation that I find most underrated is building your own annotation interface. It takes a few hours today, and he claims teams with a custom tool iterate about 10 times faster. Which makes sense: annotation is where human time is spent, and a generic interface does not know the shape of your trace.

The harness is 80% process. The tool is the part you swap out painlessly.

Recommendations off the top of my head

Well, here go atomic summaries of my studies and conclusions on the topic:

  1. Start with error analysis. Infrastructure comes later. Thirty minutes reading 20 outputs after every relevant change already changes the game.
  2. One person decides what quality is, and that person understands the domain. Quality committees don’t converge.
  3. Binary per criterion, one criterion per call, and a criterion you can explain in one sentence.
  4. Measure the judge before believing the judge. True positive rate and true negative rate on a labeled holdout.
  5. Twenty to fifty real cases are worth more than five hundred synthetic ones. Scale up later.
  6. A simple regression gate in CI is worth more than a pretty dashboard nobody opens.
  7. Never report an eval score without someone having read transcripts. Anthropic has an explicit rule about this: they don’t take an eval score at face value until someone digs into the details and reads a few transcripts.
  8. If your pass rate is 100%, the problem is your eval.

The mechanism of the mistake

I think I said in the opening that I gave the bad answer for too long. In case I forgot to, here it is: I have also given bad answers to “so, how well does that thing actually work?”. It is worth explaining the mechanism, because my mistake has a structure, and I bet it is the same structure as yours.

I come mostly from experimentation. And experimentation has a property that took me years to realize was a luxury: randomization gives you the answer key for free. The control group is the counterfactual, the business metric exists before the experiment starts, and the only question is whether the variant moved the number. You never need to define what a good answer is. The experiment design defines it for you.

Evals invert the problem. No control, no counterfactual, no ready-made metric. The answer key does not exist anywhere waiting to be discovered. You build it, case by case, with someone who understands the domain. And building an answer key is less statistics and more measurement instrument engineering. My entire repertoire taught me to analyze measurements, never to manufacture the measuring device.

So, when I shipped my first system with an LLM deciding things, I applied the wrong instinct with all the confidence of someone with trained instincts. I read the outputs, thought they were good, and treated that as evidence. But “I read it and it looks good” is a prior, and a prior is no sample. There were no labels, no denominator, no error rate. The system ran fine for a few weeks, a case showed up outside the distribution I had imagined, the model answered with its usual confidence, and I found out through a complaint rather than a metric. The error itself did not bother me. What bothered me was realizing that I had no instrument capable of detecting it, at any point, not even by luck. Errors get fixed. Instrumented blindness never even gets seen.

Tooling did not change afterwards, and neither did the model. What changed was accepting that the first deliverable of any such system is the measuring device, and that the measuring device is built by reading bad data with a notepad next to you.

And what does the right answer to the title question sound like? It impresses no one. It is boring, and it goes more or less like this:

“I looked at about a hundred real production cases and catalogued the failure modes. Three categories account for most of them. I have forty cases with human verdicts, about ten of which are edge cases I picked on purpose. The automated judge agrees with my labels at a true positive rate and a true negative rate that I measure on a holdout, and I recalibrate when I swap the judge model. The current baseline is at such and such number. It runs in CI, and if it drops, it doesn’t ship. And I know which failures I still can’t measure.”

Notice the last piece, because it is what separates people. Knowing the number is expected. Knowing where the number doesn’t reach is what gives away who has actually measured.

And getting to that answer requires nothing this text hasn’t already given you. A hundred traces, a notepad next to you, one Monday. It is less glamorous than swapping models, and worth ten times more.

And the title question remains, now pointed at your team: which stage are you at, honestly? Forget what is on the roadmap. The question is what ran last week.

A good AI system is one you can prove gets it right, in the week someone asks.

References

#evaluation#agents#applied-ai