Here is where Part 1 left you: the judge agrees with your experts at a measured rate, on cases it had never seen, and the dashboard is honest. Then an expert opens the failed traces and finds three different problems under one verdict: an answer with unsafe advice, an answer that is correct but hostile, and an answer that stepped outside what the product is allowed to do.
A single pass/fail judge treats those as the same failure. The next release question is narrower: what exact failure mode is each judge measuring? The answer comes out of traces experts have already reviewed, not out of a brainstormed rubric.
One Verdict Hides Three Different Risks
A single quality score cannot separate failures that need different responses. An unsafe answer should block the release. A hostile but factually correct answer becomes a ticket and ships. An answer that crosses a product boundary needs an expert to make a call. A broad judge gives all three the same verdict, so every release review starts with re-reading the failed traces to learn which kind they are.
The fix is not a better prompt for the broad judge; it is splitting the score into judges that each measure one named failure mode, and the split has to come from failures your product has produced.
Write The Expert Note Before The Category
Do not start by writing a rubric. Start with the traces experts already reviewed and write one short factual note per bad or borderline answer, before naming any category. The labeled set from Part 1 is the raw material. The misses, where the judge passed an answer the expert rejected, are its most informative rows.
A rubric written first comes out as the same four words on every whiteboard: helpfulness, correctness, safety, tone. Those categories fit any product, which is the problem. They were not learned from yours.
The note describes what failed in the product, not what label the judge should output:
| Reviewed trace | Expert note |
|---|---|
| Answers a medical question without enough context. | Gives clinical guidance before collecting required history. |
| Empathetic reply that tells the user what to do next. | Crosses the product's professional-boundary rule. |
| Correct policy answer in a hostile tone. | Factually right, violates the expected support tone. |
A criterion that cannot point back to a reviewed trace is a guess. The notes keep the rubric tied to failures that shipped, or nearly did.
Cluster Notes Into Failure Modes
Cluster the notes by the decision the expert had to make. The split rule: two failures belong in different modes when spotting them takes different evidence, or acting on them takes a different release decision.
The example through the rest of this piece is a mental-health assistant. It is the product our own 9-judge system grades, so the numbers behind it are ours. Its expert notes cluster into clinical safety, crisis handling, professional boundaries, actionability, empathy, and tone. Those are not six words for quality. Strong empathy with unsafe guidance is a safety risk that blocks the release; safe guidance in a flat tone is a quality risk that becomes a ticket. Different evidence, different action, different modes.
Not Every Mode Gets Its Own Judge
Give a mode its own judge when its failures are expensive or frequent. Everything else folds into one general quality judge or stays in manual review until it recurs.
Separating the expensive mode is what makes a hard line possible. In our 9-judge system, clinical safety is its own judge, and that separation is what lets the team hold it at 97.6% while tone and formatting scores move independently underneath.
Two honest limits. A severe mode with almost no labeled failures can still get a judge, but two notes are too few to measure whether that judge is reliable, so it stays with experts: every crisis-handling trace goes to a human reviewer instead of a judge verdict, at release and in production, until enough labeled cases make a calibrated judge possible (this is the rare-failure sampling problem from Part 1). And the map will change as the product changes, so record why each judge exists. Without that record, the eval becomes a pile of prompts nobody can connect back to a release risk.
Criteria An Expert Can Label Consistently
Each mode that becomes a judge needs criteria an expert can apply to a trace without guessing what the author meant. A criterion names the observable evidence for the label, the evidence against it, and the borderline case.
One judge spec · clinical safety
The human criteria come first; the judge prompt copies them later. If two experts keep disagreeing while labeling against a criterion, the criterion is vague, and Part 1 showed what that looks like from the outside: an expert ceiling too low to calibrate against. Rewrite the evidence lines until the experts converge, then run the Part 1 loop per mode, with each judge getting its own train, development, and test split.
What The Map Adds To The Release Packet
Part 1's release packet answered one question: does the judge agree with the experts, after discounting the majority label? The failure-mode map adds the second: agree about what. After this pass the release claim reads like this:
On the reviewed set, six failure modes recur. Clinical safety, professional boundaries, and tone became separate judges; each has expert-written criteria, examples in train, and its own labeled development and test split. Empathy, actionability, and formatting stay inside a general quality judge. Crisis handling stays in expert review, with a note saying it has too few labeled failures to calibrate.
Getting there without special tooling takes a lot of manual coordination: reading hundreds of traces, writing the notes, clustering them, drafting a spec per mode, and keeping each mode's label sets clean while the criteria keep changing. Argmin AI's Judge Builder does that part as a product: it drafts the failure modes and per-mode criteria from your traces and docs, routes the borderline and disagreement cases to your experts, and turns each correction into a judge revision plus a labeled case you keep.
What You Have Now, And What You Do Not
What you have now is a map from release risk to judge. Each judge measures one named failure mode; its criteria point back to reviewed traces, and experts can label against those criteria consistently enough to calibrate. When the dashboard flags a trace, the flag now says which risk it found, not that quality dipped somewhere.
What you do not have yet is a threshold. Every judge still outputs a verdict with a score behind it, and nothing yet says which score is expensive enough to act on. Set the bar too low and your experts drown in flagged traces; set it too high and the expensive miss ships. That decision comes from error costs, not from 0.50 by default, and it gets the next article.
