You shipped an AI feature, and every release now comes with the same question: is it actually any good? Reading the outputs by hand doesn't scale, so you do what most teams do, point another LLM at them and let it grade. One number per release, cheap and automatic. The whole thing rests on an assumption nobody tests: that the judge is right.
So the real question is narrow. Before you trust a single score, does that judge actually agree with the experts it replaced? Here is what the research found when people checked.
The evidence, measured
None of this means LLM judges are useless. It means you can't trust one you haven't tested against real human judgment. That test is calibration, and the rest of this article is how to run it, and how to keep it honest.
What does it mean to calibrate an LLM judge?
Calibrating an LLM judge means measuring how often it agrees with your human experts on the same examples, then improving it until that agreement is high enough to trust. It is not prompt-tweaking for its own sake.
The output is a number you can defend: on this kind of case, the judge and your experts reach the same verdict, and you know how often.
Most "judge tuning" is people editing a prompt until the scores look nice, with no ground truth to check against. That's a vibe check with extra steps. Calibration measures agreement against real humans, and tells you when a change made things worse instead of better.
How do you build the labeled set you calibrate against?
Have your actual domain experts label a batch of real examples, roughly 100 or more, as good or bad. That set is your ground truth, and everything downstream measures the judge against it.
Use real production traces, not invented ones. Invented cases skew easy and miss the failure modes that actually show up. If two experts disagree on a label, that is signal, not noise: it marks a boundary where "good" is genuinely contested.
Labeling is real work, and it is the part teams most want to skip. It also rots: change the system prompt once and half your expected answers are suddenly wrong, and nobody wants to re-label 400 rows by hand. Skip it anyway and you have no way to know if the judge agrees with anyone.
How do you split the data so the final number is honest?
Split the labeled set three ways: train, development, and test. Train is a few examples you may show the judge. Development is what you iterate against. Test is blind, touched once, for the final number.
The reason for the blind test set is the same reason it exists in machine learning. If you tune the judge against the same examples you grade it on, you measure how well you fit those examples, not how well the judge generalizes. Keep the test set sealed until you think you're done, then run it once. That number is the one you report.
How do you measure whether the judge agrees?
Run the judge on the development set and compare its verdicts to the expert labels. Report agreement in a way that separates the two kinds of error, for example true-positive and true-negative rate, not one blended pass rate.
One overall pass rate hides which direction the judge is wrong in. A useful calibration set includes cases your experts scored as bad, and a good judge catches them. If nothing ever fails, you haven't measured discrimination, you've measured optimism.
And if the scores wobble from run to run, people quit trusting them: failures get waved off as "the judge just being weird," which is how a broken judge quietly stays in your pipeline.
A 100% pass rate is a red flag, not a win. It usually means the test is too easy to fail anything.
How do you actually get the judge to agree more?
Iterate on three things and re-measure after each: sharpen the criteria so "good" is defined in concrete terms, add in-context examples from your train split, and split one vague overall score into separate judges, one per quality dimension.
The last move does the heavy lifting. A single judge asked to weigh empathy, safety, and correctness at once blurs them into one soft number, and it happily rewards answers that are right for the wrong reasons, since it only ever reads the final text. Give each dimension its own judge with its own criteria, and each one gets a sharper, more checkable task.
For a mental-health chatbot, we built a nine-judge system, one judge per quality dimension, and held clinical safety at 97.6% across a 400-item edge-case suite. These are custom judges, built for one product's definition of quality, a generic judge has no way to know what "professional boundaries" means for your users.
Why isn't calibration a one-time task?
Because the underlying model changes and takes your agreement with it. When your judge model updates, the same prompt can produce different verdicts on the same inputs, so a judge you calibrated in the spring can quietly drift by summer.
The instability isn't only across versions. The same model re-scoring the same comparison already disagrees with itself run to run, which sets a floor on how stable any single-run verdict can be, and swapping the model moves that floor further without warning. Calibration is a standing check: keep the labeled set, re-run it whenever the model changes, and confirm the number still holds.
Our approach
How ArgminAI does this for you
Everyone else hands you a framework and a blank rubric, then leaves the hard part, trusting the judge, to you. We build the judge instead, calibrated, from three inputs: your data, your definition of quality, and your experts' labels.
It comes out as a multi-agent judge, one specialized judge per quality dimension, each measured against your experts until agreement is high enough to bet on. You stay in the loop: you review cases, correct the judge, and watch that feedback change how it scores. What you get is a judge that measures what "good" means for your product, with the agreement numbers to prove it, not a generic scorer you have to babysit.
What does calibration not fix?
Calibration raises agreement; it does not make a judge omniscient. Some qualities stay hard. Tone, subtlety, and taste resist scoring, and a calibrated judge will still miss cases an expert catches on instinct.
The honest limits, stated plainly: labeling costs real expert hours up front. Agreement is measured on the kinds of cases in your set, so a new failure mode outside it is still uncharted. And every model update reopens the question, which means the labeled set is an asset you maintain, not a chore you finish. None of this is a reason to skip calibration. It's the reason to keep the labels around.
The takeaway
An uncalibrated judge gives you a number with no error bar and no owner. Calibrate it against your experts and you replace "the judge said 8" with "the judge agrees with my experts on cases like this, and here's the sealed test set that shows how often." One of those you can stake a release on.
