> ## Documentation Index
> Fetch the complete documentation index at: https://docs.dataframer.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Judges & Calibration

> Build an automated judge from your rubric and calibrate it against human review until you can trust it.

Judges turn a handful of human reviews into quality scores across all your traces. Once a judge aligns with your experts, you can measure accuracy at scale across new traces, catch regressions before they reach users, and keep evaluating at scale without adding reviewers.

DataFramer tracks this with three metrics: **Human Graded Accuracy**, **Judge Accuracy**, and **Judge-Human Alignment**, which shows how well your judges track your reviewers.

There are two sides to this: the judge you build and calibrate here, and the [datasets and evaluations](/judges/datasets-and-evaluations) you run it against.

## Judge Calibration

A **judge** is an LLM that grades traces against a rubric, the same rubric your reviewers used in [Human Reviews](/reviews/overview). You build a judge from a rubric and a model, and it is versioned, with each run recording the one it used.

Before you trust a judge, you **calibrate** it, which means checking that it agrees with your human reviewers.

DataFramer runs the judge on traces your team already reviewed and reports an **alignment** score: how often the judge's verdict matches the human verdict. A high score means the judge can grade on its own. A low score means it needs work first.

Every verdict also comes with a confidence score, so you can spot the calls the judge was unsure about and send the shaky ones back to a human.

## Calibration tasks

If alignment is low, improve the judge and check again:

* **Clarify what good means.** If your reviewers disagreed too, the rubric is ambiguous. Fix its dimensions, definitions, and examples in [Rubric Studio](/reviews/rubric-studio), then re-generate the judge's prompt from the new rubric version.
* **Sharpen the grading instructions.** If the rubric is fine but the judge reads it wrong, edit the **Judge Prompt** sections by hand on the judge page.
* **Add more ground truth.** Route more traces to a review queue, from a [finding](/findings/discovery) or straight from the [trace list](/findings/overview), review them, and add them to the dataset, so alignment rests on more reviewed traces.
* **Try a better model.** Change the judge's model on the judge page and keep the one whose grades align best with your reviewers.

Then run the [evaluation](/judges/datasets-and-evaluations) again and read the traces where the judge and your reviewers still disagree.

## Judge prompt

When you build a judge, DataFramer writes its grading prompt for you from the rubric you set up in [Rubric Studio](/reviews/rubric-studio). You can open the prompt and read it as structured sections or as raw text, and edit the wording by hand to calibrate/tune how the judge grades.

## Next steps

<Card title="Datasets & Evaluations" icon="chart-bar" href="/judges/datasets-and-evaluations">
  Build benchmark datasets and run judges across them at scale
</Card>