> ## Documentation Index
> Fetch the complete documentation index at: https://docs.dataframer.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Judges & Evaluation

> Calibrate automated judges against human review, then evaluate and generate at scale

Once you have reviewed traces, the last step is scaling that judgment up: with an automated judge you can trust, and with synthetic data to test it against.

## Datasets: benchmarks tied to a rubric

**Judge Datasets** are benchmark sets built from the same rubric your human reviewers used in [Reviews](/reviews/overview). This is what a judge is calibrated and scored against.

## Judges: build and calibrate

A **Judge** is an LLM configured to grade traces against a rubric. Calibration measures how well the judge agrees with your human reviewers before you trust it to grade traces on its own.

## Evaluations: run and measure

**Evaluations** runs a judge against a dataset and reports results per rubric dimension, so you can see where the judge is confident and where it still disagrees with humans.

<Warning>
  Judges is a UI-only feature today, there's no public API or SDK access to judge creation, calibration, or judge evaluation runs yet. (Don't confuse this with the separate `/api/dataframer/evaluations/` API below, which evaluates a *generation run's* output quality, not a judge's agreement with human reviewers.)
</Warning>

## Generating data to evaluate against

Separately, DataFramer can generate synthetic datasets from the failure patterns you've found, useful for building regression sets or expanding thin edge cases. This part does have a public API and Python SDK:

```bash theme={null}
pip install pydataframer
```

See [Core Concepts](/concepts) for how seed data → specs → generation works, and [API & MCP](/api-and-mcp) for the SDK.

## Next steps

<Card title="API & MCP" icon="code" href="/api-and-mcp">
  Programmatic access to datasets, specs, generation, and evaluations
</Card>