Skip to main content
Once you have reviewed traces, the last step is scaling that judgment up: with an automated judge you can trust, and with synthetic data to test it against.

​
Datasets: benchmarks tied to a rubric

Judge Datasets are benchmark sets built from the same rubric your human reviewers used in Reviews. This is what a judge is calibrated and scored against.

​
Judges: build and calibrate

A Judge is an LLM configured to grade traces against a rubric. Calibration measures how well the judge agrees with your human reviewers before you trust it to grade traces on its own.

​
Evaluations: run and measure

Evaluations runs a judge against a dataset and reports results per rubric dimension, so you can see where the judge is confident and where it still disagrees with humans.
Judges is a UI-only feature today, there’s no public API or SDK access to judge creation, calibration, or judge evaluation runs yet. (Don’t confuse this with the separate /api/dataframer/evaluations/ API below, which evaluates a generation run’s output quality, not a judge’s agreement with human reviewers.)

​
Generating data to evaluate against

Separately, DataFramer can generate synthetic datasets from the failure patterns you’ve found, useful for building regression sets or expanding thin edge cases. This part does have a public API and Python SDK:
See Core Concepts for how seed data → specs → generation works, and API & MCP for the SDK.

​
Next steps

API & MCP

Programmatic access to datasets, specs, generation, and evaluations