Datasets
An evaluation dataset is a set of real traces you use as a benchmark. You build one from your reviewed traces, so each trace carries the human verdict as ground truth. A judge runs against this set, and its scores are compared to what your reviewers said. Create a dataset here, or add traces straight from the Findings trace list, so a failure pattern you already found there becomes a benchmark in a few clicks and your judge is tested on the exact behaviors you care about. When you start an evaluation, DataFramer shows how many of the dataset’s traces already have human verdicts. Calibration is only as strong as that overlap, so the more reviewed traces a dataset has, the more the judge has to check itself against. To cover more cases, consider synthetic generation grounded in your real data to create additional inputs for regressions and rare edge cases, then run and review them so they combine with the datasets with ground truth.Evaluations
An evaluation runs a judge across an evaluation dataset and scores every trace. Results break down per rubric dimension, with the judge’s grades and the human consensus side by side, so you can see where the judge agrees with your reviewers and where it still disagrees. This is how you check a judge at scale before you rely on it. You can also go trace by trace to see each judge verdict next to the human consensus, filter to just the disagreements, and open the original trace to dig in. This is how you find exactly where the judge and your reviewers part ways.Generating data to evaluate against
Separately, DataFramer can generate synthetic datasets from the failure patterns you found, useful for building regression sets or filling in thin edge cases. This part has a public API and Python SDK:Next steps
API & MCP
Programmatic access to datasets, specs, generation, and evaluations