Assign traces to reviewers
Queues control which traces go to which reviewers. Findings can push flagged traces straight into a queue. From there the work spreads itself. A queue carries the rubric to grade against and the reviewers on the team, and it hands traces out round-robin, so no one assigns reviews by hand and the load stays even across the team.Do the review
Each reviewer sees their My Reviews tab: a queue of traces to grade against the rubric. They can submit corrections on different sections, rationales, and other grades. They work in a grading workbench that sits right next to the trace. They score each dimension, set their confidence and severity, add a short rationale, and can edit the model’s answer side by side to capture what the right response should have been. A built-in copilot drafts the correction and the rationale for each grade, so reviewers check and approve instead of writing from scratch.See where reviewers agreed and disagreed
All Reviews shows completed reviews across all reviewers, so you can see what they found across the board. Because more than one expert can grade the same trace, you can also see where they agree and where they don’t, which is the same signal that tells you how solid your ground truth is.Feedback gets reused automatically
A reviewer’s correction isn’t a one-off. It’s distilled into memory that speeds up future reviews of similar cases, and feeds directly into judge calibration and regression datasets, so the same mistake, once caught, doesn’t need catching twice.Next steps
Judges & Datasets
Calibrate a judge against this review data