Reviews

Reviews is the human layer on top of traces. It turns “an expert looked at this and said it was wrong” into a structured, reusable signal, not a comment that dies in Slack.

​

Rubric Studio: define what “good” means

Before reviewing, an admin sets up a rubric in Rubric Studio: the dimensions to grade on, required fields, positive/negative/ambiguous examples, severity and confidence levels, and the outcomes a reviewer can pick. The same rubric a human uses is later reused to calibrate a judge.

​

Queues: route work to reviewers

Queues control which traces go to which reviewers. Findings can push flagged traces straight into a queue.

​

Assignments: do the review

Each reviewer sees their Assignments: a queue of traces to grade against the rubric. They submit a correction, a severity, and an outcome.

​

Submissions: see what came back

Submissions shows completed reviews, so you can see what reviewers found across the board.

​

Feedback gets reused automatically

A reviewer’s correction isn’t a one-off. It’s distilled into memory that speeds up future reviews of similar cases, and feeds directly into judge calibration and regression datasets, so the same mistake, once caught, doesn’t need catching twice.

​

Next steps