The most common failure in a preference-data programme is not bad raters. It is an ambiguous rubric that nobody noticed was ambiguous, because the only metric being watched was throughput.
A reward model trained on preferences where raters agree 60% of the time is learning the rubric’s ambiguity as much as the underlying preference. Adding examples does not fix this. It encodes the noise more confidently.
Measure agreement per dimension, not per dataset
A single headline kappa hides the thing you need to act on. We report agreement per rubric dimension, because the pattern is almost always uneven: raters agree strongly on correctness and safety, and diverge on the softer dimensions like code quality or helpfulness.
That distribution is diagnostic. High variance on one dimension means the definition is underspecified, not that the raters are careless. The fix is a rubric clarification, versioned, not a coaching conversation.
Low agreement on one dimension is a specification bug. Treat it like one.
Gate production on it
Agreement measured after delivery is an autopsy. We run calibration rounds before any production item is scored, and the pod does not start until it clears the agreement target on two consecutive rounds.
Two rounds matters. One round can be cleared by luck or by raters converging on a shared misreading. The second round, after the clarifications have been folded in, tests whether the convergence was real.
What to ask a vendor
A vendor who cannot answer the fourth question is not running a calibration process. They are running a headcount.
- Which agreement statistic, computed on what sample, at what frequency
- Agreement broken out per rubric dimension, not a single figure
- Whether production is gated on it, or whether it is only reported afterwards
- What happens when a dimension falls below target mid-batch