Crowd platforms are efficient when a task can be decomposed into short, independent units that a briefly trained worker can complete with a stable error rate. Preference ranking fits that shape. Terminal-agent evaluation does not.
The task has state
An agent trajectory is a sequence where step four is only wrong because of what step two did. Grading it means holding the whole run in your head and knowing which deviation actually caused the failure.
That is not a rating judgement. It is debugging. The grader needs to read the diff, understand why the test passed for the wrong reason, and decide whether the agent got lucky. A rater who has never shipped code cannot do this reliably at any price.
The environment is part of the task
A benchmark task is not the prompt. It is the prompt plus a pinned container, a deterministic test harness and a gold solution that provably passes from a clean state. Authoring one is engineering work with a review step.
The failure mode here is quiet. A task that admits a shortcut solution still produces a number. It just produces the wrong number, and it keeps producing it every time the benchmark is run, which is worse than no benchmark at all.
A benchmark that admits shortcuts still produces a number. That is what makes it dangerous.
Adversarial review is not optional
Every task we author is attacked by a second engineer whose job is to break it: find the shortcut, find the ambiguous success criterion, find the environment leakage that lets the agent read the answer. Tasks that survive enter the suite.
This roughly doubles authoring cost. It is the difference between a benchmark that survives external scrutiny and one that gets quietly retired after someone publishes a thread about it.
What this means for sourcing
Agentic evaluation should be staffed like an engineering function, with the pay, review discipline and retention that implies. If your vendor is sourcing it through the same funnel as general rating work, the trajectories are being graded by people who cannot read them.