Evaluation vendors are hard to compare because everyone reports a high acceptance rate. The rate is not the signal. How it is produced, and what happens when it slips, is the signal.
On quality
- What share of items receives a second review, and who performs it
- Is the audit function independent of the delivery line, or the same people
- What is the escaped-defect rate, defined as defects the client found
- How are blind gold tasks seeded, and at what rate
- What happens to a batch that internal QA predicts will miss the bar
On people
- What proportion of applicants reach deployable status
- Is training paid, and is calibration time billed to the client
- What is 90-day contributor retention
- Is the reviewer-to-contributor ratio fixed, or does it stretch under load
On accountability
The last question is the one that matters most, because it is the only one whose answer costs the vendor something. A firm that carries rework cost has priced its own quality risk. A firm that does not has moved that risk onto you, whatever the acceptance rate on the slide says.
- Are acceptance criteria and rejection definitions written before signature
- Who bears the cost of rework against a missed bar
The only quality claim worth much is the one the vendor pays for when it fails.