Ten questions that separate a vendor who will hold a quality bar from one who will hand you output and leave verification with you. Written from the delivery side, so it includes the questions we would rather not be asked.
Ambiguity here is the single largest cause of failed evaluation programmes. If nobody has written down what counts as a rejection, disputes get settled by whoever argues hardest at delivery.
Agreement is a ceiling on the signal your reward model can learn. A vendor who cannot quote a statistic is not measuring it.
Coverage is the difference between a QA process and a QA claim. Ask for the shape, not the adjective.
If quality data does not arrive with the delivery, you are verifying the vendor yourself, which is the cost you were trying to outsource.
Rubric context compounds. A team that has worked your programme for eight weeks is meaningfully better than an equally qualified team that has not.
A quality commitment only means something if missing it is expensive for the vendor rather than for you.
Most labs and platforms mandate their own tooling and access controls. A vendor who needs you to adopt their platform is selling a platform.
A company-level NDA does not bind the contributor who sees your unreleased model output.
Domain evaluation is only worth its premium if the domain qualification is real.
The most revealing question on the list. A vendor who will take anything will take work they cannot staff to your bar, and you find out at delivery.
Ask for numbers, not adjectives. Ask what arrives alongside the batch. Ask what it costs the vendor to miss. Any partner worth signing will have answers written down already, because the questions are not unusual and the answers do not change per client.