How much does the ranking depend on who judges?
The discovery scores average many judge models. Here the same judgments are re-scored with different panels: without the most lenient judge, with only the strictest judges, and leaving out one judge at a time. No new model calls; just the existing ratings recombined.
Why not weight judges by the controls? Every judge scored the controls perfectly (classics and banal claims 10 → 10, wrong claims and nonsense 1 → 1), so the controls can't separate careful judges from lenient ones. Future runs need harder controls for that.
loading…