BeveridgeHow much of the work can a decision model take over?
Every system answered the same held-out questions and said how sure it was. Hand over the most confident answers first and stop when the accuracy would fall below your target. Move the target and see the result recalculated from the real answers of Beveridge, Jev and five LLMs.
Settings
The automated decisions must be at least this accurate. Answers given with the same confidence are handed over together.
| System | Automated | To a person, per 1,000 | Wrong automated decisions, per 1,000 |
|---|
Try it on real decisions
Six documents answered by Beveridge-Large-C (real outputs, run on a laptop). Move the confidence threshold: answers at or above it are taken automatically, the rest go to a person.
What is being computed
For each system, its answers are sorted from most to least confident (confidence = the probability it gave its own answer). Handing over the top X% gives an accuracy on those X% = correct ÷ handed over. The automated share is the largest X whose accuracy is still at or above the target. Example: if the 6,000 most confident of 10,000 answers are 95.3% correct and the next group brings it to 94.9%, the share at 95% is 60%. Wrong automated decisions per 1,000 = share × (1 − accuracy) × 1,000. Example: Beveridge-Large-C hands over 76.4% at 95.0% accuracy, so 0.764 × 0.050 × 1,000 ≈ 38 wrong automated decisions per 1,000. The data are the systems' real answers on Beveridge-Bench, typed decisions and Aegis 2.0 (October 2026). Beveridge itself does not run in your browser: it is a 1.6 GB model you run on your own machine (get Beveridge).
Open the calculation code