Public model leaderboards test abstracted skills. None of them measure operations work: extracts with blank columns, joins that lie, renewal dates that disagree between systems. BonuslyBench closes that gap with 40 tasks drawn from a real revenue organization — then grades every answer with deterministic code.
When language models are pointed at the actual daily work of revenue operations — reconciling a rollforward, auditing stage hygiene, building a weighted forecast — what actually happens? Not on vendor benchmarks. On real, messy data, under a grading standard that cannot be charmed.
Every GTM organization is being asked to believe that pointing an LLM at a CRM produces reliable analysis. This run replaces belief with measurement: 103 models, 40 tasks, one deterministic rubric.
CRM and data hygiene, deal intelligence, rep performance, reporting and analytics, customer success, marketing, executive communication, ops maintenance. Each task hands the model real CSV extracts and asks for the deliverable an operator produces weekly.
Owner columns blank on most rows. Renewal dates that disagree across systems. Call mappings keyed to dead deal ids. "LONG(n)" placeholders. A correct answer navigates the trap — or says the data cannot support the claim.
Expected answers were computed programmatically from the same data the models saw. Responses are graded by fixed code: numeric tolerance bands, regex presence checks, alias whitelists. No LLM graded any response. The same rubric scored all 103 models.
Company identities are aliased; personal names were replaced with consistent pseudonyms across every test, response, and artifact — verified to be score-neutral (zero grading drift across all 4,576 answers).
The top of the leaderboard is open weights. The best frontier entry ties with an open model that costs a fraction as much. Premium pricing bought speed and polish here, not measurable accuracy on operational tasks.
The best score was 36 of 40 perfect tasks. The spread opened on tasks that mix arithmetic with judgment: which deals to exclude, what stale means, when to say the data cannot answer.
20 of 103 models invented an entity at least once. A deterministic alias whitelist caught every instance. The failure mode is real, and it is guardrail-able.
Five scoring defects initially made six tasks "impossible" for every model. A verification pass read real answers against real truth and fixed the rubric — the audit protocol is itself a finding.
Median response time ranged from under 20 seconds to over 40 minutes per task. The pool holds fast-and-accurate and slow-and-accurate profiles at similar cost — a real choice for embedded workflows.
| What | Detail |
|---|---|
| Run | 2026-09-05, one execution of the full 40-task set |
| Models | 103 · 96 with full 40-task coverage · 7 partial (timeout-tier, disclosed) |
| Reference model | anthropic/claude-sonnet-5 |
| Grading | Deterministic Python against frozen truth files · no LLM judge |
| Data | One company's GTM stack, redacted · company aliases · personal pseudonyms |
| Comparability | Scores are comparable within this run and rubric version only |
Basis: 40 tasks, deterministic rubric, one run. A fabrication flags the response; disclosed partials are never scored on unrun tasks.
Scores describe operational skill on this distribution of work, not general capability. Treat the rankings as a strong prior and verify against your own data before committing spend — the test specification is included so that's a week of work, not a mystery.