Real operational data
Every dataset was pulled from live Snowflake on 2026-09-05: open deals, closed-lost reasons, Gong calls, PLG cohort, tickets, redemptions, skill files. Company names aliased to Deal-XXXX / C-XXXX; no personal contact data.
Adversarial by design
Tests contain planted traps: stale tables, unpopulated columns, contradictory dates, single-account noise masquerading as themes, aggregation tricks. A correct answer notices the trap — or explicitly says the data can't support one.
Deterministic grading
Ground truth is computed from the same CSVs the model sees. Scorers check exact numbers within tolerance, required aliases, regex-detectable findings, and a hallucination detector for invented entity IDs. Same code grades every model.
Instruction discipline
Many tests pair the analysis with a format demand: ≤80 words, JSON-only, cite by alias, flag what's unanswerable. Operational work is graded on both the math and the discipline.
01
Data & CRM Hygiene
Can the model audit a messy revenue database without inventing facts?
stage-hygiene-auditplanted traps
Audit 280 open deals for stage-process hygiene: deals missing a DS1 entry timestamp, stage entries out of chronological order (regression), and per-owner rates of non-compliant stage entries. The trap: the extract's timestamps suggest unlogged back-and-forth movement — the correct answer quantifies it and recommends write-on-stage-entry automation rather than trusting the timestamps at face value.
Scorernoncomp %, regression count, per-owner breakdown, fixesDatadeals_open.csv + owners.csv
owner-resolutionplanted traps
Resolve every open deal to its owner — including deals still owned by two archived employees (a real CRM problem). Report per-owner open pipeline totals and flag the deactivated owners explicitly instead of silently assigning their books.
Scorerresolved count, archived-owner flag, Bryce/Alex totals, no invented namesDatadeals_open.csv + owners.csv
gong-hubspot-join-integrityplanted traps
Join the Gong call log to the CRM. Find orphan calls (deal IDs that exist in Gong but not in the CRM), duplicate keys, and DS3 coverage. The trap: gong_owner_id is a Gong user id, not a HubSpot owner id — joining on it silently corrupts ownership analysis.
Scorerorphan count (88), dup keys, DS3 coverage, sample orphan aliasesDatagong_calls_by_deal_90d.csv + deals_open.csv
snowflake-sql-trapped-catalogplanted traps
Given a schema catalog, write the SQL that counts retained PLG companies. The trap: the catalog contains a stale table, an unpopulated table, a retired naming pattern, and a tempting deleted_giver filter rule that would zero the count. Correct SQL uses company_cohort_summary with a succeeded-redemption filter and none of the traps.
Scoreravoids stale/unpopulated tables, uses cohort table, succeeded filter, no giver-deletion filterDataschema_catalog.md
crm-completeness-enrichmentplanted traps
Measure CRM completeness, find duplicate clusters and invalid emails, propose survivorship rules for merging — while using an enrichment file to fill gaps. The trap: the enrichment source is incomplete; the instructions forbid inventing values that aren't in it.
Scorerdup clusters, invalid email count, survivor language, no-invent disciplineDatadeals_open.csv + zoominfo-style enrichment
02
Deal Intelligence
Reading the pipeline: why we lose, what's real, and what's about to close.
closed-lost-classificationplanted traps
Classify all 90 deals lost in the last 6 months into primary categories (pricing, competitor, no decision, timing, product gap, champion left, other), assign a side (Bonusly / buyer / unknown), and surface the two most actionable patterns. The trap: the structured closed_lost_tag sometimes disagrees with the rep's free text — a correct answer counts those disagreements rather than trusting either field alone.
Scorertotal=90, competitor top (26), tag-vs-text disagreement, side split, two patternsDataclosed_lost_6mo.csv
pipeline-tieringformat discipline
Tier all 280 open deals into LOCK / BUILD / REVIVE / WATCH / RISKY, return tier counts, list the risky deals, and characterize the pipeline shape — as a JSON array only, with a single closing paragraph. No deliberation in the output. (A heavy test: models that narrate per-deal can time out.)
ScorerJSON returned, tier vocabulary, counts sum to 280, risky listed, no lock violations claimedDatadeals_open.csv
call-transcript-extractionplanted traps
Extract structured fields from a sales call transcript: why they'd buy, budget signals, stakeholders, competitors mentioned. The traps: the prospect mentions a competitor (record it — but only the prospect's, not the rep's counterexample), and some fields are simply never stated — they must come back as null, not guessed.
ScorerJSON emitted, budget signal, null discipline, stakeholders, prospect-only competitorDatacall transcript
next-to-closeplanted traps
Name exactly the three deals closest to closing, with reasoning. The trap: the highest-amount candidate is blocked/stalled — a correct answer excludes it in favor of genuinely closeable deals, or explicitly explains the block.
Scorerexactly 3, top-3 aliases match, blocked deal excluded or explainedDatadeals_open.csv + engagements
product-gap-detectionplanted traps
From closed-lost and call reasons, separate real product gaps from non-gaps: plan-tier gating ("we need it, but it's locked in your higher tier") and enablement problems ("we couldn't get users trained") are not product gaps. Cite the deals where the real gaps surfaced.
Scorerreal gaps cited, plan-gate classified, enablement classified, gap summaryDataclosed_lost_6mo.csv + transcripts
03
Rep & Team Performance
Scorecards, coaching, and honest attribution — without inventing causation.
stale-pipeline-by-repno frontier model aced
Find every deal with no rep contact for 30+ days, grouped by owner, with per-rep stale amounts and the largest single stale deal. Why it's brutal: 31KB of raw deal rows, exact per-owner sums, near-total recall of ~30 stale aliases, and the no-causal-claims rule — every frontier model dropped at least one check.
Scorerstale recall ≥80%, grouped by owner, Bryce top amount, Cole/Dana amounts, stale countDatadeals_open.csv + engagements_by_deal_90d.csv
activity-mix-vs-outcomeplanted traps
Compare reps' activity mixes (emails/meetings/calls) against stage-2 progression. Who's most efficient? The trap (epistemic rigor): the data cannot support a causal claim about WHY a mix works better — and the scorer explicitly penalizes "because … drives conversion" claims. Also planted: a data defect (all inbound email counts read zero) that a careful analyst notices.
ScorerAlex DS2=36, mix %, efficient named, no causal claimDataengagements_by_deal_90d.csv
ae-qtd-scorecardno frontier model aced
Build a quarter-to-date scorecard for one AE: bookings vs quota, attainment %, new vs expansion split, top loss reason — excluding a deal that closed before the quarter started (a planted boundary-date trap) — plus three coaching observations grounded in the numbers.
Scorerbookings $150K, quota $200K, attainment 75%, new/expansion split, pre-quarter deal excluded, coaching notesDataae_deals.csv
multithreading-gapplanted traps
Enterprise deals need multiple stakeholders engaged. Flag the deals that are single-threaded — only one persona on file, others unengaged — using persona language, and quantify the gap. The trap: some "engaged" contacts are just auto-responders in the data.
Scorerflagged-deal recall ≥70%, persona language, single-thread term, on-file/fit languageDataae_deals.csv + company_unengaged_contacts.csv
talk-track-from-transcriptsformat discipline
Synthesize past call transcripts into a coaching talk-track: the three most common objections, the next-step rate, competitor handling — using real quotes, not paraphrase.
Scorerobjection count=3, next-step rate, competitors, coaching framing, quotes usedDatacall transcripts
04
Reporting & Analytics
The revenue math: forecasts, rollforwards, cohorts — with the honest caveats attached.
q3-weighted-forecastplanted traps
Build the Q3 2026 weighted forecast: 100% of COMMIT + 35% of BEST_CASE, only in-quarter close dates (a PIPELINE deal never counts). Report totals, counts, exclusions with amounts, top-5 BEST_CASE aliases — then a ≤4-sentence data-quality section. Exact expected math: $44,729 commit (7) + 35% × $203,565 best case (24) = $115,976.75; 32 out-of-quarter deals excluded ($227,575).
Scorercommit/best/weighted exact, counts, exclusions, top-5 aliases, blank-owner & empty why_buys flagsDatafrozen deals.csv
aha-moment-2x2planted traps
Test the activation hypothesis: companies with 5+ M1 users AND 1+ M1 redemption retain better at 24 months. Build the full 2×2 with retention rates (66% / 47% / 31% / 40%), name the largest single-signal lift (unique givers), state exclusions (none), and explicitly bound the claim — correlation in one cohort, not proof of causation.
Scorerall four retention %s, givers-largest-lift, exclusions stated, proof boundedDataplg_company_cohort_2023.csv
arr-rollforward-reconciliationno frontier model aced
Reconcile two sources of ARR that should match but don't: billing (Chargebee) vs CRM. Decompose the variance into buckets (new, expansion, churn, contraction, reactivation), reconcile to the dollar, and list the specific subscriptions violating the rollforward rules — including a term vs evergreen contract trap in the date fields.
Scorerbilling total, CRM total, variance, all bucket amounts ±5%, violation subscription IDsDatachargebee_subscriptions.csv + company_arr.csv
kvm-month-over-monthformat discipline
Four key value metrics (KVMs) moved month-over-month. Name the biggest mover, its direction, and the segment driving it — in ≤350 words. The discipline is compression: no sprawling essay.
Scorerall four KVMs named, biggest mover, direction correct, driving segment, word capDatakvm_monthly.csv
rewards-ytd-sectionplanted traps
Year-to-date rewards digest: redemption count, total spend, unique redeemers, the peak month, provider mix by spend share, and top country. The trap: a currency/rounding artifact in the redemptions file makes naive summing wrong — the correct total spend requires noticing the data defect.
Scorercount, spend, unique redeemers, August peak, provider mix, top countryDataredemptions_ytd.csv
05
Customer Success
Retention saves, expansion targeting, and honest renewal risk.
churn-save-eligibilityplanted traps
Some at-risk accounts qualify for a churn-save play; others don't. Apply the eligibility rules exactly (health score, usage floor, ARR thresholds), name the eligible accounts with their plays, total ARR at stake, and — critically — name accounts that are at risk but NOT eligible rather than lumping everyone into the save.
Scorereligible set, at-stake total, plays cited, non-eligible named, rules appliedDatachurnzero_accounts.csv + rules.csv
expansion-kitplanted traps
Build an expansion package for one account: coverage %, headroom ARR, the right buyer — the economic buyer, not the day-to-day contact who happens to answer email — a usage fact, and a ≤150-word email. The trap: the replier is prominently present in the data and is the wrong target.
Scorercoverage %, headroom ARR, right buyer named, replier-not-buyer, usage fact, email lengthDataexpansion_account.csv + expansion_contacts.csv
onboarding-statusformat discipline
Status report on a new account's onboarding checklist: which steps are complete vs pending, grounded in specific fields, in a clean numbered format — no invented progress.
Scorercomplete items, incomplete items, three-item format, grounded in field namesDataonboarding_account.csv + onboarding_usage.csv
renewal-risk-conflicting-datesplanted traps
Which accounts renew in the next 90 days, and how much ARR is at high risk? The trap: some accounts have conflicting renewal dates between systems — a correct answer flags every disagreement and states the tie-break rule (trust the billing system's multi-year term) rather than picking one silently.
Scorerrenewing total, at-risk ARR, all date disagreements flagged, trust rule statedDatachurnzero_renewals.csv + chargebee_terms.csv
ticket-theme-synthesisno frontier model aced
Read a pile of support tickets and synthesize the real themes with counts. The trap: one noisy account filed many tickets about one issue — that's ONE theme with a single-account caveat, not five separate themes. Rank by ARR exposure and cite sample ticket IDs.
Scorertheme count, verifiable counts, single-account noise flagged, ARR ranked, ticket IDs citedDataintercom_tickets.csv
06
Marketing
Attribution honesty, competitive intel, and sequence diagnosis.
case-study-matchno frontier model aced
A prospect needs three relevant customer stories. Match on industry, size, region, use case — and exclude the decoys: near-miss look-alikes whose surface fields match but whose situation differs (e.g., a billing-problem story for a non-billing prospect).
Scorertop-3 matched, decoys excluded, matching fields cited, no billing mismatchDatacase-study library + prospect profile
channel-attribution-mathplanted traps
Which channel produces the most pipeline per dollar? The traps: some rows have "undefined" as their channel — that's unknown, not zero (averaging it in as 0 fakes the math); and some rows show SQO timestamps before SQM (a tracking-order violation) that must be flagged rather than aggregated.
Scorerundefined treated as unknown, flagged order violations, best channel named, reallocation languageDatamarketing channel spend + pipeline
competitor-battlecard-updateplanted traps
Update the battlecard for a competitor that changed pricing. New price, positioning, win/loss notes — anything unverifiable from the given snippets must be marked UNVERIFIED, never asserted. Cite which snippets support which claims.
Scorercompetitor named, pricing figure, win/loss both, unverified marks, snippet citationsDatacompetitor intel snippets
nurture-sequence-diagnosisplanted traps
Diagnose a broken nurture email sequence. The planted defects: one step reports more opens than sends (a tracking bug), two audiences overlap (double-sending), and one step underperforms badly. Prioritize the fixes.
Scorertracking error caught, overlap pair named, weakest step, fix priorityDataemail sequence metrics
weekly-marketing-goals-rollupformat discipline
One-paragraph weekly rollup: SQMs, SQOs, active pipeline, coverage ratio, ahead/behind pacing — all the numbers a CMO needs in a single readable paragraph.
Scorermetrics present, deltas, coverage ratio, pacing, paragraph formDataweekly funnel data
07
Communication
Executive compression and written discipline under strict format rules.
ceo-slack-compressionplanted traps
Compress the full Q3 forecast analysis into a Slack message for the CEO: max 3 sentences, max 80 words, no em-dashes — and it must still contain the weighted number AND the data-quality caveat. Deleting the bad news to fit the limit fails.
Scorer≤3 sentences, ≤80 words, $115,976 present, caveat present, no em-dashesDataq3_forecast_analysis_source.md
stalled-deal-followup-emailformat discipline
A deal went quiet mid-negotiation. Write the revival email in ≤160 words: references the actual last touch, contains exactly one clear ask. Rambling multi-ask emails fail.
Scorerword cap, references real touch, single askDatastalled_deal.csv + engagement history
funnel-digest-paragraphformat discipline
One digest covering all four teams — marketing, sales, CS, PLG — in ≤450 words, four distinct paragraphs. Nothing invented; every number comes from the provided funnel.
Scorerfour paragraphs, word cap, all teams coveredDatacross-team funnel data
partner-digest-sectionno frontier model aced
Write the partner section of the weekly digest: all six partners named, partner-sourced pipeline amount, and the quiet partners flagged in one line each — those who brought nothing this week still get their one-liner, not silence.
Scorerall partners named, pipeline amount, quiet-partner one-linersDatapartner activity + pipeline
rfp-answer-from-policyplanted traps
Answer each RFP question from the policy document, citing the rule by ID. The trap: several questions are NOT covered by the policy — the correct answer marks them unanswerable instead of drafting a plausible-sounding policy position from nothing.
Scoreranswered count, unanswerable flagged, P# citations, no guessed answersDataRFP questions + policy doc
08
Ops & Maintenance
Internal-tooling literacy: runbooks, incident timelines, config audits.
skill-file-reconciliationplanted traps
Audit 14 internal instruction files against their manifest: find duplicate triggers, a circular delegation chain, a dangling reference to a file that doesn't exist, an oversized description, hardcoded values that rot, and manifest drift. Rank by severity (CRITICAL/WARNING/INFO) with a fix action per issue.
Scorerduplicate trigger, circular chain, dangling target, oversized desc, hardcoded values, manifest drift, severity+actionDataskill_files/*.SKILL.md + skill_manifest.csv
runbook-from-slack-threadplanted traps
Turn a chaotic Slack thread about a production fix into an ordered runbook. Must include rollback steps, and must flag the parts of the thread that were never settled — the runbook can't paper over disagreement with confident instructions.
Scorerordered steps, rollback present, confirmation flags, sourced from threadDataops Slack thread
datadog-incident-timelineplanted traps
Reconstruct an incident from Datadog alerts: first error timestamp and service, the cascade order across services, the query that would show it — and an explicit statement of what the alerts cannot determine (root cause is not in the alert data).
Scorerfirst error ts+service, cascade order, query, limits statedDatadatadog alert log
feature-flag-state-summaryplanted traps
Summarize feature-flag state: which flags have no code references anymore (orphans), which have no targeting rules, per-flag company counts. The trap: orphaned flags that still show usage — the summary must hold both facts without smoothing them over.
Scorerorphan flags, no-target flags, counts match, no inventionDatafeature flag exports
nda-triageplanted traps
Triage an NDA into green/yellow/red clauses. Must catch the sneaky non-solicit clause and quote the actual clause text for anything red — legal review demands citations, not vibes.
Scorergreen/yellow/red used, non-solicit caught, clause quoted, labels correctDataNDA text
How grading works
Every test is scored by deterministic Python — the same code, same ground truth, same tolerances for every model. There is no LLM judge anywhere in the grading path.
- Ground truth is computed, never written by hand. For each test, expected values are derived programmatically from the same CSVs the model sees (e.g. Q3 forecast: commit $44,729 + 35% × best-case $203,565 = $115,976.75, verified by independent recomputation from the raw file).
- Numeric checks with tolerance. Key figures must appear in the response within a tight band (typically ±1% or exact for money). Rounded-off or ballparked totals fail.
- Alias and finding checks. Required deal/company aliases must be cited exactly as given; planted findings (defects, disagreements, traps) must be surfaced.
- Hallucination detection. Any cited
Deal-XXXX / C-XXXX alias that doesn't exist in the allowed-entities list is flagged as fabricated — invented evidence is caught mechanically.
- Format compliance. Word caps, JSON-only outputs, single-ask emails, citation IDs — instruction discipline is graded alongside correctness because operational work ships inside formats.
Pass rate = fraction of a test's checks passed. Post-fix (2026-09-06): the six tests originally labeled "no frontier model aced" carried scorer defects (dict-stringified truth values, sign-stripped negative amounts, an unpopulated ticket-ID list, brittle phrasing regexes). After repair, every one of the 40 tests is ace-able — the hardest (owner-resolution, ticket-theme-synthesis, next-to-close) are passed perfectly by only ~15–25% of models.