home · ← back to all models
openai

openai/gpt-5.4-mini

Mean 0.935 · 30/40 perfect tests · $3.21 total · each card: the question → the correct answer (gold) → this model's actual answer

data-crm
stage-hygiene-auditPASS 1.00

? The Question

Audit stage hygiene on the open pipeline below. Rules: a deal at DS2 or later must have a substantive why-buys (15+ characters, not a placeholder such as tbd, n/a, see notes). A why_buys value of the form LONG(n) means the real text was present but truncated for display; treat it as present and substantive. Every deal should have a DS1 entry timestamp. A deal whose stage entry timestamps are out of order regressed (an earlier-stage timestamp is later than a later-stage timestamp). Report: percent of DS2+ deals non-compliant on why-buys overall and by owner (use the owners table for names), count of deals with no DS1 timestamp, count that regressed, and three process fixes. State definitions before numbers.
cost $0.1005 · 80s · in 40,949 / out 15,084 tok
✓ noncomp_pct✓ no_ds1_0✓ regressed_0✓ per_owner_breakdown✓ definitions_first✓ three_fixes

✓ Correct Answer (ground truth)

noncomp_pct: 81.5
ds2plus: 124
justin_noncomp: 15
no_ds1: 0
regressed: 0
n_deals: 156
graded checks: noncomp_pct · no_ds1_0 · regressed_0 · per_owner_breakdown · definitions_first · three_fixes

Model's actual answer

show the model's full answer
Definitions:
- DS2+ = deals in DS2, DS3, DS4, or DS5.
- Non-compliant why-buys = DS2+ deal with no substantive why_buys; LONG(n) counts as present; placeholders like tbd / n/a / see notes do not.
- Regressed = stage-entry timestamps are out of order (an earlier stage timestamp is later than a later-stage timestamp).

Overall why-buys non-compliance:
- 101 / 124 = 81.45%

By owner:
- Bryce Harmon: 15 / 18 = 83.33%
- Dana Mercer: 18 / 22 = 81.82%
- Alex Franklin: 48 / 61 = 78.69%
- Cole Ingram: 19 / 22 = 86.36%
- Farid Osman: 1 / 1 = 100.00%

Other hygiene counts:
- Deals with no DS1 timestamp: 0
- Deals that regressed: 0

Three process fixes:
- Block DS2 entry unless why_buys is substantive.
- Make DS1 timestamp required on every deal record, with validation on save.
- Add an automated stage-order check to flag any timestamp regressions immediately.
owner-resolutionPASS 1.00

? The Question

The deal extract below has an owner name field that is blank on nearly every row and an owner id field that is populated. Using the owners table, resolve each open deal to an owner name. Report: number of deals resolved, deals whose owner id has no match in the owners table (list them), the owner ids that map to archived/deactivated owners (list their names), and the total pipeline amount per resolved owner. Do not guess a name for any unmatched id.
cost $0.0939 · 49s · in 39,694 / out 13,810 tok
✓ resolved_count✓ unmatched_none✓ deactivated_flagged✓ jake_amount✓ justin_amount✓ no_invented

✓ Correct Answer (ground truth)

resolved: 156
jake_amount: 1054144.0
justin_amount: 624310.0
per_owner: Bryce Harmon=1054144.0; Alex Franklin=624310.0; Dana Mercer=341195.0; Cole Ingram=288161.43; Farid Osman=4134.0; Elena Sinclair=2100.0
deactivated:
graded checks: resolved_count · unmatched_none · deactivated_flagged · jake_amount · justin_amount · no_invented

Model's actual answer

show the model's full answer
Resolved deals: 156 / 156

Unmatched owner ids: none

Owner ids that map to archived/deactivated owners: none in the deal extract
- The archived owners in owners.csv are Gavin Porter (1520255671) and Hugo Lindqvist (77260721), but neither owner_id appears on any deal row.

Total pipeline amount by resolved owner (sum of matched deal amounts):
- Bryce Harmon (119337721): 35 deals; 1,054,144.00
- Dana Mercer (83155923): 24 deals; 341,195.00
- Alex Franklin (84342457): 67 deals; 624,310.00
- Cole Ingram (83155924): 22 deals; 288,161.43
- Farid Osman (716654662): 7 deals; 4,134.00
- Elena Sinclair (701163055): 1 deal; 2,100.00

Arithmetic check:
- 35 + 24 + 67 + 22 + 7 + 1 = 156 resolved deals
- 1,054,144.00 + 341,195.00 + 624,310.00 + 288,161.43 + 4,134.00 + 2,100.00 = 2,314,044.43 total pipeline
gong-hubspot-join-integrity0.75

? The Question

Check the integrity of the call-to-deal mapping below. The gong table lists calls in the last 90 days keyed to HubSpot deal ids; deals_open lists the current open default-pipeline deals. Report: (a) count of gong rows whose deal id has NO match in deals_open (orphans; list a few sample aliases), (b) duplicate conversation keys (rows where calls_90d exceeds distinct_conversation_keys), (c) among the open deals that ARE at stage DS3 or later, the share that have at least one logged call. Give sample aliases for the orphan finding, all taken from the data. Note: gong_owner_id is a Gong user id, not a HubSpot owner id.
cost $0.1029 · 59s · in 32,487 / out 16,734 tok
✓ orphan_count✗ dup_keys_0✓ ds3_coverage✓ sample_ids_present

✓ Correct Answer (ground truth)

orphans: 30
orphan_aliases: Deal-8FA85D, Deal-8FC3F9, Deal-3B7945, Deal-42B265, Deal-9CCC42, Deal-36EA09
dup: 0
ds3_cov_pct: 29.4
graded checks: orphan_count · dup_keys_0 · ds3_coverage · sample_ids_present

Model's actual answer

show the model's full answer
(a) Orphans:
- Count = 30 gong rows with hs_deal_id not found in deals_open.
- Sample aliases from the data: Deal-8FA85D, Deal-8FC3F9, Deal-3B7945, Deal-42B265, Deal-9CCC42, Deal-36EA09, Deal-9A43B4, Deal-605F3C.

(b) Duplicate conversation keys:
- Count = 0 rows.
- Arithmetic check: for every gong row, calls_90d = distinct_conversation_keys, so there are no rows where calls_90d > distinct_conversation_keys.

(c) Open deals at stage DS3 or later with at least one logged call:
- Open DS3+ deals = 85
- DS3+ deals with ≥1 gong row = 25
- Share = 25 / 85 = 0.2941176471 = 29.4%
snowflake-sql-trapped-catalogPASS 1.00

? The Question

Using ONLY the catalog provided, write one Snowflake SQL query returning, per customer company, for its first calendar month as a customer: unique givers, recognition count, successful redemption count. Read the catalog notes carefully: some tables are stale or unpopulated and must be avoided, and one documented filter (excluding deleted givers) must NOT be applied to historical giving counts. Put a comment above each FROM or JOIN explaining the table choice. Return only fenced SQL.
cost $0.0237 · 20s · in 15,197 / out 2,726 tok
✓ avoids_stale_deals✓ avoids_unpopulated_live✓ uses_cohort_summary✓ succeeded_filter✓ no_giver_deletion_filter✓ returns_sql

✓ Correct Answer (ground truth)

graded checks: avoids_stale_deals · avoids_unpopulated_live · uses_cohort_summary · succeeded_filter · no_giver_deletion_filter · returns_sql

Model's actual answer

show the model's full answer
```sql
SELECT
  c.COMPANY_ID AS customer_company,
  DATE_TRUNC('month', c.FIRST_SUB_PAYMENT_DATE) AS first_calendar_month_as_customer,
  c.M1_USERS AS unique_givers,
  c.M1_GIVES AS recognition_count,
  c.M1_REDEMPTIONS AS successful_redemption_count
FROM PRODUCTION.PLG.COMPANY_COHORT_SUMMARY c
-- Use the one-row-per-company cohort summary because it is the only provided source with first-subscription timing and precomputed month-1 metrics; it avoids stale/unpopulated tables.
WHERE c.FIRST_SUB_PAYMENT_DATE IS NOT NULL;
```
crm-completeness-enrichmentPASS 1.00

? The Question

Audit the CRM extract for completeness and propose fixes. Deals need owner, stage, amount, close date, why-buys; companies need industry, employee count, HQ country; contacts need email, title, persona. Report completeness percent per field. Identify duplicate company clusters (name variants, shared domain) with a survivor per cluster. List invalid emails and domain mismatches. Fill missing company fields only where the enrichment export has a matching row; where CRM and enrichment disagree list both and recommend a source. Never invent a value. End with the 10 fixes with the most pipeline amount at stake.
cost $0.0571 · 55s · in 10,026 / out 10,219 tok
✓ dup_clusters✓ invalid_emails✓ survivor_language✓ completeness_pct✓ no_invent

✓ Correct Answer (ground truth)

n_companies: 34
n_contacts: 52
company_missing: industry=0; employee_count=9; hq_country=6
invalid_email_count: 4
invalid_emails: CT-0010, CT-0080, CT-0081, CT-0192
domain_mismatch_count: 1
domain_mismatches: CT-0011
duplicate_clusters: ['C-0A092931', 'C-0A092932'], ['C-0A092933', 'C-0A092934']
n_dup_clusters: 2
enrichment_matches: 25
graded checks: dup_clusters · invalid_emails · survivor_language · completeness_pct · no_invent

Model's actual answer

show the model's full answer
No deals extract was provided, so I cannot audit deal completeness or rank fixes by pipeline amount at stake from the supplied data.

Companies completeness
- industry: 34/34 = 100.00%
- employee_count: 25/34 = 73.53%
- hq_country: 28/34 = 82.35%

Contacts completeness
- email: 52/52 = 100.00%
- title: 39/52 = 75.00%
- persona: 37/52 = 71.15%

Duplicate company clusters, using shared domain only
- acme-corp.com: C-0A092931, C-0A092932
  - survivor: C-0A092931
- globex.io: C-0A092933, C-0A092934
  - survivor: C-0A092934

Invalid emails
- CT-0010, C-66D1FC, user0@
- CT-0080, C-92D97D, user0@
- CT-0081, C-92D97D, user1@
- CT-0192, C-425E2A, user2@

Domain mismatch
- CT-0011, C-66D1FC: email domain other-domain.com does not match contact domain 66d1fc.com

Company fields that can be filled from the enrichment export where there is a matching row
- C-EC3025, ec3025.com: employee_count blank -> 400
- C-96039F, 96039f.com: employee_count blank -> 400
- C-44EA29, 44ea29.com: employee_count blank -> 400
- C-D04904, d04904.com: employee_count blank -> 400
- C-B23205, b23205.com: employee_count blank -> 400
- C-60C75F, 60c75f.com: employee_count blank -> 400
- C-7BBDFA, 7bbdfa.com: employee_count blank -> 400
- C-50D386, 50d386.com: employee_count blank -> 400

CRM vs enrichment disagreements where both have values
Recommend source: ZoomInfo enrichment, because it is the only normalized external enrichment file provided.
- C-66D1FC, 66d1fc.com
  - industry: CRM tech vs enrichment Computer Software
  - hq_country: CRM US vs enrichment United States
- C-950043, 950043.com
  - hq_country: CRM US vs enrichment United States
- C-EC3025, ec3025.com
  - industry: CRM Technology vs enrichment Computer Software
  - hq_country: CRM USA vs enrichment United States
- C-96039F, 96039f.com
  - hq_country: CRM USA vs enrichment United States
- C-44EA29, 44ea29.com
  - industry: CRM tech vs enrichment Computer Software
- C-92D97D, 92d97d.com
  - industry: CRM Technology vs enrichment Computer Software
- C-D04904, d04904.com
  - industry: CRM Technology vs enrichment Computer Software
- C-77A95A, 77a95a.com
  - industry: CRM Technology vs enrichment Computer Software
  - hq_country: CRM US vs enrichment United States
- C-AA8DDA, aa8dda.com
  - industry: CRM Technology vs enrichment Computer Software
- C-B23205, b23205.com
  - hq_country: CRM US vs enrichment United States
- C-E51FB7, e51fb7.com
  - hq_country: CRM USA vs enrichment United States
- C-D0662E, d0662e.com
  - hq_country: CRM US vs enrichment United States
- C-B25F40, b25f40.com
  - industry: CRM Technology vs enrichment Computer Software
- C-60C75F, 60c75f.com
  - industry: CRM tech vs enrichment Computer Software
- C-425E2A, 425e2a.com
  - industry: CRM Tech  vs enrichment Computer Software
  - hq_country: CRM USA vs enrichment United States
- C-2D7423, 2d7423.com
  - hq_country: CRM USA vs enrichment United States

10 highest-priority fixes I can identify from the provided files, but not by pipeline amount because no deal data was supplied
1. Fix invalid emails for CT-0010, CT-0080, CT-0081, CT-0192.
2. Resolve the domain mismatch on CT-0011.
3. Populate missing employee_count on C-EC3025 from enrichment.
4. Populate missing employee_count on C-96039F from enrichment.
5. Populate missing employee_count on C-44EA29 from enrichment.
6. Populate missing employee_count on C-D04904 from enrichment.
7. Populate missing employee_count on C-B23205 from enrichment.
8. Populate missing employee_count on C-60C75F from enrichment.
9. Populate missing employee_count on C-7BBDFA from enrichment.
10. Populate missing employee_count on C-50D386 from enrichment.
deal-intelligence
closed-lost-classification0.80

? The Question

Classify every deal lost in the last 6 months below into one primary category: pricing, competitor, no decision, timing, product gap, champion left, or other. Also assign a side (Bonusly, buyer, unknown). Then summarize: category counts, the side split, the number of deals where the structured closed_lost_tag clearly disagrees with the free-text reason, and the two patterns most worth acting on. Base everything only on the tag and free-text provided.
cost $0.4137 · 698s · in 64,606 / out 72,588 tok
✓ total_90✗ competitor_top✓ category_counts_present✓ side_split✓ two_patterns fabricated: Deal-50E5D

✓ Correct Answer (ground truth)

graded checks: total_90 · competitor_top · category_counts_present · side_split · two_patterns

Model's actual answer

show the model's full answer
Excluded from the 6-month summary: Deal-5885B9 — the row is malformed and close_date is missing, so I can’t safely confirm it belongs in the last-6-month set.

Classifications for the 89 classifiable deals:

Timing — buyer
Deal-DB0AAC, Deal-91A056, Deal-29326C, Deal-831B7B, Deal-39E25C, Deal-B6AC09, Deal-E6E80A, Deal-B038F0, Deal-175756, Deal-15DA99, Deal-F4AF5D, Deal-79B7A1, Deal-69CF3D, Deal-ECBF89, Deal-D1A623, Deal-9F176A, Deal-55867E, Deal-B3ABED, Deal-50E5D, Deal-7FBAC6, Deal-F325A5

Competitor — buyer
Deal-F7F635, Deal-422BA6, Deal-381C8C, Deal-F1E8A6, Deal-DDAB52, Deal-ACE061, Deal-0F96AA, Deal-242273, Deal-A2C349, Deal-C7156E, Deal-8A0992, Deal-D0C698, Deal-EECC02, Deal-47F1A1, Deal-BF2A98, Deal-1E7DA9, Deal-286F9C, Deal-369281, Deal-64B19A

No decision — buyer
Deal-AC944F, Deal-214060, Deal-21B045, Deal-988493, Deal-F308CA, Deal-70F704, Deal-4664E1, Deal-583ADB, Deal-7CB44D, Deal-AFA56C, Deal-3F86A0, Deal-096750, Deal-ABD14C, Deal-79E61A, Deal-AE7C4E, Deal-DAB4F1, Deal-B4B50F, Deal-2A292B, Deal-D1AABF, Deal-FEDBCB, Deal-2BBA21, Deal-386F6E, Deal-D48E0B, Deal-5DB9B0, Deal-5885B9

Pricing — Bonusly
Deal-7ED004, Deal-ED9AE7, Deal-F97C37, Deal-13E9CF, Deal-1BCA50, Deal-5AD03E, Deal-5E64CE, Deal-7B2236, Deal-C33D91, Deal-DAFB82, Deal-FAC17C, Deal-8A119B, Deal-DC77FE, Deal-2FEDDB

Product gap — Bonusly
Deal-8E27DA, Deal-242273, Deal-A2C349, Deal-9048EB, Deal-3618CC, Deal-981AD4, Deal-BB78F3, Deal-DC77FE

Champion left — buyer
Deal-E0441F, Deal-413C56, Deal-F325A5

Other — unknown
Deal-5DB9B0

Summary:

Category counts
timing: 20/89
competitor: 19/89
no decision: 25/89
pricing: 14/89
product gap: 8/89
champion left: 3/89
other: 1/89

Side split
buyer: 66/89
Bonusly: 22/89
unknown: 1/89

Structured closed_lost_tag vs free-text disagreements
22/89

Arithmetic:
20 + 19 + 25 + 14 + 8 + 3 + 1 = 90, but one row is malformed and excluded from the 6-month summary, so the classifiable set is 89.
Buyer side = 20 + 19 + 25 + 3 = 67 before excluding the malformed row; after excluding Deal-5885B9 it is 66.
Bonusly side = 14 + 8 = 22.
Unknown side = 1.

Two patterns most worth acting on:
1. Delay/no-decision is the biggest block.
Timing + no decision = 20 + 25 = 45 of 89 classifiable deals.
That is 50.6% of the set.

2. The text shows repeatable commercial/product friction inside competitor losses.
Clear pricing + product-gap pressure = 14 + 8 = 22 deals.
Recurring themes in the free text are budget approval, budget cuts, budget access, surveys, customization, and broader offerings.
pipeline-tieringPASS 1.00

? The Question

Score the open pipeline below on these signals: forecast category, stage, meetings_30d, contact count, and engagement recency. Assign exactly one tier per deal: LOCK, ACTION, BUILD, REVIVE, WATCH, or RISKY (RISKY = the forecast category disagrees with the engagement evidence). Never tier a deal LOCK if it has zero meetings_30d. Do NOT list every deal. Return ONLY a compact JSON object with exactly these keys: - "tier_counts": object mapping each tier to the number of deals in it (must sum to the total deal count), - "tier_examples": object mapping each tier to up to 3 example deal aliases from the data, - "risky_deals": array of the deal aliases you tiered RISKY, - "lock_violations": number of deals tiered LOCK that have zero meetings_30d (should be 0), - "pipeline_shape": one short paragraph on the overall pipeline shape. Note: inbound_emails_30d is 0 for every row due to a data defect, so treat meetings_30d as the inbound signal.
cost $0.5221 · 292s · in 247,804 / out 29,403 tok
✓ json_returned✓ tier_vocabulary✓ tier_counts_sum✓ risky_deals_listed✓ no_lock_violation_claimed

✓ Correct Answer (ground truth)

total: 156
graded checks: json_returned · tier_vocabulary · tier_counts_sum · risky_deals_listed · no_lock_violation_claimed

Model's actual answer

show the model's full answer
{"tier_counts":{"LOCK":3,"ACTION":20,"BUILD":39,"WATCH":80,"REVIVE":7,"RISKY":7},"tier_examples":{"LOCK":["Deal-D348E1","Deal-C26D20","Deal-403845"],"ACTION":["Deal-25F752","Deal-547B2B","Deal-944310"],"BUILD":["Deal-D73B89","Deal-523604","Deal-2B39B0"],"WATCH":["Deal-C6FE92","Deal-93C8BF","Deal-036E80"],"REVIVE":["Deal-2D1F1B","Deal-F0EBBB","Deal-1BEEBF"],"RISKY":["Deal-B7EBD1","Deal-0660B4","Deal-BA571A"]},"risky_deals":["Deal-B7EBD1","Deal-0660B4","Deal-BA571A","Deal-7BBDFA","Deal-4A13AD","Deal-690476","Deal-635B8E"],"lock_violations":0,"pipeline_shape":"Bottom-heavy: 3 LOCKs are the only clearly healthy late-stage deals, 20 ACTION deals are active but still missing the meeting signal, 39 BUILD deals show real momentum in earlier stages, and the largest share sits in 80 WATCH deals with mostly thin engagement. 7 REVIVE deals look dormant, while 7 RISKY deals have optimistic forecast categories that are not supported by the engagement evidence."}
call-transcript-extractionPASS 1.00

? The Question

For each transcript, extract CRM write-back fields as JSON: why-buys (prospect statements only), pain points, stakeholders from the speaker list, budget signal (prospect-stated or null), timeline signal, competitor mentioned (only if the prospect raised it), next step (only if explicitly agreed), objections, confidence. Never fill a field from what the rep said or hoped.
cost $0.0347 · 31s · in 16,059 / out 5,025 tok
✓ json_emitted✓ budget_signal✓ null_discipline✓ stakeholders✓ competitor_prospect_only

✓ Correct Answer (ground truth)

rep: Alex Franklin
per_transcript: {"TX-001": {"why_buys": ["automating anniversary and birthday awards"], "pain_points": ["HR team of three cannot keep up with it manually", "track everything in a spreadsheet, and people slip through
rep_sourced_nulls: {"TX-003": ["budget_signal \u2014 only the rep mentioned pricing/budget"], "TX-004": ["next_step \u2014 rep proposed, prospect said 'no promises'"], "TX-006": ["budget_signal \u2014 rep decoy, prospec
allowed_new: TX-001, TX-002, TX-003, TX-004, TX-005, TX-006, Achievers, Bucketlist, Nectar
graded checks: json_emitted · budget_signal · null_discipline · stakeholders · competitor_prospect_only

Model's actual answer

show the model's full answer
[
  {
    "transcript_id": "TX-001",
    "deal_alias": "Deal-CFE7F4",
    "stakeholders": [
      "Prospect (VP People)",
      "Prospect (HR Admin)"
    ],
    "why-buys": [
      "The big win for us would be automating anniversary and birthday awards."
    ],
    "pain_points": [
      "Our HR team of three cannot keep up with it manually.",
      "Right now we track everything in a spreadsheet.",
      "People slip through the cracks.",
      "We need SSO and audit logs for IT to sign off."
    ],
    "budget_signal": "We have about $40k earmarked for engagement tools this fiscal year.",
    "timeline_signal": "Ideally we would have this live before open enrollment in November.",
    "competitor_mentioned": "Achievers",
    "next_step": "Let's do the security review on September 12.",
    "objections": [
      "We looked at Achievers last year, but it was too heavy for a team our size.",
      "We need SSO and audit logs for IT to sign off."
    ],
    "confidence": {
      "score": "100%",
      "arithmetic": "8/8 directly stated extracted fields"
    }
  },
  {
    "transcript_id": "TX-002",
    "deal_alias": "Deal-70BB30",
    "stakeholders": [
      "Prospect (Head of Total Rewards)",
      "Prospect (CFO)"
    ],
    "why-buys": [
      "We want to tie recognition to retention for our hourly workforce."
    ],
    "pain_points": [
      "Regretted turnover there is over 30%."
    ],
    "budget_signal": "Finance has approved a $25k pilot budget for this quarter.",
    "timeline_signal": "We want a decision by end of September.",
    "competitor_mentioned": null,
    "next_step": "Yes — send the pilot agreement and we'll route it to legal this week.",
    "objections": [
      "Integration with Workday has to be rock solid."
    ],
    "confidence": {
      "score": "100%",
      "arithmetic": "7/7 directly stated extracted fields"
    }
  },
  {
    "transcript_id": "TX-003",
    "deal_alias": "Deal-530B50",
    "stakeholders": [
      "Prospect (People Ops Manager)"
    ],
    "why-buys": [
      "We need to make recognition visible across our 12 retail locations."
    ],
    "pain_points": [
      "Store managers have zero budget autonomy for on-the-spot recognition today.",
      "There is no rush on our side until Q1."
    ],
    "budget_signal": null,
    "timeline_signal": "Honestly there's no rush on our side until Q1.",
    "competitor_mentioned": "Bucketlist",
    "next_step": "Yes, let's schedule a call with our CEO — I'll send two times.",
    "objections": [
      "The CEO has to be sold first — she decides anything people-related."
    ],
    "confidence": {
      "score": "100%",
      "arithmetic": "6/6 directly stated extracted fields"
    }
  },
  {
    "transcript_id": "TX-004",
    "deal_alias": "Deal-180D02",
    "stakeholders": [
      "Prospect (VP People)",
      "Prospect (IT Security Lead)"
    ],
    "why-buys": [
      "We want to consolidate three separate recognition tools into one."
    ],
    "pain_points": [
      "We're paying for three tools and none of them talk to our HRIS."
    ],
    "budget_signal": "If it's under $15k annually, I can approve it without going to the board.",
    "timeline_signal": "Our procurement cycle runs six to eight weeks minimum.",
    "competitor_mentioned": null,
    "next_step": null,
    "objections": [
      "Our procurement cycle runs six to eight weeks minimum.",
      "The security review took three months for our last vendor."
    ],
    "confidence": {
      "score": "100%",
      "arithmetic": "6/6 directly stated extracted fields"
    }
  },
  {
    "transcript_id": "TX-005",
    "deal_alias": "Deal-F8767A",
    "stakeholders": [
      "Prospect (HR Director)",
      "Prospect (People Ops Coordinator)"
    ],
    "why-buys": [
      "Two things: automate service milestones, and give us analytics on recognition equity across departments."
    ],
    "pain_points": [
      "Our night-shift teams feel invisible.",
      "Their engagement scores run 20 points lower.",
      "Our exec team is skeptical after a failed rollout two years ago."
    ],
    "budget_signal": "We have $12k approved under our engagement line.",
    "timeline_signal": "We need this running before our January all-hands.",
    "competitor_mentioned": "Nectar",
    "next_step": "Yes — come present to our exec team on October 2.",
    "objections": [
      "We're mid-pilot with Nectar right now, so you'd need to beat that experience.",
      "Our exec team is skeptical after a failed rollout two years ago."
    ],
    "confidence": {
      "score": "100%",
      "arithmetic": "7/7 directly stated extracted fields"
    }
  },
  {
    "transcript_id": "TX-006",
    "deal_alias": "Deal-EE195F",
    "stakeholders": [
      "Prospect (HR Manager)"
    ],
    "why-buys": [
      "I'd love to cut the admin time on service awards."
    ],
    "pain_points": [
      "I personally spend five hours a month ordering and shipping plaques.",
      "Budget isn't the issue — time is.",
      "Our COO usually prefers building things in-house."
    ],
    "budget_signal": null,
    "timeline_signal": "Our fiscal year resets in February, so a Q1 start is realistic.",
    "competitor_mentioned": "just doing it internally",
    "next_step": "Yes — send the one-page overview and I'll forward it to our COO this week.",
    "objections": [
      "Our COO usually prefers building things in-house."
    ],
    "confidence": {
      "score": "100%",
      "arithmetic": "6/6 directly stated extracted fields"
    }
  }
]
next-to-close0.75

? The Question

Which 3 deals are closest to signature right now? Use the CRM data and the Slack thread; Slack often has the real status before the CRM does. For each: deal, amount, why it is close, what is left. Under 250 words.
cost $0.0123 · 19s · in 1,561 / out 2,248 tok
✓ exactly_three✗ top3_named✓ slack_first✓ blocked_excluded

✓ Correct Answer (ground truth)

top3: rank=1; deal_alias=Deal-547B2B; amount=11200; stage=DS5; forecast_category=COMMIT; close_date=2026-09-11 | rank=2; deal_alias=Deal-B7EBD1; amount=9000; stage=DS5; forecast_category=COMMIT; close_date=2026-09-10 | rank=3; deal_alias=Deal-403845; amount=9000; stage=DS5; forecast_category=COMMIT; close_date=2026-09-11
exactly_three: True
blocked_excluded: deal_alias=Deal-2465CE; reason=champion left + procurement freeze (Slack #sales-sparring 2026-09-04)
ranking_rule: Slack signature-imminent deal ranks first; remaining ranked by stage (DS5>DS4), forecast (COMMIT>BEST_CASE>PIPELINE), close_date asc, amount desc
graded checks: exactly_three · top3_named · slack_first · blocked_excluded

Model's actual answer

show the model's full answer
1) Deal-547B2B — $11,200. Why close: Slack says “redlines came back clean,” “signing page is out,” and “signing tomorrow” = signature-imminent. What’s left: signature only.

2) Deal-403845 — $9,000. Why close: Slack says it’s “moving fine” and “the order form is with their finance team.” What’s left: finance-team review/approval and then signature.

3) Deal-A2B47C — $6,360. Why close: Slack says it’s “still warm” and on “normal legal-review pace,” and CRM has it at DS5/COMMIT. What’s left: legal review.

Arithmetic: $11,200 + $9,000 + $6,360 = $26,560.
product-gap-detectionPASS 1.00

? The Question

Surface true product gaps raised by prospects in active deals. For each candidate: quote the prospect line, classify as real gap, plan gate, or rollout/enablement issue using the product docs excerpt, and name the deal and amount. Only real gaps go in the summary. Rep language does not count as prospect voice.
cost $0.0176 · 13s · in 14,983 / out 1,412 tok
✓ real_gaps✓ plan_gate_classified✓ enablement_classified✓ gap_summary

✓ Correct Answer (ground truth)

real_gaps: transcript_id=TG-001; deal_alias=Deal-EC3025; amount=62000; feature=SCIM user provisioning; quote=We need SCIM user provisioning — our security team will not approve any tool that requires manual account management.; why_real=not listed in product docs at any tier | transcript_id=TG-002; deal_alias=Deal-D0D6B5; amount=23800; feature=ADP Workforce Now integration; quote=Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us.; why_real=supported HRIS list is Workday/BambooHR/Gusto/Rippling only
plan_gates: transcript_id=TG-003; deal_alias=Deal-CFE7F4; amount=32000; feature=Custom report builder; tier_required=Enterprise; quote=I tried to build a custom report combining recognition frequency with turnover and couldn't — is that possible on our current plan?
enablement_issues: transcript_id=TG-004; deal_alias=Deal-84DBA6; amount=16000; feature=Slack integration recognition adoption; quote=We turned on the Slack integration in July, but half our managers still haven't given a single recognition — nobody ever trained them on it.; why_enablement=Slack integration exists on all plans; managers were never trained
excluded_rep_voice: transcript_id=TG-005; line=Full transparency — we don't have a native mobile app for hourly workers yet, though it's on the roadmap.; reason=rep voice, not prospect-raised
allowed_new: TG-001, TG-002, TG-003, TG-004, TG-005, SCIM user provisioning, ADP Workforce Now integration, Custom report builder
graded checks: real_gaps · plan_gate_classified · enablement_classified · gap_summary

Model's actual answer

show the model's full answer
Using only the provided data:

Classification of prospect-raised candidates

1) Deal-EC3025
- Prospect line: “We need SCIM user provisioning — our security team will not approve any tool that requires manual account management.”
- Classification: real gap
- Why: product docs say “SCIM user provisioning ... [is] NOT currently listed as supported capabilities.”
- Deal amount: not provided

2) Deal-D0D6B5
- Prospect line: “Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us.”
- Classification: real gap
- Why: product docs say “ADP Workforce Now integrations are NOT currently listed as supported capabilities.”
- Deal amount: not provided

3) Deal-CFE7F4
- Prospect line: “I tried to build a custom report combining recognition frequency with turnover and couldn't — is that possible on our current plan?”
- Classification: plan gate
- Why: product docs say “Custom report builder” is Enterprise only.
- Deal amount: not provided

4) Deal-84DBA6
- Prospect line: “We turned on the Slack integration in July, but half our managers still haven't given a single recognition — nobody ever trained them on it.”
- Classification: rollout/enablement issue
- Why: product docs say Slack integration is supported on all plans; the issue is adoption/training, not capability.
- Deal amount: not provided

Summary: true product gaps only
- Deal-EC3025 — SCIM user provisioning
- Deal-D0D6B5 — ADP Workforce Now integration

Arithmetic:
- Prospect-voiced candidates reviewed: 4
- Real gaps: 2
- 2 / 4 = 0.5 = 50%
rep-performance
stale-pipeline-by-rep0.50

? The Question

The snapshot date is 2026-09-05. Using the open deals and the engagements-by-deal table, list every open deal with no logged email, call, or meeting in the last 7 days. Compute recency from the engagements table's last_email/last_call/last_meeting fields, NOT from any date field on the deal (the deal's last_contacted_field is unreliable). For each stale deal give: deal alias, owner name, stage, amount, days since last contact. Group by owner, order by amount descending within each owner. Then state, per owner, how many stale deals they carry and the total stale amount. (Do not write outreach text.)
cost $0.1565 · 89s · in 91,880 / out 13,999 tok
✓ stale_recall_ge_80✓ grouped_by_owner✓ jake_top_amount✗ kevin_stale_amt✗ jenn_stale_amt✗ stale_count

✓ Correct Answer (ground truth)

stale_count: 65
jake_amt: 626243.0
kevin_amt: 252905.03
jenn_amt: 261645.0
top_alias: Deal-2D1F1B
top_amount: 240000.0
graded checks: stale_recall_ge_80 · grouped_by_owner · jake_top_amount · kevin_stale_amt · jenn_stale_amt · stale_count

Model's actual answer

show the model's full answer
Days since last contact = 2026-09-05 minus the latest of last_email, last_call, last_meeting. If all three are blank, days since last contact is missing.

Bryce Harmon
Stale deals: 23
Total stale amount = 240000 + 99000 + 70000 + 57600 + 48000 + 45000 + 37440 + 36000 + 33600 + 32400 + 31500 + 24000 + 23400 + 21000 + 18700 + 16170 + 11500 + 11400 + 7560 + 6000 + 6000 + 5502 + 1 = 881773.00
- Deal-2D1F1B | DS1 | 240000 | 81
- Deal-66D1FC | DS1 | 99000 | 16
- Deal-950043 | DS1 | 70000 | 19
- Deal-31ED2A | DS1 | 57600 | 17
- Deal-77A95A | DS1 | 48000 | 8
- Deal-B23205 | DS1 | 45000 | 16
- Deal-7BBDFA | DS3 | 37440 | 46
- Deal-332637 | DS2 | 36000 | 9
- Deal-C9BB20 | DS1 | 33600 | 24
- Deal-91DAAF | DS1 | 32400 | 22
- Deal-1BEEBF | DS1 | 31500 | 19
- Deal-D56743 | DS3 | 24000 | 18
- Deal-C5658B | DS1 | 23400 | 16
- Deal-40522D | DS3 | 21000 | 19
- Deal-2D1F2C | DS3 | 18700 | 16
- Deal-B02199 | DS1 | 16170 | 26
- Deal-383A99 | DS3 | 11500 | 16
- Deal-F0EBBB | DS3 | 11400 | 24
- Deal-CC3E7F | DS1 | 7560 | 9
- Deal-72301F | DS1 | 6000 | 17
- Deal-E25A09 | DS1 | 6000 | 9
- Deal-C9C286 | DS2 | 5502 | 9
- Deal-012CB1 | DS1 | 1 | 23

Dana Mercer
Stale deals: 26
Total stale amount = 60000 + 60000 + 43875 + 27000 + 23400 + 20000 + 18900 + 16250 + 15000 + 10000 + 10000 + 9000 + 9000 + 8100 + 8000 + 7740 + 5125 + 5000 + 5000 + 4680 + 4200 + 3654 + 3000 + 3000 + 2310 + 1920 = 384154.00
- Deal-44EA29 | DS2 | 60000 | 10
- Deal-96039F | DS1 | 60000 | 19
- Deal-E51FB7 | DS2 | 43875 | 12
- Deal-B42F46 | DS1 | 27000 | 19
- Deal-BA3DDC | DS3 | 23400 | 15
- Deal-9DDE86 | DS2 | 20000 | 15
- Deal-215CCA | DS3 | 18900 | 17
- Deal-5EED42 | DS3 | 16250 | 11
- Deal-57887A | DS2 | 15000 | 8
- Deal-76FDB5 | DS2 | 10000 | 11
- Deal-A6FD51 | DS2 | 10000 | 11
- Deal-3974EB | DS4 | 9000 | 8
- Deal-B7EBD1 | DS5 | 9000 | 16
- Deal-F40F04 | DS2 | 8100 | 15
- Deal-B7314F | DS3 | 8000 | 15
- Deal-D4BFC5 | DS2 | 7740 | 9
- Deal-14FEC4 | DS3 | 5125 | 9
- Deal-798903 | DS2 | 5000 | 15
- Deal-87DDD1 | DS1 | 5000 | 19
- Deal-46ECC7 | DS1 | 4680 | 9
- Deal-F336B6 | DS3 | 4200 | 15
- Deal-341F2C | DS3 | 3654 | 11
- Deal-334A0B | DS2 | 3000 | 9
- Deal-CAF1D9 | DS1 | 3000 | 8
- Deal-681C6B | DS3 | 2310 | 11
- Deal-0660B4 | DS4 | 1920 | 16

Cole Ingram
Stale deals: 26
Total stale amount = 58529.25 + 45630 + 40000 + 32175 + 31750 + 18000 + 14946.75 + 12168 + 11193 + 10000 + 9360 + 7781.2 + 7225.4 + 7000 + 6947.5 + 5616 + 5616 + 4779.88 + 4212 + 4140 + 3360 + 3334.8 + 2700 + 1875 + 1330 + 700 = 350369.78
- Deal-D04904 | DS2 | 58529.25 | 11
- Deal-AA8DDA | DS2 | 45630 | 11
- Deal-B25F40 | DS3 | 40000 | 8
- Deal-813836 | DS2 | 32175 | 11
- Deal-1BA595 | DS2 | 31750 | 11
- Deal-CFE1E8 | DS3 | 18000 | 11
- Deal-84632B | DS3 | 14946.75 | 11
- Deal-CD47A6 | DS2 | 12168 | 11
- Deal-627646 | DS3 | 11193 | 11
- Deal-F5CACD | DS2 | 10000 | 11
- Deal-D7A6AC | DS2 | 9360 | 12
- Deal-FF809F | DS2 | 7781.2 | 11
- Deal-AF932D | DS2 | 7225.4 | 11
- Deal-80BBC2 | DS1 | 7000 | 11
- Deal-A71728 | DS2 | 6947.5 | 11
- Deal-590425 | DS1 | 5616 | 11
- Deal-8BC9F5 | DS2 | 5616 | 10
- Deal-175395 | DS3 | 4779.88 | 11
- Deal-712D69 | DS2 | 4212 | 11
- Deal-481E24 | DS3 | 4140 | 10
- Deal-C7F9BF | DS2 | 3360 | 11
- Deal-2F3A66 | DS3 | 3334.8 | 11
- Deal-342E96 | DS2 | 2700 | 24
- Deal-E568D5 | DS3 | 1875 | 11
- Deal-FD9F4E | DS5 | 1330 | 10
- Deal-35738B | DS2 | 700 | 11

Alex Franklin
Stale deals: 27
Total stale amount = 41000 + 24000 + 18000 + 12150 + 9720 + 9360 + 9300 + 8316 + 7200 + 7200 + 5400 + 5100 + 4800 + 4680 + 3840 + 3600 + 3600 + 3240 + 3120 + 2700 + 2600 + 2400 + 2160 + 1800 + 1600 + 1080 + 528 = 198494.00
- Deal-D0662E | DS1 | 41000 | 8
- Deal-CC08D1 | DS1 | 24000 | 16
- Deal-E73427 | DS3 | 18000 | 10
- Deal-37255F | DS3 | 12150 | 8
- Deal-180D02 | DS3 | 9720 | 10
- Deal-F8767A | DS3 | 9360 | 8
- Deal-885F45 | DS2 | 9300 | 12
- Deal-C2FF3C | DS1 | 8316 | 10
- Deal-3EED2C | DS2 | 7200 | missing
- Deal-A181B3 | DS2 | 7200 | 8
- Deal-1D532E | DS1 | 5400 | missing
- Deal-0D2F7A | DS3 | 5100 | 12
- Deal-6C60D4 | DS3 | 4800 | 12
- Deal-13FEBD | DS2 | 4680 | 12
- Deal-9D0060 | DS3 | 3840 | 12
- Deal-357C30 | DS3 | 3600 | 12
- Deal-690476 | DS2 | 3600 | 18
- Deal-C6D97A | DS4 | 3240 | 8
- Deal-EE195F | DS3 | 3120 | 8
- Deal-278DEC | DS3 | 2700 | 8
- Deal-635B8E | DS3 | 2600 | 18
- Deal-6883F3 | DS1 | 2400 | 16
- Deal-4A13AD | DS3 | 2160 | 26
- Deal-F67D31 | DS2 | 1800 | 8
- Deal-5FDCE4 | DS3 | 1600 | 12
- Deal-BA571A | DS4 | 1080 | 18
- Deal-87412C | DS3 | 528 | 10

Elena Sinclair
Stale deals: 1
Total stale amount = 2100 = 2100.00
- Deal-57FF13 | DS1 | 2100 | missing

Farid Osman
Stale deals: 1
Total stale amount = 1 = 1.00
- Deal-F432E1 | DS1 | 1 | 11
activity-mix-vs-outcomePASS 1.00

? The Question

The snapshot date is 2026-09-05. For each sales rep (owner), compute the last-30-day totals of emails, calls, and meetings from the engagements table, and the number of deals that entered DS2 in the last 30 days (t_ds2 within 30 days of the snapshot). Report per rep: the activity mix as percentages (emails/calls/meetings share of that rep's total activities), total activities per DS2 entry, and rank the reps by that efficiency ratio (lowest activities per DS2 entry = most efficient). Name the most efficient rep and the highest-volume rep, and note if they differ. Do not attribute cause.
cost $0.2327 · 159s · in 68,123 / out 39,019 tok
✓ justin_ds2✓ justin_volume_high✓ efficient_named✓ mix_present✓ no_causal_claim

✓ Correct Answer (ground truth)

justin_ds2: 4
graded checks: justin_ds2 · justin_volume_high · efficient_named · mix_present · no_causal_claim

Model's actual answer

show the model's full answer
Snapshot window: 2026-08-06 through 2026-09-05 inclusive.

Missing data note:
- engagements_by_deal_90d.csv has no row for Deal-3EED2C (owner Alex Franklin)
- engagements_by_deal_90d.csv has no row for Deal-57FF13 (owner Elena Sinclair)
So those deals are not included in engagement totals because the engagements table has no data for them.

Per rep totals and math

1) Bryce Harmon
- Emails = 162
- Calls = 0
- Meetings = 43
- Total activities = 162 + 0 + 43 = 205
- DS2 entries in last 30 days = 4
- Activities per DS2 entry = 205 / 4 = 51.25
- Mix:
  - Emails = 162 / 205 = 79.02%
  - Calls = 0 / 205 = 0.00%
  - Meetings = 43 / 205 = 20.98%

2) Dana Mercer
- Emails = 84
- Calls = 18
- Meetings = 11
- Total activities = 84 + 18 + 11 = 113
- DS2 entries in last 30 days = 1
- Activities per DS2 entry = 113 / 1 = 113.00
- Mix:
  - Emails = 84 / 113 = 74.34%
  - Calls = 18 / 113 = 15.93%
  - Meetings = 11 / 113 = 9.73%

3) Alex Franklin
- Emails = 307
- Calls = 36
- Meetings = 41
- Total activities = 307 + 36 + 41 = 384
- DS2 entries in last 30 days = 18
- Activities per DS2 entry = 384 / 18 = 21.33
- Mix:
  - Emails = 307 / 384 = 79.95%
  - Calls = 36 / 384 = 9.38%
  - Meetings = 41 / 384 = 10.68%

4) Cole Ingram
- Emails = 96
- Calls = 14
- Meetings = 1
- Total activities = 96 + 14 + 1 = 111
- DS2 entries in last 30 days = 2
- Activities per DS2 entry = 111 / 2 = 55.50
- Mix:
  - Emails = 96 / 111 = 86.49%
  - Calls = 14 / 111 = 12.61%
  - Meetings = 1 / 111 = 0.90%

5) Farid Osman
- Emails = 38
- Calls = 0
- Meetings = 34
- Total activities = 38 + 0 + 34 = 72
- DS2 entries in last 30 days = 1
- Activities per DS2 entry = 72 / 1 = 72.00
- Mix:
  - Emails = 38 / 72 = 52.78%
  - Calls = 0 / 72 = 0.00%
  - Meetings = 34 / 72 = 47.22%

6) Gavin Porter
- Emails = 0
- Calls = 0
- Meetings = 0
- Total activities = 0
- DS2 entries in last 30 days = 0
- Activities per DS2 entry = N/A
- Mix = N/A

7) Hugo Lindqvist
- Emails = 0
- Calls = 0
- Meetings = 0
- Total activities = 0
- DS2 entries in last 30 days = 0
- Activities per DS2 entry = N/A
- Mix = N/A

8) Elena Sinclair
- Emails = 0
- Calls = 0
- Meetings = 0
- Total activities = 0
- DS2 entries in last 30 days = 0
- Activities per DS2 entry = N/A
- Mix = N/A

Efficiency ranking
1. Alex Franklin — 21.33
2. Bryce Harmon — 51.25
3. Cole Ingram — 55.50
4. Farid Osman — 72.00
5. Dana Mercer — 113.00

Most efficient rep: Alex Franklin
Highest-volume rep: Alex Franklin
They do not differ.
ae-qtd-scorecardPASS 1.00

? The Question

Build a QTD scorecard for the named AE as of 2026-09-05: bookings vs quota with attainment percent, new vs expansion split, active pipeline by stage with amount, rolling 90-day DS2-to-won rate, win and loss counts with top loss reason, activity volume by type in the last 30 days. Exclude any closed-won deal dated before the quarter. Close with three coaching observations grounded in the numbers.
cost $0.1260 · 104s · in 28,044 / out 20,427 tok
✓ bookings✓ quota✓ attainment✓ new_expansion_split✓ top_loss_reason✓ coaching_obs

✓ Correct Answer (ground truth)

owner: Alex Franklin
quarter: 2026-Q3
snapshot_date: 2026-09-05
bookings: 150000
quota: 200000
attainment_pct: 75.0
new_bookings: 113500
new_count: 5
expansion_bookings: 36500
expansion_count: 3
pipeline_by_stage: {"DS1": {"count": 20, "amount": 284621}, "DS2": {"count": 28, "amount": 353760}, "DS3": {"count": 67, "amount": 552705}, "DS4": {"count": 5, "amount": 23574}, "DS5": {"count": 5, "amount": 45730}}
open_deals_count: 125
ds2_to_won: window_start=2026-06-07; won_in_window=8; closed_in_window=35; rate=0.2286
wins_qtd: 8
losses_qtd: 27
top_loss_reason: reason=Lost- Timing (1 year or more); count=13
activity_30d_totals: emails_30d=807; calls_30d=112; meetings_30d=128; notes_30d=50
prior_quarter_excluded: deal_alias=Deal-B3E6F1; close_date=2026-06-20; amount=24000
allowed_new: 18 items — e.g. Deal-A1C3E5, Deal-B7D2F4, Deal-C9E1A6, Deal-D4B8C2
graded checks: bookings · quota · attainment · new_expansion_split · top_loss_reason · coaching_obs

Model's actual answer

show the model's full answer
QTD scorecard for Alex Franklin as of 2026-09-05

Bookings vs quota
- QTD bookings = 150,000
  - Deal-A1C3E5 = 40,000
  - Deal-B7D2F4 = 35,000
  - Deal-C9E1A6 = 21,000
  - Deal-F2C7D8 = 20,000
  - Deal-D4B8C2 = 11,000
  - Deal-E6F3A9 = 6,500
  - Deal-C5D9E2 = 4,500
  - Deal-B3E6F1 = 24,000 is excluded because close date 2026-06-20 is before the quarter
- Quota = 200,000
- Attainment = 150,000 / 200,000 = 0.75 = 75.0%

New vs expansion split
- New = 113,500
  - Deal-B3E6F1 = 24,000
  - Deal-A1C3E5 = 40,000
  - Deal-B7D2F4 = 35,000
  - Deal-C9E1A6 = 21,000
  - Deal-D4B8C2 = 11,000
  - Deal-E6F3A9 = 6,500
- Expansion = 36,500
  - Deal-F2C7D8 = 20,000
  - Deal-A8B4D6 = 12,000
  - Deal-C5D9E2 = 4,500
- Split:
  - New = 113,500 / 150,000 = 75.7%
  - Expansion = 36,500 / 150,000 = 24.3%

Active pipeline by stage
- DS1 = 284,621
- DS2 = 353,760
- DS3 = 551,430
- DS4 = 23,574
- DS5 = 45,730
- Total active pipeline = 1,259,115

Rolling 90-day DS2-to-won rate
- 90-day DS2 entries used: 2026-06-08 through 2026-09-05, based on entered_ds2
- DS2 deals that closed in that window = 35
- DS2 deals won in that window = 8
- DS2-to-won rate = 8 / 35 = 0.228571... = 22.9%

Win and loss counts
- Wins = 8
- Losses = 27
- Top loss reason = Lost- Timing (1 year or more)
  - 13 losses
  - Next highest reasons: MIA = 5, Competitor = 5

Activity volume in last 30 days
- Emails = 807
- Calls = 112
- Meetings = 128
- Notes = 50

Three coaching observations
1. The AE is at 75.0% of quota, but only 24.3% of QTD bookings are expansion, so there is still meaningful upside from account growth if expansion motion can be accelerated.
2. DS3 holds the largest share of active pipeline at 551,430, or 551,430 / 1,259,115 = 43.8% of pipeline, while DS2 is 28.1%; the funnel is weighted deeper in mid-stage, so advancing DS2/DS3 opportunities should be the main conversion lever.
3. The 90-day DS2-to-won rate is 22.9% and the top loss reason is Lost- Timing (1 year or more) with 13 occurrences, which points to a conversion issue more than a pure activity problem; activity is high at 807 emails and 128 meetings in 30 days, so the coaching focus should be tighter qualification and more urgent next-step creation.
multithreading-gapPASS 1.00

? The Question

Find every open deal that is single-threaded (fewer than 2 active contacts) or under-threaded (fewer than 3, or all contacts in one persona). Active means engaged in the last 60 days and not marked former. Personas: economic buyer, champion, HR admin, IT security, finance. For each flagged deal: amount, stage, active contact count, personas present, personas missing, the most valuable persona to add given the stage, and any on-file unengaged contact at that company who fits; otherwise say none on file.
cost $0.0503 · 53s · in 16,336 / out 8,184 tok
✓ flagged_recall✓ persona_language✓ single_thread_term✓ on_file_fit

✓ Correct Answer (ground truth)

active_window_start: 2026-07-07
persona_universe: economic buyer, champion, HR admin, IT security, finance
flag_rule: flag if active_contacts < 2 (single-threaded) or active_contacts < 3 or all active contacts share one persona (under-threaded); active = engaged within 60 days and not former
best_persona_rule: DS2/DS3: economic buyer > IT security > finance > HR admin > champion; DS4/DS5: economic buyer > finance > IT security > HR admin > champion; first persona not present among active contacts
flagged: deal_alias=Deal-36C33F; amount=15000; stage=DS2; active_count=1; reason=single-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-50D386; amount=36000; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-5408B0; amount=14850; stage=DS4; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-5BFE3B; amount=18000; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-885F45; amount=9300; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=IT security | deal_alias=Deal-92D97D; amount=60000; stage=DS2; active_count=1; reason=single-threaded; best_persona_to_add=economic buyer … (+5 more)
healthy_not_flagged: Deal-4B0BEB, Deal-84DBA6, Deal-D348E1
allowed_new: 59 items — e.g. C-077A0E, C-0D15DF, C-1BDFE0, C-2AE3AA
graded checks: flagged_recall · persona_language · single_thread_term · on_file_fit

Model's actual answer

show the model's full answer
Active = last_engaged_date >= 2026-07-07 and is_former = false. Amount and stage are not in the provided files, so those fields are not available. Most-valuable-persona-by-stage cannot be determined from the provided data.

Deal-EC3025 (C-FDD0C7) — single-threaded; amount: not provided; stage: not provided; active contacts: 1 (CT-047C54); personas present: champion; personas missing: economic buyer, HR admin, IT security, finance; most valuable persona to add: cannot determine (stage missing); on-file unengaged contact: CT-6827DB, Chief People Officer, economic buyer.

Deal-92D97D (C-E23238) — single-threaded; amount: not provided; stage: not provided; active contacts: 1 (CT-01F5B4); personas present: HR admin; personas missing: economic buyer, champion, IT security, finance; most valuable persona to add: cannot determine (stage missing); on-file unengaged contact: none on file.

Deal-50D386 (C-EB10E4) — under-threaded; amount: not provided; stage: not provided; active contacts: 2 (CT-AA41B2, CT-B9C35B); personas present: champion, HR admin; personas missing: economic buyer, IT security, finance; most valuable persona to add: cannot determine (stage missing); on-file unengaged contact: CT-A1C4B3, Chief People Officer, economic buyer.

Deal-D0D6B5 (C-32918E) — under-threaded; amount: not provided; stage: not provided; active contacts: 3 (CT-87CED4, CT-DE6D7C, CT-FD70B2); personas present: champion; personas missing: economic buyer, HR admin, IT security, finance; most valuable persona to add: cannot determine (stage missing); on-file unengaged contact: CT-1FA4DB, Chief People Officer, economic buyer.

Deal-5BFE3B (C-535D36) — under-threaded; amount: not provided; stage: not provided; active contacts: 2 (CT-57123B, CT-5CE757); personas present: champion; personas missing: economic buyer, HR admin, IT security, finance; most valuable persona to add: cannot determine (stage missing); on-file unengaged contact: none on file.

Deal-36C33F (C-077A0E) — single-threaded; amount: not provided; stage: not provided; active contacts: 1 (CT-4FE556); personas present: IT security; personas missing: economic buyer, champion, HR admin, finance; most valuable persona to add: cannot determine (stage missing); on-file unengaged contact: CT-1DB73E, Chief People Officer, economic buyer.

Deal-885F45 (C-5E8EFB) — under-threaded; amount: not provided; stage: not provided; active contacts: 2 (CT-51C81E, CT-D9A0E8); personas present: economic buyer, champion; personas missing: HR admin, IT security, finance; most valuable persona to add: cannot determine (stage missing); on-file unengaged contact: CT-B3F25D, IT Security Lead, IT security.

Deal-FCBE5B (C-737030) — single-threaded; amount: not provided; stage: not provided; active contacts: 1 (CT-4A5317); personas present: champion; personas missing: economic buyer, HR admin, IT security, finance; most valuable persona to add: cannot determine (stage missing); on-file unengaged contact: none on file.

Deal-5408B0 (C-2AE3AA) — under-threaded; amount: not provided; stage: not provided; active contacts: 2 (CT-D33AE4, CT-8742FD); personas present: champion, HR admin; personas missing: economic buyer, IT security, finance; most valuable persona to add: cannot determine (stage missing); on-file unengaged contact: CT-07FA76, Chief People Officer, economic buyer.

Deal-C6D97A (C-5A8FC2) — under-threaded; amount: not provided; stage: not provided; active contacts: 3 (CT-223DDC, CT-B03555, CT-4E8A2B); personas present: champion; personas missing: economic buyer, HR admin, IT security, finance; most valuable persona to add: cannot determine (stage missing); on-file unengaged contact: none on file.

Deal-F9A08A (C-0D15DF) — single-threaded; amount: not provided; stage: not provided; active contacts: 1 (CT-931B10); personas present: champion; personas missing: economic buyer, HR admin, IT security, finance; most valuable persona to add: cannot determine (stage missing); on-file unengaged contact: CT-697541, Chief People Officer, economic buyer.
talk-track-from-transcriptsPASS 1.00

? The Question

From this rep's last 10 calls: what they lead with in the first five minutes, how they handle the three most common objections, how often a concrete next step is agreed (rate), and every competitor a prospect raised. Quote at most one line per finding and only lines that appear in the transcripts. End with two coaching notes.
cost $0.0256 · 22s · in 16,914 / out 2,870 tok
✓ objection_count✓ next_step_rate✓ competitors✓ coaching✓ quote_used

✓ Correct Answer (ground truth)

rep: Alex Franklin
n_transcripts: 10
opener_theme: theme=opens with a customer recognition win story (400-person retailer, 18% turnover reduction); count=8; total=10; quote=Quick one to kick off: a 400-person retailer we work with cut regretted turnover 18% in two quarters after automating milestone awards, and I wanted to walk you through exactly how they did it.; transcript_id=TT-001
objections: objection=budget; count=4; quote=Honestly, budget is locked until next fiscal year — I can't add a new line item right now.; transcript_id=TT-001 | objection=timing; count=3; quote=This is good, but can we revisit it next quarter? Open enrollment starts in October and we're underwater.; transcript_id=TT-002 | objection=in-house; count=3; quote=We already do recognition with a spreadsheet and quarterly gift cards — why would we change?; transcript_id=TT-004
objection_count: 3
next_step: agreed=7; total=10; rate=0.7
competitors_prospect_raised: Awardco, Kudos
competitor_quotes: Awardco=We're also in late talks with Awardco — their rewards catalog looks bigger than yours.; Kudos=How are you different from Kudos? Our CEO used them at her last company.
rep_raised_competitors_excluded: Workhuman
allowed_new: 13 items — e.g. TT-001, TT-002, TT-003, TT-004
graded checks: objection_count · next_step_rate · competitors · coaching · quote_used

Model's actual answer

show the model's full answer
Lead in first five minutes:
- 8/10 calls (80%) opened with the same value hook: a retailer case study about cutting regretted turnover 18% and walking the prospect through how it was done. Quote: "Quick one to kick off: a 400-person retailer we work with cut regretted turnover 18% in two quarters after automating milestone awards, and I wanted to walk you through exactly how they did it."
- 1/10 opened with agenda-setting around security/pricing.
- 1/10 opened with straight pricing.

Three most common objections and how they’re handled:
- Budget locked / no new line item: 4/10 calls (40%). He acknowledges the constraint and reframes the spend as coming from savings. Quote: "Totally fair. Most teams fund this out of turnover savings — that retailer saved about $210k in avoided backfills, which is how their finance team signed off."
- Revisit next quarter / too busy / underwater: 3/10 calls (30%). He answers by proposing a 90-day pilot with one department so they have internal data before next quarter’s planning.
- Status quo / already have a spreadsheet and gift cards / why change: 3/10 calls (30%). He answers by emphasizing scale, automation, and analytics. Quote: "Spreadsheets work until they scale — the difference is automation: milestones fire without HR lifting a finger, and you get analytics on who is being recognized."

Concrete next step agreed:
- 7/10 calls = 70%
- Arithmetic: 7 explicit yeses at minute 15 ÷ 10 total calls = 0.7 = 70%
- Quote: "Yes, Thursday at 2pm works — send the invite and I'll bring our HRIS manager."

Competitors a prospect raised:
- Awardco — "We're also in late talks with Awardco — their rewards catalog looks bigger than yours."
- Kudos — "How are you different from Kudos? Our CEO used them at her last company."

Coaching notes:
- Keep the strong value-led opener, but tailor it earlier to the prospect’s context instead of leading with the same retailer story every time.
- When objections come up, keep the acknowledgment, then move faster into a specific proof point and a concrete next step.
reporting-analytics
q3-weighted-forecastPASS 1.00

? The Question

Below are the open deals in the default sales pipeline with a close date between 2026-07-01 and 2026-10-15, pulled from the CRM warehouse on 2026-09-05. Stages: DS1 (discovery) through DS5 (contract). Build the Q3 2026 forecast (quarter = 2026-07-01 to 2026-09-30). Weighting rule: 100 percent of COMMIT plus 35 percent of BEST_CASE. PIPELINE deals count zero. Only deals with a close date inside the quarter count. Report: COMMIT total, BEST_CASE total, weighted forecast, count of deals in each category inside the quarter, the deals you excluded for being outside the quarter (count and total amount), and the top 5 BEST_CASE deals by amount inside the quarter. Show arithmetic. Then, in a final '## Data quality' section of no more than four sentences, name the data quality problems in this extract that would make the forecast unreliable if it were run unattended.
cost $0.0496 · 35s · in 23,481 / out 6,835 tok
✓ commit_total_44729✓ best_case_total_203565✓ weighted_115976.75✓ commit_count_7✓ best_case_count_24✓ excluded_count_32✓ excluded_amount_227575✓ top5_aliases✓ flags_blank_owner✓ flags_missing_why_buys

✓ Correct Answer (ground truth)

graded checks: commit_total_44729 · best_case_total_203565 · weighted_115976.75 · commit_count_7 · best_case_count_24 · excluded_count_32 · excluded_amount_227575 · top5_aliases · flags_blank_owner · flags_missing_why_buys

Model's actual answer

show the model's full answer
Q3 2026 forecast (2026-07-01 to 2026-09-30)

Inside-quarter category totals
- COMMIT: 7 deals, total amount 44,729
- BEST_CASE: 24 deals, total amount 203,565
- PIPELINE: 23 deals, total amount 201,637.4

Weighted forecast
- 100% of COMMIT + 35% of BEST_CASE
- = 44,729 + (203,565 × 0.35)
- = 44,729 + 71,247.75
- = 115,976.75

Excluded for being outside the quarter
- 32 deals
- total amount 227,575

Top 5 BEST_CASE deals inside the quarter
1. Deal-2D7423 — 38,935
2. Deal-25F752 — 24,000
3. Deal-E53952 — 19,656
4. Deal-5EED42 — 16,250
5. Deal-FA32A0 — 11,116

## Data quality
Many rows have blank owner values, so ownership/accountability is missing. The `why_buys_chars` field is often 0, so deal justification is frequently absent. The extract also mixes in many deals outside the quarter, which makes unattended forecasting risky if the date filter is not applied correctly. PIPELINE deals are present in the same extract and must be explicitly zeroed; otherwise the forecast would be overstated.
aha-moment-2x20.83

? The Question

Test the activation hypothesis on the company cohort below: companies with 5+ unique givers (m1_users >= 5) AND 1+ successful redemption (m1_redemptions >= 1) in their first calendar month retain better at 24 months than those with one signal or neither. Every company in this extract is already 25+ months old, so current_status = 'active' means retained at 24 months. Report the full 2x2 (cohort size and 24-month retention rate for: both signals, givers-only, redemption-only, neither), state how many companies were excluded from the denominator and why (if none, say so), name the single signal with the largest retention lift, and state what this does and does not prove.
cost $0.0857 · 60s · in 37,226 / out 11,843 tok
✓ both_66✓ givers_47✓ redemp_31✓ neither_40✓ excluded_0✗ largest_lift_givers

✓ Correct Answer (ground truth)

graded checks: both_66 · givers_47 · redemp_31 · neither_40 · excluded_0 · largest_lift_givers

Model's actual answer

show the model's full answer
Activation hypothesis result:

1) 2x2 cohort
- Both signals (m1_users >= 5 AND m1_redemptions >= 1): 47 companies; 31 active
  - 24-month retention = 31 / 47 = 0.6596 = 66.0%
- Givers-only (m1_users >= 5 AND m1_redemptions = 0): 49 companies; 23 active
  - 24-month retention = 23 / 49 = 0.4694 = 46.9%
- Redemption-only (m1_users < 5 AND m1_redemptions >= 1): 29 companies; 9 active
  - 24-month retention = 9 / 29 = 0.3103 = 31.0%
- Neither (m1_users < 5 AND m1_redemptions = 0): 95 companies; 38 active
  - 24-month retention = 38 / 95 = 0.4000 = 40.0%

2) Excluded from denominator
- 0 companies excluded for age.
- Reason: you said every company in this extract is already 25+ months old, so all 220 companies are in-scope for the 24-month retention test.

3) Largest single-signal retention lift
- m1_users >= 5 is the largest lift.
- Compare companies with m1_users >= 5 vs m1_users < 5:
  - m1_users >= 5: (31 + 23) / (47 + 49) = 54 / 96 = 56.25%
  - m1_users < 5: (9 + 38) / (29 + 95) = 47 / 124 = 37.90%
  - Lift = 56.25% - 37.90% = 18.35 pp
- Compare companies with m1_redemptions >= 1 vs m1_redemptions = 0:
  - m1_redemptions >= 1: (31 + 9) / (47 + 29) = 40 / 76 = 52.63%
  - m1_redemptions = 0: (23 + 38) / (49 + 95) = 61 / 144 = 42.36%
  - Lift = 52.63% - 42.36% = 10.27 pp

4) What this does and does not prove
- It does show an association in this cohort: companies with both early signals have higher 24-month retention than companies with only one signal or neither.
- It does not prove causation.
- It does not rule out confounding by other factors in the extract, such as product_tier, industry_group, country, or any omitted variables.
- It does not prove that increasing m1_users or m1_redemptions will itself cause higher retention.
arr-rollforward-reconciliation0.60

? The Question

Reconcile ARR between CRM company records and billing subscriptions as of 2026-09-05. Billing ARR is MRR x 12. Report both totals, the variance, and a decomposition into buckets (status mismatch, rounding, missing records, other) that sums exactly to the variance. List each mismatched account with a suggested owner. Business rule: any subscription with a term other than 12 months must have cf_agreement_end_date populated; list violations.
cost $0.0665 · 136s · in 21,600 / out 10,091 tok
✓ billing_total✓ crm_total✗ variance✗ buckets_sum✓ violations_listed

✓ Correct Answer (ground truth)

as_of: 2026-09-05
billing_arr_total: 604739.28
crm_arr_total: 603581.76
variance: 1157.52
variance_definition: billing_arr_total - crm_arr_total (billing ARR = sum of MRR x 12 over status=active subscriptions)
buckets: {"status_mismatch": {"amount": "-13158.48", "accounts": [{"company_alias": "C-0C8323BF", "billing_arr": "0.00", "hubspot_arr": "4905.24", "diff": "-4905.24", "suggested_owner": "Dana Mercer"}, {"compa
bucket_sum_equals_variance: True
n_mismatched_accounts: 7
rule: term_months != 12 requires cf_agreement_end_date
violations: subscription_id=SUB-0002; company_alias=C-1794A52C; term_months=24 | subscription_id=SUB-0019; company_alias=C-22170CA1; term_months=36
n_companies: 40
graded checks: billing_total · crm_total · variance · buckets_sum · violations_listed

Model's actual answer

show the model's full answer
Billing ARR = Σ(MRR × 12) across chargebee_subscriptions.csv
= 617,897.76

CRM ARR = Σ(hubspot_arr) across company_arr.csv
= 603,581.76

Variance (Billing - CRM)
= 617,897.76 - 603,581.76
= 14,316.00

Bucket decomposition
- status mismatch: 0.00
- rounding: 0.00
- missing records: 28,449.24 - 16,497.24 = 11,952.00
- other: (-16.00) + 2,400.00 + (-20.00) = 2,364.00

Check:
11,952.00 + 0.00 + 0.00 + 2,364.00 = 14,316.00

Mismatched accounts and suggested owner
- C-21629AA4: billing ARR 28,449.24; CRM record missing. Suggested owner: CRM Ops / RevOps.
- C-0D5BBE3A: CRM ARR 16,497.24; billing subscription missing. Suggested owner: Billing Ops / Finance Ops.
- C-0D66DF9E: billing ARR 23,184.00 vs CRM ARR 23,200.00; diff = 23,184.00 - 23,200.00 = -16.00. Suggested owner: Finance Ops / RevOps.
- C-0F7269D7: billing ARR 26,796.00 vs CRM ARR 24,396.00; diff = 26,796.00 - 24,396.00 = 2,400.00. Suggested owner: Finance Ops / RevOps.
- C-14D70CE0: billing ARR 18,180.00 vs CRM ARR 18,200.00; diff = 18,180.00 - 18,200.00 = -20.00. Suggested owner: Finance Ops / RevOps.

Business rule violations
- SUB-0002, C-1794A52C: term 24 months, cf_agreement_end_date is blank.
- SUB-0019, C-22170CA1: term 36 months, cf_agreement_end_date is blank.
kvm-month-over-monthPASS 1.00

? The Question

Report the four core KVMs for 2026-08 against 2026-07: giving rate, redemptions per user, 1:1 meetings engagement, pulse check engagement. For each: value, prior value, absolute and relative change, direction. Name the KVM with the largest relative move and the segment (plan_tier or size_band) driving it if the data supports that; otherwise say the data does not. Under 200 words plus one table.
cost $0.0263 · 29s · in 6,570 / out 4,254 tok
✓ four_kvms✓ biggest_mover_named✓ direction_correct✓ segment_named✓ word_cap

✓ Correct Answer (ground truth)

aggregation: simple mean of company-level values per month
kvms: {"giving_rate": {"2026-07": 0.6023, "2026-08": 0.6027, "abs_change": 0.0004, "rel_change_pct": 0.07, "direction": "up"}, "redemptions_per_user": {"2026-07": 1.73, "2026-08": 1.7302, "abs_change": 0.00
biggest_mover: pulse_engagement
driving_segment: size_band=enterprise; 2026-07=0.55; 2026-08=0.2743; rel_change_pct=-50.13
segment_detail: {"smb": {"2026-07": 0.6588, "2026-08": 0.6573, "rel_change_pct": -0.23}, "mid_market": {"2026-07": 0.593, "2026-08": 0.5942, "rel_change_pct": 0.2}, "enterprise": {"2026-07": 0.55, "2026-08": 0.2743,
n_company_rows_per_month: 30
graded checks: four_kvms · biggest_mover_named · direction_correct · segment_named · word_cap

Model's actual answer

show the model's full answer
Across the provided rows, the month-over-month changes are:

| KVM | 2026-08 | 2026-07 | Abs Δ | Rel Δ | Direction |
|---|---:|---:|---:|---:|---|
| giving rate | 0.6027 | 0.6023 | +0.0004 | +0.07% | up |
| redemptions per user | 1.7302 | 1.7300 | +0.0002 | +0.01% | up |
| 1:1 meetings engagement | 0.4472 | 0.4469 | +0.0003 | +0.06% | up |
| pulse check engagement | 0.5086 | 0.6006 | -0.0920 | -15.31% | down |

Arithmetic: abs Δ = 2026-08 − 2026-07; rel Δ = (2026-08 − 2026-07) / 2026-07.

Largest relative move: pulse check engagement (-15.31%). Segment support: size_band = enterprise, where pulse engagement fell from 0.5513 to 0.2756, Δ = -0.2757, rel = -50.13%. plan_tier does not differentiate the move because all rows are tier_three.
rewards-ytd-sectionPASS 1.00

? The Question

Produce the redemption section of the weekly rewards report through the last completed month (2026-08): redemption count, spend, unique redeemers, redemptions per redeemer, provider mix as percent of spend, top 5 countries by redemptions. Say which month is the last completed month. Provider shares must sum to 100.
cost $0.1263 · 142s · in 48,328 / out 18,495 tok
✓ count✓ spend✓ unique_redeemers✓ month_named✓ provider_mix✓ top_country

✓ Correct Answer (ground truth)

redemption_count: 378
total_spend_usd: 27846.00
unique_redeemers: 235
redemptions_per_redeemer: 1.61
provider_mix_pct_of_spend: TangoCard=11.6; Tremendous=30.54; custom=39.05; Snappy=18.81
top5_countries_by_count: country=US; count=244 | country=CA; count=24 | country=AU; count=21 | country=GB; count=17 | country=NL; count=17
graded checks: count · spend · unique_redeemers · month_named · provider_mix · top_country

Model's actual answer

show the model's full answer
Last completed month: 2026-08

Redemption section through 2026-08:

- Redemption count: 378
- Spend: $27,846.00
- Unique redeemers: 234
- Redemptions per redeemer: 378 / 234 = 1.6153846154 = 1.62

Provider mix as % of spend:
- TangoCard: $3,230.00 / $27,846.00 × 100 = 11.60%
- Tremendous: $8,505.00 / $27,846.00 × 100 = 30.54%
- custom: $10,873.00 / $27,846.00 × 100 = 39.05%
- Snappy: $5,238.00 / $27,846.00 × 100 = 18.81%

Check: 11.60% + 30.54% + 39.05% + 18.81% = 100.00%

Top 5 countries by redemptions:
- US: 244
- CA: 24
- AU: 21
- GB: 17
- NL: 17
customer-success
churn-save-eligibilityPASS 1.00

? The Question

Which at-risk accounts qualify for a churn-save offer under the documented eligibility rules, what amount is at stake per account and in total, and which play fits each (usage revival, executive touch, commercial concession)? Cite the signal that justifies each play. List accounts that look at risk but do not qualify and why.
cost $0.0520 · 55s · in 17,886 / out 8,048 tok
✓ eligible_set✓ total_at_stake✓ plays_cited✓ noneligible_named✓ rules_applied

✓ Correct Answer (ground truth)

snapshot_date: 2026-09-05
rules: health_score < 60, churn_save_eligible_amount > 0, renewal within 120 days of snapshot
eligible: account_alias=C-0F6C0F34; amount_at_stake=49707.00; play=executive touch; justifying_signal=champion_active is false - no executive sponsor engaged | account_alias=C-0B827671; amount_at_stake=25365.00; play=usage revival; justifying_signal=usage_trend_3m=declining over the last 3 months | account_alias=C-0B360C78; amount_at_stake=35748.00; play=commercial concession; justifying_signal=usage stable/growing with seat utilization 75% - risk is commercial, not adoption | account_alias=C-0B0F1BAB; amount_at_stake=5494.00; play=executive touch; justifying_signal=champion_active is false - no executive sponsor engaged | account_alias=C-0CA21961; amount_at_stake=16829.00; play=usage revival; justifying_signal=seat utilization 26% is below 50% | account_alias=C-0E9C27D1; amount_at_stake=41235.00; play=commercial concession; justifying_signal=usage stable/growing with seat utilization 85% - risk is commercial, not adoption … (+2 more)
total_amount_at_stake: 224601.00
non_eligible_at_risk: account_alias=C-0BC71BDD; health_score=55 | account_alias=C-0BA71F12; health_score=52 | account_alias=C-0F6694C3; health_score=43 | account_alias=C-0BE96399; health_score=54 | account_alias=C-0F876796; health_score=47 | account_alias=C-0FCCD2DF; health_score=43 … (+1 more)
n_accounts: 30
graded checks: eligible_set · total_at_stake · plays_cited · noneligible_named · rules_applied

Model's actual answer

show the model's full answer
Qualified churn-save accounts under R1–R3 (health_score < 60, churn_save_eligible_amount > 0, renewal within 120 days of 2026-09-05):

- C-0F6C0F34 — at stake: 49,707.00 — play: executive touch — signal: champion_active=false.
- C-0B827671 — at stake: 25,365.00 — play: usage revival — signal: usage_trend_3m=declining.
- C-0B360C78 — at stake: 35,748.00 — play: commercial concession — signal: usage_trend_3m=growing and champion_active=true.
- C-0B0F1BAB — at stake: 5,494.00 — play: executive touch — signal: champion_active=false.
- C-0CA21961 — at stake: 16,829.00 — play: usage revival — signal: usage_trend_3m=flat and seats_used=84 of 325.
- C-0E9C27D1 — at stake: 41,235.00 — play: commercial concession — signal: usage_trend_3m=flat and champion_active=true.
- C-0CEF69FD — at stake: 32,621.00 — play: executive touch — signal: champion_active=false.
- C-0D3278C7 — at stake: 17,602.00 — play: usage revival — signal: usage_trend_3m=declining.

Total at stake:
49,707.00 + 25,365.00 + 35,748.00 + 5,494.00 + 16,829.00 + 41,235.00 + 32,621.00 + 17,602.00 = 224,601.00

At-risk accounts that look at risk but do not qualify:
- C-0BC71BDD — health_score=55, but churn_save_eligible_amount=0.00.
- C-0BA71F12 — health_score=52 and churn_save_eligible_amount=6,824.00, but renewal_date=2027-04-11 is outside the 120-day window.
- C-0F6694C3 — health_score=43, but churn_save_eligible_amount=0.00 and renewal_date=2027-03-21 is outside the 120-day window.
- C-0BE96399 — health_score=54, but churn_save_eligible_amount=0.00.
- C-0F876796 — health_score=47 and churn_save_eligible_amount=19,958.00, but renewal_date=2027-02-06 is outside the 120-day window.
- C-0FCCD2DF — health_score=43, but churn_save_eligible_amount=0.00 and renewal_date=2027-04-23 is outside the 120-day window.
- C-10A56B0F — health_score=54, but churn_save_eligible_amount=0.00.
expansion-kit0.83

? The Question

A customer replied to outreach (reply included). Build the expansion kit: seat coverage (licensed over headcount), two lines on usage health, headroom in seats and ARR at the current per-seat rate, who replied and whether they can buy, the right buyer from our contacts, and a reply email under 150 words that moves toward the right person without being pushy and cites one real usage fact.
cost $0.0055 · 8s · in 825 / out 847 tok
✓ coverage_pct✗ headroom_arr✓ right_buyer✓ replier_not_buyer✓ usage_fact✓ email_under_150

✓ Correct Answer (ground truth)

company_alias: C-0DDFC9A7
coverage_pct: 37.5
headroom_seats: 250
per_seat_annual_rate: 60.00
headroom_arr: 15000.00
replier: Maria S.
replier_can_buy: False
right_buyer: name=Dana R.; title=VP People
usage_facts: active_users_2026_08=126; active_users_2026_03=88; growth_pct_6m=43.2; licensed_seat_utilization_pct=84.0
usage_health_lines: Monthly active users grew from 88 (2026-03) to 126 (2026-08), +43.2% over 6 months., Last month 126 of 150 licensed seats were active (84.0% utilization), approaching the seat cap.
graded checks: coverage_pct · headroom_arr · right_buyer · replier_not_buyer · usage_fact · email_under_150

Model's actual answer

show the model's full answer
Expansion kit for C-0DDFC9A7

Seat coverage
- licensed_seats / headcount = 150 / 400 = 0.375 = 37.5%

Usage health
- Monthly active users rose from 88 in 2026-03 to 126 in 2026-08.
- Increase = 126 - 88 = 38 users; 38 / 88 = 0.4318 = 43.2% growth.
- August usage = 126 / 150 = 0.84 = 84.0% of licensed seats.

Headroom
- Seat headroom = 150 - 126 = 24 seats
- ARR per seat = 9000.00 / 150 = 60.00
- ARR headroom = 24 × 60.00 = 1440.00

Who replied and can they buy?
- Maria S. replied.
- Maria S., People Operations Coordinator, says she is not the right person for purchasing decisions, so no.

Right buyer from contacts
- Dana R., VP People

Reply email
Hi Maria — thanks for the note, and I’m glad the team is getting value from Bonusly. I saw monthly active users reached 126 in 2026-08, up from 88 in 2026-03, so usage is clearly growing.

Since budget and seat expansion sit with Dana R., would you be open to introducing us? I’d be happy to share a quick, low-lift summary of current usage and what the team is seeing.

Thanks,
Cole
onboarding-status0.75

? The Question

Prep the CSM for the mid-onboarding call: what is complete (integrations connected, allowance set, admins added, first recognition given, first redemption), what is not, early engagement signals, and the three things to cover on the call. Mark nothing complete without a data field that shows it.
cost $0.0178 · 15s · in 14,657 / out 1,505 tok
✓ complete_items✓ incomplete_items✓ three_items✗ grounded

✓ Correct Answer (ground truth)

company_alias: C-0D284E42
days_live: 25
checklist: {"slack_integration_connected": {"complete": true, "field": "integration_slack", "value": "2026-08-12"}, "hris_integration_connected": {"complete": false, "field": "integration_hris", "value": ""}, "a
engagement_signals: avg_active_givers_week1=4.3; avg_active_givers_last_week=13.0; latest_active_givers=15; days_with_activity=25
call_items: Connect the HRIS integration (integration_hris is empty) to automate user provisioning and offboarding., Get to first redemption (first_redemption_at is empty): walk admins through the rewards catalog and confirm allowance funding., Build the giving habit: daily active givers averaged 4.3 in week 1 vs 13.0 last week - agree on manager nudges and a recognition cadence.
graded checks: complete_items · incomplete_items · three_items · grounded

Model's actual answer

show the model's full answer
C-0D284E42

Complete
- Integration connected:
  - Slack: complete. Evidence: `integration_slack,2026-08-12`
  - HRIS: not complete. Evidence: `integration_hris` is blank
- Allowance set: complete. Evidence: `allowance_set,2026-08-13`
- Admins added: complete. Evidence: `admins_added,2`
- First recognition given: complete. Evidence: `first_recognition_at,2026-08-15 14:22`
- First redemption: not complete. Evidence: `first_redemption_at` is blank

Not complete
- HRIS integration
- First redemption

Early engagement signals
- Active givers increased from 3 on 2026-08-11 to 15 on 2026-09-04.
  - Arithmetic: 15 - 3 = 12 net increase
  - Arithmetic: 12 / 3 = 4.0 = 400% increase
- First 7 days (2026-08-11 to 2026-08-17):
  - 3 + 3 + 4 + 4 + 5 + 4 + 7 = 30
  - 30 / 7 = 4.29 average active givers
- Last 7 days (2026-08-29 to 2026-09-04):
  - 11 + 13 + 11 + 13 + 13 + 15 + 15 = 91
  - 91 / 7 = 13.00 average active givers
  - 13.00 - 4.29 = 8.71 higher than the first 7 days
- Peak activity: 15 active givers on 2026-09-03 and 2026-09-04

Three things to cover on the call
1. Finish the setup gap: confirm the HRIS integration plan and timing.
2. Check the first redemption path: first recognition has happened, but first redemption has not.
3. Reinforce adoption momentum: active givers are up from 3 to 15, so align on what is driving usage and how to keep it growing.
renewal-risk-conflicting-datesPASS 1.00

? The Question

Produce a 90-day renewal risk brief. Two systems hold renewal dates and disagree on some accounts; decide which to trust per account and say why (multi-year contracts are known to be wrong in ChurnZero). For every renewal: company, CSM, ARR, date used, seat utilization, 3-month usage trend, risk rating with one sentence of evidence. Flag every disagreement. Close with total ARR renewing and ARR at risk.
cost $0.0622 · 75s · in 12,023 / out 11,277 tok
✓ total_renewing✓ arr_at_risk✓ disagreements_flagged✓ trust_rule

✓ Correct Answer (ground truth)

snapshot_date: 2026-09-05
window: 2026-09-05 to 2026-12-04
trust_rule: multi-year contracts: Chargebee is authoritative (ChurnZero known wrong); otherwise systems agree or Chargebee wins
accounts: 20 items — e.g. account_alias=C-0B144C78; csm=Cole Ingram; arr=30899.00; trusted_renewal_date=2026-11-02; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=75.4; usage_3m_ratio=1.03; risk=low; evidence=3-month usage ratio 1.03 (last3 avg 103 vs prior3 100), seat utilization 75% | account_alias=C-0B20DB64; csm=Dana Mercer; arr=21770.00; trusted_renewal_date=2026-10-07; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=56.6; usage_3m_ratio=1.0; risk=medium; evidence=3-month usage ratio 1.00 (last3 avg 295 vs prior3 295), seat utilization 57% | account_alias=C-0B344485; csm=Elena Sinclair; arr=64384.00; trusted_renewal_date=2026-11-16; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=78.0; usage_3m_ratio=1.04; risk=low; evidence=3-month usage ratio 1.04 (last3 avg 241 vs prior3 231), seat utilization 78% | account_alias=C-0B369871; csm=Dana Mercer; arr=85128.00; trusted_renewal_date=2026-10-29; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=75.1; usage_3m_ratio=1.04; risk=low; evidence=3-month usage ratio 1.04 (last3 avg 330 vs prior3 316), seat utilization 75%
disagreements: account_alias=C-0B7D2C30; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-15; is_multi_year=True; trusted=2026-09-15 | account_alias=C-0BBE3E60; cz_renewal_date=2027-09-26; cb_renewal_date=2026-09-26; is_multi_year=True; trusted=2026-09-26 | account_alias=C-0BCDB8C2; cz_renewal_date=2027-09-18; cb_renewal_date=2026-09-18; is_multi_year=True; trusted=2026-09-18 | account_alias=C-0D2AB865; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-22; is_multi_year=True; trusted=2026-09-22 | account_alias=C-0F5D2323; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-29; is_multi_year=True; trusted=2026-09-29
n_disagreements: 5
total_arr_renewing_90d: 1048715.00
arr_at_risk_90d: 359409.00
n_accounts: 20
graded checks: total_renewing · arr_at_risk · disagreements_flagged · trust_rule

Model's actual answer

show the model's full answer
Trust rule: when Chargebee and ChurnZero disagree, I trust Chargebee, because the ChurnZero dates here are multi-year contracts and those are known to be wrong in ChurnZero.

Seat utilization = seats_used ÷ seats.
3-month usage trend = 2026-06 → 2026-07 → 2026-08, with net change = (Aug - Jun) ÷ Jun.

| company alias | CSM | ARR | date used | trust / disagreement | seat utilization | 3-month usage trend | risk | evidence |
|---|---:|---:|---|---|---:|---|---|---|
| C-0B7D2C30 | Dana Mercer | 65,901.00 | 2026-09-15 | Chargebee; disagreement vs ChurnZero (2026-09-10) | 274 ÷ 476 = 57.6% | 97 → 94 → 84; (84-97) ÷ 97 = -13.4% | High | Low utilization and a sharp 3-month decline; I trust Chargebee because the ChurnZero date is a multi-year renewal. |
| C-0BCDB8C2 | Cole Ingram | 54,427.00 | 2026-09-18 | Chargebee; disagreement vs ChurnZero (2027-09-18) | 232 ÷ 424 = 54.7% | 127 → 118 → 110; (110-127) ÷ 127 = -13.4% | High | Low utilization and sustained decline; I trust Chargebee because the ChurnZero date is a multi-year renewal. |
| C-0D2AB865 | Elena Sinclair | 38,022.00 | 2026-09-22 | Chargebee; disagreement vs ChurnZero (2026-09-10) | 250 ÷ 407 = 61.4% | 125 → 117 → 109; (109-125) ÷ 125 = -12.8% | High | Utilization is only moderate and usage is falling fast; I trust Chargebee because the ChurnZero date is a multi-year renewal. |
| C-0BBE3E60 | Dana Mercer | 30,993.00 | 2026-09-26 | Chargebee; disagreement vs ChurnZero (2027-09-26) | 74 ÷ 114 = 64.9% | 39 → 35 → 33; (33-39) ÷ 39 = -15.4% | High | Moderate utilization with continued decline; I trust Chargebee because the ChurnZero date is a multi-year renewal. |
| C-0F5D2323 | Cole Ingram | 90,647.00 | 2026-09-29 | Chargebee; disagreement vs ChurnZero (2026-09-10) | 111 ÷ 390 = 28.5% | 20 → 21 → 18; (18-20) ÷ 20 = -10.0% | High | Very low utilization and usage is down over 3 months; I trust Chargebee because the ChurnZero date is a multi-year renewal. |
| C-0EC6999D | Elena Sinclair | 79,419.00 | 2026-10-03 | No disagreement; both systems match | 31 ÷ 112 = 27.7% | 17 → 16 → 15; (15-17) ÷ 17 = -11.8% | High | Very low utilization and continued decline; no date conflict to resolve. |
| C-0B20DB64 | Dana Mercer | 21,770.00 | 2026-10-07 | No disagreement; both systems match | 214 ÷ 378 = 56.6% | 294 → 298 → 294; (294-294) ÷ 294 = 0.0% | Medium | Utilization is below 60% but usage is flat, so this is risk but not a steep deterioration. |
| C-0BBC4E7A | Cole Ingram | 56,374.00 | 2026-10-10 | No disagreement; both systems match | 228 ÷ 337 = 67.7% | 142 → 141 → 139; (139-142) ÷ 142 = -2.1% | Medium | Moderate utilization with a slight downtrend keeps this in the middle of the pack. |
| C-0FD551AB | Elena Sinclair | 48,815.00 | 2026-10-14 | No disagreement; both systems match | 210 ÷ 376 = 55.9% | 123 → 122 → 126; (126-123) ÷ 123 = 2.4% | Medium | Utilization is below 60%, but usage edged up in August, so risk is present but not severe. |
| C-0F9F8F13 | Dana Mercer | 46,230.00 | 2026-10-18 | No disagreement; both systems match | 199 ÷ 352 = 56.5% | 185 → 185 → 182; (182-185) ÷ 185 = -1.6% | Medium | Utilization is below 60% and usage is slightly down, indicating moderate renewal risk. |
| C-0BC34584 | Cole Ingram | 16,740.00 | 2026-10-22 | No disagreement; both systems match | 327 ÷ 494 = 66.2% | 104 → 104 → 106; (106-104) ÷ 104 = 1.9% | Medium | Utilization is midrange and usage is basically flat to slightly up. |
| C-0B7A7546 | Elena Sinclair | 35,062.00 | 2026-10-25 | No disagreement; both systems match | 182 ÷ 205 = 88.8% | 64 → 65 → 63; (63-64) ÷ 64 = -1.6% | Low | Very high utilization and essentially flat usage make this low risk. |
| C-0B369871 | Dana Mercer | 85,128.00 | 2026-10-29 | No disagreement; both systems match | 317 ÷ 422 = 75.1% | 326 → 330 → 333; (333-326) ÷ 326 = 2.1% | Low | Utilization is just over 75% and usage is rising slightly, so renewal risk looks low. |
| C-0B144C78 | Cole Ingram | 30,899.00 | 2026-11-02 | No disagreement; both systems match | 169 ÷ 224 = 75.4% | 101 → 101 → 106; (106-101) ÷ 101 = 5.0% | Low | Utilization is above 75% and usage increased in August. |
| C-0FC4DBB8 | Elena Sinclair | 94,732.00 | 2026-11-05 | No disagreement; both systems match | 356 ÷ 464 = 76.7% | 189 → 191 → 193; (193-189) ÷ 189 = 2.1% | Low | Strong utilization and continued usage growth indicate low risk. |
| C-0D5BBE3A | Dana Mercer | 39,740.00 | 2026-11-09 | No disagreement; both systems match | 85 ÷ 102 = 83.3% | 88 → 90 → 91; (91-88) ÷ 88 = 3.4% | Low | High utilization and steady growth make this a low-risk renewal. |
| C-0FB9D5AF | Cole Ingram | 63,158.00 | 2026-11-13 | No disagreement; both systems match | 144 ÷ 199 = 72.4% | 173 → 173 → 176; (176-173) ÷ 173 = 1.7% | Medium | Utilization is below 75%, so even with slight growth this stays in the middle. |
| C-0B344485 | Elena Sinclair | 64,384.00 | 2026-11-16 | No disagreement; both systems match | 224 ÷ 287 = 78.0% | 238 → 240 → 244; (244-238) ÷ 238 = 2.5% | Low | Healthy utilization and improving usage support a low-risk view. |
| C-0CB2C1B4 | Dana Mercer | 40,628.00 | 2026-11-20 | No disagreement; both systems match | 386 ÷ 473 = 81.6% | 47 → 48 → 49; (49-47) ÷ 47 = 4.3% | Low | High utilization and a small usage increase indicate low risk. |
| C-22170CA1 | Cole Ingram | 45,646.00 | 2026-11-24 | No disagreement; both systems match | 251 ÷ 294 = 85.4% | 143 → 148 → 146; (146-143) ÷ 143 = 2.1% | Low | Very strong utilization and broadly stable usage keep this low risk. |

Total ARR renewing = 65,901.00 + 54,427.00 + 38,022.00 + 30,993.00 + 90,647.00 + 79,419.00 + 21,770.00 + 56,374.00 + 48,815.00 + 46,230.00 + 16,740.00 + 35,062.00 + 85,128.00 + 30,899.00 + 94,732.00 + 39,740.00 + 63,158.00 + 64,384.00 + 40,628.00 + 45,646.00 = 1,048,715.00

ARR at risk (High only) = 65,901.00 + 54,427.00 + 38,022.00 + 30,993.00 + 90,647.00 + 79,419.00 = 359,409.00
ticket-theme-synthesis0.80

? The Question

Synthesize the quarter's support tickets into 4 to 7 themes. Do not trust existing tags; read the text. Per theme: name, count, share, distinct accounts, ARR affected, two ticket ids, one-line recommendation. Separate broad patterns from single-account noise and rank by ARR exposure, not volume.
cost $0.2701 · 242s · in 69,734 / out 48,177 tok
✗ theme_count_range✓ counts_verifiable✓ single_account_flagged✓ arr_ranked✓ ticket_ids

✓ Correct Answer (ground truth)

quarter_window: 2026-06-01 through 2026-08-31
total_tickets: 80
themes: {"points_not_posting": {"count": 20, "share_pct": 25.0, "distinct_accounts": 9, "arr_affected": "31100.00", "example_ticket_ids": ["IC-460001", "IC-460002"], "single_account": false}, "slack_sync": {"
ranking_by_arr_exposure: hris_provisioning, redemption_failures, billing_dispute, points_not_posting, slack_sync
single_account_noise_theme: billing_dispute
note: existing tags are noisy/unreliable; classify by text
sample_ticket_ids: 80 items — e.g. IC-460004, IC-460016, IC-460001, IC-460041
graded checks: theme_count_range · counts_verifiable · single_account_flagged · arr_ranked · ticket_ids

Model's actual answer

show the model's full answer
Total tickets: 80. Total ARR affected: $1,598,900.

Single-account noise
1) Invoice / seat-count / renewal pricing
- Count: 13; share: 13/80 = 16.25%
- Distinct accounts: 1 (C-0E9C27D1 only)
- ARR affected: 13 × $52,000 = $676,000
- Ticket ids: IC-460069, IC-460078
- Recommendation: Treat as an account-specific billing issue; audit C-0E9C27D1’s seat-count, renewal-tier, and approval mapping before broadening to product.

Broad patterns, ranked by ARR exposure
2) HRIS provisioning / sync
- Count: 12; share: 12/80 = 15.00%
- Distinct accounts: 3
- ARR affected: $456,000 (sum of the 12 ticket ARRs)
- Ticket ids: IC-460059, IC-460055
- Recommendation: Prioritize HRIS sync reliability, backfill/retry behavior, and clearer provisioning failure visibility.

3) Points not posting / missing points
- Count: 23; share: 23/80 = 28.75%
- Distinct accounts: 10
- ARR affected: $226,200 (sum of the 23 ticket ARRs)
- Ticket ids: IC-460004, IC-460016
- Recommendation: Investigate recognition posting latency/failures and the balance update pipeline end to end.

4) Redemption / checkout / gift card fulfillment
- Count: 18; share: 18/80 = 22.50%
- Distinct accounts: 7
- ARR affected: $177,300 (sum of the 18 ticket ARRs)
- Ticket ids: IC-460025, IC-460030
- Recommendation: Fix checkout timeout and gift-card fulfillment failures, and make errors visible instead of silent.

5) Slack integration / slash command failures
- Count: 14; share: 14/80 = 17.50%
- Distinct accounts: 4
- ARR affected: $63,400 (sum of the 14 ticket ARRs)
- Ticket ids: IC-460041, IC-460047
- Recommendation: Stabilize Slack auth, sync persistence, and slash-command reliability; verify re-auth stickiness.
marketing
case-study-matchPASS 1.00

? The Question

Given the prospect, rank the three most similar existing customers with public case studies for use as social proof. Similarity on industry, size band, use case, region. Cite the fields that drove each match. Do not name any customer without a case study and never include billing data or contact names.
cost $0.0260 · 20s · in 14,921 / out 3,289 tok
✓ top3✓ decoys_excluded✓ cites_fields✓ no_billing

✓ Correct Answer (ground truth)

top3: company_alias=C-11C31562; similarity_score=3 | company_alias=C-64171065; similarity_score=3 | company_alias=C-A13C193D; similarity_score=3
decoys_excluded: company_alias=C-D6217CAA; similarity_score=4; reason=no_case_study | company_alias=C-C153868F; similarity_score=4; reason=no_case_study
constraints: no_billing_data, no_contact_names, only_customers_with_has_case_study_true_named
prospect_fields_used: industry, size_band, use_case, region
prospect_alias: C-82AF3719
graded checks: top3 · decoys_excluded · cites_fields · no_billing

Model's actual answer

show the model's full answer
Similarity score = exact matches across 4 fields (industry, size_band, use_case, region).

No public-case-study customer matches all 4 fields.

1) C-64171065 — 3/4
   Matches: industry=Technology, size_band=Mid-Market, use_case=employee_recognition
   Mismatch: region=NA-East vs prospect NA-West
   Arithmetic: 1+1+1+0 = 3

2) C-11C31562 — 3/4
   Matches: size_band=Mid-Market, use_case=employee_recognition, region=NA-West
   Mismatch: industry=Manufacturing vs prospect Technology
   Arithmetic: 0+1+1+1 = 3

3) C-A13C193D — 3/4
   Matches: industry=Technology, size_band=Mid-Market, region=NA-West
   Mismatch: use_case=retention vs prospect employee_recognition
   Arithmetic: 1+1+0+1 = 3
channel-attribution-mathPASS 1.00

? The Question

Compute channel performance for the trailing 6 months. Paid: spend, SQMs, SQOs, cost per SQM, cost per SQO, SQM-to-SQO rate, pipeline amount, pipeline per dollar. Organic: volume, SQO rate, pipeline. Report a channel with spend and zero SQMs as undefined, not zero. Flag rows where SQO date precedes SQM date. Finish with a reallocation recommendation and your confidence given sample sizes.
cost $0.0676 · 51s · in 25,284 / out 10,196 tok
✓ undefined_channel✓ undefined_not_zero✓ flagged_rows✓ best_channel✓ reallocation

✓ Correct Answer (ground truth)

months: 2026-03, 2026-04, 2026-05, 2026-06, 2026-07, 2026-08
per_channel: {"linkedin_ads": {"spend_usd": 24000, "sqms": 25, "sqos": 8, "pipeline_amount": 96000, "cost_per_sqm": 960.0, "cost_per_sqo": 3000.0, "sqm_to_sqo_rate": 0.32, "pipeline_per_dollar": 4.0}, "organic_sea
undefined_channels: paid_social
flagged_rows_sqo_before_sqm: CT-000041, CT-000044
best_pipeline_per_dollar_channel: paid_search
reallocation_recommendation: from=paid_social; to=paid_search; rationale=paid_social has spend but zero SQMs (cost per SQM undefined, not zero); paid_search has the highest pipeline per dollar.
confidence_note: webinars and referral have small SQM samples (<20); treat rates as directional.
graded checks: undefined_channel · undefined_not_zero · flagged_rows · best_channel · reallocation

Model's actual answer

show the model's full answer
Trailing 6 months (2026-03 to 2026-08)

Paid = paid_search + linkedin_ads + paid_social + webinars
- Spend: 36,000 + 24,000 + 18,000 + 9,000 = 87,000
- SQMs: 40 + 25 + 0 + 12 = 77
- SQOs: 18 + 8 + 0 + 5 = 31
- Cost per SQM: 87,000 / 77 = 1,129.87
- Cost per SQO: 87,000 / 31 = 2,806.45
- SQM-to-SQO rate: 31 / 77 = 40.26%
- Pipeline: 720,000 + 96,000 + 0 + 60,000 = 876,000
- Pipeline per dollar: 876,000 / 87,000 = 10.07

Organic = organic_search
- Volume (SQMs): 30
- SQO rate: 10 / 30 = 33.33%
- Pipeline: 90,000

Undefined row
- paid_social: spend 18,000, SQMs 0 -> cost per SQM, cost per SQO, SQM-to-SQO rate, and pipeline per dollar are undefined, not zero

SQO-before-SQM flags
- CT-000044, linkedin_ads: SQM 2026-07-23, SQO 2026-07-18
- CT-000041, linkedin_ads: SQM 2026-06-14, SQO 2026-06-09

Reallocation recommendation
- Move budget out of paid_social first, since it has spend with 0 SQMs.
- Next, favor paid_search over linkedin_ads and webinars on efficiency: paid_search has the best pipeline per dollar at 10.07 and the largest pipeline total at 720,000.
- If you need a secondary recipient, webinars is materially better than linkedin_ads on pipeline per dollar (6.67 vs 4.00).

Confidence
- Paid_search: moderate-high confidence; 40 SQMs and 18 SQOs.
- LinkedIn_ads: moderate confidence; 25 SQMs and 8 SQOs, plus 2 date-order anomalies.
- Webinars: moderate-low confidence; 12 SQMs and 5 SQOs.
- Organic_search: moderate confidence; 30 SQMs and 10 SQOs.
- Overall confidence in the reallocation is moderate, because sample sizes are still limited and there is no incrementality/control data.
competitor-battlecard-update0.80

? The Question

Update the battlecard for the competitor. Sections: one-line positioning, pricing with source and date (newer source wins, note the conflict), where they win, where we win, objections and responses, recent changes, our 12-month win/loss record against them. Cite a snippet id for every factual claim. Rep opinion on a call is not a fact about the competitor. Anything from the old card you cannot re-source gets marked unverified.
cost $0.0390 · 50s · in 3,593 / out 7,319 tok
✓ competitor_named✓ pricing✓ win_loss✓ unverified_marked✗ citations

✓ Correct Answer (ground truth)

competitor: Rivally
pricing: value_per_user_per_month=7; source_snippet=S17; source_date=2026-08-12
win_loss_12mo: wins=13; losses=7
unverified_old_claims: Rivally lacks a Slack integration., Rivally was acquired by WorkHuman in 2025.
rep_opinion_snippet_ids_not_facts: S09, S21
citation_ids_available: 25 items — e.g. S01, S02, S03, S04
graded checks: competitor_named · pricing · win_loss · unverified_marked · citations

Model's actual answer

show the model's full answer
# Battlecard: Rivally

One-line positioning
- Rivally is a points-based recognition product with quick setup, Slack support, and an expanding EU footprint. [S02][S04][S12][S15]

Pricing
- Current public pricing: Recognition Starter is $7 per user/month, annual billing required (2026-08-12). [S17]
- Conflict: older public pricing pages showed $5 per user/month, annual billing on 2026-01-20 and again on 2026-04-01. [S03][S08]
- Additional pricing signals: a 2026-06-02 deal note said Rivally quoted $6.50/user/mo to a 500-seat prospect on an annual term, and a 2026-08-14 deal note said list was $7/user/mo with a 15% discount for a 3-year term. [S13][S18]
- Newer source wins: $7 per user/month, annual billing required. [S17]

Where they win
- Strong when the buyer wants a simple points-based recognition feed. [S02][S16]
- Fast rollout / low-friction admin: one reviewer said setup took under a week and Slack integration worked out of the box. [S04]
- EU narrative: they pitched EU data residency, hired an ex-Workday VP EMEA, later made EU data residency generally available, and reviewers praised them for distributed EU teams and multi-language support. [S05][S11][S12][S15]
- Support can be a positive: one reviewer praised response time under 4 hours. [S22]

Where we win
- Analytics depth: reviewers called Rivally’s analytics/reporting limited or basic, and one migration review said exports are CSV-only. [S02][S07][S20]
- Admin/provisioning: Rivally lacks SCIM provisioning, manual user management is painful, and bulk recognition editing is still missing. [S10][S24]
- EMEA rewards depth: one review said the rewards catalog in EMEA is thinner than the US catalog. [S14]

Objections and responses
- “They’re cheaper.” Response: current public pricing is $7/user/mo annual; older $5 pricing is stale, and later deal notes show $6.50/user/mo on an annual term and $7/user/mo list with a 15% discount for a 3-year term. [S17][S03][S08][S13][S18]
- “They’re easy to implement.” Response: yes, but the review set also flags limited/basic analytics, no SCIM, painful manual admin, and CSV-only exports. [S04][S02][S07][S10][S20][S24]
- “They’re strong in Europe.” Response: they do have EU data residency and multi-language praise, but the EMEA rewards catalog is thinner than the US catalog. [S12][S15][S14]
- Do not use rep opinions on UI or discounting as facts; those are explicitly unverified. [S09][S21]

Recent changes
- 2026-03-05: launched Rivally Pulse as a lightweight engagement survey add-on. [S06]
- 2026-07-01: opened a Dublin office and announced EU data residency generally available. [S15]
- 2026-08-12: pricing page moved Recognition Starter to $7 per user/month, annual billing required. [S17]
- 2026-08-20: announced Microsoft Teams app v2 in public preview. [S19]
- 2026-09-01: Pulse exited beta and is now priced as an add-on, not bundled. [S23]

Our 12-month win/loss record against them
- 2025-09 to 2026-08 total: 20 deals = 13 wins + 7 losses.
- Arithmetic: 13 / 20 = 0.65 = 65% win rate.
- Month-by-month:
  - 2025-09: Deal-7767F5 loss, Deal-072E31 win
  - 2025-10: Deal-A9FD43 win, Deal-F65C8F win
  - 2025-11: Deal-7AA785 win, Deal-D263E0 loss
  - 2025-12: Deal-44C524 win, Deal-935746 loss
  - 2026-01: Deal-0D0CD6 win, Deal-E46EAB win
  - 2026-02: Deal-D5B790 win, Deal-1D2392 win
  - 2026-03: Deal-5C636E win, Deal-9066A6 loss
  - 2026-04: Deal-5645A5 loss, Deal-72A02F loss
  - 2026-05: Deal-C6FFAA loss
  - 2026-06: Deal-67BE14 win
  - 2026-07: Deal-1B6969 win
  - 2026-08: Deal-F03E7B win
nurture-sequence-diagnosisPASS 1.00

? The Question

Review the sequences. Per sequence: sent, open rate, reply rate, meeting rate, weakest step. Identify tracking errors (opened above sent), audience overlap between sequences, and the failure mode for anything under 2 percent reply. One change per weak sequence and which to fix first. Under 350 words.
cost $0.0920 · 60s · in 24,244 / out 15,758 tok
✓ tracking_error✓ overlap_pair✓ weakest_step✓ fix_priority

✓ Correct Answer (ground truth)

per_sequence: {"New Logo Nurture": {"sent": 1386, "opened": 490, "replied": 90, "meetings": 27, "open_rate": 0.3535, "reply_rate": 0.0649, "meeting_rate": 0.0195, "weakest_step": 3}, "Expansion Nurture": {"sent": 8
tracking_error: sequence=Expansion Nurture; step=2; sent=300; opened=340; issue=opened exceeds sent
audience_overlap_pair: Cold Outbound - HR Leaders, Cold Outbound - People Ops
all_pairwise_overlaps: ['Cold Outbound - HR Leaders', 'Cold Outbound - People Ops'], ['Expansion Nurture', 'New Logo Nurture']
under_2pct_reply: Cold Outbound - HR Leaders
failure_mode: Cold Outbound - HR Leaders=opens healthy (~37%) but replies near zero from step 1 -> body/CTA not landing, not a deliverability or subject-line problem
fix_priority_first: Cold Outbound - HR Leaders
graded checks: tracking_error · overlap_pair · weakest_step · fix_priority

Model's actual answer

show the model's full answer
New Logo Nurture: sent 1,386 (=500+458+428); open rate 490/1,386=35.35%; reply rate 90/1,386=6.49%; meeting rate 27/1,386=1.95%; weakest step = 3 (18/428=4.21% reply).

Expansion Nurture: sent 875 (=300+300+275); open rate 565/875=64.57%; reply rate 59/875=6.74%; meeting rate 12/875=1.37%; weakest step = 3 (12/275=4.36% reply). Tracking error: step 2 opened 340 > sent 300.

Cold Outbound - HR Leaders: sent 1,785 (=600+595+590); open rate 545/1,785=30.53%; reply rate 8/1,785=0.45%; meeting rate 0/1,785=0.00%; weakest step = 3 (1/590=0.17% reply). Failure mode under 2% reply: opens are not converting to replies at all.

Cold Outbound - People Ops: sent 1,163 (=400+386+377); open rate 340/1,163=29.23%; reply rate 29/1,163=2.49%; meeting rate 6/1,163=0.52%; weakest step = 3 (6/377=1.59% reply).

Audience overlap:
- Cold Outbound - HR Leaders ↔ Cold Outbound - People Ops: 21 shared contact_keys.
- Expansion Nurture ↔ New Logo Nurture: 2 shared contact_keys (CT-000301, CT-000624).
- Total shared contact_keys across sequences: 23.

One change per weak sequence:
- New Logo Nurture: fix step 3 messaging/CTA.
- Expansion Nurture: fix the step 2 tracking error first.
- Cold Outbound - HR Leaders: change the audience/fit first.
- Cold Outbound - People Ops: fix step 3 messaging/CTA.

Fix first: Cold Outbound - HR Leaders, because 0.45% reply is the worst and below 2%.
weekly-marketing-goals-rollupPASS 1.00

? The Question

Produce the weekly marketing goals update for the active quarter: SQMs, SQOs, DS2s, closed-lost MIA rate, same-quarter close count, active pipeline coverage against target. For each: QTD actual, target, delta, pace (on/behind/ahead given days elapsed). One paragraph on what moved this week.
cost $0.0102 · 16s · in 707 / out 1,927 tok
✓ metrics_present✓ deltas✓ coverage_ratio✓ pace✓ paragraph

✓ Correct Answer (ground truth)

days_elapsed: 66
days_in_quarter: 92
elapsed_fraction: 0.7174
metrics: {"SQMs": {"qtd_actual": 230, "target": 300, "delta": -70, "expected_at_day_elapsed": 215.22, "pace": "ahead"}, "SQOs": {"qtd_actual": 84, "target": 120, "delta": -36, "expected_at_day_elapsed": 86.09,
coverage_ratio_active_pipeline: 0.75
pace_rule: expected = target * days_elapsed/days_in_quarter; within +/-5% of expected = on, above = ahead, below = behind; lower_better metrics invert.
this_week_paragraph_required: True
graded checks: metrics_present · deltas · coverage_ratio · pace · paragraph

Model's actual answer

show the model's full answer
Quarter: Q3-2026
Days elapsed: 66/92 = 71.7% of quarter

Metric | QTD actual | Target | Delta | Pace
SQMs | 230 | 300 | 230 - 300 = -70 | ahead (230/300 = 76.7%, above 71.7% elapsed)
SQOs | 84 | 120 | 84 - 120 = -36 | behind (84/120 = 70.0%, below 71.7% elapsed)
DS2s | 40 | 75 | 40 - 75 = -35 | behind (40/75 = 53.3%, below 71.7% elapsed)
closed-lost MIA rate | 5/25 = 0.20 = 20.0% | 0.10 = 10.0% | 20.0% - 10.0% = +10.0 pp | behind (actual rate is 2.0x target rate)
same-quarter close count | 10 | 20 | 10 - 20 = -10 | behind (10/20 = 50.0%, below 71.7% elapsed)
active pipeline coverage against target | 3,000,000/4,000,000 = 0.75 = 75.0% | 4,000,000 | 3,000,000 - 4,000,000 = -1,000,000 (-25.0%) | ahead (75.0% of target, above 71.7% elapsed)

What moved this week: no week-over-week data was provided, so I can’t determine what moved this week from the files alone; this is only a QTD status readout.
communication
ceo-slack-compressionPASS 1.00

? The Question

Compress the forecast analysis below into a single Slack message for the CEO. Rules: at most three sentences; under 80 words; lead with the decision or the so-what; keep every distinct number that matters and the one caveat that changes the conclusion; cut any sentence that does not carry a claim, a number, or a directive. No em dashes, no throat-clearing phrases.
cost $0.0132 · 6s · in 14,814 / out 470 tok
✓ max_3_sentences✓ under_80_words✓ weighted_present✓ caveat_present✓ no_em_dash

✓ Correct Answer (ground truth)

graded checks: max_3_sentences · under_80_words · weighted_present · caveat_present · no_em_dash

Model's actual answer

show the model's full answer
Q3 forecast is 115,976.75 from 54 of 86 in-quarter deals: 44,729 COMMIT + 0.35 x 203,565 BEST_CASE = 115,976.75; PIPELINE is 0. Caveat: owner is blank on 85 of 86 and why-buys is empty on all 7 COMMIT deals, so the number is precise but not defensible.
stalled-deal-followup-emailPASS 1.00

? The Question

Write a follow-up email for this stalled deal. Under 120 words. Reference one specific prior touch from the engagement history, make one clear ask, and match the tone of the prior thread. No invented details.
cost $0.0034 · 8s · in 812 / out 386 tok
✓ word_cap✓ references_touch✓ single_ask

✓ Correct Answer (ground truth)

deal_alias: Deal-0D2F7A
amount: 5100
stage: DS3
owner: Alex Franklin
days_since_last_engagement: 31
word_cap: 120
must_reference: engagement_id=ENG-00002; date=2026-07-28; type=meeting; why=most substantive prior touch; concrete prospect reactions to milestone awards and points catalog
single_cta: True
tone: warm, concise, professional (matches prior thread)
allowed_facts: 14 items — e.g. Deal-0D2F7A, $5,100, DS3, Alex Franklin
allowed_new: ENG-00001, ENG-00002, ENG-00003, ENG-00004, ENG-00005
graded checks: word_cap · references_touch · single_ask

Model's actual answer

show the model's full answer
Hi Alex,

Following up on the July 28 demo with the People team — glad the automated milestone awards and points catalog resonated. On August 5, I sent a recap with pricing for 150 seats and haven’t heard back.

Would you like me to resend the pricing or set up a quick 15-minute follow-up to answer any questions?

Best,
[Your Name]
funnel-digest-paragraphPASS 1.00

? The Question

Write the weekly GTM digest: one paragraph per team (Marketing, Sales, CS, PLG) in Bonusly brand voice. Every number must trace to the extracts. No customer contact names, no employee performance commentary, no compensation. Four paragraphs, under 400 words total.
cost $0.0074 · 12s · in 996 / out 1,005 tok
✓ four_paragraphs✓ word_cap✓ teams

✓ Correct Answer (ground truth)

required_paragraphs: Marketing, Sales, CS, PLG
allowed_numbers: 16 items — e.g. 2, 3, 9, 08
word_cap: 400
banned: customer contact names, employee performance commentary, compensation
graded checks: four_paragraphs · word_cap · teams

Model's actual answer

show the model's full answer
Marketing: Nice work closing the week at 46 SQMs vs 52 target, which is 46 ÷ 52 = 88.5% of goal, leaving a gap of 6 SQMs. The webinar channel contributed 18 SQMs, and paid search held steady at $150 per SQM.

Sales: Strong execution with 14 SQOs converted, 9 DS2 meetings set, $310,000 in new pipeline created, and 3 same-quarter closes. No extra math needed here beyond the raw weekly output, and the momentum is showing up across the funnel.

CS: The team saved 2 renewals, moved Team NPS to 61, and enters next week with 3 open red-flag accounts. That’s a clear week of retention work, with the risk list kept visible and manageable.

PLG: PLG added 412 new signups with activation at 31%, which is 412 × 31% = 127.72, or about 128 activated signups. On top of that, 38 companies reached the aha moment of 10 recognition gives.
partner-digest-sectionPASS 1.00

? The Question

Write the partner digest for the period. Every partner in the roster appears once: active plays get status with numbers, quiet partners get one line. Partner-sourced pipeline count and amount must match the deal data. Nothing omitted, nothing added.
cost $0.0090 · 12s · in 1,609 / out 1,265 tok
✓ all_partners✓ pipeline_amount✓ quiet_one_line

✓ Correct Answer (ground truth)

roster: partner_name=Apex Rewards Co; status=active | partner_name=HRCloud Partners; status=active | partner_name=CultureBridge; status=active | partner_name=WorkWell Group; status=active | partner_name=Recogniq; status=quiet | partner_name=KudosWave; status=quiet … (+2 more)
per_partner_pipeline: {"Apex Rewards Co": {"count": 2, "amount": 275000}, "HRCloud Partners": {"count": 1, "amount": 140000}, "CultureBridge": {"count": 2, "amount": 135000}}
partner_sourced_pipeline: count=5; amount=550000
quiet_partners: Recogniq, KudosWave, PeopleFirst Advisors, TotalPerk
active_no_deals: WorkWell Group
rule: every roster partner named exactly once; quiet partners get one line; no partners outside roster.
graded checks: all_partners · pipeline_amount · quiet_one_line

Model's actual answer

show the model's full answer
Partner digest for 2026-08-24 to 2026-09-04

Apex Rewards Co — active
- 2 partner-sourced deals: Deal-DDAAF2 ($180,000) + Deal-2CF33E ($95,000) = $275,000.
- From Slack: co-webinar locked for 09-15; both sourced opps logged with UTM Source = Partner.

HRCloud Partners — active
- 1 partner-sourced deal: Deal-F1CDA5 ($140,000) = $140,000.
- From Slack: integration referral closed the security review; opp moved to DS2; one sourced opp this period.

CultureBridge — active
- 2 partner-sourced deals: Deal-096E1D ($60,000) + Deal-067213 ($75,000) = $135,000.
- From Slack: lunch-and-learn produced two sourced opps; both early stage.

WorkWell Group — active
- 0 partner-sourced deals = $0.
- From Slack: restart the joint playbook in Q4; no sourced deals this period; planning call booked for 09-09.

Recogniq — quiet
- No partner-sourced deals or activity provided.

KudosWave — quiet
- No partner-sourced deals or activity provided.

PeopleFirst Advisors — quiet
- No partner-sourced deals or activity provided.

TotalPerk — quiet
- No partner-sourced deals or activity provided.

Partner-sourced pipeline total: 5 deals = $550,000
- $180,000 + $95,000 + $140,000 + $60,000 + $75,000 = $550,000
rfp-answer-from-policyPASS 1.00

? The Question

Answer the 10 questionnaire items from the policy excerpts only. Cite the excerpt id for every answer. Where the excerpts do not answer the question, mark it unanswerable and say what document would; do not guess. Tone: vendor response, direct.
cost $0.0045 · 6s · in 861 / out 623 tok
✓ answered_count✓ unanswerable_flagged✓ citations✓ no_guess

✓ Correct Answer (ground truth)

answered_count: 7
answer_map: {"Q1": ["P1"], "Q2": ["P1"], "Q3": ["P2"], "Q4": ["P3"], "Q5": ["P4"], "Q6": ["P5"], "Q7": ["P6"]}
unanswerable: Q8, Q9, Q10
valid_citation_ids: P1, P2, P3, P4, P5, P6, P7, P8
rule: unanswerable items must be flagged and the document that would answer them named; do not guess.
graded checks: answered_count · unanswerable_flagged · citations · no_guess

Model's actual answer

show the model's full answer
Q1: Yes. Customer data is encrypted at rest using AES-256. [P1]

Q2: Yes. Data in transit is encrypted with TLS 1.2 or higher. [P1]

Q3: Yes. Bonusly maintains a SOC 2 Type II report, renewed annually, and the report is available to prospects under NDA. [P2]

Q4: Customer data is retained for the duration of the contract plus 90 days, then deleted from production systems. [P3]

Q5: The current subprocessor list is published at trust.bonusly.com, and customers receive 30 days’ notice before a new subprocessor is added. [P4]

Q6: Yes. SAML 2.0 single sign-on is available on the Enterprise plan. [P5]

Q7: Production databases are backed up daily. Backups are encrypted and retained for 35 days. [P6]

Q8: Unanswerable from the excerpts. No ISO 27001 certification statement is provided. A security certification or compliance document would be needed. [P1–P8]

Q9: Unanswerable from the excerpts. No contractual uptime SLA percentage is provided. The MSA/SLA terms would be needed. [P1–P8]

Q10: Unanswerable from the excerpts. No HIPAA Business Associate Agreement terms are provided. A HIPAA/BAA addendum or legal terms document would be needed. [P1–P8]
ops-maintenance
skill-file-reconciliationPASS 1.00

? The Question

Reconcile the skill set below against its manifest. Find and report (with severity CRITICAL/WARNING/INFO and an action type TRIM_DESC, UPDATE_BODY, DELETE_SKILL, MERGE, or REVIEW): (1) any two skills whose ALWAYS-trigger phrases overlap/duplicate, (2) any circular delegation chain between skills (name it), (3) any delegation target that does not exist (dangling, name it), (4) any version conflict and which skill should survive, (5) how many manifest descriptions exceed 1,024 characters, (6) any hardcoded page ids, dates, or person names in skill bodies, (7) manifest drift in BOTH directions (files with no manifest row, and manifest rows with no file). Make one proposal per finding; rewrite nothing.
cost $0.1717 · 60s · in 162,909 / out 11,012 tok
✓ duplicate_trigger_found✓ circular_chain✓ dangling_target✓ oversized_desc_count✓ hardcoded_values✓ manifest_drift✓ severity_and_action

✓ Correct Answer (ground truth)

graded checks: duplicate_trigger_found · circular_chain · dangling_target · oversized_desc_count · hardcoded_values · manifest_drift · severity_and_action

Model's actual answer

show the model's full answer
1) WARNING / UPDATE_BODY
Overlap / duplicate ALWAYS-trigger phrases:
- comms-drafter and email-drafter both trigger on the same email-writing language, including “write me an email,” “draft a follow-up,” “review this email,” and “what should I say.”
Proposal: keep email-drafter as the email-specific executor and trim comms-drafter’s email triggers or route them to email-drafter.

2) CRITICAL / UPDATE_BODY
Circular delegation chain:
- deal-strategy-coach ↔ email-drafter
- deal-strategy-coach sends drafting work to email-drafter, and email-drafter sends strategy work back to deal-strategy-coach.
Proposal: make the handoff one-way only so the loop cannot recur.

3) CRITICAL / UPDATE_BODY
Dangling delegation target(s):
- prospect-research-multithreading is referenced in deal-strategy-coach and email-drafter, but it does not appear in the manifest.
- bonusly-brand is also referenced in comms-drafter, email-drafter, and sales-forecast, but it does not appear in the manifest.
Proposal: add the missing skills or remove those references.

4) WARNING / UPDATE_BODY
Version conflict:
- analysis-validator’s body says version 3.6, but the validation trail template still says “analysis-validator v3.2.”
Survivor: analysis-validator v3.6.
Proposal: update the trail template to v3.6 everywhere.

5) INFO / REVIEW
Manifest descriptions over 1,024 characters:
- 0.
Arithmetic: max(description_chars) = 1006, so 1006 < 1024 and 0/14 exceed the limit.

6) WARNING / UPDATE_BODY
Hardcoded page ids, dates, and person names in skill bodies:
- analysis-validator: hardcoded dates and names in the body/changelog.
- deal-strategy-coach: hardcoded page id 2257879045, dates, and person names.
- partner-digest: hardcoded page ids 2286616609, 2265382925, 2236940297, 2237825028, 2239365136, 2238283777, plus names.
- sales-forecast: hardcoded page ids 2232811524 and 2232582148, plus names.
- weekly-pipeline-report: hardcoded person name Ben Lavin and dated references.
- signalforge-feedback: hardcoded page ids 2295136266, 2234417154, 2247295002.
Proposal: move IDs, dates, and names into references or runtime lookups.

7) INFO / REVIEW
Manifest drift in both directions:
- Files with no manifest row: 0/14.
- Manifest rows with no file: 0/14.
Proposal: no change needed; the supplied set is in sync.
runbook-from-slack-threadPASS 1.00

? The Question

Turn the resolved incident thread into a runbook: ordered steps with the exact command or action taken, who ran it, how success was verified, and rollback for each step that changed state. Every step must trace to a message in the thread. Add nothing the thread does not contain; where the thread is unclear, mark the step as needing confirmation.
cost $0.0078 · 13s · in 1,035 / out 1,336 tok
✓ ordered_steps✓ rollback✓ needs_confirmation✓ sourced

✓ Correct Answer (ground truth)

steps: order=1; msg=M01; actor=Bryce Harmon; action=acknowledge PagerDuty alert, take IC; state_change=False; verified_by=M01 | order=2; msg=M02; actor=Farid Osman; action=check queue depth; command=bundle exec rake sidekiq:queue_depth; state_change=False; verified_by=M02 (48,213 pending) | order=3; msg=M04; actor=Farid Osman; action=pause auto-enqueue via feature flag; command=bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'; state_change=True; rollback=bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'; verified_by=M07 (queue depth falling) | order=4; msg=M05; actor=Elena Sinclair; action=cleared the dead set; state_change=True; needs_confirmation=thread gives no command or method for clearing the dead set | order=5; msg=M06; actor=Bryce Harmon; action=scale reward-worker to 6 replicas; command=kubectl scale deployment/reward-worker --replicas=6; state_change=True; rollback=kubectl scale deployment/reward-worker --replicas=3; verified_by=M07/M08 | order=6; msg=M08; actor=Cole Ingram; action=verify queue drained and error rate baseline; command=bundle exec rake sidekiq:queue_depth; state_change=False; verified_by=M08 (returns 0) … (+2 more)
ambiguous_step: order=4; msg=M05; mark=needs-confirmation
rule: no steps beyond what the thread contains; every step traces to a message id.
graded checks: ordered_steps · rollback · needs_confirmation · sourced

Model's actual answer

show the model's full answer
Runbook: reward-worker queue incident

1) Acknowledge incident / take IC
- Trace: [M01]
- Who: Bryce Harmon
- Action: PagerDuty alert fired for reward-worker queue depth > 10k; Bryce Harmon acknowledged and took IC.
- Success verification: not stated in the thread.
- Rollback: not applicable.

2) Check queue depth
- Trace: [M02]
- Who: Farid Osman
- Command: `bundle exec rake sidekiq:queue_depth`
- Result: reward queue at 48,213 pending jobs; normal is under 500.
- Success verification: the command returned the queue depth value above.
- Rollback: not applicable.

3) Check dead jobs
- Trace: [M03]
- Who: Farid Osman
- Action: inspected the dead set.
- Result: 112 dead jobs, all Redis::TimeoutError from around 13:58.
- Success verification: dead set contents and error type were observed.
- Rollback: not applicable.

4) Pause enqueue to stop the bleed
- Trace: [M04]
- Who: Farid Osman
- Command: `bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'`
- Result: enqueue paused.
- Success verification: not explicitly stated in the thread.
- Rollback: `bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'`
- Notes: this is a state-changing step and its rollback is explicitly given in the thread.

5) Clear the dead set
- Trace: [M05]
- Who: Elena Sinclair
- Action: “cleared out the dead set” while in the console.
- Exact command: not stated in the thread.
- Success verification: not stated in the thread.
- Rollback: not stated in the thread; needs confirmation.

6) Scale workers up
- Trace: [M06]
- Who: Bryce Harmon
- Command: `kubectl scale deployment/reward-worker --replicas=6` (was 3)
- Result: worker replicas increased from 3 to 6.
- Success verification: later queue depth dropped to 9,400 and falling ~1,200/min [M07].
- Rollback: `kubectl scale deployment/reward-worker --replicas=3`
- Notes: this is a state-changing step and its rollback is explicitly given in the thread.

7) Verify recovery progress
- Trace: [M07]
- Who: Farid Osman
- Action: observed queue depth down to 9,400 and falling ~1,200/min.
- Success verification: the reported decrease itself was the verification.
- Rollback: not applicable.

8) Verify full recovery
- Trace: [M08]
- Who: Cole Ingram
- Command: `bundle exec rake sidekiq:queue_depth`
- Result: returns 0.
- Additional verification: error rate in Datadog back to baseline.
- Rollback: not applicable.

9) Re-enable enqueue
- Trace: [M09]
- Who: Bryce Harmon
- Command: `bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'`
- Result: enqueue re-enabled; 40 new jobs processed cleanly in the next 3 minutes.
- Success verification: “40 new jobs processed cleanly in the next 3 minutes.”
- Rollback: `bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'`
- Notes: the rollback is the inverse of the earlier disable command in [M04].

10) Scale workers back down
- Trace: [M10]
- Who: Bryce Harmon
- Command: `kubectl scale deployment/reward-worker --replicas=3`
- Result: queue stable at 0.
- Success verification: “Queue stable at 0.”
- Rollback: not stated in the thread; needs confirmation.
- Notes: incident resolved.
datadog-incident-timelinePASS 1.00

? The Question

From the log slice, sequence the failure: first error with timestamp, the cascade in order, the service and job involved, and the Datadog query that would confirm the first error. State what the logs do not show.
cost $0.0183 · 21s · in 3,575 / out 2,987 tok
✓ first_error_ts✓ service_named✓ cascade_ordered✓ query✓ gaps_stated

✓ Correct Answer (ground truth)

first_error: timestamp=2026-09-03T14:01:12Z; service=reward-service; message=Redis::TimeoutError: Connection to Redis at redis-primary:6379 timed out after 5s
cascade_order: service=reward-service; first_error_at=2026-09-03T14:01:12Z | service=sidekiq; first_error_at=2026-09-03T14:01:40Z | service=api-gateway; first_error_at=2026-09-03T14:03:05Z | service=web-app; first_error_at=2026-09-03T14:03:30Z
job_class_involved: RewardGiveJob
confirming_query: service:reward-service status:error "@message:Redis::TimeoutError"
logs_do_not_show: root cause of the Redis timeout, redis-server's own logs, infrastructure metrics (CPU/memory/network), any deploy or config change preceding the incident, who applied mitigation or when (no actor fields in logs)
clean_services: postgres
graded checks: first_error_ts · service_named · cascade_ordered · query · gaps_stated

Model's actual answer

show the model's full answer
First error:
2026-09-03T14:01:12Z — reward-service ERROR — Redis::TimeoutError: Connection to Redis at redis-primary:6379 timed out after 5s

Cascade in order:
1) 14:01:12Z reward-service ERROR: Redis connection timeout
2) 14:01:20Z reward-service ERROR: retry exhausted for RewardGiveJob
3) 14:01:30Z reward-service ERROR: retry exhausted for RewardGiveJob
4) 14:01:40Z reward-service ERROR: retry exhausted for RewardGiveJob
5) 14:01:40Z sidekiq ERROR: RewardGiveJob failed: Redis::TimeoutError; retrying in 60s
6) 14:02:28Z sidekiq ERROR: RewardGiveJob failed: Redis::TimeoutError; retrying
7) 14:02:30Z sidekiq WARN: Queue reward depth above 10,000
8) 14:03:05Z api-gateway ERROR: 502 upstream timeout calling reward-service /gives
9) 14:03:30Z web-app ERROR: Give form submission failed: upstream 502 from api-gateway
10) 14:03:31Z sidekiq ERROR: RewardGiveJob failed: Redis::TimeoutError; retrying
11) 14:03:48Z api-gateway ERROR: 502 upstream timeout calling reward-service
12) 14:04:13Z api-gateway ERROR: 502 upstream timeout calling reward-service
13) 14:04:22Z sidekiq ERROR: RewardGiveJob failed: Redis::TimeoutError; retrying
14) 14:04:45Z web-app ERROR: Give form submission failed: upstream 502
15) 14:05:16Z api-gateway ERROR: 502 upstream timeout calling reward-service
16) 14:05:26Z sidekiq ERROR: RewardGiveJob failed: Redis::TimeoutError; retrying
17) 14:05:42Z web-app ERROR: Give form submission failed: upstream 502
18) 14:06:47Z sidekiq ERROR: RewardGiveJob failed: Redis::TimeoutError; retrying
19) 14:06:49Z web-app ERROR: Give form submission failed: upstream 502
20) 14:06:52Z api-gateway ERROR: 502 upstream timeout calling reward-service
21) 14:22:10Z reward-service INFO: Redis connection restored; resuming job processing
22) 14:24:45Z sidekiq INFO: Queue reward depth below 500

Service and job involved:
reward-service and RewardGiveJob

Arithmetic from first error:
14:01:12 -> 14:01:20 = 8s
14:01:12 -> 14:02:30 = 1m 18s
14:01:12 -> 14:03:05 = 1m 53s
14:01:12 -> 14:03:30 = 2m 18s
14:01:12 -> 14:22:10 = 20m 58s
14:01:12 -> 14:24:45 = 23m 33s

Datadog query to confirm the first error:
service:reward-service "Redis::TimeoutError: Connection to Redis at redis-primary:6379 timed out after 5s"

What the logs do not show:
- No pre-timeout root cause for why Redis became unreachable
- No network, Redis, or host-level diagnostic detail
- No stack trace
- No request/customer impact counts
- No specific job enqueue/start timestamp for the first failed RewardGiveJob
feature-flag-state-summaryPASS 1.00

? The Question

Summarize the state of the listed feature flags: what each controls per the code excerpt, which companies or segments have it on, targeting rules, and any flag with no code reference or no targets. Company counts must match the export. Do not describe a flag that is not in the export.
cost $0.0065 · 9s · in 899 / out 1,058 tok
✓ orphans✓ no_targets✓ counts_match✓ no_invented

✓ Correct Answer (ground truth)

flags: {"recognition_streaks_v2": {"state": "on", "targeting_rules": "segment:beta_companies", "company_count": 42}, "points_budget_guardrails": {"state": "on", "targeting_rules": "all_companies", "company_c
code_referenced: recognition_streaks_v2, points_budget_guardrails, slack_dm_nudges, redeem_flow_redesign, analytics_dashboard_v3, ms_teams_app_v2
orphan_flags_no_code_reference: legacy_give_modal, survey_boosters_q3, paused_offboard_cleanup
flags_with_no_targets: paused_offboard_cleanup
rule: company counts must equal the export; do not describe flags absent from the export.
graded checks: orphans · no_targets · counts_match · no_invented

Model's actual answer

show the model's full answer
Here is the summary from the export and code excerpt only.

recognition_streaks_v2
- State: on
- What it controls: records streaks after a give via StreakTracker.record(give)
- Targeting rules: segment:beta_companies
- In scope: 42 companies
- No missing code reference / no missing targets

points_budget_guardrails
- State: on
- What it controls: enforces budget guardrails via BudgetService.new(company).enforce!(giver, points)
- Targeting rules: all_companies
- In scope: 220 companies
- No missing code reference / no missing targets

slack_dm_nudges
- State: on
- What it controls: sends a Slack DM nudge via SlackDm.send_nudge(user)
- Targeting rules: segment:region_na
- In scope: 87 companies
- No missing code reference / no missing targets

redeem_flow_redesign
- State: off
- What it controls: switches redeem rendering to RedeemV2Component.new(company) instead of RedeemV1Component.new(company)
- Targeting rules: targeted_list
- In scope: 12 companies
- No missing code reference / no missing targets

analytics_dashboard_v3
- State: on
- What it controls: uses AnalyticsV3.new(company) for the dashboard
- Targeting rules: segment:tier_three
- In scope: 65 companies
- No missing code reference / no missing targets

ms_teams_app_v2
- State: off
- What it controls: installs TeamsAppV2 via TeamsAppV2.install(company)
- Targeting rules: targeted_list
- In scope: 9 companies
- No missing code reference / no missing targets

legacy_give_modal
- State: off
- What it controls: no code reference provided
- Targeting rules: segment:legacy_plan
- In scope: 14 companies
- Missing code reference

survey_boosters_q3
- State: on
- What it controls: no code reference provided
- Targeting rules: segment:legacy_plan
- In scope: 7 companies
- Missing code reference

paused_offboard_cleanup
- State: off
- What it controls: no code reference provided
- Targeting rules: none given
- In scope: 0 companies
- Missing code reference and no targets

Arithmetic check from the export:
42 + 220 + 87 + 12 + 65 + 9 + 14 + 7 + 0 = 456 exported company counts total.
nda-triagePASS 1.00

? The Question

Triage each NDA as GREEN (standard approval), YELLOW (counsel review), or RED (full legal review). Cite the clause that drives each classification by quoting it. Catch embedded non-solicits, non-competes, missing carve-outs, and unusual term or governing law. One line of reasoning per NDA.
cost $0.0042 · 6s · in 939 / out 542 tok
✓ green✓ yellow✓ red✓ non_solicit_caught✓ clause_quoted✓ correct_labels

✓ Correct Answer (ground truth)

classifications: {"NDA-1": {"classification": "GREEN", "driving_reason": "standard mutual NDA: 2-year term, standard carve-outs, Delaware law", "driving_clause_quote": "Carve-outs: information that (a) is or becomes p
non_solicit_in: NDA-2
checks: classification_per_nda, clause_quotes_exist_in_document, non_solicit_caught, carveouts_checked
graded checks: green · yellow · red · non_solicit_caught · clause_quoted · correct_labels

Model's actual answer

show the model's full answer
NDA-1 — GREEN
Reasoning: Standard mutual NDA with ordinary carve-outs and Delaware law; no non-solicit, non-compete, exclusivity, or license grant. Driving clause: “Carve-outs: information that (a) is or becomes publicly available through no breach, (b) was known prior to disclosure, (c) is received from a third party without duty of confidence, (d) is independently developed, or (e) must be disclosed by law or court order.” Also: “No license, no obligation to proceed, no exclusivity.”

NDA-2 — YELLOW
Reasoning: The embedded non-solicitation is the review trigger; otherwise the NDA looks standard and uses Delaware law with standard carve-outs. Driving clause: “Non-solicitation: ‘During the term of this Agreement and for eighteen (18) months thereafter, neither party shall solicit for employment or hire any employee of the other party with whom it came into contact under this Agreement.’”

NDA-3 — RED
Reasoning: Full legal review is required because it includes a broad non-compete, lacks any confidentiality carve-outs, and uses foreign governing law/exclusive jurisdiction. Driving clauses: “Non-compete: ‘For a period of three (3) years following the Effective Date, Recipient shall not, directly or indirectly, engage in or provide services to any business that competes with Discloser’s business.’” “This Agreement shall be governed by and construed in accordance with the laws of the Republic of Ireland, and the parties submit to the exclusive jurisdiction of its courts.” “No confidentiality carve-outs are stated in this Agreement.”