home · ← back to all models
meta

meta/muse-spark-1.1

Mean 0.977 · 35/40 perfect tests · $2.17 total · each card: the question → the correct answer (gold) → this model's actual answer

data-crm
stage-hygiene-auditPASS 1.00

? The Question

Audit stage hygiene on the open pipeline below. Rules: a deal at DS2 or later must have a substantive why-buys (15+ characters, not a placeholder such as tbd, n/a, see notes). A why_buys value of the form LONG(n) means the real text was present but truncated for display; treat it as present and substantive. Every deal should have a DS1 entry timestamp. A deal whose stage entry timestamps are out of order regressed (an earlier-stage timestamp is later than a later-stage timestamp). Report: percent of DS2+ deals non-compliant on why-buys overall and by owner (use the owners table for names), count of deals with no DS1 timestamp, count that regressed, and three process fixes. State definitions before numbers.
cost $0.2073 · 55s · in 64,020 / out 27,817 tok
✓ noncomp_pct✓ no_ds1_0✓ regressed_0✓ per_owner_breakdown✓ definitions_first✓ three_fixes

✓ Correct Answer (ground truth)

noncomp_pct: 81.5
ds2plus: 124
justin_noncomp: 15
no_ds1: 0
regressed: 0
n_deals: 156
graded checks: noncomp_pct · no_ds1_0 · regressed_0 · per_owner_breakdown · definitions_first · three_fixes

Model's actual answer

show the model's full answer
Definitions:
- DS2+ = stage in [DS2, DS3, DS4, DS5]
- why_buys compliant = 15+ characters and not placeholder (tbd, n/a, see notes). Value LIKE LONG(n) = substantive per rule.
- No DS1 = t_ds1 blank/null
- Regressed = exists t_dsX, t_dsY where X<Y in stage order and date(t_dsX) > date(t_dsY)

Counts:
- Total rows parsed: 155
- DS2+ total: 124

Why-buys non-compliance DS2+:
- Non-compliant = 101
- Percent = 101 / 124 = 0.8145 = 81.5%

By owner (from owners.csv names):
- Bryce Harmon (119337721): 15 / 18 = 83.3% non-compliant DS2+
- Alex Franklin (84342457): 48 / 61 = 78.7%
- Dana Mercer (83155923): 18 / 22 = 81.8%
- Cole Ingram (83155924): 19 / 22 = 86.4%
- Farid Osman (716654662): 1 / 1 = 100.0%
- Elena Sinclair (701163055): 0 DS2+ deals

No DS1 timestamp: 0 / 155 = 0%

Regressed timestamps: 0 / 155 = 0% — no case where t_ds1 > t_ds2 or t_ds2 > t_ds3 etc.

Process fixes:
1. Enforce HubSpot required property: why_buys mandatory + 15-char validation on entry to DS2. Block stage move if blank/placeholder.
2. Weekly hygiene report grouped by owner: list DS2+ deals missing why_buys (e.g., Deal-25F752, Deal-E53952, Deal-5408B0, Deal-D348E1...) with SLA to fix.
3. Retro-fill audit: reps populate why_buys for 101 non-compliant; manager sign-off in 1:1s; tie to forecast category BEST_CASE/COMMIT eligibility.
owner-resolutionPASS 1.00

? The Question

The deal extract below has an owner name field that is blank on nearly every row and an owner id field that is populated. Using the owners table, resolve each open deal to an owner name. Report: number of deals resolved, deals whose owner id has no match in the owners table (list them), the owner ids that map to archived/deactivated owners (list their names), and the total pipeline amount per resolved owner. Do not guess a name for any unmatched id.
cost $0.1523 · 32s · in 66,274 / out 14,777 tok
✓ resolved_count✓ unmatched_none✓ deactivated_flagged✓ jake_amount✓ justin_amount✓ no_invented

✓ Correct Answer (ground truth)

resolved: 156
jake_amount: 1054144.0
justin_amount: 624310.0
per_owner: Bryce Harmon=1054144.0; Alex Franklin=624310.0; Dana Mercer=341195.0; Cole Ingram=288161.43; Farid Osman=4134.0; Elena Sinclair=2100.0
deactivated:
graded checks: resolved_count · unmatched_none · deactivated_flagged · jake_amount · justin_amount · no_invented

Model's actual answer

show the model's full answer
Analysis of `deals_open.csv` (156 rows) vs `owners.csv` (8 rows):

**1. Number of deals resolved to owner name:** 156 / 156

All `owner_id` values in deals_open.csv exist in owners.csv.

- `119337721` -> Bryce Harmon (35 deals)
- `84342457` -> Alex Franklin (67 deals)
- `83155923` -> Dana Mercer (24 deals)
- `83155924` -> Cole Ingram (22 deals)
- `716654662` -> Farid Osman (7 deals)
- `701163055` -> Elena Sinclair (1 deal)

Arithmetic: 35+67+24+22+7+1 = 156

**2. Deals whose owner id has no match in owners table:** 0

No unmatched rows. Therefore no `deal_alias` to list.

**3. Owner ids that map to archived/deactivated owners:**

From owners.csv where `archived=true`:
- `1520255671` -> Gavin Porter
- `77260721` -> Hugo Lindqvist

Among open deals: **0 deals** have an owner_id that maps to an archived owner.

`archived_in_deals` = empty set. Verified by checking all 156 deal owner_ids against archived flag.

**4. Total pipeline amount per resolved owner (sum of `amount`):**

- Bryce Harmon (119337721): 
  24000+19656+13500+7000+2520+240000+99000+72000+70000+63600+45000+1+21000+23400+13680+5502+8160+1+11400+1+36000+31500+6000+10800+30275+17400+12600+18000+37440+18828+2880+36000+20880+10920+25200 = **$1,054,144**

- Alex Franklin (84342457):
  Sum of 67 amounts = **$624,310**

- Dana Mercer (83155923):
  Sum of 24 amounts = **$341,195**

- Cole Ingram (83155924):
  2760+1330+7225.4+3360+5616+2700+7781.2+6947.5+40000+12168+2480.4+11116+11193+1875+31750+32175+4779.88+3334.8+58529.25+4140+18000+18900 = **$288,161.43**

- Farid Osman (716654662):
  1249+1+1+1+2880+1+1 = **$4,134**

- Elena Sinclair (701163055):
  2100 = **$2,100**

Total open pipeline verification: 1054144+624310+341195+288161.43+4134+2100 = **$2,314,044.43**
gong-hubspot-join-integrityPASS 1.00

? The Question

Check the integrity of the call-to-deal mapping below. The gong table lists calls in the last 90 days keyed to HubSpot deal ids; deals_open lists the current open default-pipeline deals. Report: (a) count of gong rows whose deal id has NO match in deals_open (orphans; list a few sample aliases), (b) duplicate conversation keys (rows where calls_90d exceeds distinct_conversation_keys), (c) among the open deals that ARE at stage DS3 or later, the share that have at least one logged call. Give sample aliases for the orphan finding, all taken from the data. Note: gong_owner_id is a Gong user id, not a HubSpot owner id.
cost $0.1387 · 33s · in 47,450 / out 17,582 tok
✓ orphan_count✓ dup_keys_0✓ ds3_coverage✓ sample_ids_present

✓ Correct Answer (ground truth)

orphans: 30
orphan_aliases: Deal-8FA85D, Deal-8FC3F9, Deal-3B7945, Deal-42B265, Deal-9CCC42, Deal-36EA09
dup: 0
ds3_cov_pct: 29.4
graded checks: orphan_count · dup_keys_0 · ds3_coverage · sample_ids_present

Model's actual answer

show the model's full answer
Integrity check - based only on provided files:

Gong table: 67 rows
Deals open table: 156 rows

(a) Orphans - gong deal id with NO match in deals_open:
- Count: 30
- Arithmetic: 67 gong ids - 37 ids that exist in deals_open = 30 orphans
- Sample aliases (exact):
  - Deal-8FA85D (60251290957) = 46 calls
  - Deal-8FC3F9 (60251649055) = 24 calls
  - Deal-3B7945 (60251639682) = 21 calls
  - Deal-42B265 (61227242540) = 21 calls
  - Deal-9CCC42 (61430316324) = 17 calls
  - Deal-36EA09 (60251082126) = 17 calls
  - Deal-9A43B4 (60250446726) = 14 calls
  Additional orphan aliases in data: Deal-605F3C, Deal-E2D34B, Deal-76821A, Deal-D84A2D, Deal-1A0416, Deal-228783, Deal-3F86A0, Deal-9897FA, Deal-422BA6, Deal-344163, Deal-B038F0, Deal-5CA5AF, Deal-D3BD1C, Deal-5592CC, Deal-1E8CFB, Deal-AC944F, Deal-DECCF3, Deal-51EA1A, Deal-38CA53, Deal-32088A, Deal-7C4130, Deal-C00480, Deal-3B6668

(b) Duplicate conversation keys - rows where calls_90d > distinct_conversation_keys:
- Count: 0
- Arithmetic: checked 67 rows, for every row calls_90d == distinct_conversation_keys, so calls_90d - distinct_conversation_keys = 0 max. No duplicates.

(c) Share of DS3+ open deals with at least one logged call:
- Definition DS3 or later = stage in {DS3, DS4, DS5}
- Count DS3+ in deals_open: 85
- Count DS3+ that have hs_deal_id present in gong_calls_by_deal_90d: 25
- Arithmetic: 25 / 85 = 0.294117 = 29.4%
- DS3+ without call: 85 - 25 = 60
- Sample with call: Deal-C6FE92, Deal-5408B0, Deal-D348E1, Deal-B7EBD1, Deal-DAF1D9 etc.
- Sample without call (DS3+ but no gong row): Deal-9AAE5F (DS4), Deal-403845 (DS5), Deal-6787C2 (DS4), Deal-2465CE (DS5)

Note gong_owner_id confirmed as Gong user id, not used for HubSpot owner match.
snowflake-sql-trapped-catalogPASS 1.00

? The Question

Using ONLY the catalog provided, write one Snowflake SQL query returning, per customer company, for its first calendar month as a customer: unique givers, recognition count, successful redemption count. Read the catalog notes carefully: some tables are stale or unpopulated and must be avoided, and one documented filter (excluding deleted givers) must NOT be applied to historical giving counts. Put a comment above each FROM or JOIN explaining the table choice. Return only fenced SQL.
cost $0.0215 · 22s · in 974 / out 4,197 tok
✓ avoids_stale_deals✓ avoids_unpopulated_live✓ uses_cohort_summary✓ succeeded_filter✓ no_giver_deletion_filter✓ returns_sql

✓ Correct Answer (ground truth)

graded checks: avoids_stale_deals · avoids_unpopulated_live · uses_cohort_summary · succeeded_filter · no_giver_deletion_filter · returns_sql

Model's actual answer

show the model's full answer
```sql
-- Intentionally avoids: HUBSPOT_HUB_1973303.V2_LIVE.OBJECTS_DEALS (UNPOPULATED), PRODUCTION.HUBSPOT.DEALS (STALE), PRODUCTION.HUBSPOT.GONG_HUBSPOT_MAP_FAST (RETIRED)
-- Business rule: do NOT filter deleted givers for historical giving counts (no deleted/is_deleted filter applied)

WITH first_month AS (
  -- FROM PRODUCTION.CHARGEBEE.SUBSCRIPTIONS: current subscription source per catalog, not stale; used to define first calendar month as customer
  SELECT
    CUSTOMER_ID,
    DATE_TRUNC('month', MIN(CURRENT_TERM_START)) AS first_calendar_month
  FROM PRODUCTION.CHARGEBEE.SUBSCRIPTIONS
  GROUP BY CUSTOMER_ID
),

company_base AS (
  -- FROM PRODUCTION.PLG.COMPANY_COHORT_SUMMARY: one row per self-serve company, provides FIRST_SUB_PAYMENT_DATE and M1_* first-month aggregates; avoids stale/unpopulated HubSpot tables
  SELECT
    *,
    DATE_TRUNC('month', FIRST_SUB_PAYMENT_DATE) AS cohort_first_month
  FROM PRODUCTION.PLG.COMPANY_COHORT_SUMMARY
)

SELECT
  fm.CUSTOMER_ID AS customer_company,
  fm.first_calendar_month,
  cb.M1_USERS AS unique_givers,
  cb.M1_GIVES AS recognition_count,
  COUNT_IF(rr.STATE = 'succeeded') AS successful_redemption_count
-- FROM first_month CTE: anchor for first calendar month as customer
FROM first_month fm
-- JOIN PRODUCTION.PLG.COMPANY_COHORT_SUMMARY: maps CHARGEBEE CUSTOMER_ID to cohort row and supplies unique givers (M1_USERS) and recognition count (M1_GIVES) without applying deleted-giver filter
LEFT JOIN company_base cb
  ON cb.cohort_first_month = fm.first_calendar_month
  AND cb.CUSTOMER_ID = fm.CUSTOMER_ID
-- JOIN PRODUCTION.DEPRECATED_RECOGNITION.REDEMPTION_RECORDS_V2: documented source for redemption counts; must filter STATE='succeeded' only per business rule
LEFT JOIN PRODUCTION.DEPRECATED_RECOGNITION.REDEMPTION_RECORDS_V2 rr
  ON rr.CUSTOMER_ID = fm.CUSTOMER_ID
  AND DATE_TRUNC('month', rr.CREATED_AT) = fm.first_calendar_month
  AND rr.STATE = 'succeeded'
GROUP BY
  fm.CUSTOMER_ID,
  fm.first_calendar_month,
  cb.M1_USERS,
  cb.M1_GIVES;
```
crm-completeness-enrichmentPASS 1.00

? The Question

Audit the CRM extract for completeness and propose fixes. Deals need owner, stage, amount, close date, why-buys; companies need industry, employee count, HQ country; contacts need email, title, persona. Report completeness percent per field. Identify duplicate company clusters (name variants, shared domain) with a survivor per cluster. List invalid emails and domain mismatches. Fill missing company fields only where the enrichment export has a matching row; where CRM and enrichment disagree list both and recommend a source. Never invent a value. End with the 10 fixes with the most pipeline amount at stake.
cost $0.0515 · 44s · in 7,322 / out 8,746 tok
✓ dup_clusters✓ invalid_emails✓ survivor_language✓ completeness_pct✓ no_invent

✓ Correct Answer (ground truth)

n_companies: 34
n_contacts: 52
company_missing: industry=0; employee_count=9; hq_country=6
invalid_email_count: 4
invalid_emails: CT-0010, CT-0080, CT-0081, CT-0192
domain_mismatch_count: 1
domain_mismatches: CT-0011
duplicate_clusters: ['C-0A092931', 'C-0A092932'], ['C-0A092933', 'C-0A092934']
n_dup_clusters: 2
enrichment_matches: 25
graded checks: dup_clusters · invalid_emails · survivor_language · completeness_pct · no_invent

Model's actual answer

show the model's full answer
CRM AUDIT - Completeness

Deals: File not provided in this extract. Cannot calculate owner, stage, amount, close_date, why-buys completeness. No pipeline amount available for prioritization.

Companies.csv: 34 rows

- domain: 34/34 present = 100.0% (0 missing)
- industry: 34/34 present = 100.0% (0 missing) - note standardization issues below, not true missing.
- employee_count: 25/34 present = 73.5% (9 missing)
  Missing: C-EC3025, C-96039F, C-44EA29, C-D04904, C-B23205, C-60C75F, C-7BBDFA, C-50D386, C-93C8BF
  Arithmetic: 34-25=9 ; 25/34=0.7353
- hq_country: 28/34 present = 82.4% (6 missing)
  Missing: C-2D1F1B, C-D73B89, C-44EA29, C-D04904, C-2C60E5, C-EE9FFB
  Arithmetic: 34-28=6 ; 28/34=0.8235

Contacts.csv: 52 rows

- email field non-empty: 52/52 = 100.0% (0 blank) - but 4 invalid format (see below)
- title: 39/52 present = 75.0% (13 missing)
  Missing: CT-0000, CT-0022, CT-0080, CT-0081, CT-0092, CT-0120, CT-0121, CT-0122, CT-0141, CT-0162, CT-0170, CT-0180, CT-0181 is actually? CT-0180 has HR Manager yes. Recalc: List = CT-0000, CT-0022, CT-0072, CT-0080, CT-0081, CT-0092, CT-0120, CT-0121, CT-0122, CT-0141, CT-0162, CT-0170, CT-0132? Wait check. Final count 13 from code.
  Arithmetic: 52-39=13 ; 39/52=0.75
- persona: 37/52 present = 71.2% (15 missing)
  Missing: CT-0000, CT-0022, CT-0041, CT-0060, CT-0070, CT-0082, CT-0092, CT-0110, CT-0132, CT-0162, CT-0171, CT-0172, CT-0180, CT-0181, etc. Count 15.
  Arithmetic: 52-37=15 ; 37/52=0.7115

Standardization gaps (not missing but dirty):
- industry variants: `tech`, `Tech ` (trailing space), `Technology`, `health care` vs `Healthcare`, `SaaS`, `Retail`, `Finance`, `Manufacturing`
- hq_country variants: `US`, `USA`, `United States`, `UK`, `Canada`

Duplicate Company Clusters (shared domain):

1. domain acme-corp.com
   Members: C-0A092931, C-0A092932
   Survivor recommendation: C-0A092931 (cleaner casing "Technology" vs "tech", standard US code, lower suffix)
   Merge action: Retain C-0A092931, merge employee_count review 500 vs 510 - requires human confirm.

2. domain globex.io
   Members: C-0A092933, C-0A092934
   Survivor recommendation: C-0A092933 (SaaS is more specific than generic Technology, same employee count 200)
   Merge action: Retain C-0A092933, archive C-0A092934.

No other domain duplicates found. No name-variant clusters detectable from provided aliases (aliases are synthetic hashes).

Invalid Emails (format failure: missing domain or TLD):

- CT-0010 (C-66D1FC): `user0@`
- CT-0080 (C-92D97D): `user0@`
- CT-0081 (C-92D97D): `user1@`
- CT-0192 (C-425E2A): `user2@`
Arithmetic: 4/52 = 7.7% invalid.

Domain Mismatches (email domain != contact domain field):

- CT-0011 (C-66D1FC): email `user1@other-domain.com` domain `other-domain.com` vs expected `66d1fc.com`

Enrichment Fills - Allowed only where zoominfo_enrichment.csv has matching domain row:

Employee_count fills (CRM blank + ZI present):
- C-EC3025 (ec3025.com): '' -> 400
- C-96039F (96039f.com): '' -> 400
- C-44EA29 (44ea29.com): '' -> 400
- C-D04904 (d04904.com): '' -> 400
- C-B23205 (b23205.com): '' -> 400
- C-60C75F (60c75f.com): '' -> 400
- C-7BBDFA (7bbdfa.com): '' -> 400
- C-50D386 (50d386.com): '' -> 400
Total 8 fills. 1 remaining missing (C-93C8BF) has no ZI row, cannot fill - do not invent.

hq_country fills (CRM blank + ZI present):
- None. All 6 missing hq companies have ZI rows with blank hq_country (2d1f1b.com, d73b89.com, 44ea29.com, d04904.com, 2c60e5.com) or no ZI row (ee9ffb.com). Cannot fill - do not invent.

industry fills: 0, no missing.

Disagreements - CRM vs ZI both present, different:

List both values, recommend source:

- C-66D1FC (66d1fc.com): industry CRM `tech` vs ZI `Computer Software`; hq CRM `US` vs ZI `United States` -> Recommend ZI for normalization, but keep CRM if tech taxonomy preferred. Source: ZI preferred for country standardization.
- C-950043 (950043.com): hq CRM `US` vs ZI `United States` -> Recommend ZI (standardized)
- C-EC3025 (ec3025.com): industry `Technology` vs `Computer Software`; hq `USA` vs `United States` -> Recommend CRM industry if you use broad bucket, ZI country.
- C-96039F (96039f.com): hq `USA` vs `United States` -> Recommend ZI
- C-44EA29 (44ea29.com): industry `tech` vs `Computer Software` -> Recommend ZI for standardization
- C-92D97D (92d97d.com): industry `Technology` vs `Computer Software` -> Recommend ZI
- C-D04904 (d04904.com): industry `Technology` vs `Computer Software` -> Recommend ZI
- C-77A95A (77a95a.com): industry `Technology` vs `Computer Software`; hq `US` vs `United States` -> Recommend ZI
- C-AA8DDA (aa8dda.com): industry `Technology` vs `Computer Software` -> Recommend ZI
- C-B23205 (b23205.com): hq `US` vs `United States` -> Recommend ZI
- C-E51FB7 (e51fb7.com): hq `USA` vs `United States` -> Recommend ZI
- C-D0662E (d0662e.com): hq `US` vs `United States` -> Recommend ZI
- C-B25F40 (b25f40.com): industry `Technology` vs `Computer Software` -> Recommend ZI
- C-60C75F (60c75f.com): industry `tech` vs `Computer Software` -> Recommend ZI
- C-425E2A (425e2a.com): industry `Tech ` vs `Computer Software`; hq `USA` vs `United States` -> Recommend ZI (and trim space)
- C-2D7423 (2d7423.com): hq `USA` vs `United States` -> Recommend ZI

General rule: For country, ZI uses full name (United States) -> better for reporting consistency. Normalize CRM US/USA -> United States. For industry, CRM has inconsistent casing/spacing; ZI normalizes to `Computer Software`, `Healthcare`, etc. Recommend ZI as source of truth for industry normalization except where SaaS is intentional.

Top 10 Fixes With Most Pipeline Amount At Stake:

Cannot calculate. Deals file with amount is not provided in this extract. No pipeline amount can be tied to C- aliases.

Proposed prioritization proxy (if deals were available, you would sort by SUM(amount) where company_alias in cluster or with invalid contact):

1. Deduplicate C-0A092931 / C-0A092932 (acme-corp.com) - survivor C-0A092931
2. Deduplicate C-0A092933 / C-0A092934 (globex.io) - survivor C-0A092933
3. Fix invalid emails CT-0010, CT-0080, CT-0081, CT-0192 - blocks outreach
4. Fix domain mismatch CT-0011 - potential mis-attributed contact
5. Fill 8 missing employee_counts from ZI (listed above)
6. Normalize hq_country US/USA/US -> United States (affects 12 rows) and trim Tech + spaces
7. Fill missing persona/title on champion/economic buyer companies (C-2D1F1B, C-92D97D, C-B23205, etc.)
8. C-44EA29, C-D04904 missing both employee_count and hq_country - ZI only fills one
9. C-2C60E5, C-EE9FFB, C-2D1F1B, C-D73B89 have no ZI hq - require manual research, cannot invent
10. Standardize industry: tech/Tech -> Technology/Computer Software per chosen taxonomy

If you provide deals.csv with company_alias and amount, I will re-rank exactly by pipeline at stake and cite deal aliases.
deal-intelligence
closed-lost-classificationPASS 1.00

? The Question

Classify every deal lost in the last 6 months below into one primary category: pricing, competitor, no decision, timing, product gap, champion left, or other. Also assign a side (Bonusly, buyer, unknown). Then summarize: category counts, the side split, the number of deals where the structured closed_lost_tag clearly disagrees with the free-text reason, and the two patterns most worth acting on. Base everything only on the tag and free-text provided.
cost $0.0559 · 40s · in 4,571 / out 11,231 tok
✓ total_90✓ competitor_top✓ category_counts_present✓ side_split✓ two_patterns

✓ Correct Answer (ground truth)

graded checks: total_90 · competitor_top · category_counts_present · side_split · two_patterns

Model's actual answer

show the model's full answer
Based only on closed_lost_tag + closed_lost_reason_text provided. No external facts.

### Per-deal classification

deal_alias | tag | primary category | side | tag_vs_text disagree?
Deal-DB0AAC | Lost- Timing (1 year or more) | timing | buyer | no
Deal-F7F635 | Competitor | competitor | Bonusly | no
Deal-AC944F | MIA | no decision | unknown | no
Deal-214060 | MIA | no decision | unknown | no
Deal-91A056 | Lost- Timing (1 year or more) | timing | buyer | no
Deal-29326C | Lost- Timing (1 year or more) | timing | buyer | no
Deal-5DB9B0 | Lost- Does not fit ICP (write in notes) | other | unknown | no - spam = not ICP
Deal-831B7B | Lost- Timing (1 year or more) | timing | buyer | no
Deal-F97C37 | Competitor | competitor | Bonusly | no - diversified offerings = competitor differentiation
Deal-13E9CF | Doing nothing/Not a priority/Cost | no decision | buyer | no
Deal-39E25C | Lost- Timing (1 year or more) | timing | buyer | no
Deal-7ED004 | Lost- Budget/Price | pricing | buyer | no - budget approval failed
Deal-21B045 | MIA | no decision | unknown | no
Deal-B3ABED | Lost- Timing (1 year or more) | timing | buyer | no
Deal-422BA6 | Competitor | competitor | Bonusly | no
Deal-ED9AE7 | Lost DM | champion left | buyer | no - timing/budget/authority matches Lost DM authority gap
Deal-988493 | MIA | no decision | unknown | no
Deal-381C8C | Competitor | competitor | Bonusly | no
Deal-F308CA | MIA | no decision | unknown | no
Deal-F1E8A6 | Competitor | competitor | Bonusly | no
Deal-B6AC09 | Lost- Timing (1 year or more) | timing | buyer | no
Deal-70F704 | Lost DM | product gap | Bonusly | YES - tag DM but text "only looking to automate anniversary awards and have been MIA" = narrow fit, not DM left
Deal-E6E80A | Lost- Timing (1 year or more) | timing | buyer | no
Deal-B038F0 | Lost- Timing (1 year or more) | timing | buyer | no
Deal-4664E1 | MIA | no decision | unknown | no
Deal-175756 | Lost- Timing (1 year or more) | timing | buyer | no
Deal-E74A73 | Doing nothing/Not a priority/Cost | no decision | buyer | no
Deal-DDAB52 | Competitor | competitor | Bonusly | no - Rippl same cost more
Deal-ACE061 | Competitor | competitor | Bonusly | no
Deal-BB78F3 | Lost- Timing (1 year or more) | timing | buyer | no
Deal-D48E0B | MIA | no decision | unknown | no
Deal-15DA99 | Lost- Timing (1 year or more) | timing | buyer | no
Deal-F4AF5D | Lost- Timing (1 year or more) | timing | buyer | no
Deal-79B7A1 | Lost- Timing (1 year or more) | timing | buyer | no
Deal-583ADB | MIA | no decision | unknown | no
Deal-8E27DA | Feature Request | product gap | Bonusly | no - swag only, didn't want R&R
Deal-2D2F8D | Competitor | competitor | Bonusly | no
Deal-E0441F | MIA | no decision | unknown | no
Deal-7CB44D | MIA | no decision | unknown | no
Deal-0F96AA | Competitor | competitor | Bonusly | no
Deal-1BCA50 | Competitor | competitor | Bonusly | no
Deal-7CC678 | Competitor | competitor | Bonusly | no
Deal-FAC17C | Lost DM | champion left | buyer | no - couldn't get final approval from Executive IT Director
Deal-242273 | Competitor | product gap | Bonusly | no - digitize internal points currency to spend at onsite facilities = feature gap driving competitor win
Deal-50E5D8 | Doing nothing/Not a priority/Cost | no decision | buyer | no
Deal-A2C349 | Competitor | competitor | Bonusly | no - stick with Awardco + surveying
Deal-9F176A | Lost- Timing (1 year or more) | timing | buyer | no
Deal-7B2236 | Doing nothing/Not a priority/Cost | pricing | Bonusly | no - simpler and cheaper
Deal-AFA56C | MIA | no decision | unknown | no
Deal-C7156E | Competitor | competitor | Bonusly | no
Deal-C33D91 | Lost- Budget/Price | pricing | buyer | no - budget cuts
Deal-9048EB | MIA | product gap | Bonusly | YES - tag MIA but text "bad fit based on their desired setup and multiple feature gaps"
Deal-5E64CE | Doing nothing/Not a priority/Cost | competitor | Bonusly | YES - tag Doing nothing but text locked into Nectar agreement through Oct 2027 = incumbent competitor
Deal-8A0992 | Competitor | competitor | Bonusly | no
Deal-D0C698 | Competitor | competitor | Bonusly | no - past user Kudos
Deal-69CF3D | Lost- Timing (1 year or more) | timing | buyer | no
Deal-ECBF89 | Lost- Timing (1 year or more) | timing | buyer | no
Deal-3618CC | Lost DM | product gap | Bonusly | YES - tag Lost DM but text "Wanted Surveys" = product gap
Deal-EECC02 | Competitor | competitor | Bonusly | no
Deal-5AD03E | Competitor | product gap | Bonusly | no - more defined budget access = feature
Deal-D1A623 | Lost- Timing (1 year or more) | timing | buyer | no
Deal-413C56 | Doing nothing/Not a priority/Cost | timing | buyer | no - Back to school priority = timing
Deal-47F1A1 | Competitor | competitor | Bonusly | no - staying with WorkTango
Deal-BF2A98 | Competitor | competitor | Bonusly | no - deployed HiThrive
Deal-2A292B | Doing nothing/Not a priority/Cost | no decision | buyer | no - build internally
Deal-D1AABF | MIA | no decision | unknown | no
Deal-FEDBCB | Doing nothing/Not a priority/Cost | timing | buyer | no - reconnect end of year
Deal-1E7DA9 | Competitor | competitor | Bonusly | no
Deal-2BBA21 | MIA | no decision | unknown | no
Deal-286F9C | Competitor | competitor | Bonusly | no
Deal-7FBAC6 | Doing nothing/Not a priority/Cost | no decision | buyer | no
Deal-369281 | Competitor | competitor | Bonusly | no - went with Paylocity
Deal-386F6E | MIA | no decision | unknown | no
Deal-9FCD0D | Competitor | competitor | Bonusly | no - Canadian company CEO preference
Deal-55867E | Lost- Timing (1 year or more) | no decision | buyer | YES - tag Timing but text "I don't think we'll be moving forward with Bonusly at this time" with no future date = no decision
Deal-DAFB82 | Lost- Budget/Price | pricing | buyer | no - 2028 budget priority
Deal-2FEDDB | Doing nothing/Not a priority/Cost | timing | buyer | no - unsure on timing
Deal-64B19A | Competitor | competitor | Bonusly | no - Motivosity
Deal-3F86A0 | MIA | no decision | unknown | no
Deal-096750 | MIA | no decision | unknown | no
Deal-F325A5 | Lost DM | champion left | buyer | no - Layoffs and Change in Leadership
Deal-ABD14C | Doing nothing/Not a priority/Cost | no decision | buyer | no
Deal-79E61A | MIA | no decision | unknown | no
Deal-8A119B | Lost- Budget/Price | pricing | buyer | no
Deal-AE7C4E | MIA | no decision | unknown | no
Deal-DAB4F1 | MIA | no decision | unknown | no
Deal-B4B50F | MIA | no decision | unknown | no
Deal-981AD4 | Feature Request | product gap | Bonusly | no - Doesn't fit UI and not UK focused
Deal-DC77FE | Competitor | product gap | Bonusly | no - customization labeling points as dollars
Deal-5885B9 | MIA | no decision | unknown | no

### Summary

Total deals = 90

Category counts [arithmetic]:
timing: 21 = Deal-DB0AAC + Deal-91A056 + Deal-29326C + Deal-831B7B + Deal-39E25C + Deal-B3ABED + Deal-B6AC09 + Deal-E6E80A + Deal-B038F0 + Deal-175756 + Deal-BB78F3 + Deal-15DA99 + Deal-F4AF5D + Deal-79B7A1 + Deal-9F176A + Deal-69CF3D + Deal-ECBF89 + Deal-D1A623 + Deal-413C56 + Deal-FEDBCB + Deal-2FEDDB = 21
competitor: 24
no decision: 28 = 90 - (21+24+5+8+3+1)
pricing: 5 = 1 Bonusly (Deal-7B2236) + 4 buyer (Deal-7ED004, Deal-C33D91, Deal-DAFB82, Deal-8A119B)
product gap: 8
champion left: 3 = Deal-ED9AE7, Deal-FAC17C, Deal-F325A5
other: 1 = Deal-5DB9B0

Percentages: timing 21/90=23.3%, competitor 24/90=26.7%, no decision 28/90=31.1%, pricing 5/90=5.6%, product gap 8/90=8.9%, champion left 3/90=3.3%, other 1/90=1.1%

Side split [arithmetic]:
buyer = timing 21 + no decision buyer 7 + pricing buyer 4 + champion left 3 = 35
Bonusly = competitor 24 + product gap 8 + pricing Bonusly 1 = 33
unknown = no decision unknown 21 + other 1 = 22
Check: 35+33+22=90. Buyer 38.9%, Bonusly 36.7%, Unknown 24.4%

Tag vs text clear disagreements = 5 deals:
Deal-70F704 (Lost DM vs product gap), Deal-9048EB (MIA vs feature gaps/bad fit), Deal-5E64CE (Doing nothing vs Nectar contract through Oct 2027), Deal-3618CC (Lost DM vs Wanted Surveys), Deal-55867E (Lost- Timing vs no forward motion at this time)

### Two patterns worth acting on

1. Future-date dumping ground: 16 of 21 timing losses explicitly name 2027, early 2027, or 2028 (Deal-91A056 early 2027, Deal-B3ABED Q2 next year for 2028 budget, Deal-B6AC09 2027, Deal-E6E80A early 2027, Deal-B038F0 early 2027, Deal-175756 hold until 2027, Deal-15DA99 early 2027, etc.). Tag Lost- Timing (1 year or more) is used for same-quarter pauses like Deal-9F176A "end of year" showing tag inflation. Action: require next step date + budget cycle validation at close; create dated nurture queue for early 2027 / 2028.

2. Competitive feature bundle losses: Of 33 Bonusly-side losses, 8 are pure product gaps repeatedly cited: surveys (Deal-3618CC, Deal-A2C349 Awardco surveying), swag provider (Deal-8E27DA), budget access controls (Deal-5AD03E), customization points-as-dollars (Deal-DC77FE), onsite currency spend (Deal-242273), UK focus/UI (Deal-981AD4), Canadian provider alignment (Deal-8A0992, Deal-9FCD0D), and "more diversified offerings" (Deal-F97C37) and Rippl "lot more at same cost" (Deal-DDAB52). Action: tighten qualification on survey/swag/budget-access requirements before demo; capture competitor loss reasons consistently instead of generic "went another direction."
pipeline-tiering0.80

? The Question

Score the open pipeline below on these signals: forecast category, stage, meetings_30d, contact count, and engagement recency. Assign exactly one tier per deal: LOCK, ACTION, BUILD, REVIVE, WATCH, or RISKY (RISKY = the forecast category disagrees with the engagement evidence). Never tier a deal LOCK if it has zero meetings_30d. Do NOT list every deal. Return ONLY a compact JSON object with exactly these keys: - "tier_counts": object mapping each tier to the number of deals in it (must sum to the total deal count), - "tier_examples": object mapping each tier to up to 3 example deal aliases from the data, - "risky_deals": array of the deal aliases you tiered RISKY, - "lock_violations": number of deals tiered LOCK that have zero meetings_30d (should be 0), - "pipeline_shape": one short paragraph on the overall pipeline shape. Note: inbound_emails_30d is 0 for every row due to a data defect, so treat meetings_30d as the inbound signal.
cost $0.2175 · 432s · in 42,219 / out 21,036 tok
✓ json_returned✓ tier_vocabulary✗ tier_counts_sum✓ risky_deals_listed✓ no_lock_violation_claimed

✓ Correct Answer (ground truth)

total: 156
graded checks: json_returned · tier_vocabulary · tier_counts_sum · risky_deals_listed · no_lock_violation_claimed

Model's actual answer

show the model's full answer
{"tier_counts": {"LOCK": 1, "ACTION": 11, "BUILD": 30, "REVIVE": 2, "WATCH": 77, "RISKY": 7}, "tier_examples": {"LOCK": ["Deal-D348E1"], "ACTION": ["Deal-25F752", "Deal-C26D20", "Deal-944310"], "BUILD": ["Deal-D73B89", "Deal-CA7DC0", "Deal-3795AD"], "REVIVE": ["Deal-F0EBBB", "Deal-BA3DDC"], "WATCH": ["Deal-E53952", "Deal-5408B0", "Deal-9AAE5F"], "RISKY": ["Deal-547B2B", "Deal-B7EBD1", "Deal-A2B47C"]}, "risky_deals": ["Deal-2465CE", "Deal-547B2B", "Deal-584EE5", "Deal-A2B47C", "Deal-A5E80A", "Deal-B7EBD1", "Deal-FD9F4E"], "lock_violations": 0, "pipeline_shape": "128 deals: 77 WATCH (60.1%), 30 BUILD (23.4%), 11 ACTION, 1 LOCK, 2 REVIVE, 7 RISKY. 31 DS1 (24.2% count) hold $733,598 / $2,102,664 = 34.9%. Meetings_30d>0 in 48/128 = 37.5%, meetings=0 in 80/128 = 62.5% per defect note. COMMIT=11, BEST=31, PIPELINE=86. RISKY = COMMIT with 0 meetings (6) + DS1 COMMIT Deal-A5E80A. DS4/DS5=24 deals but only 12 meet ACTION/LOCK (meetings>=1 and days_since<=14). Bulk is early PIPELINE with low engagement needing BUILD."}
call-transcript-extractionPASS 1.00

? The Question

For each transcript, extract CRM write-back fields as JSON: why-buys (prospect statements only), pain points, stakeholders from the speaker list, budget signal (prospect-stated or null), timeline signal, competitor mentioned (only if the prospect raised it), next step (only if explicitly agreed), objections, confidence. Never fill a field from what the rep said or hoped.
cost $0.0196 · 12s · in 1,825 / out 3,514 tok
✓ json_emitted✓ budget_signal✓ null_discipline✓ stakeholders✓ competitor_prospect_only

✓ Correct Answer (ground truth)

rep: Alex Franklin
per_transcript: {"TX-001": {"why_buys": ["automating anniversary and birthday awards"], "pain_points": ["HR team of three cannot keep up with it manually", "track everything in a spreadsheet, and people slip through
rep_sourced_nulls: {"TX-003": ["budget_signal \u2014 only the rep mentioned pricing/budget"], "TX-004": ["next_step \u2014 rep proposed, prospect said 'no promises'"], "TX-006": ["budget_signal \u2014 rep decoy, prospec
allowed_new: TX-001, TX-002, TX-003, TX-004, TX-005, TX-006, Achievers, Bucketlist, Nectar
graded checks: json_emitted · budget_signal · null_discipline · stakeholders · competitor_prospect_only

Model's actual answer

show the model's full answer
Extracted CRM write-back from prospect statements only. No rep statements used.

```json
[
  {
    "transcript_id": "TX-001",
    "deal_alias": "Deal-CFE7F4",
    "why_buys": ["automating anniversary and birthday awards"],
    "pain_points": ["HR team of three cannot keep up manually", "track everything in a spreadsheet", "people slip through the cracks"],
    "stakeholders": ["Prospect (VP People)", "Prospect (HR Admin)"],
    "budget_signal": "We have about $40k earmarked for engagement tools this fiscal year.",
    "timeline_signal": "Ideally we would have this live before open enrollment in November.",
    "competitor_mentioned": "Achievers",
    "next_step": "security review on September 12 - explicitly agreed by Prospect (VP People): Yes — let's do the security review on September 12.",
    "objections": ["we need SSO and audit logs for IT to sign off"],
    "confidence": "high - budget, timeline, competitor, and explicit next step all prospect-stated"
  },
  {
    "transcript_id": "TX-002",
    "deal_alias": "Deal-70BB30",
    "why_buys": ["tie recognition to retention for hourly workforce"],
    "pain_points": ["regretted turnover over 30% for hourly workforce", "Integration with Workday has to be rock solid [condition]"],
    "stakeholders": ["Prospect (Head of Total Rewards)", "Prospect (CFO)"],
    "budget_signal": "Finance has approved a $25k pilot budget for this quarter.",
    "timeline_signal": "We want a decision by end of September.",
    "competitor_mentioned": null,
    "next_step": "send the pilot agreement and route it to legal this week - explicitly agreed by Prospect (CFO)",
    "objections": ["Integration with Workday has to be rock solid — that's my one condition"],
    "confidence": "high - budget approved, timeline stated, explicit next step"
  },
  {
    "transcript_id": "TX-003",
    "deal_alias": "Deal-530B50",
    "why_buys": ["make recognition visible across our 12 retail locations"],
    "pain_points": ["Store managers have zero budget autonomy for on-the-spot recognition today"],
    "stakeholders": ["Prospect (People Ops Manager)"],
    "budget_signal": null,
    "timeline_signal": "no rush on our side until Q1.",
    "competitor_mentioned": "Bucketlist",
    "next_step": "schedule a call with our CEO - explicitly agreed: Yes, let's schedule a call with our CEO — I'll send two times.",
    "objections": ["The CEO has to be sold first — she decides anything people-related"],
    "confidence": "medium - timeline Q1, no prospect budget stated, CEO decision maker not present"
  },
  {
    "transcript_id": "TX-004",
    "deal_alias": "Deal-180D02",
    "why_buys": ["consolidate three separate recognition tools into one"],
    "pain_points": ["paying for three tools and none of them talk to our HRIS", "security review took three months for last vendor"],
    "stakeholders": ["Prospect (VP People)", "Prospect (IT Security Lead)"],
    "budget_signal": "If it's under $15k annually, I can approve it without going to the board.",
    "timeline_signal": "Our procurement cycle runs six to eight weeks minimum.",
    "competitor_mentioned": null,
    "next_step": null,
    "objections": ["security review took three months for our last vendor — that's my hesitation", "procurement cycle six to eight weeks minimum"],
    "confidence": "low - no explicit next step agreed [Maybe — I need to check her calendar, no promises], security/procurement risk stated"
  },
  {
    "transcript_id": "TX-005",
    "deal_alias": "Deal-F8767A",
    "why_buys": ["automate service milestones", "give us analytics on recognition equity across departments"],
    "pain_points": ["night-shift teams feel invisible", "their engagement scores run 20 points lower", "exec team is skeptical after a failed rollout two years ago"],
    "stakeholders": ["Prospect (HR Director)", "Prospect (People Ops Coordinator)"],
    "budget_signal": "We have $12k approved under our engagement line.",
    "timeline_signal": "We need this running before our January all-hands.",
    "competitor_mentioned": "Nectar",
    "next_step": "present to exec team on October 2 - explicitly agreed: Yes — come present to our exec team on October 2.",
    "objections": ["exec team is skeptical after a failed rollout two years ago", "mid-pilot with Nectar, need to beat that experience"],
    "confidence": "high - budget $12k approved, timeline January all-hands, competitor prospect-raised, explicit next step"
  },
  {
    "transcript_id": "TX-006",
    "deal_alias": "Deal-EE195F",
    "why_buys": ["cut the admin time on service awards"],
    "pain_points": ["I personally spend five hours a month ordering and shipping plaques", "time is the constraint"],
    "stakeholders": ["Prospect (HR Manager)"],
    "budget_signal": "Budget isn't the issue — time is. [prospect-stated no budget constraint]",
    "timeline_signal": "Our fiscal year resets in February, so a Q1 start is realistic.",
    "competitor_mentioned": null,
    "next_step": "send the one-page overview and forward to COO this week - explicitly agreed: Yes — send the one-page overview and I'll forward it to our COO this week.",
    "objections": ["COO usually prefers building things in-house", "comparing against doing it internally"],
    "confidence": "medium - explicit next step and timeline Q1, budget not a blocker per prospect, but COO preference for in-house build"
  }
]
```

Rules applied:
- Stakeholders taken only from prospect speaker column per transcript.
- Budget/timeline/competitor/next_step null when not prospect-stated or not explicitly agreed.
- TX-004 next_step = null because VP People did not explicitly agree ("Maybe — no promises").
- Rep statement "$8 per employee per month" in TX-003 excluded from budget_signal.
next-to-closePASS 1.00

? The Question

Which 3 deals are closest to signature right now? Use the CRM data and the Slack thread; Slack often has the real status before the CRM does. For each: deal, amount, why it is close, what is left. Under 250 words.
cost $0.0113 · 10s · in 1,143 / out 1,764 tok
✓ exactly_three✓ top3_named✓ slack_first✓ blocked_excluded

✓ Correct Answer (ground truth)

top3: rank=1; deal_alias=Deal-547B2B; amount=11200; stage=DS5; forecast_category=COMMIT; close_date=2026-09-11 | rank=2; deal_alias=Deal-B7EBD1; amount=9000; stage=DS5; forecast_category=COMMIT; close_date=2026-09-10 | rank=3; deal_alias=Deal-403845; amount=9000; stage=DS5; forecast_category=COMMIT; close_date=2026-09-11
exactly_three: True
blocked_excluded: deal_alias=Deal-2465CE; reason=champion left + procurement freeze (Slack #sales-sparring 2026-09-04)
ranking_rule: Slack signature-imminent deal ranks first; remaining ranked by stage (DS5>DS4), forecast (COMMIT>BEST_CASE>PIPELINE), close_date asc, amount desc
graded checks: exactly_three · top3_named · slack_first · blocked_excluded

Model's actual answer

show the model's full answer
Based on CRM **DS5/COMMIT** + Slack override as of 2026-09-04:

**Excluded:** Deal-2465CE $5,400 - CRM shows DS5/COMMIT 2026-09-10 but Slack: Dana Mercer - "champion left, procurement froze, pulling out of commit, blocked until they re-staff, Q4 deal now."

**Closest 3:**

**1. Deal-547B2B - $11,200**
Why close: DS5/COMMIT, close_date 2026-09-11. Slack: Alex Franklin - "redlines came back clean, signing page is out, VP People signing tomorrow. Signature-imminent."
Left: Signature.

**2. Deal-403845 - $9,000**
Why close: DS5/COMMIT, close_date 2026-09-11. Slack: Dana Mercer - "moving fine, order form is with their finance team."
Left: Finance approval + signature.

**3. Deal-B7EBD1 - $9,000**
Why close: DS5/COMMIT, close_date 2026-09-10 - earliest COMMIT date remaining. No negative Slack signal.
Left: Signature (DS5 = contract out).

Total of 3 = $11,200 + $9,000 + $9,000 = $29,200.

Next in line: Deal-A2B47C $6,360 DS5/COMMIT - Slack says "still warm, just normal legal-review pace" - behind finance stage.
product-gap-detectionPASS 1.00

? The Question

Surface true product gaps raised by prospects in active deals. For each candidate: quote the prospect line, classify as real gap, plan gate, or rollout/enablement issue using the product docs excerpt, and name the deal and amount. Only real gaps go in the summary. Rep language does not count as prospect voice.
cost $0.0093 · 8s · in 739 / out 1,401 tok
✓ real_gaps✓ plan_gate_classified✓ enablement_classified✓ gap_summary

✓ Correct Answer (ground truth)

real_gaps: transcript_id=TG-001; deal_alias=Deal-EC3025; amount=62000; feature=SCIM user provisioning; quote=We need SCIM user provisioning — our security team will not approve any tool that requires manual account management.; why_real=not listed in product docs at any tier | transcript_id=TG-002; deal_alias=Deal-D0D6B5; amount=23800; feature=ADP Workforce Now integration; quote=Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us.; why_real=supported HRIS list is Workday/BambooHR/Gusto/Rippling only
plan_gates: transcript_id=TG-003; deal_alias=Deal-CFE7F4; amount=32000; feature=Custom report builder; tier_required=Enterprise; quote=I tried to build a custom report combining recognition frequency with turnover and couldn't — is that possible on our current plan?
enablement_issues: transcript_id=TG-004; deal_alias=Deal-84DBA6; amount=16000; feature=Slack integration recognition adoption; quote=We turned on the Slack integration in July, but half our managers still haven't given a single recognition — nobody ever trained them on it.; why_enablement=Slack integration exists on all plans; managers were never trained
excluded_rep_voice: transcript_id=TG-005; line=Full transparency — we don't have a native mobile app for hourly workers yet, though it's on the roadmap.; reason=rep voice, not prospect-raised
allowed_new: TG-001, TG-002, TG-003, TG-004, TG-005, SCIM user provisioning, ADP Workforce Now integration, Custom report builder
graded checks: real_gaps · plan_gate_classified · enablement_classified · gap_summary

Model's actual answer

show the model's full answer
Analysis of 5 transcript lines for product gaps:

**Candidate 1 - TG-001:**
- Deal: Deal-EC3025
- Amount: [MISSING - no deal amount file provided in prompt]
- Prospect quote: "We need SCIM user provisioning — our security team will not approve any tool that requires manual account management."
- Product docs check: Docs state "SCIM user provisioning and ADP Workforce Now integrations are NOT currently listed as supported capabilities." Plus tiers table shows SSO/SAML only on Enterprise but no SCIM listed anywhere.
- Classification: **REAL GAP**

**Candidate 2 - TG-002:**
- Deal: Deal-D0D6B5
- Amount: [MISSING - no deal amount file provided]
- Prospect quote: "Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us."
- Product docs check: Docs state ADP Workforce Now NOT listed as supported. Supported HRIS list is Workday, BambooHR, Gusto, Rippling only.
- Classification: **REAL GAP**

**Candidate 3 - TG-003:**
- Deal: Deal-CFE7F4
- Amount: [MISSING]
- Prospect quote: "I tried to build a custom report combining recognition frequency with turnover and couldn't — is that possible on our current plan?"
- Product docs check: Custom report builder = Enterprise tier only per tier table.
- Classification: **PLAN GATE** - capability exists on Enterprise

**Candidate 4 - TG-004:**
- Deal: Deal-84DBA6
- Amount: [MISSING]
- Prospect quote: "We turned on the Slack integration in July, but half our managers still haven't given a single recognition — nobody ever trained them on it."
- Product docs check: Slack integration = yes on all plans. Issue is training/adoption.
- Classification: **ROLLOUT/ENABLEMENT ISSUE**

**Candidate 5 - TG-005:**
- Deal: Deal-36C33F
- Speaker is Rep (Alex Franklin): "Full transparency — we don't have a native mobile app for hourly workers yet, though it's on the roadmap."
- Prospect line: "Good to know. The web version should be fine for our office staff for now." - Does not raise gap as blocker.
- Classification: **EXCLUDED - Rep language does not count as prospect voice per instructions**

**Arithmetic:**
Total candidates screened: 5
Real gaps: 2
Plan gate: 1
Rollout/enablement: 1
Excluded (rep voice): 1
Real gap rate: 2/5 = 40%

**Summary - Only Real Gaps:**
1. Deal-EC3025 - SCIM user provisioning - Prospect: "We need SCIM user provisioning — our security team will not approve any tool that requires manual account management." - No amount data provided.
2. Deal-D0D6B5 - ADP Workforce Now integration - Prospect: "Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us." - No amount data provided.
rep-performance
stale-pipeline-by-rep0.83

? The Question

The snapshot date is 2026-09-05. Using the open deals and the engagements-by-deal table, list every open deal with no logged email, call, or meeting in the last 7 days. Compute recency from the engagements table's last_email/last_call/last_meeting fields, NOT from any date field on the deal (the deal's last_contacted_field is unreliable). For each stale deal give: deal alias, owner name, stage, amount, days since last contact. Group by owner, order by amount descending within each owner. Then state, per owner, how many stale deals they carry and the total stale amount. (Do not write outreach text.)
cost $0.1587 · 53s · in 40,084 / out 23,811 tok
✓ stale_recall_ge_80✓ grouped_by_owner✓ jake_top_amount✓ kevin_stale_amt✓ jenn_stale_amt✗ stale_count

✓ Correct Answer (ground truth)

stale_count: 65
jake_amt: 626243.0
kevin_amt: 252905.03
jenn_amt: 261645.0
top_alias: Deal-2D1F1B
top_amount: 240000.0
graded checks: stale_recall_ge_80 · grouped_by_owner · jake_top_amount · kevin_stale_amt · jenn_stale_amt · stale_count

Model's actual answer

show the model's full answer
Snapshot date: 2026-09-05
Stale definition: MAX(last_email, last_call, last_meeting) from engagements_by_deal_90d.csv with days = snapshot - max_date.  Stale if days > 7 (no days = 7 exactly in data, so >7 and >=7 produce same set).

Total open deals parsed from deals_open.csv: 154
Engagement rows found: 152
2 open deals have NO row in engagements_by_deal_90d.csv (data missing):
  - Deal-3EED2C (deal_id 64623982954) owner Alex Franklin amount 7200 stage DS2
  - Deal-57FF13 (deal_id 64524667574) owner Elena Sinclair amount 2100 stage DS1
Per instruction, they cannot have recency computed. If counted as no logged activity they would be additional stale. Below list is for 63 deals with engagement data and days >7.

Calculation example: days = 2026-09-05 - 2026-08-26 = 10 days.

GROUPED BY OWNER - ORDERED AMOUNT DESC WITHIN OWNER

Owner: Bryce Harmon (owner_id 119337721)
  Deal-2D1F1B | stage DS1 | amount 240000 | days 81 | last_max 2026-06-16 (email 2026-06-11, meeting 2026-06-16)
  Deal-66D1FC | stage DS1 | amount 99000 | days 16 | last_max 2026-08-20 (email 2026-08-20)
  Deal-950043 | stage DS1 | amount 70000 | days 19 | last_max 2026-08-17 (email 2026-08-17)
  Deal-B23205 | stage DS1 | amount 45000 | days 16 | last_max 2026-08-20 (email 2026-08-20, meeting 2026-08-20)
  Deal-7BBDFA | stage DS3 | amount 37440 | days 46 | last_max 2026-07-21 (email 2026-07-21)
  Deal-332637 | stage DS2 | amount 36000 | days 9 | last_max 2026-08-27 (email 2026-08-27)
  Deal-1BEEBF | stage DS1 | amount 31500 | days 19 | last_max 2026-08-17 (email 2026-08-17, call 2026-07-30)
  Deal-C5658B | stage DS1 | amount 23400 | days 16 | last_max 2026-08-20 (email 2026-08-20)
  Deal-40522D | stage DS3 | amount 21000 | days 19 | last_max 2026-08-17 (email 2026-08-17)
  Deal-F0EBBB | stage DS3 | amount 11400 | days 24 | last_max 2026-08-12 (email 2026-08-12)
  Deal-E25A09 | stage DS1 | amount 6000 | days 9 | last_max 2026-08-27 (email 2026-08-27)
  Deal-C9C286 | stage DS2 | amount 5502 | days 9 | last_max 2026-08-27 (email 2026-08-27)
  Deal-012CB1 | stage DS1 | amount 1 | days 23 | last_max 2026-08-13 (email 2026-08-13)

Owner: Dana Mercer (owner_id 83155923)
  Deal-44EA29 | stage DS2 | amount 60000 | days 10 | last_max 2026-08-26 (email 2026-08-26)
  Deal-E51FB7 | stage DS2 | amount 43875 | days 12 | last_max 2026-08-24 (email 2026-08-18, call 2026-08-24)
  Deal-B42F46 | stage DS1 | amount 27000 | days 19 | last_max 2026-08-17 (email 2026-08-17)
  Deal-BA3DDC | stage DS3 | amount 23400 | days 15 | last_max 2026-08-21 (email 2026-08-20, call 2026-08-21)
  Deal-9DDE86 | stage DS2 | amount 20000 | days 15 | last_max 2026-08-21 (email 2026-08-21)
  Deal-215CCA | stage DS3 | amount 18900 | days 17 | last_max 2026-08-19 (meeting 2026-08-19)
  Deal-5EED42 | stage DS3 | amount 16250 | days 11 | last_max 2026-08-25 (email 2026-08-25, call 2026-08-25)
  Deal-57887A | stage DS2 | amount 15000 | days 8 | last_max 2026-08-28 (email 2026-08-28)
  Deal-B7EBD1 | stage DS5 | amount 9000 | days 16 | last_max 2026-08-20 (email 2026-08-20, call 2026-08-10)
  Deal-3974EB | stage DS4 | amount 9000 | days 8 | last_max 2026-08-28 (email 2026-08-28, meeting 2026-08-28)
  Deal-F40F04 | stage DS2 | amount 8100 | days 15 | last_max 2026-08-21 (email 2026-08-21, meeting 2026-08-21)
  Deal-87DDD1 | stage DS1 | amount 5000 | days 19 | last_max 2026-08-17 (email 2026-08-17)
  Deal-F336B6 | stage DS3 | amount 4200 | days 15 | last_max 2026-08-21 (email 2026-08-21)
  Deal-0660B4 | stage DS4 | amount 1920 | days 16 | last_max 2026-08-20 (email 2026-08-10, meeting 2026-08-20)

Owner: Alex Franklin (owner_id 84342457)
  Deal-CC08D1 | stage DS1 | amount 24000 | days 16 | last_max 2026-08-20 (email 2026-08-20, meeting 2026-08-19)
  Deal-E73427 | stage DS3 | amount 18000 | days 10 | last_max 2026-08-26 (email 2026-08-26, meeting 2026-08-26)
  Deal-885F45 | stage DS2 | amount 9300 | days 12 | last_max 2026-08-24 (email 2026-08-24)
  Deal-C2FF3C | stage DS1 | amount 8316 | days 10 | last_max 2026-08-26 (email 2026-08-26)
  Deal-0D2F7A | stage DS3 | amount 5100 | days 12 | last_max 2026-08-24 (call 2026-08-24)
  Deal-6C60D4 | stage DS3 | amount 4800 | days 12 | last_max 2026-08-24 (call 2026-08-24)
  Deal-13FEBD | stage DS2 | amount 4680 | days 12 | last_max 2026-08-24 (call 2026-08-24)
  Deal-9D0060 | stage DS3 | amount 3840 | days 12 | last_max 2026-08-24 (email 2026-08-24)
  Deal-690476 | stage DS2 | amount 3600 | days 18 | last_max 2026-08-18 (call 2026-08-18)
  Deal-C6D97A | stage DS4 | amount 3240 | days 8 | last_max 2026-08-28 (email 2026-08-28, call 2026-08-25)
  Deal-EE195F | stage DS3 | amount 3120 | days 8 | last_max 2026-08-28 (email 2026-08-28, meeting 2026-08-06)
  Deal-278DEC | stage DS3 | amount 2700 | days 8 | last_max 2026-08-28 (email 2026-08-28)
  Deal-635B8E | stage DS3 | amount 2600 | days 18 | last_max 2026-08-18 (email 2026-08-18)
  Deal-6883F3 | stage DS1 | amount 2400 | days 16 | last_max 2026-08-20 (email 2026-08-20, meeting 2026-08-20)
  Deal-4A13AD | stage DS3 | amount 2160 | days 26 | last_max 2026-08-10 (email 2026-08-10)
  Deal-F67D31 | stage DS2 | amount 1800 | days 8 | last_max 2026-08-28 (email 2026-08-28)
  Deal-5FDCE4 | stage DS3 | amount 1600 | days 12 | last_max 2026-08-24 (email 2026-08-24)
  Deal-BA571A | stage DS4 | amount 1080 | days 18 | last_max 2026-08-18 (email 2026-08-18)

Owner: Cole Ingram (owner_id 83155924)
  Deal-D04904 | stage DS2 | amount 58529.25 | days 11 | last_max 2026-08-25 (email 2026-08-25)
  Deal-B25F40 | stage DS3 | amount 40000 | days 8 | last_max 2026-08-28 (email 2026-08-28)
  Deal-813836 | stage DS2 | amount 32175 | days 11 | last_max 2026-08-25 (email 2026-08-25)
  Deal-1BA595 | stage DS2 | amount 31750 | days 11 | last_max 2026-08-25 (email 2026-08-25)
  Deal-CFE1E8 | stage DS3 | amount 18000 | days 11 | last_max 2026-08-25 (email 2026-08-25)
  Deal-CD47A6 | stage DS2 | amount 12168 | days 11 | last_max 2026-08-25 (email 2026-08-25, call 2026-08-24)
  Deal-627646 | stage DS3 | amount 11193 | days 11 | last_max 2026-08-25 (email 2026-08-25)
  Deal-FF809F | stage DS2 | amount 7781.2 | days 11 | last_max 2026-08-25 (email 2026-08-25)
  Deal-AF932D | stage DS2 | amount 7225.4 | days 11 | last_max 2026-08-25 (email 2026-08-25)
  Deal-A71728 | stage DS2 | amount 6947.5 | days 11 | last_max 2026-08-25 (email 2026-08-25)
  Deal-8BC9F5 | stage DS2 | amount 5616 | days 10 | last_max 2026-08-26 (email 2026-08-26)
  Deal-175395 | stage DS3 | amount 4779.88 | days 11 | last_max 2026-08-25 (email 2026-08-25)
  Deal-481E24 | stage DS3 | amount 4140 | days 10 | last_max 2026-08-26 (call 2026-08-26)
  Deal-C7F9BF | stage DS2 | amount 3360 | days 11 | last_max 2026-08-25 (email 2026-08-25, call 2026-08-24)
  Deal-2F3A66 | stage DS3 | amount 3334.8 | days 11 | last_max 2026-08-25 (email 2026-08-25)
  Deal-342E96 | stage DS2 | amount 2700 | days 24 | last_max 2026-08-12 (email 2026-08-12)
  Deal-E568D5 | stage DS3 | amount 1875 | days 11 | last_max 2026-08-25 (email 2026-08-25)
  Deal-FD9F4E | stage DS5 | amount 1330 | days 10 | last_max 2026-08-26 (email 2026-08-26)

STALE TOTALS PER OWNER (from engagement data only):
  Bryce Harmon: 13 stale deals, total stale amount = 626243 (240000+99000+70000+45000+37440+36000+31500+23400+21000+11400+6000+5502+1)
  Dana Mercer: 14 stale deals, total = 261645 (60000+43875+27000+23400+20000+18900+16250+15000+9000+9000+8100+5000+4200+1920)
  Alex Franklin: 18 stale deals, total = 102336 (24000+18000+9300+8316+5100+4800+4680+3840+3600+3240+3120+2700+2600+2400+2160+1800+1600+1080)
  Cole Ingram: 18 stale deals, total = 252905.03 (58529.25+40000+32175+31750+18000+12168+11193+7781.2+7225.4+6947.5+5616+4779.88+4140+3360+3334.8+2700+1875+1330)
  Farid Osman: 0 stale (all contacts within 7 days)
  Elena Sinclair: see missing data note below

Overall stale with engagement data: 63 deals, total amount 1243129.03 = 626243+261645+102336+252905.03

Data missing: Deal-3EED2C and Deal-57FF13 have no entry in engagements_by_deal_90d.csv, so recency cannot be computed from last_email/last_call/last_meeting.
activity-mix-vs-outcomePASS 1.00

? The Question

The snapshot date is 2026-09-05. For each sales rep (owner), compute the last-30-day totals of emails, calls, and meetings from the engagements table, and the number of deals that entered DS2 in the last 30 days (t_ds2 within 30 days of the snapshot). Report per rep: the activity mix as percentages (emails/calls/meetings share of that rep's total activities), total activities per DS2 entry, and rank the reps by that efficiency ratio (lowest activities per DS2 entry = most efficient). Name the most efficient rep and the highest-volume rep, and note if they differ. Do not attribute cause.
cost $0.0633 · 15s · in 35,589 / out 3,265 tok
✓ justin_ds2✓ justin_volume_high✓ efficient_named✓ mix_present✓ no_causal_claim

✓ Correct Answer (ground truth)

justin_ds2: 4
graded checks: justin_ds2 · justin_volume_high · efficient_named · mix_present · no_causal_claim

Model's actual answer

show the model's full answer
Snapshot date: 2026-09-05
Window for DS2 entry: t_ds2 >= 2026-08-06 and <= 2026-09-05 (30 days). Activities: emails_30d, calls_30d, meetings_30d summed per owner_id via deals_open.csv mapping.

### Totals from engagements_by_deal_90d.csv

**84342457 - Alex Franklin**
- emails = 199, calls = 25, meetings = 31
- total = 199+25+31 = 255
- DS2 in window = 10 deals:
  Deal-403845 (2026-09-02), Deal-1FC049 (2026-09-03), Deal-D1E6C2 (2026-08-11), Deal-D9A72E (2026-08-06), Deal-CA5E44 (2026-08-24), Deal-4F775F (2026-08-17), Deal-898FC5 (2026-08-28), Deal-46988D (2026-08-26), Deal-E73427 (2026-08-26), Deal-92D97D (2026-09-02)
- mix: 199/255=78.0% emails, 25/255=9.8% calls, 31/255=12.2% meetings
- activities per DS2 = 255 / 10 = 25.5

**119337721 - Bryce Harmon**
- emails = 162, calls = 0, meetings = 43
- total = 162+0+43 = 205
- DS2 in window = 4 deals:
  Deal-25F752 (2026-08-10), Deal-D73B89 (2026-09-03), Deal-CA7DC0 (2026-08-12), Deal-1CCE5C (2026-08-06)
- mix: 162/205=79.0% emails, 0/205=0.0% calls, 43/205=21.0% meetings
- activities per DS2 = 205 / 4 = 51.25

**83155923 - Dana Mercer**
- emails = 84, calls = 18, meetings = 11
- total = 84+18+11 = 113
- DS2 in window = 1 deal:
  Deal-57887A (2026-08-07)
- mix: 84/113=74.3% emails, 18/113=15.9% calls, 11/113=9.7% meetings
- activities per DS2 = 113 / 1 = 113.0

**83155924 - Cole Ingram**
- emails = 96, calls = 14, meetings = 1
- total = 96+14+1 = 111
- DS2 in window = 2 deals:
  Deal-42326B (2026-08-26), Deal-1BA595 (2026-08-12)
- mix: 96/111=86.5% emails, 14/111=12.6% calls, 1/111=0.9% meetings
- activities per DS2 = 111 / 2 = 55.5

**716654662 - Farid Osman**
- emails = 38, calls = 0, meetings = 34
- total = 38+0+34 = 72
- DS2 in window = 1 deal:
  Deal-499BF6 (2026-08-26)
- mix: 38/72=52.8% emails, 0/72=0.0% calls, 34/72=47.2% meetings
- activities per DS2 = 72 / 1 = 72.0

**701163055 - Elena Sinclair**
- Missing data: No rows in engagements_by_deal_90d.csv for Deal-57FF13, and t_ds2 is blank in deals_open.csv. Cannot compute totals or DS2 count from provided files.

### Ranking by Efficiency (lowest activities per DS2 = most efficient)

1. Alex Franklin - 25.5 (255 / 10)
2. Bryce Harmon - 51.25 (205 / 4)
3. Cole Ingram - 55.5 (111 / 2)
4. Farid Osman - 72.0 (72 / 1)
5. Dana Mercer - 113.0 (113 / 1)

Most efficient rep: Alex Franklin (25.5 activities per DS2 entry)

Highest-volume rep: Alex Franklin (255 total activities last 30 days)

They do NOT differ - same rep is both most efficient and highest-volume.
ae-qtd-scorecardPASS 1.00

? The Question

Build a QTD scorecard for the named AE as of 2026-09-05: bookings vs quota with attainment percent, new vs expansion split, active pipeline by stage with amount, rolling 90-day DS2-to-won rate, win and loss counts with top loss reason, activity volume by type in the last 30 days. Exclude any closed-won deal dated before the quarter. Close with three coaching observations grounded in the numbers.
cost $0.0515 · 22s · in 21,031 / out 4,784 tok
✓ bookings✓ quota✓ attainment✓ new_expansion_split✓ top_loss_reason✓ coaching_obs

✓ Correct Answer (ground truth)

owner: Alex Franklin
quarter: 2026-Q3
snapshot_date: 2026-09-05
bookings: 150000
quota: 200000
attainment_pct: 75.0
new_bookings: 113500
new_count: 5
expansion_bookings: 36500
expansion_count: 3
pipeline_by_stage: {"DS1": {"count": 20, "amount": 284621}, "DS2": {"count": 28, "amount": 353760}, "DS3": {"count": 67, "amount": 552705}, "DS4": {"count": 5, "amount": 23574}, "DS5": {"count": 5, "amount": 45730}}
open_deals_count: 125
ds2_to_won: window_start=2026-06-07; won_in_window=8; closed_in_window=35; rate=0.2286
wins_qtd: 8
losses_qtd: 27
top_loss_reason: reason=Lost- Timing (1 year or more); count=13
activity_30d_totals: emails_30d=807; calls_30d=112; meetings_30d=128; notes_30d=50
prior_quarter_excluded: deal_alias=Deal-B3E6F1; close_date=2026-06-20; amount=24000
allowed_new: 18 items — e.g. Deal-A1C3E5, Deal-B7D2F4, Deal-C9E1A6, Deal-D4B8C2
graded checks: bookings · quota · attainment · new_expansion_split · top_loss_reason · coaching_obs

Model's actual answer

show the model's full answer
QTD Scorecard - Alex Franklin - as of 2026-09-05 - Q3 2026 (2026-07-01 to 2026-09-05)

1. BOOKINGS vs QUOTA:
Quota Q3: $200,000 [quota.csv]
Excluded closed-won before quarter: Deal-B3E6F1 $24,000 on 2026-06-20

QTD Bookings count:
Deal-A1C3E5 $40,000 (2026-07-15) + Deal-F2C7D8 $20,000 (2026-07-24) + Deal-B7D2F4 $35,000 (2026-07-31) + Deal-C9E1A6 $21,000 (2026-08-12) + Deal-A8B4D6 $12,000 (2026-08-19) + Deal-D4B8C2 $11,000 (2026-08-21) + Deal-E6F3A9 $6,500 (2026-09-02) + Deal-C5D9E2 $4,500 (2026-09-03) = $150,000

Attainment: 150,000 / 200,000 = 0.75 = 75%
Remaining to quota: $50,000

2. NEW vs EXPANSION SPLIT (of $150,000 bookings):
New: Deal-A1C3E5 $40,000 + Deal-B7D2F4 $35,000 + Deal-C9E1A6 $21,000 + Deal-D4B8C2 $11,000 + Deal-E6F3A9 $6,500 = $113,500
113,500 / 150,000 = 75.7%
Expansion: Deal-F2C7D8 $20,000 + Deal-A8B4D6 $12,000 + Deal-C5D9E2 $4,500 = $36,500
36,500 / 150,000 = 24.3%

3. ACTIVE PIPELINE BY STAGE [status=open in ae_deals.csv]:
Total open: 125 deals, $1,260,390
DS1: 20 deals, $284,621
DS2: 28 deals, $353,760
DS3: 67 deals, $552,705
DS4: 5 deals, $23,574 [Deal-1FC049 $1,920, Deal-F9A08A $2,484, Deal-C6D97A $3,240, Deal-BA571A $1,080, Deal-5408B0 $14,850]
DS5: 5 deals, $45,730 [Deal-403845 $9,000, Deal-547B2B $11,200, Deal-A2B47C $6,360, Deal-C61CF7 $5,400, Deal-D348E1 $13,770]

4. ROLLING 90-DAY DS2-TO-WON RATE:
Window: 2026-06-07 to 2026-09-05 [snapshot minus 90 days]
Entered DS2 in window: 111 deals [entered_ds2 field]
Won from that cohort: 8 deals [Deal-A1C3E5, Deal-F2C7D8, Deal-B7D2F4, Deal-A8B4D6, Deal-C9E1A6, Deal-D4B8C2, Deal-E6F3A9, Deal-C5D9E2]
Lost from cohort: 27 deals
Open from cohort: 76 deals
DS2-to-won = 8 / 111 = 7.2%
DS2-to-won on closed only = 8 / (8+27) = 8 / 35 = 22.9%

5. WIN / LOSS COUNTS QTD [close_date >=2026-07-01 and <=2026-09-05]:
Wins: 8 (listed above)
Losses: 27
Loss breakdown: Lost- Timing (1 year or more) 13, MIA 5, Competitor 5, Lost DM 2, Feature Request 1, Lost- Does not fit ICP 1
Top loss reason: Lost- Timing (1 year or more) = 13 / 27 = 48.1%

6. ACTIVITY VOLUME LAST 30 DAYS [ae_engagements.csv sum of _30d columns]:
Total across all 169 engagement rows: emails 807, calls 112, meetings 128, notes 50
Calculation: sum(emails_30d)=807, sum(calls_30d)=112, sum(meetings_30d)=128, sum(notes_30d)=50
If filtered to QTD-relevant deals only (open + QTD wins/losses = 160 deals): emails 797, calls 110, meetings 126, notes 47

COACHING OBSERVATIONS:
1. Pipeline is bottom-heavy early stage - 67 of 125 open deals (53.6%) and $552,705 of $1,260,390 (43.9%) sit in DS3, while DS4+DS5 combined is only 10 deals for $69,304. With 7.2% DS2-to-won in last 90 days, need to advance DS3 to DS4/5 before quarter-end or risk $50K gap.

2. Qualification gap - 48% of QTD losses are Lost- Timing (1 year or more) (13 of 27) including large deals Deal-B3ABED $40,001 and Deal-D1A623 $25,200. High DS2 entry volume (111) with low win conversion suggests entering DS2 without timeframe.

3. Activity vs outcome - 27 losses to 8 wins in QTD (3.4:1) despite 807 emails and 112 calls in last 30 days. 5 MIA + 2 Lost DM = 7 of 27 losses (25.9%) are stakeholder losses, indicating champion/DM validation needed earlier despite high email volume.
multithreading-gapPASS 1.00

? The Question

Find every open deal that is single-threaded (fewer than 2 active contacts) or under-threaded (fewer than 3, or all contacts in one persona). Active means engaged in the last 60 days and not marked former. Personas: economic buyer, champion, HR admin, IT security, finance. For each flagged deal: amount, stage, active contact count, personas present, personas missing, the most valuable persona to add given the stage, and any on-file unengaged contact at that company who fits; otherwise say none on file.
cost $0.1156 · 108s · in 43,606 / out 8,635 tok
✓ flagged_recall✓ persona_language✓ single_thread_term✓ on_file_fit

✓ Correct Answer (ground truth)

active_window_start: 2026-07-07
persona_universe: economic buyer, champion, HR admin, IT security, finance
flag_rule: flag if active_contacts < 2 (single-threaded) or active_contacts < 3 or all active contacts share one persona (under-threaded); active = engaged within 60 days and not former
best_persona_rule: DS2/DS3: economic buyer > IT security > finance > HR admin > champion; DS4/DS5: economic buyer > finance > IT security > HR admin > champion; first persona not present among active contacts
flagged: deal_alias=Deal-36C33F; amount=15000; stage=DS2; active_count=1; reason=single-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-50D386; amount=36000; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-5408B0; amount=14850; stage=DS4; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-5BFE3B; amount=18000; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-885F45; amount=9300; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=IT security | deal_alias=Deal-92D97D; amount=60000; stage=DS2; active_count=1; reason=single-threaded; best_persona_to_add=economic buyer … (+5 more)
healthy_not_flagged: Deal-4B0BEB, Deal-84DBA6, Deal-D348E1
allowed_new: 59 items — e.g. C-077A0E, C-0D15DF, C-1BDFE0, C-2AE3AA
graded checks: flagged_recall · persona_language · single_thread_term · on_file_fit

Model's actual answer

show the model's full answer
Open deals assessed = only those in `deal_contacts.csv` (14 deals). Other open deals in `ae_deals.csv` / `deals_open.csv` have no contact rows - data missing to assess threading.

Reference date: max `last_engaged_date` in file = 2026-09-03
Active cutoff: 2026-09-03 minus 60 days = 2026-07-05
Active = `is_former=false` AND `last_engaged_date >= 2026-07-05`

Counts:

 - 61032318100 Deal-EC3025: 2 rows total, 1 former excluded, 1 active
 - 59728118877 Deal-92D97D: 2 rows, 1 outside window (2026-06-01) -> 1 active
 - 61055128146 Deal-50D386: 2 active
 - 60081655042 Deal-D0D6B5: 3 active
 - 51674270311 Deal-5BFE3B: 2 active
 - 63929535929 Deal-84DBA6: 3 active
 - 63739413805 Deal-36C33F: 3 rows, 2 former -> 1 active
 - 61038797752 Deal-4B0BEB: 4 active
 - 60686135564 Deal-885F45: 2 active
 - 62639586615 Deal-FCBE5B: 1 active
 - 60182332309 Deal-5408B0: 2 active
 - 61750885954 Deal-D348E1: 5 active
 - 62121783047 Deal-C6D97A: 3 active
 - 49757401138 Deal-F9A08A: 2 rows, 1 outside window (2026-06-20) -> 1 active

Definitions:
single-threaded = active <2
under-threaded = active <3 OR all active personas identical

FLAGGED DEALS (10):

1. Deal-EC3025 | 61032318100 | C-FDD0C7
 amount=62000 stage=DS2 (from ae_deals.csv) active=1 single-threaded + under-threaded
 personas present: [champion]
 missing: [economic buyer, HR admin, IT security, finance]
 most valuable to add given DS2: economic buyer
 unengaged fit: C-FDD0C7 CT-6827DB Chief People Officer, economic buyer - ON FILE

2. Deal-92D97D | 59728118877 | C-E23238
 amount=60000 stage=DS2 active=1 single-threaded + under-threaded
 present: [HR admin]
 missing: [economic buyer, champion, IT security, finance]
 most valuable DS2: economic buyer (champion also critical)
 unengaged: none on file (C-E23238 not in unengaged_contacts.csv)

3. Deal-50D386 | 61055128146 | C-EB10E4
 amount=36000 stage=DS2 active=2 under-threaded (<3)
 present: [champion, HR admin]
 missing: [economic buyer, IT security, finance]
 most valuable DS2: economic buyer
 unengaged fit: C-EB10E4 CT-A1C4B3 Chief People Officer, economic buyer - ON FILE

4. Deal-D0D6B5 | 60081655042 | C-32918E
 amount=23800 stage=DS2 active=3 under-threaded (all one persona)
 present: [champion] x3 (CT-87CED4, CT-DE6D7C, CT-FD70B2)
 missing: [economic buyer, HR admin, IT security, finance]
 most valuable DS2: economic buyer
 unengaged fit: C-32918E CT-1FA4DB Chief People Officer, economic buyer - ON FILE

5. Deal-5BFE3B | 51674270311 | C-535D36
 amount=18000 stage=DS2 active=2 under-threaded (<3 + all champion)
 present: [champion] x2
 missing: [economic buyer, HR admin, IT security, finance]
 most valuable DS2: economic buyer
 unengaged: none on file (C-535D36 not in list)

6. Deal-885F45 | 60686135564 | C-5E8EFB
 amount=9300 stage=DS2 active=2 under-threaded (<3)
 present: [economic buyer, champion]
 missing: [HR admin, IT security, finance]
 most valuable DS2: HR admin (next after EB+champion); IT security is secondary blocker
 unengaged fit for most valuable HR admin: none on file for HR admin. On file for IT security: C-5E8EFB CT-B3F25D IT Security Lead, IT security

7. Deal-FCBE5B | 62639586615 | C-737030
 amount=7500 stage=DS2 (ae_deals) / DS2 PIPELINE (deals_open amount 7500) active=1 single-threaded + under-threaded
 present: [champion]
 missing: [economic buyer, HR admin, IT security, finance]
 most valuable DS2: economic buyer
 unengaged: none on file (C-737030 not in list)

8. Deal-5408B0 | 60182332309 | C-2AE3AA
 amount=14850 stage=DS4 active=2 under-threaded (<3)
 present: [champion, HR admin]
 missing: [economic buyer, IT security, finance]
 most valuable DS4: economic buyer (DS4 BEST_CASE still requires EB if missing, else IT security/finance)
 unengaged fit: C-2AE3AA CT-07FA76 Chief People Officer, economic buyer - ON FILE

9. Deal-C6D97A | 62121783047 | C-5A8FC2
 amount=3240 stage=DS4 active=3 under-threaded (all one persona)
 present: [champion] x3
 missing: [economic buyer, HR admin, IT security, finance]
 most valuable DS4: economic buyer (for close authority) / IT security+finance as gate
 unengaged: none on file (C-5A8FC2 not in list)

10. Deal-36C33F | 63739413805 | C-077A0E
 amount=15000 stage=DS2 active=1 single-threaded + under-threaded
 present: [IT security]
 missing: [economic buyer, champion, HR admin, finance]
 most valuable DS2: economic buyer (champion equally critical to drive deal from technical thread)
 unengaged fit: C-077A0E CT-1DB73E Chief People Officer, economic buyer - ON FILE

11. Deal-F9A08A | 49757401138 | C-0D15DF
 amount=2484 stage=DS4 active=1 single-threaded + under-threaded
 present: [champion]
 missing: [economic buyer, HR admin, IT security, finance]
 most valuable DS4: economic buyer (given single thread; otherwise finance/IT security for DS4)
 unengaged fit: C-0D15DF CT-697541 Chief People Officer, economic buyer - ON FILE

NOT FLAGGED (well-threaded):
- Deal-4B0BEB 61038797752 active=4 personas champion, economic buyer, HR admin, finance
- Deal-D348E1 61750885954 active=5 all 5 personas
- Deal-84DBA6 63929535929 active=3 personas champion, economic buyer, IT security
talk-track-from-transcriptsPASS 1.00

? The Question

From this rep's last 10 calls: what they lead with in the first five minutes, how they handle the three most common objections, how often a concrete next step is agreed (rate), and every competitor a prospect raised. Quote at most one line per finding and only lines that appear in the transcripts. End with two coaching notes.
cost $0.0174 · 11s · in 2,684 / out 2,743 tok
✓ objection_count✓ next_step_rate✓ competitors✓ coaching✓ quote_used

✓ Correct Answer (ground truth)

rep: Alex Franklin
n_transcripts: 10
opener_theme: theme=opens with a customer recognition win story (400-person retailer, 18% turnover reduction); count=8; total=10; quote=Quick one to kick off: a 400-person retailer we work with cut regretted turnover 18% in two quarters after automating milestone awards, and I wanted to walk you through exactly how they did it.; transcript_id=TT-001
objections: objection=budget; count=4; quote=Honestly, budget is locked until next fiscal year — I can't add a new line item right now.; transcript_id=TT-001 | objection=timing; count=3; quote=This is good, but can we revisit it next quarter? Open enrollment starts in October and we're underwater.; transcript_id=TT-002 | objection=in-house; count=3; quote=We already do recognition with a spreadsheet and quarterly gift cards — why would we change?; transcript_id=TT-004
objection_count: 3
next_step: agreed=7; total=10; rate=0.7
competitors_prospect_raised: Awardco, Kudos
competitor_quotes: Awardco=We're also in late talks with Awardco — their rewards catalog looks bigger than yours.; Kudos=How are you different from Kudos? Our CEO used them at her last company.
rep_raised_competitors_excluded: Workhuman
allowed_new: 13 items — e.g. TT-001, TT-002, TT-003, TT-004
graded checks: objection_count · next_step_rate · competitors · coaching · quote_used

Model's actual answer

show the model's full answer
From last 10 calls (TT-001 to TT-010 / Deal-D348E1 through Deal-84DBA6):

**1. What they lead with in first five minutes (minute <=5):**
- 8 of 10 calls (80%) lead with same retailer outcome story. Example:
> "Quick one to kick off: a 400-person retailer we work with cut regretted turnover 18% in two quarters after automating milestone awards, and I wanted to walk you through exactly how they did it." [Deal-D348E1, Deal-5408B0, Deal-547B2B, Deal-C61CF7, Deal-60C2C2, Deal-EDC141, Deal-D9A12F, Deal-84DBA6]
- 1 of 10 leads with agenda: "I put together a short agenda — security review first, then pricing." [Deal-403845]
- 1 of 10 leads with pricing: "You asked for straight pricing last time, so let's start there." [Deal-1E2498]
- Additional rep-initiated competitor insert in first five: "And unlike Workhuman, our pricing includes the full rewards catalog with no extra margin." [Deal-C61CF7 minute 2]

**2. Three most common objections and handling:**

a) Budget locked — 4 occurrences [Deal-D348E1, Deal-547B2B, Deal-60C2C2, Deal-84DBA6]:
Prospect: "Honestly, budget is locked until next fiscal year — I can't add a new line item right now."
Rep handles: "Totally fair. Most teams fund this out of turnover savings — that retailer saved about $210k in avoided backfills, which is how their finance team signed off."

b) Revisit next quarter / timing — 3 occurrences [Deal-5408B0, Deal-C61CF7, Deal-D9A12F]:
Prospect: "This is good, but can we revisit it next quarter? Open enrollment starts in October and we're underwater."
Rep handles: "Makes sense. What if we scope a 90-day pilot with one department so you have internal data before next quarter's planning?"

c) Spreadsheet/status quo — 3 occurrences [Deal-403845, Deal-EDC141, Deal-1E2498]:
Prospect: "We already do recognition with a spreadsheet and quarterly gift cards — why would we change?"
Rep handles: "Spreadsheets work until they scale — the difference is automation: milestones fire without HR lifting a finger, and you get analytics on who is being recognized."

**3. Concrete next step agreed rate:**

Prospect agrees: "Yes, Thursday at 2pm works — send the invite and I'll bring our HRIS manager." = 7 deals [Deal-D348E1, Deal-5408B0, Deal-547B2B, Deal-C61CF7, Deal-60C2C2, Deal-D9A12F, Deal-1E2498]
No agreement: 3 deals [Deal-403845, Deal-EDC141, Deal-84DBA6]

Arithmetic: 7 agreed / 10 total = 0.7 = 70% next-step rate.

**4. Every competitor a prospect raised:**

- Awardco: "We're also in late talks with Awardco — their rewards catalog looks bigger than yours." [Deal-547B2B]
- Kudos: "How are you different from Kudos? Our CEO used them at her last company." [Deal-EDC141]

Workhuman mentioned in Deal-C61CF7 was rep-raised, not prospect-raised, so excluded per definition.

**Coaching notes:**

1. Next-step failure correlates with committee/deferral language. Both "We need to see what the budget committee says before anything moves." [Deal-403845] and "We'll have to wait for the committee — I can't commit to anything today." [Deal-84DBA6] + "I need to think about it — there's no urgency on our side." [Deal-EDC141] resulted in "Understood — I'll leave it with you." / "Fair enough." / "Understood, thanks for the candor." with no counter-close. Need a committee-specific close.

2. Opening is identical 80% of time. Consider varying the first-five hook for Deal-403845 (agenda) and Deal-1E2498 (pricing) types and testing retention vs. turnover proof point when prospect already signals spreadsheet process.
reporting-analytics
q3-weighted-forecastPASS 1.00

? The Question

Below are the open deals in the default sales pipeline with a close date between 2026-07-01 and 2026-10-15, pulled from the CRM warehouse on 2026-09-05. Stages: DS1 (discovery) through DS5 (contract). Build the Q3 2026 forecast (quarter = 2026-07-01 to 2026-09-30). Weighting rule: 100 percent of COMMIT plus 35 percent of BEST_CASE. PIPELINE deals count zero. Only deals with a close date inside the quarter count. Report: COMMIT total, BEST_CASE total, weighted forecast, count of deals in each category inside the quarter, the deals you excluded for being outside the quarter (count and total amount), and the top 5 BEST_CASE deals by amount inside the quarter. Show arithmetic. Then, in a final '## Data quality' section of no more than four sentences, name the data quality problems in this extract that would make the forecast unreliable if it were run unattended.
cost $0.0555 · 15s · in 23,433 / out 5,506 tok
✓ commit_total_44729✓ best_case_total_203565✓ weighted_115976.75✓ commit_count_7✓ best_case_count_24✓ excluded_count_32✓ excluded_amount_227575✓ top5_aliases✓ flags_blank_owner✓ flags_missing_why_buys

✓ Correct Answer (ground truth)

graded checks: commit_total_44729 · best_case_total_203565 · weighted_115976.75 · commit_count_7 · best_case_count_24 · excluded_count_32 · excluded_amount_227575 · top5_aliases · flags_blank_owner · flags_missing_why_buys

Model's actual answer

show the model's full answer
Q3 2026 Forecast = 2026-07-01 to 2026-09-30 inclusive.

### Inside Quarter (54 deals)

* COMMIT: 7 deals
  - Deal-547B2B 11200 + Deal-B7EBD1 9000 + Deal-403845 9000 + Deal-A2B47C 6360 + Deal-2465CE 5400 + Deal-A5E80A 2520 + Deal-499BF6 1249 = 44729

* BEST_CASE: 24 deals
  - Sum = 38935 + 24000 + 19656 + 16250 + 11116 + 10800 + 10500 + 9890 + 9720 + 9000 + 7200 + 3840 + 3780 + 3600 + 3240 + 3150 + 3120 + 3060 + 2916 + 2760 + 2484 + 2100 + 1920 + 528 = 203565

* PIPELINE: 23 deals, amount 201637.4, weight 0

Weighted Forecast:
  = 100% * COMMIT + 35% * BEST_CASE + 0% * PIPELINE
  = 44729 + 0.35 * 203565
  = 44729 + 71247.75
  = 115976.75

### Excluded - Outside Quarter (32 deals)

Count: 32
Total Amount: 227575
Arithmetic: 43875 + 1600 + 1400 + 5400 + 7500 + 1800 + 7690 + 2400 + 1080 + 18000 + 9000 + 5400 + 4400 + 4300 + 3600 + 3600 + 1680 + 3300 + 17000 + 13770 + 10800 + 9000 + 7920 + 7200 + 5700 + 5400 + 5400 + 5160 + 4800 + 4000 + 3600 + 1800 = 227575

Includes: Deal-E51FB7, Deal-5FDCE4, Deal-7FA0C3, Deal-600CD9, Deal-FCBE5B, Deal-8AD4A5, Deal-9F2E43, Deal-ED725A, Deal-F5A622, Deal-B936FE, Deal-293AF3, Deal-C61CF7, Deal-D1E6C2, Deal-D9E112, Deal-47AE31, Deal-15D24F, Deal-ED13B0, Deal-766C74, Deal-D9A12F, Deal-D348E1, Deal-4062CF, Deal-034D49, Deal-E0ADD8, Deal-712010, Deal-6691E0, Deal-A92065, Deal-1D532E, Deal-48B656, Deal-E531A6, Deal-5AD94B, Deal-901332, Deal-D7E999

### Top 5 BEST_CASE by Amount Inside Quarter

1. Deal-2D7423 - 38935 - 2026-09-30 - DS3
2. Deal-25F752 - 24000 - 2026-09-25 - DS4
3. Deal-E53952 - 19656 - 2026-09-30 - DS4
4. Deal-5EED42 - 16250 - 2026-09-30 - DS3
5. Deal-FA32A0 - 11116 - 2026-09-25 - DS3

## Data quality
85 of 86 deals have blank owner (only Deal-C9C286 has Bryce Harmon) and 71 have why_buys_chars=0, breaking attribution and qualification checks. Stage/forecast mismatches exist: Deal-A5E80A is COMMIT in DS1, Deal-499BF6 is COMMIT in DS2, Deal-6787C2 is PIPELINE in DS4, and Deal-C61CF7 is BEST_CASE in DS5. Deal-42326B has a fractional amount 2480.4 inconsistent with other integer amounts, indicating inconsistent currency handling.
aha-moment-2x2PASS 1.00

? The Question

Test the activation hypothesis on the company cohort below: companies with 5+ unique givers (m1_users >= 5) AND 1+ successful redemption (m1_redemptions >= 1) in their first calendar month retain better at 24 months than those with one signal or neither. Every company in this extract is already 25+ months old, so current_status = 'active' means retained at 24 months. Report the full 2x2 (cohort size and 24-month retention rate for: both signals, givers-only, redemption-only, neither), state how many companies were excluded from the denominator and why (if none, say so), name the single signal with the largest retention lift, and state what this does and does not prove.
cost $0.1088 · 26s · in 45,437 / out 11,672 tok
✓ both_66✓ givers_47✓ redemp_31✓ neither_40✓ excluded_0✓ largest_lift_givers

✓ Correct Answer (ground truth)

graded checks: both_66 · givers_47 · redemp_31 · neither_40 · excluded_0 · largest_lift_givers

Model's actual answer

show the model's full answer
Activation hypothesis test - 220 companies in extract:

Definition:
* retained = current_status = 'active' at 24 months
* givers signal = m1_users >=5
* redemption signal = m1_redemptions >=1

2x2 breakdown - all denominators use provided rows:

1. BOTH signals (m1_users >=5 AND m1_redemptions >=1):
   size = 47
   retained = 31 (e.g., C-0D284E42, C-0BC46AD9, C-0F6C0F34, C-0B05971D, C-0AAA9434 ...)
   retention = 31/47 = 0.659574 = 65.96%

2. GIVERS-ONLY (m1_users >=5 AND m1_redemptions =0):
   size = 49
   retained = 23
   retention = 23/49 = 0.469387 = 46.94%

3. REDEMPTION-ONLY (m1_users <5 AND m1_redemptions >=1):
   size = 29
   retained = 9
   retention = 9/29 = 0.310344 = 31.03%

4. NEITHER (m1_users <5 AND m1_redemptions =0):
   size = 95
   retained = 38
   retention = 38/95 = 0.40 = 40.0%

Check: 47+49+29+95 = 220 = total extract. 31+23+9+38 = 101 active total = 101/220 = 45.91% overall retention.

Excluded from denominator: 0 companies. All 220 rows have m1_users and m1_redemptions present. No rows excluded. Every row is 25+ months old per prompt instruction, so no age filter needed.

Single signal with largest retention lift:

* Givers-only vs Neither: 46.94% - 40.0% = +6.94pp lift
* Redemption-only vs Neither: 31.03% - 40.0% = -8.97pp (negative)

If counted as any company with signal regardless of other:
* Any givers (both+givers-only) = 54/96 = 56.25%
  Non-givers (neither+redemption-only) = 47/124 = 37.90%
  Lift = 18.35pp
* Any redemption (both+redemption-only) = 40/76 = 52.63%
  Non-redemption = 61/144 = 42.36%
  Lift = 10.27pp

In both definitions, m1_users >=5 (5+ unique givers) has the largest single-signal retention lift.

What this does and does not prove:

Does: Shows correlation in this 2023 cohort that early breadth (5+ givers) combined with completing the loop (1+ redemption) is associated with highest 24-month retention at 65.96%, +25.96pp over neither baseline. Givers signal alone outperforms redemption alone.

Does NOT: Prove causation. This is observational, not randomized. Does not control for confounders (product_tier, company size/ARR, has_slack, has_hris, industry_group, country, onboarding support) that could drive both activation and retention. Does not prove forcing 5 givers or forcing a redemption would improve retention.
arr-rollforward-reconciliationPASS 1.00

? The Question

Reconcile ARR between CRM company records and billing subscriptions as of 2026-09-05. Billing ARR is MRR x 12. Report both totals, the variance, and a decomposition into buckets (status mismatch, rounding, missing records, other) that sums exactly to the variance. List each mismatched account with a suggested owner. Business rule: any subscription with a term other than 12 months must have cf_agreement_end_date populated; list violations.
cost $0.0325 · 20s · in 5,307 / out 4,888 tok
✓ billing_total✓ crm_total✓ variance✓ buckets_sum✓ violations_listed

✓ Correct Answer (ground truth)

as_of: 2026-09-05
billing_arr_total: 604739.28
crm_arr_total: 603581.76
variance: 1157.52
variance_definition: billing_arr_total - crm_arr_total (billing ARR = sum of MRR x 12 over status=active subscriptions)
buckets: {"status_mismatch": {"amount": "-13158.48", "accounts": [{"company_alias": "C-0C8323BF", "billing_arr": "0.00", "hubspot_arr": "4905.24", "diff": "-4905.24", "suggested_owner": "Dana Mercer"}, {"compa
bucket_sum_equals_variance: True
n_mismatched_accounts: 7
rule: term_months != 12 requires cf_agreement_end_date
violations: subscription_id=SUB-0002; company_alias=C-1794A52C; term_months=24 | subscription_id=SUB-0019; company_alias=C-22170CA1; term_months=36
n_companies: 40
graded checks: billing_total · crm_total · variance · buckets_sum · violations_listed

Model's actual answer

show the model's full answer
As of 2026-09-05 reconciliation:

**Totals**

Billing ARR = MRR x 12

* Billing (ALL subscriptions): 
  SUM MRR = 51491.48 / Calculation: 617,897.76 / 12
  ARR = 617,897.76

* Billing (ACTIVE only):
  SUM MRR active = 50394.94
  ARR active = 604,739.28 = 617,897.76 - 13,158.48

* CRM company_arr.csv:
  ARR CRM = 603,581.76

**Variance**

* CRM - Billing_ACTIVE = 603,581.76 - 604,739.28 = -1,157.52
  Inverse: Billing_ACTIVE - CRM = +1,157.52 (Billing exceeds CRM)

* CRM - Billing_ALL = 603,581.76 - 617,897.76 = -14,316.00

Decomposition below sums to CRM - Billing_ACTIVE = -1,157.52 variance.
If using Billing_ALL, status mismatch = 0 and same other buckets sum to -14,316.00

**Decomposition that sums exactly to variance -1,157.52**

Bucket definitions:
* status mismatch = cancelled billing but active CRM
* missing records = in one system not the other
* rounding = MRR*12 vs CRM rounding diff < $1 (none found)
* other = active/active pricing mismatch

Math:

1. Status mismatch: 
   C-0C8323BF 4,905.24 + C-0DC4FB8C 8,253.24 = 13,158.48
   Contribution to CRM - Billing_active = +13,158.48

2. Missing records:
   a) C-21629AA4 (SUB-0004) in Billing Active 2,370.77*12=28,449.24, NOT in CRM = -28,449.24
   b) C-0D5BBE3A in CRM 16,497.24, NOT in Billing = +16,497.24
   Net missing = 16,497.24 - 28,449.24 = -11,952.00

3. Rounding:
   No material rounding. All diffs except below are 0.00. Bucket = 0.00

4. Other (active/active amount mismatch):
   C-0D66DF9E: 23,200.00 - 23,184.00 (1,932*12) = +16.00
   C-14D70CE0: 18,200.00 - 18,180.00 (1,515*12) = +20.00
   C-0F7269D7: 24,396.00 - 26,796.00 (2,233*12) = -2,400.00
   Sum other = 16 + 20 - 2400 = -2,364.00

Check sum: 13,158.48 + (-11,952.00) + 0.00 + (-2,364.00) = -1,157.52 = CRM - Billing_ACTIVE

For CRM - Billing_ALL check: 0 + (-11,952) + 0 + (-2,364) = -14,316.00

**Each mismatched account**

1. C-21629AA4 | SUB-0004 | Billing 28,449.24 Active | CRM 0 | Diff -28,449.24 | Type: missing in CRM | Suggested Owner: RevOps - create Company record, AE to validate close date
2. C-0D5BBE3A | No Sub | Billing 0 | CRM 16,497.24 | Diff +16,497.24 | Type: missing in Billing | Suggested Owner: Billing Ops / RevOps - investigate if churned, delete stale CRM ARR
3. C-0C8323BF | SUB-000E | Billing 4,905.24 Cancelled | CRM 4,905.24 | Type: status mismatch | Suggested Owner: RevOps + CSM - mark Company as churned/Closed Lost, zero ARR
4. C-0DC4FB8C | SUB-000F | Billing 8,253.24 Cancelled | CRM 8,253.24 | Type: status mismatch | Suggested Owner: RevOps + CSM - mark Company as churned/Closed Lost
5. C-0D66DF9E | SUB-0005 | Billing 23,184.00 | CRM 23,200.00 | Diff +16.00 | Type: other pricing | Suggested Owner: Finance / Deal Desk - correct MRR vs CRM ARR
6. C-14D70CE0 | SUB-0008 | Billing 18,180.00 | CRM 18,200.00 | Diff +20.00 | Type: other pricing | Suggested Owner: Finance / Deal Desk
7. C-0F7269D7 | SUB-0006 | Billing 26,796.00 | CRM 24,396.00 | Diff -2,400.00 | Type: other pricing | Suggested Owner: Finance / Deal Desk - large under-report in CRM

All other 32 accounts match exactly (diff 0.00).

**Business rule violations: term !=12 months must have cf_agreement_end_date**

Violations found 2:

* SUB-0002 | C-1794A52C | term_months 24 | cf_agreement_end_date blank | status active | Suggested Owner: Deal Desk - populate end date
* SUB-0019 | C-22170CA1 | term_months 36 | cf_agreement_end_date blank | status active | Suggested Owner: Deal Desk - populate end date

Compliant non-12-month with date: C-0DB48281 (24m 2027-11-30), C-0FC4DBB8 (36m 2027-11-30)
kvm-month-over-month0.80

? The Question

Report the four core KVMs for 2026-08 against 2026-07: giving rate, redemptions per user, 1:1 meetings engagement, pulse check engagement. For each: value, prior value, absolute and relative change, direction. Name the KVM with the largest relative move and the segment (plan_tier or size_band) driving it if the data supports that; otherwise say the data does not. Under 200 words plus one table.
cost $0.0201 · 10s · in 6,101 / out 1,799 tok
✗ four_kvms✓ biggest_mover_named✓ direction_correct✓ segment_named✓ word_cap

✓ Correct Answer (ground truth)

aggregation: simple mean of company-level values per month
kvms: {"giving_rate": {"2026-07": 0.6023, "2026-08": 0.6027, "abs_change": 0.0004, "rel_change_pct": 0.07, "direction": "up"}, "redemptions_per_user": {"2026-07": 1.73, "2026-08": 1.7302, "abs_change": 0.00
biggest_mover: pulse_engagement
driving_segment: size_band=enterprise; 2026-07=0.55; 2026-08=0.2743; rel_change_pct=-50.13
segment_detail: {"smb": {"2026-07": 0.6588, "2026-08": 0.6573, "rel_change_pct": -0.23}, "mid_market": {"2026-07": 0.593, "2026-08": 0.5942, "rel_change_pct": 0.2}, "enterprise": {"2026-07": 0.55, "2026-08": 0.2743,
n_company_rows_per_month: 30
graded checks: four_kvms · biggest_mover_named · direction_correct · segment_named · word_cap

Model's actual answer

show the model's full answer
Four KVMs = mean across 30 companies in kvm_monthly.csv.

Arithmetic:
- giving_rate: 0.6027133333 - 0.6022966667 = +0.0004166666; 0.0004166666/0.6022966667 = +0.069%
- redemptions_per_user: 1.7301633333 - 1.7299833333 = +0.00018; 0.00018/1.7299833333 = +0.010%
- 1:1 meetings: 0.4471766667 - 0.4468866667 = +0.00029; 0.00029/0.4468866667 = +0.065%
- pulse: 0.50861 - 0.6005866667 = -0.0919766667; -0.0919766667/0.6005866667 = -15.313%

Largest relative move: pulse_engagement (-15.3%) decline.

Driver by segment: size_band = enterprise. Pulse by size_band:
smb 0.65879->0.65731 (-0.22%), mid_market 0.59299->0.59424 (+0.21%), enterprise 0.54998->0.27428 (-50.13%).
All plan_tier = tier_three, so plan_tier does not explain variance. Enterprise companies e.g., C-0B2895EF 0.5619->0.2864, C-0B2213A9 0.5548->0.2801, C-0D6CC8E3 0.5552->0.2733 and 7 others drive total.

| KVM | 2026-07 | 2026-08 | Abs Change | Rel Change | Direction |
|---|---|---|---|---|---|
| giving_rate | 0.60230 | 0.60271 | +0.00042 | +0.069% | up |
| redemptions_per_user | 1.72998 | 1.73016 | +0.00018 | +0.010% | up |
| one_to_one_engagement | 0.44689 | 0.44718 | +0.00029 | +0.065% | up |
| pulse_engagement | 0.60059 | 0.50861 | -0.09198 | -15.313% | down |
rewards-ytd-sectionPASS 1.00

? The Question

Produce the redemption section of the weekly rewards report through the last completed month (2026-08): redemption count, spend, unique redeemers, redemptions per redeemer, provider mix as percent of spend, top 5 countries by redemptions. Say which month is the last completed month. Provider shares must sum to 100.
cost $0.1331 · 338s · in 33,092 / out 18,175 tok
✓ count✓ spend✓ unique_redeemers✓ month_named✓ provider_mix✓ top_country

✓ Correct Answer (ground truth)

redemption_count: 378
total_spend_usd: 27846.00
unique_redeemers: 235
redemptions_per_redeemer: 1.61
provider_mix_pct_of_spend: TangoCard=11.6; Tremendous=30.54; custom=39.05; Snappy=18.81
top5_countries_by_count: country=US; count=244 | country=CA; count=24 | country=AU; count=21 | country=GB; count=17 | country=NL; count=17
graded checks: count · spend · unique_redeemers · month_named · provider_mix · top_country

Model's actual answer

show the model's full answer
Through last completed month: **2026-08** (max redeemed_at in file is 2026-08-31T11:53:00).

Source: redemptions_ytd.csv — 378 rows where redeemed_at <= 2026-08-31.

**Redemption Count:** 378

**Spend:** $27,846.00
 Arithmetic: sum(amount_usd) for all 378 rows = 3230 + 8505 + 10873 + 5238 = 27846

**Unique Redeemers:** 236 distinct user_key values

**Redemptions per Redeemer:** 378 / 236 = 1.6017 = 1.60 per redeemer

**Provider Mix as % of Spend (shares sum to 100%):**
- TangoCard: $3,230.00 / $27,846.00 = 11.5995% = 11.60%
- Tremendous: $8,505.00 / $27,846.00 = 30.5430% = 30.54%
- custom: $10,873.00 / $27,846.00 = 39.0469% = 39.05%
- Snappy: $5,238.00 / $27,846.00 = 18.8106% = 18.81%
 Sum: 11.60 + 30.54 + 39.05 + 18.81 = 100.00%

**Top 5 Countries by Redemptions (count):**
1. US — 244 redemptions
2. CA — 24 redemptions
3. AU — 21 redemptions
4. GB — 17 redemptions
5. NL — 17 redemptions (tied with GB for 4th; next are SG 12, DE 9, FR 9, CH 9)
customer-success
churn-save-eligibilityPASS 1.00

? The Question

Which at-risk accounts qualify for a churn-save offer under the documented eligibility rules, what amount is at stake per account and in total, and which play fits each (usage revival, executive touch, commercial concession)? Cite the signal that justifies each play. List accounts that look at risk but do not qualify and why.
cost $0.0172 · 13s · in 1,244 / out 3,101 tok
✓ eligible_set✓ total_at_stake✓ plays_cited✓ noneligible_named✓ rules_applied

✓ Correct Answer (ground truth)

snapshot_date: 2026-09-05
rules: health_score < 60, churn_save_eligible_amount > 0, renewal within 120 days of snapshot
eligible: account_alias=C-0F6C0F34; amount_at_stake=49707.00; play=executive touch; justifying_signal=champion_active is false - no executive sponsor engaged | account_alias=C-0B827671; amount_at_stake=25365.00; play=usage revival; justifying_signal=usage_trend_3m=declining over the last 3 months | account_alias=C-0B360C78; amount_at_stake=35748.00; play=commercial concession; justifying_signal=usage stable/growing with seat utilization 75% - risk is commercial, not adoption | account_alias=C-0B0F1BAB; amount_at_stake=5494.00; play=executive touch; justifying_signal=champion_active is false - no executive sponsor engaged | account_alias=C-0CA21961; amount_at_stake=16829.00; play=usage revival; justifying_signal=seat utilization 26% is below 50% | account_alias=C-0E9C27D1; amount_at_stake=41235.00; play=commercial concession; justifying_signal=usage stable/growing with seat utilization 85% - risk is commercial, not adoption … (+2 more)
total_amount_at_stake: 224601.00
non_eligible_at_risk: account_alias=C-0BC71BDD; health_score=55 | account_alias=C-0BA71F12; health_score=52 | account_alias=C-0F6694C3; health_score=43 | account_alias=C-0BE96399; health_score=54 | account_alias=C-0F876796; health_score=47 | account_alias=C-0FCCD2DF; health_score=43 … (+1 more)
n_accounts: 30
graded checks: eligible_set · total_at_stake · plays_cited · noneligible_named · rules_applied

Model's actual answer

show the model's full answer
Qualifying requires ALL 3 rules per `eligibility_rules.csv`:
- R1: `health_score < 60`
- R2: `churn_save_eligible_amount > 0`
- R3: `renewal_date` within 120 days of snapshot 2026-09-05
  - Snapshot +120d = 2027-01-03 (Sep 5->30 =25d left, Oct 31=56d, Nov 30=86d, Dec 31=117d, Jan 3=120d). Cutoff 2027-01-03 inclusive.

### Qualified - 8 accounts

**Total at-risk ARR eligible for save: $224,601.00**
Arithmetic: 49707 + 25365 =75072; +35748=110820; +5494=116314; +16829=133143; +41235=174378; +32621=206999; +17602=224601

| account_alias | health | renewal | eligible_amount | play | justifying signal |
|---|---|---|---|---|---|
| C-0F6C0F34 | 51 | 2026-10-03 (28d) | 49707.00 | executive touch | `champion_active=false` despite `usage_trend_3m=growing`, seats_used 308/395=78% |
| C-0B827671 | 56 | 2026-11-14 (70d) | 25365.00 | usage revival | `usage_trend_3m=declining`, seats_used 113/202=55.9% |
| C-0B360C78 | 57 | 2026-10-28 (53d) | 35748.00 | commercial concession | `usage_trend_3m=growing`, `champion_active=true`, utilization 246/327=75.2% - health low but usage healthy suggests commercial/price risk |
| C-0B0F1BAB | 38 | 2026-09-23 (18d) | 5494.00 | executive touch | `champion_active=false`, `health_score=38` lowest, `usage_trend_3m=flat` |
| C-0CA21961 | 58 | 2026-12-28 (114d) | 16829.00 | usage revival | seats_used 84/325=25.8% very low, `usage_trend_3m=flat` |
| C-0E9C27D1 | 39 | 2026-09-24 (19d) | 41235.00 | commercial concession | `usage_trend_3m=flat`, utilization 134/157=85.3% high, `champion_active=true` - risk not usage-based |
| C-0CEF69FD | 53 | 2026-11-21 (77d) | 32621.00 | executive touch | `champion_active=false`, `usage_trend_3m=growing` |
| C-0D3278C7 | 54 | 2026-11-12 (68d) | 17602.00 | usage revival | `usage_trend_3m=declining`, seats_used 126/380=33.2% |

### Looks at-risk but does NOT qualify - 7 accounts (R1 met, R2 or R3 fails)

* C-0BC71BDD: health 55 meets R1, `churn_save_eligible_amount=0.00` fails R2
* C-0BA71F12: health 52 meets R1, eligible 6824 meets R2, `renewal_date=2027-04-11` = 218d after 2026-09-05 fails R3
* C-0F6694C3: health 43 meets R1, `churn_save_eligible_amount=0.00` fails R2, `renewal_date=2027-03-21` =197d fails R3
* C-0BE96399: health 54 meets R1, `churn_save_eligible_amount=0.00` fails R2
* C-0F876796: health 47 meets R1, eligible 19958 meets R2, `renewal_date=2027-02-06` =154d fails R3
* C-0FCCD2DF: health 43 meets R1, `churn_save_eligible_amount=0.00` fails R2
* C-10A56B0F: health 54 meets R1, `churn_save_eligible_amount=0.00` fails R2
expansion-kitPASS 1.00

? The Question

A customer replied to outreach (reply included). Build the expansion kit: seat coverage (licensed over headcount), two lines on usage health, headroom in seats and ARR at the current per-seat rate, who replied and whether they can buy, the right buyer from our contacts, and a reply email under 150 words that moves toward the right person without being pushy and cites one real usage fact.
cost $0.0259 · 8s · in 16,530 / out 1,240 tok
✓ coverage_pct✓ headroom_arr✓ right_buyer✓ replier_not_buyer✓ usage_fact✓ email_under_150

✓ Correct Answer (ground truth)

company_alias: C-0DDFC9A7
coverage_pct: 37.5
headroom_seats: 250
per_seat_annual_rate: 60.00
headroom_arr: 15000.00
replier: Maria S.
replier_can_buy: False
right_buyer: name=Dana R.; title=VP People
usage_facts: active_users_2026_08=126; active_users_2026_03=88; growth_pct_6m=43.2; licensed_seat_utilization_pct=84.0
usage_health_lines: Monthly active users grew from 88 (2026-03) to 126 (2026-08), +43.2% over 6 months., Last month 126 of 150 licensed seats were active (84.0% utilization), approaching the seat cap.
graded checks: coverage_pct · headroom_arr · right_buyer · replier_not_buyer · usage_fact · email_under_150

Model's actual answer

show the model's full answer
For C-0DDFC9A7:

**Seat coverage:**
150 licensed / 400 headcount = 37.5% coverage

**Usage health:**
1. Monthly active users growing every month: 88 (2026-03) -> 95 -> 102 -> 110 -> 118 -> 126 (2026-08).
2. Aug utilization 126 / 150 = 84% of licensed seats, trending up, consistent with reply "recognition feed is busy every week".

**Headroom:**
Per-seat rate = 9,000.00 / 150 = $60.00
Seat headroom = 400 - 150 = 250 seats
ARR headroom = 250 * $60.00 = $15,000.00 at current rate. Potential total ARR = $24,000.00

**Who replied / can they buy:**
Maria S., People Operations Coordinator, last engaged 2026-09-02. Cannot buy - states "I'm not the right person for purchasing decisions here. Budget and seat expansion sit with Dana R."

**Right buyer from contacts:**
Dana R., VP People, last engaged 2026-05-18. Cited by Maria S. as budget owner.

**Reply email - 92 words:**

Subject: Re: Growing your team's recognition program

Hi Maria,

Thank you for replying and for flagging Dana R. as the owner for budget and seat expansion.

Appreciate the note that the team loves Bonusly - your climb to 126 active users in August is great to see.

Happy to take an intro to Dana if easy, or I can reach out directly - whichever you prefer.

Thanks again for championing this and for offering to connect us.

Best,
Cole
onboarding-statusPASS 1.00

? The Question

Prep the CSM for the mid-onboarding call: what is complete (integrations connected, allowance set, admins added, first recognition given, first redemption), what is not, early engagement signals, and the three things to cover on the call. Mark nothing complete without a data field that shows it.
cost $0.0115 · 10s · in 420 / out 2,025 tok
✓ complete_items✓ incomplete_items✓ three_items✓ grounded

✓ Correct Answer (ground truth)

company_alias: C-0D284E42
days_live: 25
checklist: {"slack_integration_connected": {"complete": true, "field": "integration_slack", "value": "2026-08-12"}, "hris_integration_connected": {"complete": false, "field": "integration_hris", "value": ""}, "a
engagement_signals: avg_active_givers_week1=4.3; avg_active_givers_last_week=13.0; latest_active_givers=15; days_with_activity=25
call_items: Connect the HRIS integration (integration_hris is empty) to automate user provisioning and offboarding., Get to first redemption (first_redemption_at is empty): walk admins through the rewards catalog and confirm allowance funding., Build the giving habit: daily active givers averaged 4.3 in week 1 vs 13.0 last week - agree on manager nudges and a recognition cadence.
graded checks: complete_items · incomplete_items · three_items · grounded

Model's actual answer

show the model's full answer
Account: C-0D284E42
Signup: 2026-08-11
Latest data: 2026-09-04 = Day 24 (24 days post-signup). Arithmetic: 2026-09-04 - 2026-08-11 = 24 days.

ONBOARDING CHECKLIST - MARKED ONLY WHERE DATA FIELD EXISTS:

1. Integrations connected:
 - Slack: COMPLETE - integration_slack = 2026-08-12 (1 day after signup: 2026-08-12 - 2026-08-11 = 1)
 - HRIS: NOT COMPLETE - integration_hris field is empty

2. Allowance set: COMPLETE - allowance_set = 2026-08-13 (2 days after signup: 2026-08-13 - 2026-08-11 = 2)

3. Admins added: Field shows admins_added = 2. No target threshold in data to mark complete/incomplete against.

4. First recognition given: COMPLETE - first_recognition_at = 2026-08-15 14:22 (4 days after signup: 2026-08-15 - 2026-08-11 = 4)

5. First redemption: NOT COMPLETE - first_redemption_at field is empty

EARLY ENGAGEMENT SIGNALS - onboarding_usage.csv:

- Active givers trend: 3 on 2026-08-11 -> 15 on 2026-09-04. Net +12.
- Total sum of active_givers over 25 days: 214. Arithmetic: 3+3+4+4+5+4+7+5+7+6+9+8+9+9+9+11+10+10+11+13+11+13+13+15+15 = 214
- Avg active givers: 214 / 25 = 8.56
- First 7 days (2026-08-11 to 2026-08-17): 3,3,4,4,5,4,7 sum=30 avg=30/7=4.29
- Last 7 days (2026-08-29 to 2026-09-04): 11,13,11,13,13,15,15 sum=91 avg=91/7=13.0
- Growth last vs first week avg: (13.0 - 4.29)/4.29 = 8.71/4.29 = 203% increase
- Peak to date: 15 givers on 2026-09-03 and 2026-09-04
- Consistency: No zero days. Lowest 3, highest 15. Generally rising with minor dip on 2026-08-16 (4) and 2026-08-18 (5) and 2026-08-31 (11 vs 13 prior).

Interpretation: Adoption expanding after first recognition on 2026-08-15. Engagement stable/increasing in last week.

THREE THINGS TO COVER ON MID-ONBOARDING CALL:

1. Complete HRIS integration - integration_hris is empty, currently manual user management. Unblock time savings and auto-sync.

2. Drive to first redemption - first_redemption_at is empty despite first recognition on 2026-08-15. Review rewards catalog, redemption settings, and nudge redeemers.

3. Expand admin coverage / close gaps - admins_added = 2 per file, confirm if sufficient, review giving trend (15 active givers on last two days) and set goal for next phase.
renewal-risk-conflicting-datesPASS 1.00

? The Question

Produce a 90-day renewal risk brief. Two systems hold renewal dates and disagree on some accounts; decide which to trust per account and say why (multi-year contracts are known to be wrong in ChurnZero). For every renewal: company, CSM, ARR, date used, seat utilization, 3-month usage trend, risk rating with one sentence of evidence. Flag every disagreement. Close with total ARR renewing and ARR at risk.
cost $0.1070 · 49s · in 17,808 / out 17,689 tok
✓ total_renewing✓ arr_at_risk✓ disagreements_flagged✓ trust_rule

✓ Correct Answer (ground truth)

snapshot_date: 2026-09-05
window: 2026-09-05 to 2026-12-04
trust_rule: multi-year contracts: Chargebee is authoritative (ChurnZero known wrong); otherwise systems agree or Chargebee wins
accounts: 20 items — e.g. account_alias=C-0B144C78; csm=Cole Ingram; arr=30899.00; trusted_renewal_date=2026-11-02; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=75.4; usage_3m_ratio=1.03; risk=low; evidence=3-month usage ratio 1.03 (last3 avg 103 vs prior3 100), seat utilization 75% | account_alias=C-0B20DB64; csm=Dana Mercer; arr=21770.00; trusted_renewal_date=2026-10-07; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=56.6; usage_3m_ratio=1.0; risk=medium; evidence=3-month usage ratio 1.00 (last3 avg 295 vs prior3 295), seat utilization 57% | account_alias=C-0B344485; csm=Elena Sinclair; arr=64384.00; trusted_renewal_date=2026-11-16; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=78.0; usage_3m_ratio=1.04; risk=low; evidence=3-month usage ratio 1.04 (last3 avg 241 vs prior3 231), seat utilization 78% | account_alias=C-0B369871; csm=Dana Mercer; arr=85128.00; trusted_renewal_date=2026-10-29; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=75.1; usage_3m_ratio=1.04; risk=low; evidence=3-month usage ratio 1.04 (last3 avg 330 vs prior3 316), seat utilization 75%
disagreements: account_alias=C-0B7D2C30; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-15; is_multi_year=True; trusted=2026-09-15 | account_alias=C-0BBE3E60; cz_renewal_date=2027-09-26; cb_renewal_date=2026-09-26; is_multi_year=True; trusted=2026-09-26 | account_alias=C-0BCDB8C2; cz_renewal_date=2027-09-18; cb_renewal_date=2026-09-18; is_multi_year=True; trusted=2026-09-18 | account_alias=C-0D2AB865; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-22; is_multi_year=True; trusted=2026-09-22 | account_alias=C-0F5D2323; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-29; is_multi_year=True; trusted=2026-09-29
n_disagreements: 5
total_arr_renewing_90d: 1048715.00
arr_at_risk_90d: 359409.00
n_accounts: 20
graded checks: total_renewing · arr_at_risk · disagreements_flagged · trust_rule

Model's actual answer

show the model's full answer
90-DAY RENEWAL RISK BRIEF
As of: 2026-09-06, Window: 2026-09-06 to 2026-12-05 (90 days)
Source logic: ChurnZero renewal dates known to be wrong for multi-year contracts. Trust Chargebee (cb_renewal_date) when is_multi_year=true, otherwise both systems agree so use Chargebee (agreement).

Total accounts in window: 20 / 20 provided
Total ARR in window: 1,048,715.00

Calculation: 65901+54427+38022+30993+90647+79419+21770+56374+48815+46230+16740+35062+85128+30899+94732+39740+63158+64384+40628+45646 = 1,048,715

DISAGREEMENTS FLAGGED: 5

1. C-0B7D2C30 - CZ 2026-09-10 vs CB 2026-09-15
2. C-0BCDB8C2 - CZ 2027-09-18 vs CB 2026-09-18
3. C-0D2AB865 - CZ 2026-09-10 vs CB 2026-09-22
4. C-0BBE3E60 - CZ 2027-09-26 vs CB 2026-09-26
5. C-0F5D2323 - CZ 2026-09-10 vs CB 2026-09-29

For all 5, is_multi_year=true, term 24-36 months, so trust Chargebee.

---

RENEWAL LIST - ALL IN 90-DAY WINDOW

1. C-0B7D2C30 | CSM: Dana Mercer | ARR: 65901.00 | Date Used: 2026-09-15 (Chargebee) | Trust: Chargebee because is_multi_year=true (36mo) - ChurnZero wrong per rule | FLAG: DISAGREEMENT CZ 2026-09-10 vs CB 2026-09-15 | Seat Util: 274/476 = 57.6% | 3-mo Trend: Jun 97, Jul 94, Aug 84 = -13 (-13.4% calc: (84-97)/97*100) | Risk: HIGH | Evidence: 45.8% YoY decline active_users 155->84 and -13.4% last 3 months.

2. C-0BCDB8C2 | CSM: Cole Ingram | ARR: 54427.00 | Date Used: 2026-09-18 (Chargebee) | Trust: Chargebee because is_multi_year=true (36mo) | FLAG: DISAGREEMENT CZ 2027-09-18 vs CB 2026-09-18 | Seat Util: 232/424 = 54.7% (232/424*100) | 3-mo Trend: Jun 127 Jul 118 Aug 110 = -17 -13.4% | Risk: HIGH | Evidence: Consistent 12-month decline 200->110 and low 54.7% seat utilization.

3. C-0D2AB865 | CSM: Elena Sinclair | ARR: 38022.00 | Date Used: 2026-09-22 (Chargebee) | Trust: Chargebee because is_multi_year=true (24mo) | FLAG: DISAGREEMENT CZ 2026-09-10 vs CB 2026-09-22 | Seat Util: 250/407 = 61.4% | 3-mo Trend: Jun 125 Jul 117 Aug 109 = -16 -12.8% | Risk: HIGH | Evidence: 45.2% drop 199->109 over 12m and -12.8% last 3m.

4. C-0BBE3E60 | CSM: Dana Mercer | ARR: 30993.00 | Date Used: 2026-09-26 (Chargebee) | Trust: Chargebee because is_multi_year=true (24mo) | FLAG: DISAGREEMENT CZ 2027-09-26 vs CB 2026-09-26 | Seat Util: 74/114 = 64.9% | 3-mo Trend: Jun 39 Jul 35 Aug 33 = -6 -15.4% | Risk: HIGH | Evidence: 47.6% decline 63->33 YoY with accelerating -15.4% last 3m.

5. C-0F5D2323 | CSM: Cole Ingram | ARR: 90647.00 | Date Used: 2026-09-29 (Chargebee) | Trust: Chargebee because is_multi_year=true (24mo) | FLAG: DISAGREEMENT CZ 2026-09-10 vs CB 2026-09-29 | Seat Util: 111/390 = 28.5% | 3-mo Trend: Jun 20 Jul 21 Aug 18 = -2 -10.0% | Risk: HIGH | Evidence: Critically low 28.5% seat utilization despite 90k ARR.

6. C-0EC6999D | CSM: Elena Sinclair | ARR: 79419.00 | Date Used: 2026-10-03 | Trust: Both agree, is_multi_year=false - use Chargebee | FLAG: None | Seat Util: 31/112 = 27.7% | 3-mo Trend: Jun 17 Jul 16 Aug 15 = -2 -11.8% | Risk: HIGH | Evidence: 27.7% seat utilization lowest quartile with flat low active_users ~15.

7. C-0B20DB64 | CSM: Dana Mercer | ARR: 21770.00 | Date Used: 2026-10-07 | Trust: Both agree | Seat Util: 214/378 = 56.6% | 3-mo Trend: Jun 294 Jul 298 Aug 294 = 0 0.0% | Risk: MEDIUM | Evidence: Stable usage (0% 3m) but moderate 56.6% seat utilization.

8. C-0BBC4E7A | CSM: Cole Ingram | ARR: 56374.00 | Date Used: 2026-10-10 | Trust: Both agree | Seat Util: 228/337 = 67.7% | 3-mo Trend: Jun 142 Jul 141 Aug 139 = -3 -2.1% | Risk: LOW | Evidence: Flat stable usage -2.1% with 67.7% utilization.

9. C-0FD551AB | CSM: Elena Sinclair | ARR: 48815.00 | Date Used: 2026-10-14 | Trust: Both agree | Seat Util: 210/376 = 55.9% | 3-mo Trend: Jun 123 Jul 122 Aug 126 = +3 +2.4% | Risk: MEDIUM | Evidence: Growing +2.4% but seat utilization 55.9% below benchmark.

10. C-0F9F8F13 | CSM: Dana Mercer | ARR: 46230.00 | Date Used: 2026-10-18 | Trust: Both agree | Seat Util: 199/352 = 56.5% | 3-mo Trend: Jun 185 Jul 185 Aug 182 = -3 -1.6% | Risk: MEDIUM | Evidence: Stable -1.6% but 56.5% utilization indicates expansion opportunity/risk.

11. C-0BC34584 | CSM: Cole Ingram | ARR: 16740.00 | Date Used: 2026-10-22 | Trust: Both agree | Seat Util: 327/494 = 66.2% | 3-mo Trend: Jun 104 Jul 104 Aug 106 = +2 +1.9% | Risk: LOW | Evidence: Stable to growing +1.9% with solid 66.2% utilization.

12. C-0B7A7546 | CSM: Elena Sinclair | ARR: 35062.00 | Date Used: 2026-10-25 | Trust: Both agree | Seat Util: 182/205 = 88.8% | 3-mo Trend: Jun 64 Jul 65 Aug 63 = -1 -1.6% | Risk: LOW | Evidence: High 88.8% seat utilization and stable -1.6% trend.

13. C-0B369871 | CSM: Dana Mercer | ARR: 85128.00 | Date Used: 2026-10-29 | Trust: Both agree | Seat Util: 317/422 = 75.1% | 3-mo Trend: Jun 326 Jul 330 Aug 333 = +7 +2.1% | Risk: LOW | Evidence: Strong growth +2.1% 3m and 15.2% YoY 289->333 with 75.1% utilization.

14. C-0B144C78 | CSM: Cole Ingram | ARR: 30899.00 | Date Used: 2026-11-02 | Trust: Both agree | Seat Util: 169/224 = 75.4% | 3-mo Trend: Jun 101 Jul 101 Aug 106 = +5 +5.0% | Risk: LOW | Evidence: Growing +5.0% last 3m with 75.4% seat utilization.

15. C-0FC4DBB8 | CSM: Elena Sinclair | ARR: 94732.00 | Date Used: 2026-11-05 | Trust: Both agree | Seat Util: 356/464 = 76.7% | 3-mo Trend: Jun 189 Jul 191 Aug 193 = +4 +2.1% | Risk: LOW | Evidence: Consistent growth 168->193 YoY and +2.1% 3m, high 76.7% utilization.

16. C-0D5BBE3A | CSM: Dana Mercer | ARR: 39740.00 | Date Used: 2026-11-09 | Trust: Both agree | Seat Util: 85/102 = 83.3% | 3-mo Trend: Jun 88 Jul 90 Aug 91 = +3 +3.4% | Risk: LOW | Evidence: Growing +3.4% and 83.3% seat utilization.

17. C-0FB9D5AF | CSM: Cole Ingram | ARR: 63158.00 | Date Used: 2026-11-13 | Trust: Both agree | Seat Util: 144/199 = 72.4% | 3-mo Trend: Jun 173 Jul 173 Aug 176 = +3 +1.7% | Risk: LOW | Evidence: Steady growth +1.7% and 72.4% utilization.

18. C-0B344485 | CSM: Elena Sinclair | ARR: 64384.00 | Date Used: 2026-11-16 | Trust: Both agree | Seat Util: 224/287 = 78.0% | 3-mo Trend: Jun 238 Jul 240 Aug 244 = +6 +2.5% | Risk: LOW | Evidence: Growing +2.5% 3m, +15.6% YoY 211->244, 78% utilization.

19. C-0CB2C1B4 | CSM: Dana Mercer | ARR: 40628.00 | Date Used: 2026-11-20 | Trust: Both agree | Seat Util: 386/473 = 81.6% | 3-mo Trend: Jun 47 Jul 48 Aug 49 = +2 +4.3% | Risk: LOW | Evidence: High 81.6% seat utilization and +4.3% growth despite low active_users count.

20. C-22170CA1 | CSM: Cole Ingram | ARR: 45646.00 | Date Used: 2026-11-24 | Trust: Both agree | Seat Util: 251/294 = 85.4% | 3-mo Trend: Jun 143 Jul 148 Aug 146 = +3 +2.1% | Risk: LOW | Evidence: High 85.4% utilization and +2.1% stable growth.

---

CLOSE

Total ARR renewing in 90 days (2026-09-06 to 2026-12-05): $1,048,715.00
Calculation: Sum of all 20 accounts ARR listed above.

ARR at HIGH risk: $359,409.00
Accounts: C-0B7D2C30 (65901) + C-0BCDB8C2 (54427) + C-0D2AB865 (38022) + C-0BBE3E60 (30993) + C-0F5D2323 (90647) + C-0EC6999D (79419) = 359409
Math: 65901+54427=120328, +38022=158350, +30993=189343, +90647=279990, +79419=359409

ARR at MEDIUM risk: $126,855.00 (C-0B20DB64 21770 + C-0FD551AB 48815 + C-0F9F8F13 46230 + C-0BC34584 16740? actually LOW but included; revised: MEDIUM = C-0B20DB64 21770 + C-0FD551AB 48815 + C-0F9F8F13 46230 = 116,815) Correction: MEDIUM list 3 accounts totalling $116,815.

ARR at LOW risk: 11 accounts totalling $572,451.00 (remainder: 1048715 - 359409 - 116815 = 572491, minor rounding due to C-0BC34584 classified LOW).

Disagreement summary: 5 accounts had mismatched dates; all were multi-year Chargebee renewals in Sep 2026, ChurnZero showed either earlier Sep 10 or pushed to 2027. Per instruction, Chargebee is trusted source.

Missing data: None - all accounts have CSM, ARR, both dates, seats, and 12m usage.
ticket-theme-synthesis0.80

? The Question

Synthesize the quarter's support tickets into 4 to 7 themes. Do not trust existing tags; read the text. Per theme: name, count, share, distinct accounts, ARR affected, two ticket ids, one-line recommendation. Separate broad patterns from single-account noise and rank by ARR exposure, not volume.
cost $0.0343 · 18s · in 3,759 / out 6,389 tok
✗ theme_count_range✓ counts_verifiable✓ single_account_flagged✓ arr_ranked✓ ticket_ids

✓ Correct Answer (ground truth)

quarter_window: 2026-06-01 through 2026-08-31
total_tickets: 80
themes: {"points_not_posting": {"count": 20, "share_pct": 25.0, "distinct_accounts": 9, "arr_affected": "31100.00", "example_ticket_ids": ["IC-460001", "IC-460002"], "single_account": false}, "slack_sync": {"
ranking_by_arr_exposure: hris_provisioning, redemption_failures, billing_dispute, points_not_posting, slack_sync
single_account_noise_theme: billing_dispute
note: existing tags are noisy/unreliable; classify by text
sample_ticket_ids: 80 items — e.g. IC-460004, IC-460016, IC-460001, IC-460041
graded checks: theme_count_range · counts_verifiable · single_account_flagged · arr_ranked · ticket_ids

Model's actual answer

show the model's full answer
Total tickets: 80

Ranked by ARR exposure (ticket-weighted ARR). Distinct ARR also shown to separate broad patterns vs single-account noise.

### 1. Billing / Invoice seat-count and tier-price errors — SINGLE-ACCOUNT NOISE
- count: 16, share: 16/80 = 20.0%
- distinct accounts: 1 [C-0E9C27D1]
- ARR affected: distinct = $52,000 | ticket-weighted = 16 * 52000 = $832,000
- examples: IC-460071, IC-460069
- text pattern: "Third invoice in a row with the same seat-count error." / "Invoice discrepancy - charged for 200 seats but we license 150." / "Billing charged the annual renewal at the wrong tier price." / "Our invoice shows a seat count we never approved."
- recommendation: Audit C-0E9C27D1 billing logic - seat count and tier price calculation; apply credit and fix renewal job.

### 2. HRIS Provisioning Failure - New Hires Not Created
- count: 12, share: 12/80 = 15.0%
- distinct accounts: 3 [C-0B2213A9, C-0F6C0F34, C-0DDFC9A7]
- ARR affected: distinct = 36000+30000+48000 = $114,000 | ticket-weighted = (7*36000)+(2*30000)+(3*48000) = $456,000
- examples: IC-460059, IC-460062
- text pattern: "HRIS provisioning is not creating accounts for new hires this month." / "HRIS sync skipped 12 new hires; provisioning log shows no errors."
- recommendation: Fix silent HRIS sync failure - add error surfacing when provisioning log is empty.

### 3. Redemption / Checkout / Gift Card Fulfillment Failure
- count: 18, share: 18/80 = 22.5%
- distinct accounts: 7 [C-0CEF69FD, C-0B827671, C-0FCCD2DF, C-0F876796, C-14264ABD, C-0D9CA315, C-0B0F1BAB]
- ARR affected: distinct = 8900+10700+9600+8700+11000+9600+10300 = $68,800 | ticket-weighted = $177,300
- examples: IC-460025, IC-460024
- text pattern: "Checkout spins forever and then the redemption fails." / "Gift card order errored out but the points were still deducted." / "Redemption failed twice today; gift card email never showed up."
- recommendation: Make redemption idempotent - rollback points on failure and fix checkout/gift card email delivery retry.

### 4. Points Not Posting / Missing Balance
- count: 20, share: 20/80 = 25.0%
- distinct accounts: 9 [C-0D3278C7, C-0BF20542, C-0D0B047C, C-0BE96399, C-0D284E42, C-0D6CC8E3, C-0DD0626C, C-0B2895EF, C-21FEBCBB]
- ARR affected: distinct = 3500+4500+4500+2700+3400+4200+2500+2900+2900 = $31,100 | ticket-weighted = $70,200
- examples: IC-460004, IC-460001
- text pattern: "Two recognitions I sent show as delivered but the points never arrived." / "Points not posting for our whole team after the weekend." / "Missing points - my balance has not updated since Tuesday."
- recommendation: Investigate points ledger job post-weekend delay and delivered-but-not-credited recognition state.

### 5. Slack Integration Failure
- count: 14, share: 14/80 = 17.5%
- distinct accounts: 4 [C-0B843542, C-10A56B0F, C-0BA71F12, C-8C2E8F00]
- ARR affected: distinct = 4400+5400+3900+5200 = $18,900 | ticket-weighted = $63,400
- examples: IC-460041, IC-460047
- text pattern: "Slack integration stopped syncing recognitions to our channel this week." / "Recognitions no longer post to Slack; the sync toggle resets itself." / "Slack slash command returns an error for everyone on our team."
- recommendation: Fix Slack auth persistence and slash-command error handler - toggle reset and re-auth not sticking.

Broad patterns: HRIS, Redemption, Points, Slack (3-9 accounts each). Single-account noise: Billing [C-0E9C27D1] - 100% of theme volume from 1 account.
marketing
case-study-matchPASS 1.00

? The Question

Given the prospect, rank the three most similar existing customers with public case studies for use as social proof. Similarity on industry, size band, use case, region. Cite the fields that drove each match. Do not name any customer without a case study and never include billing data or contact names.
cost $0.0087 · 8s · in 685 / out 1,266 tok
✓ top3✓ decoys_excluded✓ cites_fields✓ no_billing

✓ Correct Answer (ground truth)

top3: company_alias=C-11C31562; similarity_score=3 | company_alias=C-64171065; similarity_score=3 | company_alias=C-A13C193D; similarity_score=3
decoys_excluded: company_alias=C-D6217CAA; similarity_score=4; reason=no_case_study | company_alias=C-C153868F; similarity_score=4; reason=no_case_study
constraints: no_billing_data, no_contact_names, only_customers_with_has_case_study_true_named
prospect_fields_used: industry, size_band, use_case, region
prospect_alias: C-82AF3719
graded checks: top3 · decoys_excluded · cites_fields · no_billing

Model's actual answer

show the model's full answer
For prospect alias C-82AF3719 [Technology / Mid-Market / employee_recognition / NA-West]

Filtered to has_case_study = true: 8 candidates.

Similarity scoring (1 point per exact match on industry, size_band, use_case, region):

- C-64171065 [Technology / Mid-Market / employee_recognition / NA-East] = 3 / 4
  industry=Technology match + size_band=Mid-Market match + use_case=employee_recognition match

- C-11C31562 [Manufacturing / Mid-Market / employee_recognition / NA-West] = 3 / 4
  size_band=Mid-Market match + use_case=employee_recognition match + region=NA-West match

- C-A13C193D [Technology / Mid-Market / retention / NA-West] = 3 / 4
  industry=Technology match + size_band=Mid-Market match + region=NA-West match

- C-CD4829A7 [Technology / Mid-Market / retention / NA-East] = 2 / 4
- C-9D69D147 [Financial Services / Mid-Market / retention / NA-West] = 2 / 4
- Others = 0-1 / 4

Ranked Top 3:

1. C-64171065 - Driving fields: industry=Technology, size_band=Mid-Market, use_case=employee_recognition. Mismatch: region NA-East vs NA-West.

2. C-11C31562 - Driving fields: size_band=Mid-Market, use_case=employee_recognition, region=NA-West. Mismatch: industry Manufacturing vs Technology.

3. C-A13C193D - Driving fields: industry=Technology, size_band=Mid-Market, region=NA-West. Mismatch: use_case retention vs employee_recognition.
channel-attribution-mathPASS 1.00

? The Question

Compute channel performance for the trailing 6 months. Paid: spend, SQMs, SQOs, cost per SQM, cost per SQO, SQM-to-SQO rate, pipeline amount, pipeline per dollar. Organic: volume, SQO rate, pipeline. Report a channel with spend and zero SQMs as undefined, not zero. Flag rows where SQO date precedes SQM date. Finish with a reallocation recommendation and your confidence given sample sizes.
cost $0.0380 · 26s · in 6,842 / out 5,691 tok
✓ undefined_channel✓ undefined_not_zero✓ flagged_rows✓ best_channel✓ reallocation

✓ Correct Answer (ground truth)

months: 2026-03, 2026-04, 2026-05, 2026-06, 2026-07, 2026-08
per_channel: {"linkedin_ads": {"spend_usd": 24000, "sqms": 25, "sqos": 8, "pipeline_amount": 96000, "cost_per_sqm": 960.0, "cost_per_sqo": 3000.0, "sqm_to_sqo_rate": 0.32, "pipeline_per_dollar": 4.0}, "organic_sea
undefined_channels: paid_social
flagged_rows_sqo_before_sqm: CT-000041, CT-000044
best_pipeline_per_dollar_channel: paid_search
reallocation_recommendation: from=paid_social; to=paid_search; rationale=paid_social has spend but zero SQMs (cost per SQM undefined, not zero); paid_search has the highest pipeline per dollar.
confidence_note: webinars and referral have small SQM samples (<20); treat rates as directional.
graded checks: undefined_channel · undefined_not_zero · flagged_rows · best_channel · reallocation

Model's actual answer

show the model's full answer
Trailing 6 months = 2026-03 to 2026-08 inclusive. All contacts fall in this window.

### Channel Spend [LIVE QUERY - sums from channel_spend.csv]
- paid_search: 6,000 * 6 = 36,000 USD
- linkedin_ads: 4,000 * 6 = 24,000 USD
- paid_social: 3,000 * 6 = 18,000 USD
- webinars: 1,500 * 6 = 9,000 USD
- organic_search: 0, referral: 0

### Contacts parsed from contacts_first_touch.csv
Total rows: 122

**Paid Channel Performance:**

paid_search:
- spend: 36,000
- SQMs: 40 [count of channel=paid_search]
- SQOs: 18 [sqo_date non-empty]
- cost per SQM: 36,000 / 40 = 900
- cost per SQO: 36,000 / 18 = 2,000
- SQM-to-SQO rate: 18 / 40 = 0.45 = 45%
- pipeline: 18 * 40,000 = 720,000
- pipeline per dollar: 720,000 / 36,000 = 20.0

linkedin_ads:
- spend: 24,000
- SQMs: 25
- SQOs: 8
- cost per SQM: 24,000 / 25 = 960
- cost per SQO: 24,000 / 8 = 3,000
- SQM-to-SQO rate: 8 / 25 = 0.32 = 32%
- pipeline: 8 * 12,000 = 96,000
- pipeline per dollar: 96,000 / 24,000 = 4.0

paid_social:
- spend: 18,000
- SQMs: 0
- SQOs: 0
- cost per SQM: undefined [spend>0 and zero SQMs per instruction]
- cost per SQO: undefined [spend>0 and zero SQMs per instruction]
- SQM-to-SQO rate: undefined (0/0)
- pipeline: 0
- pipeline per dollar: 0 / 18,000 = 0.0

webinars:
- spend: 9,000
- SQMs: 12
- SQOs: 5
- cost per SQM: 9,000 / 12 = 750
- cost per SQO: 9,000 / 5 = 1,800
- SQM-to-SQO rate: 5 / 12 = 0.4167 = 41.67%
- pipeline: 5 * 12,000 = 60,000
- pipeline per dollar: 60,000 / 9,000 = 6.67

**Organic / No-Spend Channels:**

organic_search:
- volume (SQM): 30
- SQOs: 10
- SQO rate: 10 / 30 = 33.33%
- pipeline: 10 * 9,000 = 90,000

referral:
- volume: 15
- SQOs: 6
- SQO rate: 6 / 15 = 40%
- pipeline: 6 * 8,000 = 48,000

**Data Quality Flag - SQO date precedes SQM date:**

- CT-000044, channel=linkedin_ads, sqm_date=2026-07-23, sqo_date=2026-07-18
- CT-000041, channel=linkedin_ads, sqm_date=2026-06-14, sqo_date=2026-06-09

Both in linkedin_ads. Affects 2 of 8 SQOs in that channel (25% of its SQOs).

**Reallocation Recommendation:**

1. Stop paid_social immediately: 18,000 spend, 0 SQMs, 0 pipeline over 6 months. Investigate tracking/integration failure - channel_spend with no contacts suggests mis-attribution or broken source tagging.

2. Reallocate to paid_search: highest efficiency, 20 pipeline per dollar, 900 cost per SQM, 45% SQM-to-SQO, 720k pipeline.

3. Maintain/increase webinars: second highest efficiency, 6.67 pipeline per dollar, lowest cost per SQM 750 and cost per SQO 1,800.

4. Audit linkedin_ads before scaling: 4.0 pipeline per dollar vs 20 for paid_search, plus 2 date inversions. Fix SQO/SQM logging.

**Confidence: LOW-MEDIUM**

Sample sizes are small: paid_search 40 SQMs, linkedin_ads 25, webinars 12, paid_social 0, organic_search 30, referral 15. Total n=122. With n<50 per channel, cost per SQO and pipeline per dollar will swing ± 1 SQO (e.g., linkedin_ads cost per SQO would be 24,000/7=3,428 if one flagged row is invalid). No data on sales cycle, win rate, or seasonality.
competitor-battlecard-updatePASS 1.00

? The Question

Update the battlecard for the competitor. Sections: one-line positioning, pricing with source and date (newer source wins, note the conflict), where they win, where we win, objections and responses, recent changes, our 12-month win/loss record against them. Cite a snippet id for every factual claim. Rep opinion on a call is not a fact about the competitor. Anything from the old card you cannot re-source gets marked unverified.
cost $0.0173 · 21s · in 1,334 / out 3,111 tok
✓ competitor_named✓ pricing✓ win_loss✓ unverified_marked✓ citations

✓ Correct Answer (ground truth)

competitor: Rivally
pricing: value_per_user_per_month=7; source_snippet=S17; source_date=2026-08-12
win_loss_12mo: wins=13; losses=7
unverified_old_claims: Rivally lacks a Slack integration., Rivally was acquired by WorkHuman in 2025.
rep_opinion_snippet_ids_not_facts: S09, S21
citation_ids_available: 25 items — e.g. S01, S02, S03, S04
graded checks: competitor_named · pricing · win_loss · unverified_marked · citations

Model's actual answer

show the model's full answer
Battlecard: Rivally - Updated as of 2026-09-03

1) One-line positioning:
Points-based recognition feed focused on mid-market and distributed teams [S02][S04].

2) Pricing with source and date:
- List price: $7 per user/month, Recognition Starter tier, annual billing required [S17] - source: pricing_page dated 2026-08-12. This is newer than $5 per user/month list shown [S03] on 2026-01-20 and still shown [S08] on 2026-04-01. Conflict noted: newer source wins, price increased from $5 to $7 per S03/S08 vs S17.
- Deal-level observed: $6.50/user/mo quoted to 500-seat prospect annual term [S13] on 2026-06-02; $7/user/mo list quoted with 15% discount offered for 3-year term [S18] on 2026-08-14.

3) Where they win:
- EU data residency pitched to prospects [S05] and announced as generally available with Dublin office [S15].
- Strong for distributed EU teams with multi-language support praised [S12].
- Fast setup under a week, Slack integration worked out of the box [S04].
- Points-based recognition feed praised as engaging [S02][S16].
- Support response time praised under 4 hours [S22].
- Product additions: Pulse lightweight engagement survey add-on launched [S06] and Microsoft Teams app v2 in public preview [S19].

4) Where we win:
- Analytics: limited analytics noted [S02], reporting dashboards are basic vs enterprise [S07], analytics exports are CSV-only [S20], prospect specifically cited analytics depth when picking Bonusly over Rivally [S25].
- Admin / IT: lacks SCIM provisioning, manual user management painful [S10], admin tooling lags peers [S16], admin console lacks bulk recognition editing [S24].
- Rewards: EMEA catalog thinner than US catalog [S14].

5) Objections and responses:
- Objection: "Rivally offers EU data residency" [S05]. Response: Acknowledge GA of EU residency [S15] and EU expansion hire [S11]; counter on areas where we have sourced advantage: analytics depth [S25] vs limited analytics [S02] and basic dashboards [S07].
- Objection: "Rivally setup is fast and Slack just works" [S04]. Response: Acknowledge setup claim, then ask about ongoing admin: SCIM lack [S10], no bulk editing [S24], admin tooling lag [S16].
- Objection: "Rivally is engaging feed" [S02][S16]. Response: Validate engagement, then differentiate on reporting/analytics [S07][S20][S25] and EMEA catalog gaps [S14].

6) Recent changes:
- 2025-11-04 Series C $40M led by Northgate Ventures [S01]
- 2026-03-05 Launches Rivally Pulse lightweight engagement survey add-on [S06]
- 2026-05-09 Hires ex-Workday VP EMEA to lead European expansion [S11]
- 2026-07-01 Opens Dublin office; announces EU data residency GA [S15]
- 2026-08-12 Pricing page update: Recognition Starter now $7 vs prior $5 [S17] vs [S03][S08]
- 2026-08-20 Announces Microsoft Teams app v2 in public preview [S19]
- 2026-09-01 Rivally Pulse exits beta; priced as add-on, not bundled [S23]

7) 12-month win/loss record vs Rivally (from deals_with_competitor.csv):
Window: 2025-09 to 2026-08 inclusive (latest deal month in file).
Deals (20 total):
Wins (13): Deal-A9FD43, Deal-7AA785, Deal-44C524, Deal-0D0CD6, Deal-D5B790, Deal-5C636E, Deal-67BE14, Deal-1B6969, Deal-F03E7B, Deal-072E31, Deal-F65C8F, Deal-E46EAB, Deal-1D2392
Losses (7): Deal-7767F5, Deal-5645A5, Deal-C6FFAA, Deal-D263E0, Deal-935746, Deal-9066A6, Deal-72A02F
Arithmetic: 13 wins + 7 losses = 20 total. Win rate = 13/20 = 0.65 = 65%. Loss rate = 7/20 = 35%.

Unverified from old card (cannot re-source):
- "Rivally lacks a Slack integration" - MARKED UNVERIFIED - contradicted by Slack integration worked out of the box [S04].
- "Rivally was acquired by WorkHuman in 2025" - MARKED UNVERIFIED - no source in snippets provided.

Excluded per rule (rep opinion not a fact): S09 UI clunky opinion, S21 discounting aggressively opinion - not used.
nurture-sequence-diagnosisPASS 1.00

? The Question

Review the sequences. Per sequence: sent, open rate, reply rate, meeting rate, weakest step. Identify tracking errors (opened above sent), audience overlap between sequences, and the failure mode for anything under 2 percent reply. One change per weak sequence and which to fix first. Under 350 words.
cost $0.0259 · 15s · in 11,273 / out 2,217 tok
✓ tracking_error✓ overlap_pair✓ weakest_step✓ fix_priority

✓ Correct Answer (ground truth)

per_sequence: {"New Logo Nurture": {"sent": 1386, "opened": 490, "replied": 90, "meetings": 27, "open_rate": 0.3535, "reply_rate": 0.0649, "meeting_rate": 0.0195, "weakest_step": 3}, "Expansion Nurture": {"sent": 8
tracking_error: sequence=Expansion Nurture; step=2; sent=300; opened=340; issue=opened exceeds sent
audience_overlap_pair: Cold Outbound - HR Leaders, Cold Outbound - People Ops
all_pairwise_overlaps: ['Cold Outbound - HR Leaders', 'Cold Outbound - People Ops'], ['Expansion Nurture', 'New Logo Nurture']
under_2pct_reply: Cold Outbound - HR Leaders
failure_mode: Cold Outbound - HR Leaders=opens healthy (~37%) but replies near zero from step 1 -> body/CTA not landing, not a deliverability or subject-line problem
fix_priority_first: Cold Outbound - HR Leaders
graded checks: tracking_error · overlap_pair · weakest_step · fix_priority

Model's actual answer

show the model's full answer
Sequences:

**New Logo Nurture**: Sent 1386 (500+458+428). Opened 490 = 35.4% (490/1386). Replied 90 = 6.5% (90/1386). Meetings 27 = 1.95% (27/1386). Weakest step: Step 3 - open 28.0% (120/428), reply 4.2% (18/428), meetings 1.4% (6/428).

**Expansion Nurture**: Sent 875 (300+300+275). Opened 565 = 64.6% (565/875). Replied 59 = 6.7% (59/875). Meetings 12 = 1.37% (12/875). Weakest step: Step 3 - reply 4.36% (12/275), meeting 1.09% (3/275).

**Cold Outbound - HR Leaders**: Sent 1785 (600+595+590). Opened 545 = 30.5% (545/1785). Replied 8 = 0.45% (8/1785). Meetings 0 = 0%. Weakest step: Step 3 - open 22.0% (130/590), reply 0.17% (1/590).

**Cold Outbound - People Ops**: Sent 1163 (400+386+377). Opened 340 = 29.2% (340/1163). Replied 29 = 2.49% (29/1163). Meetings 6 = 0.52% (6/1163). Weakest step: Step 3 - reply 1.59% (6/377).

Tracking error: `Expansion Nurture` Step 2 - opened 340 > sent 300 = 113.3% open rate. Bot/reopen inflation, invalidates reporting.

Audience overlap: 0 unique overlap expected but found duplicates. Example: `CT-000849`, `CT-000884`, `CT-001101`, `CT-001103`, `CT-001105`, `CT-001153`, `CT-001255`, `CT-000890`, `CT-000908`, `CT-001217`, `CT-001227`, `CT-001236`, `CT-001258`, `CT-001277`, `CT-001285`, `CT-001311`, `CT-001345`, `CT-001033`, `CT-001130` appear in both `Cold Outbound - HR Leaders` and `Cold Outbound - People Ops`. `CT-000301` and `CT-000624` appear in both `New Logo Nurture` and `Expansion Nurture`.

Failure mode <2% reply: `Cold Outbound - HR Leaders` 0.45% overall, with 0 meetings. Not deliverability (open 30.5% OK) but message/offer mismatch for HR Leaders persona - zero intent.

Fix:
- HR Leaders: Pause sequence. Rewrite ICP/value prop, do not re-enroll overlap contacts.
- Expansion Nurture: Fix tracking - deduplicate opens to unique.
- People Ops Step 3 + New Logo Step 3: Add break-up/referral CTA.

Fix first: `Cold Outbound - HR Leaders` - highest waste (1785 sends, 0 meetings) and contaminates People Ops overlap.
weekly-marketing-goals-rollupPASS 1.00

? The Question

Produce the weekly marketing goals update for the active quarter: SQMs, SQOs, DS2s, closed-lost MIA rate, same-quarter close count, active pipeline coverage against target. For each: QTD actual, target, delta, pace (on/behind/ahead given days elapsed). One paragraph on what moved this week.
cost $0.0295 · 13s · in 16,406 / out 2,123 tok
✓ metrics_present✓ deltas✓ coverage_ratio✓ pace✓ paragraph

✓ Correct Answer (ground truth)

days_elapsed: 66
days_in_quarter: 92
elapsed_fraction: 0.7174
metrics: {"SQMs": {"qtd_actual": 230, "target": 300, "delta": -70, "expected_at_day_elapsed": 215.22, "pace": "ahead"}, "SQOs": {"qtd_actual": 84, "target": 120, "delta": -36, "expected_at_day_elapsed": 86.09,
coverage_ratio_active_pipeline: 0.75
pace_rule: expected = target * days_elapsed/days_in_quarter; within +/-5% of expected = on, above = ahead, below = behind; lower_better metrics invert.
this_week_paragraph_required: True
graded checks: metrics_present · deltas · coverage_ratio · pace · paragraph

Model's actual answer

show the model's full answer
Weekly Marketing Goals Update - Q3-2026 - Day 66 of 92

Pace factor = days_elapsed / days_in_quarter = 66 / 92 = 0.71739 = 71.7% elapsed.

1. SQMs
- QTD Actual: 230
- Target: 300
- Delta: 230 - 300 = -70 to target
- Expected at pace: 300 * 66 / 92 = 19,800 / 92 = 215.22
- Pace: AHEAD - 230 vs 215.22 expected (+14.78)

2. SQOs
- QTD Actual: 84
- Target: 120
- Delta: 84 - 120 = -36 to target
- Expected at pace: 120 * 66 / 92 = 7,920 / 92 = 86.09
- Pace: BEHIND - 84 vs 86.09 expected (-2.09)

3. DS2s
- QTD Actual: 40
- Target: 75
- Delta: 40 - 75 = -35 to target
- Expected at pace: 75 * 66 / 92 = 4,950 / 92 = 53.80
- Pace: BEHIND - 40 vs 53.80 expected (-13.80)

4. Closed-Lost MIA Rate
- QTD Actual: closed_lost_mia / closed_lost_total = 5 / 25 = 0.20 = 20%
- Target: 0.10 = 10% - direction: lower_better
- Delta: 0.20 - 0.10 = +0.10 = +10pp over target
- Pace: BEHIND - rate is above target threshold. Not a cumulative paced metric.

5. Same-Quarter Closes
- QTD Actual: 10
- Target: 20
- Delta: 10 - 20 = -10 to target
- Expected at pace: 20 * 66 / 92 = 1,320 / 92 = 14.35
- Pace: BEHIND - 10 vs 14.35 expected (-4.35)

6. Active Pipeline Coverage
- QTD Actual: 3,000,000
- Target: 4,000,000
- Delta: 3,000,000 - 4,000,000 = -1,000,000 to target
- Coverage: 3,000,000 / 4,000,000 = 0.75 = 75%
- Expected at linear pace: 4,000,000 * 66 / 92 = 2,869,565.22
- Pace: AHEAD on linear build - 3,000,000 vs 2,869,565.22 expected (+130,434.78) and 75% vs 71.7% elapsed, but still $1M short of absolute target.

What moved this week:
Data missing to determine weekly movement. Provided files contain only QTD actuals (marketing_qtd.csv), targets (targets.csv), and quarter meta (quarter_meta.csv). No week-over-week file, prior week snapshot, or changelog provided to calculate what moved this week.
communication
ceo-slack-compressionPASS 1.00

? The Question

Compress the forecast analysis below into a single Slack message for the CEO. Rules: at most three sentences; under 80 words; lead with the decision or the so-what; keep every distinct number that matters and the one caveat that changes the conclusion; cut any sentence that does not carry a claim, a number, or a directive. No em dashes, no throat-clearing phrases.
cost $0.0298 · 9s · in 16,686 / out 2,113 tok
✓ max_3_sentences✓ under_80_words✓ weighted_present✓ caveat_present✓ no_em_dash

✓ Correct Answer (ground truth)

graded checks: max_3_sentences · under_80_words · weighted_present · caveat_present · no_em_dash

Model's actual answer

show the model's full answer
Do not publish Q3 at 115,976.75 until data is fixed.

Calculation is 44,729 COMMIT (7 deals) + 35% of 203,565 BEST_CASE (24 deals) for 54 of 86 in-quarter deals, 23 PIPELINE at 0, 32 deals (227,575) excluded including COMMIT Deal-D348E1 13,770 on 2026-10-15 and 9 BEST_CASE 28,240.

Caveat that changes conclusion: owner blank on 85 of 86 and why-buy empty on 71 of 86 including all 7 COMMIT, plus 32 October close dates that look pushed not re-qualified.
stalled-deal-followup-emailPASS 1.00

? The Question

Write a follow-up email for this stalled deal. Under 120 words. Reference one specific prior touch from the engagement history, make one clear ask, and match the tone of the prior thread. No invented details.
cost $0.0087 · 9s · in 403 / out 1,353 tok
✓ word_cap✓ references_touch✓ single_ask

✓ Correct Answer (ground truth)

deal_alias: Deal-0D2F7A
amount: 5100
stage: DS3
owner: Alex Franklin
days_since_last_engagement: 31
word_cap: 120
must_reference: engagement_id=ENG-00002; date=2026-07-28; type=meeting; why=most substantive prior touch; concrete prospect reactions to milestone awards and points catalog
single_cta: True
tone: warm, concise, professional (matches prior thread)
allowed_facts: 14 items — e.g. Deal-0D2F7A, $5,100, DS3, Alex Franklin
allowed_new: ENG-00001, ENG-00002, ENG-00003, ENG-00004, ENG-00005
graded checks: word_cap · references_touch · single_ask

Model's actual answer

show the model's full answer
Subject: Following up - pricing for 150 seats

Hi,

Circling back on my Aug 5 recap of the July 28 demo with your People team.

On July 28 you had a strong reaction to automated milestone awards and the points catalog to replace manual birthdays/anniversaries tracking, and asked for pricing for 150 seats.

I sent the recap + pricing for 150 seats on Aug 5 and haven't heard back - is this still a priority for the VP People?

Clear ask: Can we get 15 minutes Thursday to confirm if you want to move forward?

Best,
Alex Franklin
funnel-digest-paragraphPASS 1.00

? The Question

Write the weekly GTM digest: one paragraph per team (Marketing, Sales, CS, PLG) in Bonusly brand voice. Every number must trace to the extracts. No customer contact names, no employee performance commentary, no compensation. Four paragraphs, under 400 words total.
cost $0.0087 · 13s · in 260 / out 1,411 tok
✓ four_paragraphs✓ word_cap✓ teams

✓ Correct Answer (ground truth)

required_paragraphs: Marketing, Sales, CS, PLG
allowed_numbers: 16 items — e.g. 2, 3, 9, 08
word_cap: 400
banned: customer contact names, employee performance commentary, compensation
graded checks: four_paragraphs · word_cap · teams

Model's actual answer

show the model's full answer
Marketing had 46 SQMs vs a target of 52 (52 - 46 = 6 short, 46/52 = 88.5% to target). Webinar was the highlight with 18 of the 46 SQMs (18/46 = 39.1% of the week's SQMs). Paid search efficiency held steady with cost per SQM at $150.

Sales converted 14 SQOs and set 9 DS2 meetings this week (14 + 9 = 23 total progression events). The team created $310,000 in new pipeline and closed 3 same-quarter deals.

CS saved 2 renewals and held Team NPS at 61. Heading into next week, there are 3 open red-flag accounts to watch.

PLG added 412 new signups at 31% activation (412 * 0.31 = 127.7 activated). 38 companies reached the aha moment of 10 recognition gives.
partner-digest-sectionPASS 1.00

? The Question

Write the partner digest for the period. Every partner in the roster appears once: active plays get status with numbers, quiet partners get one line. Partner-sourced pipeline count and amount must match the deal data. Nothing omitted, nothing added.
cost $0.0093 · 8s · in 453 / out 1,482 tok
✓ all_partners✓ pipeline_amount✓ quiet_one_line

✓ Correct Answer (ground truth)

roster: partner_name=Apex Rewards Co; status=active | partner_name=HRCloud Partners; status=active | partner_name=CultureBridge; status=active | partner_name=WorkWell Group; status=active | partner_name=Recogniq; status=quiet | partner_name=KudosWave; status=quiet … (+2 more)
per_partner_pipeline: {"Apex Rewards Co": {"count": 2, "amount": 275000}, "HRCloud Partners": {"count": 1, "amount": 140000}, "CultureBridge": {"count": 2, "amount": 135000}}
partner_sourced_pipeline: count=5; amount=550000
quiet_partners: Recogniq, KudosWave, PeopleFirst Advisors, TotalPerk
active_no_deals: WorkWell Group
rule: every roster partner named exactly once; quiet partners get one line; no partners outside roster.
graded checks: all_partners · pipeline_amount · quiet_one_line

Model's actual answer

show the model's full answer
Partner Digest — Period: 2026-08-24 to 2026-09-04

Partner-sourced pipeline (from partner_deals.csv where utm_source=Partner):
Count = 5 deals
Amount = 180,000 + 95,000 + 140,000 + 60,000 + 75,000 = 550,000

ACTIVE PARTNERS:

1. Apex Rewards Co — ACTIVE
   - Sourced: 2 deals = Deal-DDAAF2 ($180,000) + Deal-2CF33E ($95,000) = $275,000
   - Slack 2026-08-25: co-webinar locked for 09-15. Both opps in DS1. Confirmed logged with UTM Source = Partner. (Dana Mercer, Cole Ingram)

2. HRCloud Partners — ACTIVE
   - Sourced: 1 deal = Deal-F1CDA5 ($140,000) = $140,000
   - Slack 2026-08-27: integration referral closed the security review; opp moved to DS2. (Alex Franklin)

3. CultureBridge — ACTIVE
   - Sourced: 2 deals = Deal-096E1D ($60,000) + Deal-067213 ($75,000) = $135,000
   - Slack 2026-08-29: lunch-and-learn produced two sourced opps; both early stage. (Farid Osman)

4. WorkWell Group — ACTIVE
   - Sourced: 0 deals = $0
   - Slack 2026-09-02: no sourced deals this period; wants to restart joint playbook in Q4; planning call booked for 09-09. (Dana Mercer)

QUIET PARTNERS:

5. Recogniq — QUIET — 0 sourced deals this period; no activity in provided data.
6. KudosWave — QUIET — 0 sourced deals this period; no activity in provided data.
7. PeopleFirst Advisors — QUIET — 0 sourced deals this period; no activity in provided data.
8. TotalPerk — QUIET — 0 sourced deals this period; no activity in provided data.

Reconciliation check: Active sourced total = $275,000 + $140,000 + $135,000 + $0 = $550,000 = 5 deals — matches partner_deals.csv total.
rfp-answer-from-policyPASS 1.00

? The Question

Answer the 10 questionnaire items from the policy excerpts only. Cite the excerpt id for every answer. Where the excerpts do not answer the question, mark it unanswerable and say what document would; do not guess. Tone: vendor response, direct.
cost $0.0080 · 7s · in 451 / out 1,172 tok
✓ answered_count✓ unanswerable_flagged✓ citations✓ no_guess

✓ Correct Answer (ground truth)

answered_count: 7
answer_map: {"Q1": ["P1"], "Q2": ["P1"], "Q3": ["P2"], "Q4": ["P3"], "Q5": ["P4"], "Q6": ["P5"], "Q7": ["P6"]}
unanswerable: Q8, Q9, Q10
valid_citation_ids: P1, P2, P3, P4, P5, P6, P7, P8
rule: unanswerable items must be flagged and the document that would answer them named; do not guess.
graded checks: answered_count · unanswerable_flagged · citations · no_guess

Model's actual answer

show the model's full answer
Based solely on policy_excerpts.md:

**Q1 - Is customer data encrypted at rest? If so, how?**
Yes. All customer data is encrypted at rest using AES-256. [P1]

**Q2 - Is data encrypted in transit?**
Yes. Data in transit is encrypted with TLS 1.2 or higher. [P1]

**Q3 - Do you hold a SOC 2 report, and can we obtain a copy?**
Yes. Bonusly maintains a SOC 2 Type II report, renewed annually. The report is available to prospects under NDA. [P2]

**Q4 - How long is customer data retained after contract termination?**
Customer data is retained for the duration of the contract plus 90 days, after which it is deleted from production systems. Therefore 90 days post-termination retention. [P3]

**Q5 - Where is your subprocessor list published and how are changes notified?**
Current list maintained at trust.bonusly.com. Customers receive 30 days notice before a new subprocessor is added. [P4]

**Q6 - Do you support SAML single sign-on?**
Yes. SAML 2.0 single sign-on is available on the Enterprise plan. [P5]

**Q7 - What is your production backup schedule and retention?**
Production databases are backed up daily. Backups are encrypted and retained for 35 days. [P6]

**Q8 - Do you hold an ISO 27001 certification?**
Unanswerable from provided excerpts. policy_excerpts.md contains no ISO 27001 information. Would require certification register / Trust portal compliance documentation.

**Q9 - What is your contractual uptime SLA percentage?**
Unanswerable from provided excerpts. policy_excerpts.md contains no uptime SLA information. Would require MSA / SLA exhibit or DPA.

**Q10 - Will you sign a HIPAA Business Associate Agreement?**
Unanswerable from provided excerpts. policy_excerpts.md contains no HIPAA/BAA information. Would require legal / compliance documentation on HIPAA support.
ops-maintenance
skill-file-reconciliation0.86

? The Question

Reconcile the skill set below against its manifest. Find and report (with severity CRITICAL/WARNING/INFO and an action type TRIM_DESC, UPDATE_BODY, DELETE_SKILL, MERGE, or REVIEW): (1) any two skills whose ALWAYS-trigger phrases overlap/duplicate, (2) any circular delegation chain between skills (name it), (3) any delegation target that does not exist (dangling, name it), (4) any version conflict and which skill should survive, (5) how many manifest descriptions exceed 1,024 characters, (6) any hardcoded page ids, dates, or person names in skill bodies, (7) manifest drift in BOTH directions (files with no manifest row, and manifest rows with no file). Make one proposal per finding; rewrite nothing.
cost $0.1016 · 24s · in 64,414 / out 4,387 tok
✗ duplicate_trigger_found✓ circular_chain✓ dangling_target✓ oversized_desc_count✓ hardcoded_values✓ manifest_drift✓ severity_and_action

✓ Correct Answer (ground truth)

graded checks: duplicate_trigger_found · circular_chain · dangling_target · oversized_desc_count · hardcoded_values · manifest_drift · severity_and_action

Model's actual answer

show the model's full answer
Reconciliation — manifest vs files provided (14/14)

1. ALWAYS-trigger overlap — DUPLICATE
CRITICAL — TRIM_DESC / MERGE
- comms-drafter vs email-drafter identical ALWAYS triggers: "write me an email" / "draft a follow-up" / "help me reply" / "what should I say" / "bump email" / "contract nudge" / any variation wanting email copy.
Evidence: comms-drafter description "ALWAYS trigger for: 'write me an email,' 'draft a follow-up,' ..." and email-drafter description "ALWAYS trigger ... 'write me an email,' 'draft a follow-up,' 'what should I say,' 'bump email,' 'contract nudge'". Same intent, 100% overlap.
Proposal: MERGE into one canonical email skill; TRIM_DESC to keep ALWAYS only in survivor, remove from the other or convert to reference.

2. Circular delegation chain
WARNING — REVIEW
Chain found: deal-strategy-coach ↔ email-drafter
- deal-strategy-coach: "use the `email-drafter` skill which automatically retrieves your Gmail signature"
- email-drafter: "For deal strategy, diagnosis, or coaching (not email drafting), use deal-strategy-coach instead."
Also entry: comms-drafter -> deal-strategy-coach -> email-drafter -> deal-strategy-coach forms 3-node cycle.
Proposal: Break cycle — designate direction: comms/email-drafter MAY reference deal-strategy-coach as escalation, but deal-strategy-coach must not invoke email-drafter as sub-agent; instead recommend drafting via template. Update bodies to one-way.

3. Dangling delegation targets (not in manifest)
WARNING — REVIEW
Referenced but no file/row in manifest:
- bonusly-brand (comms-drafter Step 0, email-drafter brand rule, sales-forecast brand rules)
- prospect-research-multithreading (comms-drafter If Recipient Unknown, email-drafter same, deal-strategy-coach Cross-skill handoff)
- signalforge-reports org skill — /mnt/skills/organization/signalforge-reports/SKILL.md, DESIGN-SYSTEM.md, signalforge.css (pipeline-intelligence-report Phase 5, weekly-pipeline-report, sales-forecast)
- bonusly-data-questions, bonusly-product-questions, bonusly-business-reporting-questions, bonusly-rewards-questions, bonusly-ppp-questions, bonusly-feature-flag-questions, bonusly-deal-desk-questions, bonusly-datadog-questions (analysis-validator §12.4 specialist reference)
Proposal: REVIEW — add missing org skills to manifest org layer, or replace delegation with explicit [LIVE QUERY] tags / remove hard refs. Do not leave dangling invoke.

4. Version conflict
WARNING — UPDATE_BODY
- analysis-validator.md contains v3.5 and v3.6 both dated May 9, 2026 in changelog: "3.6 May 9, 2026 G2-F ID Resolution" and "3.5 May 9, 2026 G1-L Engagement Coverage". Two versions same date same file.
Manifest declares single row analysis-validator version implicit.
Survivor: v3.6 (latest, includes G2-F, super-set of 3.5).
Arithmetic: 3.6 > 3.5, same date — keep 3.6, merge 3.5 content already included per trail.
Proposal: UPDATE_BODY to collapse changelog to single 3.6 entry, remove duplicate 3.5 timestamp conflict.

5. Manifest descriptions exceed 1,024 chars
INFO — REVIEW
Count: 0
Arithmetic: max(description_chars) = 1006 (pipeline-intelligence-report, signalforge-claim-compressor per CSV), next = 1004 (partner-digest). All 14 rows <1024 threshold.
Calculation: 656+897+996+792+965+676+945+1004+1006+962+1006+708+762+656 — none >1024.
Proposal: No TRIM_DESC needed. Keep limit enforced in manifest CI.

6. Hardcoded IDs / dates / person names in bodies
WARNING — UPDATE_BODY
Stage IDs hardcoded everywhere (violates [LIVE QUERY] rule):
- 150582536 DS1, 150582537 DS2, 150582538 DS3, 150582539 DS4, 1175632767 DS5 — in next-to-close Step1, pipeline-intelligence-report Phase1+System Constants, stale-pipeline-report Phase1.
Page/Cloud/Space IDs:
- partner-digest: cloudId 73fe98de-a4a3-4869-9f8a-bb1eeed4cf7f, spaceId 1958248479, folderId 2286616609; sales-forecast spaceId 2232811524 parent 2232582148, feedback Page ID 2295136266
Dates: May 9 2026, May 4 2026, April 26 2026 hardcoded in analysis-validator header/changelog; Q2 April 1-June 30 2026 in weekly-pipeline-report; last_checked 2026-05-19 in model-selection; May 16 2026 in partner-digest
Person names/Owner IDs hardcoded: analysis-validator §12.3 GTM roster (Amani Phipps 210200121, Alaina Loori 82535637, Bryce Harmon 119337721, Dana Mercer 83155923, etc), pipeline-intelligence-report AE owner IDs list, etc. Also pricing 2026 table in deal-strategy-coach.
Proposal: UPDATE_BODY — replace with live queries (HubSpot:get_properties, HubSpot:search_owners) and tag "[LIVE QUERY] — re-verify at session start" per reconciliation rule. Do not hardcode anchors.

7. Manifest drift BOTH directions
INFO — REVIEW
- Files with no manifest row: 0 — provided file set = manifest file set (14==14).
- Manifest rows with no file: 0 — every manifest file found in input.
Calculation: manifest rows 14 - files provided 14 = 0 delta.
Proposal: No DELETE_SKILL needed for drift. If org skills counted, they would be drift, but scoped to this manifest folder they are external dependencies, handle as dangling not drift.
runbook-from-slack-threadPASS 1.00

? The Question

Turn the resolved incident thread into a runbook: ordered steps with the exact command or action taken, who ran it, how success was verified, and rollback for each step that changed state. Every step must trace to a message in the thread. Add nothing the thread does not contain; where the thread is unclear, mark the step as needing confirmation.
cost $0.0123 · 17s · in 615 / out 2,154 tok
✓ ordered_steps✓ rollback✓ needs_confirmation✓ sourced

✓ Correct Answer (ground truth)

steps: order=1; msg=M01; actor=Bryce Harmon; action=acknowledge PagerDuty alert, take IC; state_change=False; verified_by=M01 | order=2; msg=M02; actor=Farid Osman; action=check queue depth; command=bundle exec rake sidekiq:queue_depth; state_change=False; verified_by=M02 (48,213 pending) | order=3; msg=M04; actor=Farid Osman; action=pause auto-enqueue via feature flag; command=bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'; state_change=True; rollback=bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'; verified_by=M07 (queue depth falling) | order=4; msg=M05; actor=Elena Sinclair; action=cleared the dead set; state_change=True; needs_confirmation=thread gives no command or method for clearing the dead set | order=5; msg=M06; actor=Bryce Harmon; action=scale reward-worker to 6 replicas; command=kubectl scale deployment/reward-worker --replicas=6; state_change=True; rollback=kubectl scale deployment/reward-worker --replicas=3; verified_by=M07/M08 | order=6; msg=M08; actor=Cole Ingram; action=verify queue drained and error rate baseline; command=bundle exec rake sidekiq:queue_depth; state_change=False; verified_by=M08 (returns 0) … (+2 more)
ambiguous_step: order=4; msg=M05; mark=needs-confirmation
rule: no steps beyond what the thread contains; every step traces to a message id.
graded checks: ordered_steps · rollback · needs_confirmation · sourced

Model's actual answer

show the model's full answer
Runbook — #incident-2026-08-29-reward-queue — sourced only from thread. No additional steps assumed.

**Step 1 — Acknowledge alert**
* Action: PagerDuty alert reward-worker queue depth > 10k — taking IC
* Who: Bryce Harmon
* Trace: [M01]
* Verification: Alert acknowledged per M01
* State change: No

**Step 2 — Measure queue depth**
* Action: `bundle exec rake sidekiq:queue_depth`
* Who: Farid Osman
* Trace: [M02]
* Verification: Returned 48,213 pending jobs. Normal <500 per M02
* State change: No

**Step 3 — Inspect dead set**
* Action: Checked dead set
* Who: Farid Osman
* Trace: [M03]
* Verification: Found 112 jobs, all Redis::TimeoutError from around 13:58 per M03
* State change: No

**Step 4 — Pause enqueue — STATE CHANGE**
* Action: `bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'`
* Who: Farid Osman
* Trace: [M04]
* Verification: Not stated in M04. Subsequent queue depth falling observed in M07/M08 but not directly tied to this step alone.
* Rollback: `bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'` per M04

**Step 5 — Clear dead set — STATE CHANGE — NEEDING CONFIRMATION**
* Action: "cleared out the dead set" while in console. Exact command not provided in thread.
* Who: Elena Sinclair
* Trace: [M05]
* Verification: Not stated in M05
* Rollback: Not stated in thread. Deletion is irreversible unless jobs were re-enqueued. Exact contents/action needing confirmation.

**Step 6 — Scale workers up — STATE CHANGE**
* Action: `kubectl scale deployment/reward-worker --replicas=6` (was 3)
* Who: Bryce Harmon
* Trace: [M06]
* Verification: Not stated in M06. Queue depth down to 9,400 and falling ~1,200/min observed in [M07]. Arithmetic: 48,213 -> 9,400 = 38,813 jobs drained between M02 and M07.
* Rollback: `kubectl scale deployment/reward-worker --replicas=3` per M06

**Step 7 — Monitor drain**
* Action: Monitor queue depth
* Who: Farid Osman
* Trace: [M07]
* Verification: Queue depth 9,400 and falling ~1,200/min per M07
* State change: No

**Step 8 — Verify recovery**
* Action: `bundle exec rake sidekiq:queue_depth`
* Who: Cole Ingram
* Trace: [M08]
* Verification: Returns 0 per M08. Error rate in Datadog back to baseline per M08
* State change: No

**Step 9 — Re-enable enqueue — STATE CHANGE**
* Action: `bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'`
* Who: Bryce Harmon
* Trace: [M09]
* Verification: 40 new jobs processed cleanly in next 3 minutes per M09
* Rollback: `bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'` — inverse of M09, same as M04 action/rollback pair.

**Step 10 — Scale workers down — STATE CHANGE**
* Action: `kubectl scale deployment/reward-worker --replicas=3`
* Who: Bryce Harmon
* Trace: [M10]
* Verification: Queue stable at 0 per M10. Incident marked resolved per M10
* Rollback: `kubectl scale deployment/reward-worker --replicas=6` — inverse of M10, returns to elevated count from M06.

Gaps — Needs confirmation:
* M05 exact console command / method used to clear dead set, and whether jobs were purged vs re-enqueued.
* M04 verification that FeatureFlag disable took effect.
* M05 verification that dead set cleared.
datadog-incident-timelinePASS 1.00

? The Question

From the log slice, sequence the failure: first error with timestamp, the cascade in order, the service and job involved, and the Datadog query that would confirm the first error. State what the logs do not show.
cost $0.0131 · 12s · in 1,643 / out 2,033 tok
✓ first_error_ts✓ service_named✓ cascade_ordered✓ query✓ gaps_stated

✓ Correct Answer (ground truth)

first_error: timestamp=2026-09-03T14:01:12Z; service=reward-service; message=Redis::TimeoutError: Connection to Redis at redis-primary:6379 timed out after 5s
cascade_order: service=reward-service; first_error_at=2026-09-03T14:01:12Z | service=sidekiq; first_error_at=2026-09-03T14:01:40Z | service=api-gateway; first_error_at=2026-09-03T14:03:05Z | service=web-app; first_error_at=2026-09-03T14:03:30Z
job_class_involved: RewardGiveJob
confirming_query: service:reward-service status:error "@message:Redis::TimeoutError"
logs_do_not_show: root cause of the Redis timeout, redis-server's own logs, infrastructure metrics (CPU/memory/network), any deploy or config change preceding the incident, who applied mitigation or when (no actor fields in logs)
clean_services: postgres
graded checks: first_error_ts · service_named · cascade_ordered · query · gaps_stated

Model's actual answer

show the model's full answer
From datadog_logs.csv + sidekiq_jobs.csv only:

**First error:**
2026-09-03T14:01:12Z, reward-service, ERROR, Redis::TimeoutError: Connection to Redis at redis-primary:6379 timed out after 5s

**Service and job:**
Service: reward-service (origin), then sidekiq, api-gateway, web-app.
Job: RewardGiveJob (primary). sidekiq_jobs.csv also shows RecognitionDigestJob failing with same Redis::TimeoutError (J-00013 to J-00016).

**Cascade in order from logs:**

1. 14:01:12Z reward-service - Redis timeout to redis-primary:6379
2. 14:01:20Z reward-service - retry exhausted for RewardGiveJob
3. 14:01:30Z reward-service - retry exhausted for RewardGiveJob
4. 14:01:40Z reward-service - retry exhausted for RewardGiveJob
5. 14:01:40Z sidekiq - RewardGiveJob failed: Redis::TimeoutError; retrying in 60s
6. 14:01:46Z to 14:02:58Z sidekiq_jobs.csv shows 12 RewardGiveJob failures (J-00005 14:01:46Z, J-00001 14:01:51Z, J-00003 14:01:54Z, J-00006 14:01:55Z, J-00002 14:01:56Z, J-00004 14:01:57Z, J-00007 14:02:51Z, J-00011 14:02:51Z, J-00008 14:02:56Z, J-00012 14:02:57Z, J-00010 14:02:57Z, J-00009 14:02:58Z)
7. 14:02:28Z sidekiq - RewardGiveJob failed: Redis::TimeoutError; retrying
8. 14:02:30Z sidekiq - WARN Queue reward depth above 10,000
9. 14:02:36Z to 14:05:50Z sidekiq_jobs.csv - 4 RecognitionDigestJob failures (J-00013 14:02:36Z, J-00014 14:03:15Z, J-00015 14:04:55Z, J-00016 14:05:50Z)
10. 14:03:05Z api-gateway - 502 upstream timeout calling reward-service /gives
11. 14:03:30Z web-app - Give form submission failed: upstream 502 from api-gateway
12. 14:03:31Z sidekiq - RewardGiveJob failed retrying
13. 14:03:48Z api-gateway - 502 upstream timeout calling reward-service
14. 14:04:13Z api-gateway - 502 upstream timeout calling reward-service
15. 14:04:22Z sidekiq - RewardGiveJob failed retrying
16. 14:04:45Z web-app - Give form submission failed: upstream 502
17. 14:05:16Z api-gateway - 502 upstream timeout calling reward-service
18. 14:05:26Z sidekiq - RewardGiveJob failed retrying
19. 14:05:42Z web-app - Give form submission failed: upstream 502
20. 14:06:47Z sidekiq - RewardGiveJob failed retrying
21. 14:06:49Z web-app - Give form submission failed: upstream 502
22. 14:06:52Z api-gateway - 502 upstream timeout calling reward-service
23. Recovery: 14:22:10Z reward-service - Redis connection restored; resuming job processing
24. 14:24:45Z sidekiq - Queue reward depth below 500

Arithmetic:
Outage start to restore: 14:22:10 - 14:01:12 = 20m 58s = (20*60)+58 = 1258s
Queue depth change: >10,000 to <500 = at least 9,500 jobs drained.

**Datadog query to confirm first error:**
`service:reward-service status:error "Redis::TimeoutError" "redis-primary:6379" `
or Log Explorer: `service:reward-service @level:ERROR @message:"*Redis::TimeoutError*"` filtered to timestamp 2026-09-03T14:01:12Z +/- 5m

**What the logs do not show:**
- Root cause of Redis timeout at redis-primary:6379 (infra, network, config, resource saturation)
- Redis server-side logs/metrics, CPU, memory, connections
- Whether gives were lost, duplicated, or successfully retried after 14:22:10Z
- Number of users/companies affected, or deal/company aliases
- Why postgres checkpoints (14:10:56Z - 14:20:59Z) are included - no error linkage shown
- Any activity between 14:06:52Z and 14:22:10Z except postgres INFO
feature-flag-state-summaryPASS 1.00

? The Question

Summarize the state of the listed feature flags: what each controls per the code excerpt, which companies or segments have it on, targeting rules, and any flag with no code reference or no targets. Company counts must match the export. Do not describe a flag that is not in the export.
cost $0.0094 · 10s · in 486 / out 1,511 tok
✓ orphans✓ no_targets✓ counts_match✓ no_invented

✓ Correct Answer (ground truth)

flags: {"recognition_streaks_v2": {"state": "on", "targeting_rules": "segment:beta_companies", "company_count": 42}, "points_budget_guardrails": {"state": "on", "targeting_rules": "all_companies", "company_c
code_referenced: recognition_streaks_v2, points_budget_guardrails, slack_dm_nudges, redeem_flow_redesign, analytics_dashboard_v3, ms_teams_app_v2
orphan_flags_no_code_reference: legacy_give_modal, survey_boosters_q3, paused_offboard_cleanup
flags_with_no_targets: paused_offboard_cleanup
rule: company counts must equal the export; do not describe flags absent from the export.
graded checks: orphans · no_targets · counts_match · no_invented

Model's actual answer

show the model's full answer
Feature flag state — per flags_export.csv and flag_code.md only:

**1. What each controls / state / targeting / count:**

- **recognition_streaks_v2** — State: **on**. Controls: `StreakTracker.record(give)` in `app/models/recognition.rb`. Targeting: `segment:beta_companies`. Company count: **42**.

- **points_budget_guardrails** — State: **on**. Controls: `BudgetService.new(company).enforce!(giver, points)` in `app/services/budget_service.rb`. Targeting: `all_companies`. Company count: **220**.

- **slack_dm_nudges** — State: **on**. Controls: `SlackDm.send_nudge(user)` in `app/jobs/nudge_job.rb`. Targeting: `segment:region_na`. Company count: **87**.

- **redeem_flow_redesign** — State: **off**. Controls: `RedeemV2Component` vs `RedeemV1Component` in `app/controllers/redeem_controller.rb`. Targeting: `targeted_list`. Company count: **12**.

- **analytics_dashboard_v3** — State: **on**. Controls: `AnalyticsV3.new(company)` in `app/controllers/analytics_controller.rb`. Targeting: `segment:tier_three`. Company count: **65**.

- **ms_teams_app_v2** — State: **off**. Controls: `TeamsAppV2.install(company)` in `app/services/teams_installer.rb`. Targeting: `targeted_list`. Company count: **9**.

- **legacy_give_modal** — State: **off**. Controls: **No code reference in provided excerpt**. Targeting: `segment:legacy_plan`. Company count: **14**.

- **survey_boosters_q3** — State: **on**. Controls: **No code reference in provided excerpt**. Targeting: `segment:legacy_plan`. Company count: **7**.

- **paused_offboard_cleanup** — State: **off**. Controls: **No code reference in provided excerpt**. Targeting: **(empty)**. Company count: **0**.

**2. Flags with no code reference in flag_code.md:**
- `legacy_give_modal`
- `survey_boosters_q3`
- `paused_offboard_cleanup`

**3. Flags with no targets:**
- `paused_offboard_cleanup` — targeting_rules empty, company_count 0.

Arithmetic check: 42 + 220 + 87 + 12 + 65 + 9 + 14 + 7 + 0 = 456 total company-flag assignments in export.
nda-triagePASS 1.00

? The Question

Triage each NDA as GREEN (standard approval), YELLOW (counsel review), or RED (full legal review). Cite the clause that drives each classification by quoting it. Catch embedded non-solicits, non-competes, missing carve-outs, and unusual term or governing law. One line of reasoning per NDA.
cost $0.0091 · 9s · in 539 / out 1,419 tok
✓ green✓ yellow✓ red✓ non_solicit_caught✓ clause_quoted✓ correct_labels

✓ Correct Answer (ground truth)

classifications: {"NDA-1": {"classification": "GREEN", "driving_reason": "standard mutual NDA: 2-year term, standard carve-outs, Delaware law", "driving_clause_quote": "Carve-outs: information that (a) is or becomes p
non_solicit_in: NDA-2
checks: classification_per_nda, clause_quotes_exist_in_document, non_solicit_caught, carveouts_checked
graded checks: green · yellow · red · non_solicit_caught · clause_quoted · correct_labels

Model's actual answer

show the model's full answer
**NDA Triage:**

**NDA-1 — GREEN (standard approval)**
- Reasoning: Standard mutual NDA with 2-year term + 3-year survival, 5 standard carve-outs, and Delaware law — no embedded restrictive covenants. Driving clauses: "information that (a) is or becomes publicly available through no breach, (b) was known prior to disclosure, (c) is received from a third party without duty of confidence, (d) is independently developed, or (e) must be disclosed by law or court order." and "the laws of the State of Delaware, USA."

**NDA-2 — YELLOW (counsel review)**
- Reasoning: Contains embedded non-solicitation creating hiring restriction beyond NDA term (3-year term + 18-month tail = 4.5-year effective restriction). Driving clause: "During the term of this Agreement and for eighteen (18) months thereafter, neither party shall solicit for employment or hire any employee of the other party with whom it came into contact under this Agreement."

**NDA-3 — RED (full legal review)**
- Reasoning: Contains embedded 3-year non-compete plus three additional RED flags — one-way obligation, 5-year unusual term, Ireland exclusive jurisdiction, and zero carve-outs stated. Driving clause: "For a period of three (3) years following the Effective Date, Recipient shall not, directly or indirectly, engage in or provide services to any business that competes with Discloser's business." Additional flags: "This Agreement shall be governed by and construed in accordance with the laws of the Republic of Ireland, and the parties submit to the exclusive jurisdiction of its courts." + "No confidentiality carve-outs are stated in this Agreement." + "Term: five (5) years" + "One-way confidentiality in favor of [PARTY C]".