home · ← back to all models
z-ai

z-ai/glm-5.1

Mean 0.964 · 33/40 perfect tests · $1.99 total · each card: the question → the correct answer (gold) → this model's actual answer

data-crm
stage-hygiene-auditPASS 1.00

? The Question

Audit stage hygiene on the open pipeline below. Rules: a deal at DS2 or later must have a substantive why-buys (15+ characters, not a placeholder such as tbd, n/a, see notes). A why_buys value of the form LONG(n) means the real text was present but truncated for display; treat it as present and substantive. Every deal should have a DS1 entry timestamp. A deal whose stage entry timestamps are out of order regressed (an earlier-stage timestamp is later than a later-stage timestamp). Report: percent of DS2+ deals non-compliant on why-buys overall and by owner (use the owners table for names), count of deals with no DS1 timestamp, count that regressed, and three process fixes. State definitions before numbers.
cost $0.1317 · 287s · in 48,452 / out 23,456 tok
✓ noncomp_pct✓ no_ds1_0✓ regressed_0✓ per_owner_breakdown✓ definitions_first✓ three_fixes

✓ Correct Answer (ground truth)

noncomp_pct: 81.5
ds2plus: 124
justin_noncomp: 15
no_ds1: 0
regressed: 0
n_deals: 156
graded checks: noncomp_pct · no_ds1_0 · regressed_0 · per_owner_breakdown · definitions_first · three_fixes

Model's actual answer

show the model's full answer
STAGE HYGIENE AUDIT

Definitions:
- DS2+ deal: any deal whose current stage is DS2, DS3, DS4, or DS5.
- Substantive why-buys: a value of 15+ characters that is not a placeholder (tbd, n/a, see notes, blank). Values of the form LONG(n) are treated as present and substantive per the rules.
- Non-compliant: a DS2+ deal whose why_buys fails the substantive test.
- No DS1 timestamp: the t_ds1 field is empty/blank.
- Regressed: an earlier-stage timestamp is later than a later-stage timestamp in the t_ds1..t_ds5 chain (i.e., timestamps are not monotonically non-decreasing).

Arithmetic shown inline.

WHY-BUYS COMPLIANCE (DS2+ deals)

Total DS2+ deals: 124
Compliant: 23 (all 23 carry LONG(n) values; zero carry free-text >= 15 chars)
Non-compliant: 101 (all 101 have completely blank why_buys — not placeholders, just empty)

Overall non-compliance: 101 / 124 = 81.5%

By owner (active owners only; archived owners Gavin Porter and Hugo Lindqvist have no deals in this pipeline):

  Owner                  Non-compliant / Total    Pct
  Alex Franklin          48 / 61                  78.7%
  Dana Mercer             18 / 22                  81.8%
  Bryce Harmon             15 / 18                  83.3%
  Cole Ingram     19 / 22                  86.4%
  Farid Osman             1 /  1                 100.0%

Alex Franklin carries the largest absolute volume of non-compliant deals (48 of 101). Cole Ingram has the worst rate among owners with meaningful volume (86.4%). Farid Osman's 100% is a single deal (Deal-499BF6, DS2, $1,249).

DS1 TIMESTAMP COVERAGE

Deals with no t_ds1 timestamp: 0
Every deal in the pipeline, including all 32 DS1-only deals, has a populated t_ds1.

REGRESSION CHECK

Deals that regressed (out-of-order stage timestamps): 0
All 156 deals have monotonically non-decreasing timestamp chains across t_ds1 through t_ts5. No earlier-stage timestamp postdates a later-stage timestamp.

THREE PROCESS FIXES

1. Make why_buys a required field at the DS2 stage transition. Block stage advancement to DS2 in the CRM if the field is blank or under 15 characters. This is the single highest-impact fix — 101 of 124 DS2+ deals (81.5%) are failing purely because the field is empty, not because reps wrote placeholders. A hard gate at transition eliminates the entire class of blank-field violations at the source.

2. Run a weekly why-buys backfill sweep grouped by owner, prioritizing Alex Franklin (48 open gaps) and Cole Ingram (86.4% rate). Reps cannot fix what they are not reminded of; a weekly report listing each non-compliant deal alias with a one-click link to edit the field, sent Monday morning, drives closure on the existing backlog while the stage gate (fix 1) prevents new gaps.

3. Add a validation rule that flags any deal reaching DS4 or DS5 without a why_buys as a pipeline-health exception in the forecast review. Eight DS5 deals (Deal-D348E1, Deal-C26D20, Deal-547B2B, Deal-403845, Deal-B7EBD1, Deal-A2B47C, Deal-2465CE, Deal-FD9F4E) and twelve DS4 deals sit in forecast categories COMMIT or BEST_CASE with no recorded buying rationale — these are the deals most likely to slip or close lost with no diagnostic trail. Surfacing them in the forecast call forces the conversation before the deal stalls, not after.
owner-resolution0.67

? The Question

The deal extract below has an owner name field that is blank on nearly every row and an owner id field that is populated. Using the owners table, resolve each open deal to an owner name. Report: number of deals resolved, deals whose owner id has no match in the owners table (list them), the owner ids that map to archived/deactivated owners (list their names), and the total pipeline amount per resolved owner. Do not guess a name for any unmatched id.
cost $0.0865 · 337s · in 27,098 / out 17,218 tok
✓ resolved_count✗ unmatched_none✓ deactivated_flagged✓ jake_amount✓ justin_amount✗ no_invented

✓ Correct Answer (ground truth)

resolved: 156
jake_amount: 1054144.0
justin_amount: 624310.0
per_owner: Bryce Harmon=1054144.0; Alex Franklin=624310.0; Dana Mercer=341195.0; Cole Ingram=288161.43; Farid Osman=4134.0; Elena Sinclair=2100.0
deactivated:
graded checks: resolved_count · unmatched_none · deactivated_flagged · jake_amount · justin_amount · no_invented

Model's actual answer

show the model's full answer
**Owner Resolution Report**

Deals resolved: 156 / 156 (all owner IDs matched)

Unmatched owner IDs (no match in owners table): None

Deals mapped to archived/deactivated owners: None
(The two archived owners -- Gavin Porter (1520255671) and Hugo Lindqvist (77260721) -- appear zero times in the open pipeline.)

**Total pipeline by resolved (active) owner:**

| Owner | Deals | Pipeline |
|---|---|---|
| Bryce Harmon | 35 | $1,054,144.00 |
| Alex Franklin | 67 | $624,310.00 |
| Dana Mercer | 24 | $341,195.00 |
| Cole Ingram | 22 | $288,161.43 |
| Farid Osman | 7 | $4,134.00 |
| Elena Sinclair | 1 | $2,100.00 |

Total resolved pipeline: $2,314,044.43

Arithmetic check: 35 + 67 + 24 + 22 + 7 + 1 = 156 deals, matching the full extract.
gong-hubspot-join-integrityPASS 1.00

? The Question

Check the integrity of the call-to-deal mapping below. The gong table lists calls in the last 90 days keyed to HubSpot deal ids; deals_open lists the current open default-pipeline deals. Report: (a) count of gong rows whose deal id has NO match in deals_open (orphans; list a few sample aliases), (b) duplicate conversation keys (rows where calls_90d exceeds distinct_conversation_keys), (c) among the open deals that ARE at stage DS3 or later, the share that have at least one logged call. Give sample aliases for the orphan finding, all taken from the data. Note: gong_owner_id is a Gong user id, not a HubSpot owner id.
cost $0.1099 · 437s · in 34,955 / out 22,209 tok
✓ orphan_count✓ dup_keys_0✓ ds3_coverage✓ sample_ids_present

✓ Correct Answer (ground truth)

orphans: 30
orphan_aliases: Deal-8FA85D, Deal-8FC3F9, Deal-3B7945, Deal-42B265, Deal-9CCC42, Deal-36EA09
dup: 0
ds3_cov_pct: 29.4
graded checks: orphan_count · dup_keys_0 · ds3_coverage · sample_ids_present

Model's actual answer

show the model's full answer
Here is the integrity check:

---

**(a) Orphan gong rows (deal id absent from deals_open): 30 of 67**

44.8% of gong rows reference a deal that is not currently open. These are likely closed-won, closed-lost, or deleted deals still carrying Gong call history. Sample aliases:

- Deal-8FA85D (46 calls, Diversified Financial Services)
- Deal-8FC3F9 (24 calls, Consumer Durables & Apparel)
- Deal-3B7945 (21 calls, Pharmaceuticals/Biotech)
- Deal-42B265 (21 calls, Software & Services)
- Deal-9CCC42 (17 calls, Food/Beverage/Tobacco)
- Deal-36EA09 (17 calls, Software & Services)
- Deal-9A43B4 (14 calls, Software & Services)
- Deal-605F3C (12 calls, Software & Services)
- Deal-E2D34B (11 calls, industry blank)
- Deal-76821A (11 calls, Software & Services)

Notable: the top orphans by volume (Deal-8FA85D at 46 calls, Deal-8FC3F9 at 24) are all calls_with_ds2_date = 0, calls_with_sqo_date = 0 -- they predate the current qualification framework.

---

**(b) Duplicate conversation keys: 0 rows**

No gong row has calls_90d > distinct_conversation_keys. In every row the two values match exactly, meaning each call maps to a unique Gong conversation. No key duplication detected.

---

**(c) DS3+ open deals with call coverage: 25 of 85 = 29.4%**

85 open deals sit at DS3 or later (DS3/DS4/DS5). Only 25 of them (29.4%) have at least one logged Gong call in the last 90 days. The remaining 60 deals (70.6%) at DS3+ have zero call activity recorded in the gong table.

Arithmetic: 25 / 85 = 0.2941... = 29.4%

DS3+ deals without any logged call (first 10): Deal-9AAE5F, Deal-403845, Deal-6787C2, Deal-2465CE, Deal-62D607, Deal-7B3B0F, Deal-F9A08A, Deal-0660B4, Deal-1FC049, Deal-FD9F4E.
snowflake-sql-trapped-catalogPASS 1.00

? The Question

Using ONLY the catalog provided, write one Snowflake SQL query returning, per customer company, for its first calendar month as a customer: unique givers, recognition count, successful redemption count. Read the catalog notes carefully: some tables are stale or unpopulated and must be avoided, and one documented filter (excluding deleted givers) must NOT be applied to historical giving counts. Put a comment above each FROM or JOIN explaining the table choice. Return only fenced SQL.
cost $0.0248 · 168s · in 992 / out 6,909 tok
✓ avoids_stale_deals✓ avoids_unpopulated_live✓ uses_cohort_summary✓ succeeded_filter✓ no_giver_deletion_filter✓ returns_sql

✓ Correct Answer (ground truth)

graded checks: avoids_stale_deals · avoids_unpopulated_live · uses_cohort_summary · succeeded_filter · no_giver_deletion_filter · returns_sql

Model's actual answer

show the model's full answer
```sql
-- Per customer company, first-calendar-month metrics:
--   unique givers, recognition count, successful redemption count.
-- Scope: self-serve companies only (COMPANY_COHORT_SUMMARY is the sole
--         source for giving/recognition data in this catalog).

SELECT
    ccs.company_id,
    ccs.m1_users                  AS unique_givers,
    ccs.m1_gives                  AS recognition_count,
    COUNT(r.redemption_id)        AS successful_redemption_count

FROM production.plg.company_cohort_summary AS ccs
-- Only table in the catalog with giving/recognition metrics.
-- One row per self-serve company; M1_* columns cover month 1.
-- M1_GIVES  = recognition count in first month.
-- M1_USERS  = unique givers in first month.
--   *** Business rule: do NOT join a users table to exclude deleted givers;
--       that filter understates historical giving counts. M1_* pre-aggregated
--       values already include all givers (including since-deleted). ***
-- Covers self-serve companies only; no raw recognition-events table exists
-- in the catalog for sales-sourced companies.

LEFT JOIN production.deprecated_recognition.redemption_records_v2 AS r
-- Documented source for redemption counts despite the DEPRECATED schema name.
-- Business rule mandates STATE = 'succeeded' only; we enforce it here
-- instead of relying on M1_REDEMPTIONS from the cohort summary, which
-- may not apply this filter. Sliced to the first calendar month of
-- paid customer status (month of FIRST_SUB_PAYMENT_DATE).
    ON r.company_id = ccs.company_id
   AND r.state      = 'succeeded'
   AND r.created_at >= DATE_TRUNC('month', ccs.first_sub_payment_date)
   AND r.created_at <  DATEADD(month, 1,
                               DATE_TRUNC('month', ccs.first_sub_payment_date))

GROUP BY
    ccs.company_id,
    ccs.m1_users,
    ccs.m1_gives

ORDER BY
    ccs.company_id;
```
crm-completeness-enrichmentPASS 1.00

? The Question

Audit the CRM extract for completeness and propose fixes. Deals need owner, stage, amount, close date, why-buys; companies need industry, employee count, HQ country; contacts need email, title, persona. Report completeness percent per field. Identify duplicate company clusters (name variants, shared domain) with a survivor per cluster. List invalid emails and domain mismatches. Fill missing company fields only where the enrichment export has a matching row; where CRM and enrichment disagree list both and recommend a source. Never invent a value. End with the 10 fixes with the most pipeline amount at stake.
cost $0.1024 · 330s · in 14,277 / out 27,321 tok
✓ dup_clusters✓ invalid_emails✓ survivor_language✓ completeness_pct✓ no_invent fabricated: C-66D97D

✓ Correct Answer (ground truth)

n_companies: 34
n_contacts: 52
company_missing: industry=0; employee_count=9; hq_country=6
invalid_email_count: 4
invalid_emails: CT-0010, CT-0080, CT-0081, CT-0192
domain_mismatch_count: 1
domain_mismatches: CT-0011
duplicate_clusters: ['C-0A092931', 'C-0A092932'], ['C-0A092933', 'C-0A092934']
n_dup_clusters: 2
enrichment_matches: 25
graded checks: dup_clusters · invalid_emails · survivor_language · completeness_pct · no_invent

Model's actual answer

show the model's full answer
Here is the full audit. All figures computed from the three files provided.

════════════════════════════════════════════════════════
CRITICAL GAP: NO deals.csv PROVIDED
════════════════════════════════════════════════════════

The task specifies deals need owner, stage, amount, close date, and why-buys. No deals file was included in this extract. Deal-level completeness, pipeline amounts, and the "10 fixes with most pipeline amount at stake" cannot be computed without deal records. I flag this as the single most critical gap and proceed with what the data supports: companies (34 rows), contacts (52 rows), and ZoomInfo enrichment (25 rows).

════════════════════════════════════════════════════════
1. COMPLETENESS REPORT
════════════════════════════════════════════════════════

COMPANIES (n=34)
  company_alias     34/34  (100.0%)
  domain            34/34  (100.0%)
  industry          34/34  (100.0%)  — but 10 rows use non-standard labels (see §3)
  employee_count    25/34  ( 73.5%)  — 9 missing
  hq_country        28/34  ( 82.4%)  — 6 missing

CONTACTS (n=52)
  contact_key       52/52  (100.0%)
  company_alias     52/52  (100.0%)
  email             52/52  (100.0%)  — but 4 are malformed, 1 domain mismatch (see §4)
  title             39/52  ( 75.0%)  — 13 missing
  persona           36/52  ( 69.2%)  — 16 missing
  domain            52/52  (100.0%)

  Valid + domain-matching emails: 48/52 (92.3%)
  Malformed emails:               4
  Domain mismatches:              1

DEALS
  Not provided. Completeness: N/A. All five required fields (owner, stage, amount, close date, why-buys) are entirely absent from this extract.

════════════════════════════════════════════════════════
2. DUPLICATE COMPANY CLUSTERS
════════════════════════════════════════════════════════

Two clusters found, both by shared domain:

CLUSTER 1 — domain: acme-corp.com
  C-0A092931   industry='Technology'  emp=500   hq='US'
  C-0A092932   industry='tech'        emp=510   hq='USA'
  Survivor: C-0A092931 (lower alias number; cleaner industry label)
  Reconcile note: emp counts differ (500 vs 510). Neither has enrichment. Recommend manual confirmation of headcount before merge.

CLUSTER 2 — domain: globex.io
  C-0A092933   industry='SaaS'        emp=200   hq='US'
  C-0A092934   industry='Technology'  emp=200   hq='US'
  Survivor: C-0A092933 (lower alias number; 'SaaS' is more specific than 'Technology')
  Reconcile note: all other fields agree. Industry differs — pick 'SaaS' (more granular) unless your taxonomy doesn't support it.

No name-variant clusters were detectable: company_alias values are opaque IDs (C-XXXXXX), and no company_name field exists. Name-based dedup is impossible with the current schema. Recommend adding a company_name field to future extracts.

════════════════════════════════════════════════════════
3. NORMALIZATION ISSUES (data present but non-standard)
════════════════════════════════════════════════════════

INDUSTRY — 10 rows use non-standard labels that should normalize:
  'tech'     → 'Technology'   (C-66D1FC, C-44EA29, C-60C75F, C-0A092932)
  'Tech '    → 'Technology'   (C-425E2A, C-BA969B, C-93C8BF, C-C9BB20)  [trailing space]
  'health care' → 'Healthcare' (C-7BBDFA, C-50D386)

HQ COUNTRY — 18 rows use abbreviations instead of full names:
  'US'  → 'United States'   (10 rows)
  'USA' → 'United States'   (6 rows)
  'UK'  → 'United Kingdom'  (3 rows)

These are not missing values, but they break grouping, filtering, and joins. Recommend a one-time normalization pass plus a validation rule on CRM picklist fields.

════════════════════════════════════════════════════════
4. INVALID EMAILS AND DOMAIN MISMATCHES
════════════════════════════════════════════════════════

INVALID (malformed — missing domain after @):
  CT-0010   C-66D1FC    email='user0@'
  CT-0080   C-92D97D    email='user0@'
  CT-0081   C-92D97D    email='user1@'
  CT-0192   C-425E2A    email='user2@'

DOMAIN MISMATCH (email domain != company domain):
  CT-0011   C-66D1FC    email_domain=other-domain.com    company_domain=66d1fc.com

Total: 5 contacts with email data quality issues (4 malformed + 1 mismatch), out of 52 (9.6%).

No duplicate email addresses were found across contacts.

════════════════════════════════════════════════════════
5. ENRICHMENT CROSS-REFERENCE
════════════════════════════════════════════════════════

5a. COMPANIES WITH NO ENRICHMENT ROW (9 of 34)

  C-BA969B    ba969b.com     emp=50    hq=US
  C-332637    332637.com     emp=1500  hq=UK
  C-93C8BF    93c8bf.com     emp=      hq=Canada
  C-EE9FFB    ee9ffb.com     emp=1500  hq=
  C-C9BB20    c9bb20.com     emp=900   hq=UK
  C-0A092931  acme-corp.com  emp=500   hq=US
  C-0A092932  acme-corp.com  emp=510   hq=USA
  C-0A092933  globex.io      emp=200   hq=US
  C-0A092934  globex.io      emp=200   hq=US

These cannot be enriched from ZoomInfo. Note that both duplicate clusters (acme-corp.com, globex.io) have no enrichment — resolving duplicates will also reduce the enrichment gap from 9 to 7 companies.

5b. FILLABLE FROM ENRICHMENT (CRM empty, ZI has value)

8 companies can get employee_count filled from enrichment:

  C-EC3025   ec3025.com    employee_count ← 400
  C-96039F   96039f.com    employee_count ← 400
  C-44EA29   44ea29.com    employee_count ← 400
  C-D04904   d04904.com    employee_count ← 400
  C-B23205   b23205.com    employee_count ← 400
  C-60C75F   60c75f.com    employee_count ← 400
  C-7BBDFA   7bbdfa.com    employee_count ← 400
  C-50D386   50d386.com    employee_count ← 400

No industry or hq_country fields can be filled — wherever CRM is empty, ZI is also empty for those fields.

5c. CANNOT FILL (both CRM and ZI empty)

  C-2D1F1B   2d1f1b.com    hq_country
  C-D73B89   d73b89.com    hq_country
  C-44EA29   44ea29.com    hq_country
  C-D04904   d04904.com    hq_country
  C-2C60E5   2c60e5.com    hq_country

These 5 companies have no hq_country in either source. A value cannot be invented. Recommend manual enrichment or a third-party lookup.

5d. DISAGREEMENTS (CRM and ZI both populated, values differ)

Employee count: 0 disagreements. Where both sources have a value, they agree exactly.

HQ country: 0 real disagreements. 10 format-only differences (CRM uses 'US'/'USA', ZI uses 'United States'; same actual country). Recommend adopting 'United States' / 'United Kingdom' / 'Canada' as the standard and normalizing both sources.

Industry: 12 rows flagged, but they fall into two categories:

  REAL DISAGREEMENT (10 rows) — CRM says 'Technology'/'tech', ZI says 'Computer Software':
    C-66D1FC   66d1fc.com    CRM='tech'          ZI='Computer Software'
    C-EC3025   ec3025.com    CRM='Technology'    ZI='Computer Software'
    C-44EA29   44ea29.com    CRM='tech'          ZI='Computer Software'
    C-92D97D   92d97d.com    CRM='Technology'    ZI='Computer Software'
    C-D04904   d04904.com    CRM='Technology'    ZI='Computer Software'
    C-77A95A   77a95a.com    CRM='Technology'    ZI='Computer Software'
    C-AA8DDA   aa8dda.com    CRM='Technology'    ZI='Computer Software'
    C-B25F40   b25f40.com    CRM='Technology'    ZI='Computer Software'
    C-60C75F   60c75f.com    CRM='tech'          ZI='Computer Software'
    C-425E2A   425e2a.com    CRM='Tech'          ZI='Computer Software'

    Recommendation: These are taxonomy-level, not record-level errors. ZoomInfo uses a more granular sub-industry ('Computer Software') while CRM uses the broader 'Technology'. Recommend mapping 'Computer Software' to 'Technology' in your CRM industry picklist (or adding 'Computer Software' as a sub-industry field). Do not overwrite CRM values per-row — fix the mapping once.

  FORMAT-ONLY (2 rows) — both CRM and ZI use the non-standard 'health care':
    C-7BBDFA   7bbdfa.com    CRM='health care'   ZI='health care'
    C-50D386   50d386.com    CRM='health care'   ZI='health care'

    Not a disagreement — both sources agree but both use the non-standard form. Normalize to 'Healthcare' in both.

════════════════════════════════════════════════════════
6. ADDITIONAL DATA QUALITY ISSUES
════════════════════════════════════════════════════════

COMPANIES WITH NO CONTACTS (14 of 34 = 41.2%):
  C-2C60E5, C-B97B4E, C-2D7423, C-7BBDFA, C-50D386, C-BA969B, C-332637, C-93C8BF, C-EE9FFB, C-C9BB20, C-0A092931, C-0A092932, C-0A092933, C-0A092934

Over 40% of companies have zero contacts. If any of these have open deals, those deals have no champion, no economic buyer, and no path to engagement. This is a severe coverage gap.

TITLE/PERSONA LOGIC MISMATCH (2 contacts):
  CT-0020   C-C6FE92    title=CTO    persona=HR admin    ← CTO is not an HR admin
  CT-0130   C-B23205    title=CTO    persona=HR admin    ← CTO is not an HR admin

These personas appear mis-tagged. A CTO is typically an economic buyer or technical champion, not HR admin. Recommend manual review.

MISSING TITLE (13 contacts):
  CT-0000, CT-0022, CT-0072, CT-0080, CT-0081, CT-0092, CT-0120, CT-0121, CT-0122, CT-0132, CT-0141, CT-0162, CT-0170

MISSING PERSONA (16 contacts):
  CT-0000, CT-0022, CT-0041, CT-0060, CT-0070, CT-0072, CT-0081, CT-0082, CT-0092, CT-0110, CT-0132, CT-0162, CT-0171, CT-0172, CT-0180, CT-0181

Note: 3 contacts are missing BOTH title and persona (CT-0000, CT-0022, CT-0072, CT-0092, CT-0132, CT-0162 — 6 actually). These are effectively unclassified contacts.

════════════════════════════════════════════════════════
7. TOP 10 FIXES (PRIORITIZED BY IMPACT)
════════════════════════════════════════════════════════

Because no deals.csv was provided, pipeline dollar amounts cannot be calculated. The ranking below is by number of records affected × severity of downstream impact (forecasting, segmentation, routing). If deal data is supplied, this list should be re-ranked by sum of deal amount per affected company.

 1. OBTAIN AND AUDIT deals.csv
    Affects: entire pipeline (unknown number of deals, unknown $ amount)
    The task requires owner, stage, amount, close date, why-buys on deals. None of this exists in the extract. Without it, pipeline completeness and the dollar-weighted fix ranking are impossible. This is the #1 blocker.

 2. MERGE DUPLICATE COMPANY CLUSTERS
    Affects: 4 company rows (2 clusters: acme-corp.com, globex.io)
    Survivors: C-0A092931 (acme-corp.com), C-0A092933 (globex.io)
    Impact: double-counted companies inflate TAM, distort segment counts, and create split account ownership. If any deals are attached to the non-survivor aliases, they must be re-parented to the survivor before merge.

 3. FILL 8 MISSING employee_count VALUES FROM ENRICHMENT
    Affects: C-EC3025, C-96039F, C-44EA29, C-D04904, C-B23205, C-60C75F, C-7BBDFA, C-50D386
    Source: zoominfo_enrichment.csv (all show 400)
    Impact: employee_count drives segmentation (SMB/mid-market/enterprise), routing rules, and ICP matching. 8 of 9 missing values can be filled immediately. The 9th (C-93C8BF) has no enrichment and cannot be filled.

 4. FIX 4 MALFORMED EMAILS
    Affects: CT-0010 (C-66D97D), CT-0080 (C-92D97D), CT-0081 (C-92D97D), CT-0192 (C-425E2A)
    Issue: emails end with '@' and no domain — undeliverable
    Impact: these contacts cannot be emailed, breaking sequences and cadences. 2 of 3 contacts at C-92D97D are affected (only CT-0082 has a valid email), meaning that account is nearly unreachable.

 5. RESOLVE 1 DOMAIN MISMATCH
    Affects: CT-0011 (C-66D1FC)
    Issue: email is user1@other-domain.com but company domain is 66d1fc.com
    Impact: email may bounce or route to a wrong entity. Confirm whether this contact genuinely uses a different email domain (e.g., personal email, subsidiary) or whether the email is wrong.

 6. CLASSIFY 16 CONTACTS MISSING PERSONA
    Affects: 16 of 52 contacts (30.8%) — listed in §6
    Impact: persona drives deal-threading (economic buyer, champion, HR admin, IT security, finance). Without it, buying-committee mapping is incomplete and deals may stall without the right stakeholder identified.

 7. FILL 13 CONTACTS MISSING TITLE
    Affects: 13 of 52 contacts (25.0%) — listed in §6
    Impact: title informs persona inference, routing, and personalization. 6 contacts are missing both title and persona (effectively blank contacts).

 8. NORMALIZE INDUSTRY PICKLIST
    Affects: 10 companies with non-standard labels ('tech', 'Tech ', 'health care') + 10 CRM/ZI taxonomy disagreements ('Technology' vs 'Computer Software')
    Impact: breaks industry filtering, cohort analysis, and ICP matching. Fix the picklist once, then normalize all values. Do not resolve 'Technology' vs 'Computer Software' per-row — it is a taxonomy mapping issue.

 9. NORMALIZE HQ COUNTRY FORMAT
    Affects: 18 companies using 'US'/'USA'/'UK' instead of full names + 10 CRM/ZI format mismatches
    Impact: breaks geographic grouping and territory assignment. Adopt 'United States' / 'United Kingdom' / 'Canada' as standard, normalize both CRM and enrichment.

10. RESOLVE 5 PERMANENTLY MISSING hq_country VALUES
    Affects: C-2D1F1B, C-D73B89, C-44EA29, C-D04904, C-2C60E5 (both CRM and ZI empty)
    Impact: cannot be filled from current data. These 5 companies cannot be assigned to a geographic territory. Requires manual lookup or a third enrichment source.

════════════════════════════════════════════════════════

Summary: the extract has good structural completeness (100% on keys/domains) but significant field-level gaps (employee_count 73.5%, contact title 75.0%, persona 69.2%), 2 duplicate clusters, 5 email data quality issues, 10 non-standard industry labels, 18 non-standard country labels, and 9 companies with no enrichment coverage. The absence of deals.csv is the dominant gap — supply it and I will re-rank the top 10 fixes by actual pipeline dollars at stake.
deal-intelligence
closed-lost-classificationPASS 1.00

? The Question

Classify every deal lost in the last 6 months below into one primary category: pricing, competitor, no decision, timing, product gap, champion left, or other. Also assign a side (Bonusly, buyer, unknown). Then summarize: category counts, the side split, the number of deals where the structured closed_lost_tag clearly disagrees with the free-text reason, and the two patterns most worth acting on. Base everything only on the tag and free-text provided.
cost $0.0882 · 582s · in 5,106 / out 26,477 tok
✓ total_90✓ competitor_top✓ category_counts_present✓ side_split✓ two_patterns

✓ Correct Answer (ground truth)

graded checks: total_90 · competitor_top · category_counts_present · side_split · two_patterns

Model's actual answer

show the model's full answer
## CLOSED-LOST CLASSIFICATION (90 DEALS)

---

### FULL DEAL TABLE

| # | Deal | Category | Side | Tag-Reason Agree? |
|---|------|----------|------|-------------------|
| 1 | DB0AAC | timing | buyer | Y |
| 2 | F7F635 | competitor | unknown | Y |
| 3 | AC944F | no decision | buyer | Y |
| 4 | 214060 | no decision | buyer | Y |
| 5 | 91A056 | timing | buyer | Y |
| 6 | 29326C | timing | buyer | Y |
| 7 | 5DB9B0 | other | unknown | Y |
| 8 | 831B7B | timing | buyer | Y |
| 9 | F97C37 | competitor | Bonusly | Y |
| 10 | 13E9CF | no decision | buyer | **N** — tag includes "Cost"; reason says "Not a budget issue" |
| 11 | 39E25C | timing | buyer | Y |
| 12 | 7ED004 | pricing | buyer | Y |
| 13 | 21B045 | no decision | buyer | Y |
| 14 | B3ABED | timing | buyer | Y |
| 15 | 422BA6 | competitor | Bonusly | Y |
| 16 | ED9AE7 | no decision | buyer | **N** — tag "Lost DM"; reason = timing, budget, authority |
| 17 | 988493 | no decision | buyer | Y |
| 18 | 381C8C | competitor | unknown | **N** — tag "Competitor"; free text says only "not moving forward," no competitor named |
| 19 | F308CA | no decision | buyer | Y |
| 20 | F1E8A6 | competitor | unknown | **N** — tag "Competitor"; free text says only "not moving forward" |
| 21 | B6AC09 | timing | buyer | Y |
| 22 | 70F704 | no decision | buyer | **N** — tag "Lost DM"; reason = narrow use case + MIA |
| 23 | E6E80A | timing | buyer | Y |
| 24 | B038F0 | timing | buyer | Y |
| 25 | 4664E1 | no decision | buyer | Y |
| 26 | 175756 | timing | buyer | Y |
| 27 | E74A73 | no decision | buyer | Y |
| 28 | DDAB52 | competitor | Bonusly | Y |
| 29 | ACE061 | competitor | unknown | Y |
| 30 | BB78F3 | timing | buyer | Y |
| 31 | D48E0B | no decision | buyer | Y |
| 32 | 15DA99 | timing | buyer | Y |
| 33 | F4AF5D | timing | buyer | Y |
| 34 | 79B7A1 | timing | buyer | Y |
| 35 | 583ADB | no decision | buyer | Y |
| 36 | 8E27DA | product gap | Bonusly | Y |
| 37 | 2D2F8D | competitor | unknown | Y |
| 38 | E0441F | no decision | buyer | Y |
| 39 | 7CB44D | no decision | buyer | Y |
| 40 | 0F96AA | competitor | unknown | Y |
| 41 | 1BCA50 | pricing | Bonusly | **N** — tag "Competitor"; reason says "mostly about the budget and gift card details" |
| 42 | 7CC678 | competitor | unknown | Y |
| 43 | FAC17C | no decision | buyer | **N** — tag "Lost DM"; reason = couldn't get IT Director approval (DM still there, just not approving) |
| 44 | 242273 | product gap | Bonusly | **N** — tag "Competitor"; reason = internal points currency / onsite spend (product capability gap) |
| 45 | 50E5D8 | no decision | buyer | Y |
| 46 | A2C349 | competitor | unknown | Y |
| 47 | 9F176A | timing | buyer | Y |
| 48 | 7B2236 | pricing | Bonusly | **N** — tag "Doing nothing/Not a priority/Cost"; reason = "simpler and cheaper" (active pricing objection, not doing nothing) |
| 49 | AFA56C | no decision | buyer | Y |
| 50 | C7156E | competitor | unknown | Y |
| 51 | C33D91 | pricing | buyer | Y |
| 52 | 9048EB | product gap | Bonusly | **N** — tag "MIA"; reason explicitly says "bad fit" + "multiple feature gaps" |
| 53 | 5E64CE | timing | buyer | **N** — tag "Doing nothing/Not a priority/Cost"; reason = Nectar contract lock-in through Oct 2027, plans to switch to Bonusly |
| 54 | 8A0992 | competitor | buyer | Y |
| 55 | D0C698 | competitor | buyer | Y |
| 56 | 69CF3D | timing | buyer | Y |
| 57 | ECBF89 | timing | buyer | Y |
| 58 | 3618CC | product gap | Bonusly | **N** — tag "Lost DM"; reason = "Wanted Surveys" |
| 59 | EECC02 | competitor | unknown | Y |
| 60 | 5AD03E | product gap | Bonusly | **N** — tag "Competitor"; reason = "Wanted more defined budget access" |
| 61 | D1A623 | timing | buyer | Y |
| 62 | 413C56 | timing | buyer | **N** — tag "Doing nothing/Not a priority/Cost"; reason = back-to-school season, CEO not ready (temporal blocker) |
| 63 | 47F1A1 | competitor | buyer | Y |
| 64 | BF2A98 | competitor | buyer | Y |
| 65 | 2A292B | no decision | buyer | Y |
| 66 | D1AABF | no decision | buyer | Y |
| 67 | FEDBCB | timing | buyer | **N** — tag "Doing nothing/Not a priority/Cost"; reason = wants to reconnect closer to end of year |
| 68 | 1E7DA9 | competitor | unknown | Y |
| 69 | 2BBA21 | no decision | buyer | Y |
| 70 | 286F9C | competitor | Bonusly | Y — "not really a good fit for us" |
| 71 | 7FBAC6 | no decision | buyer | Y |
| 72 | 369281 | competitor | buyer | Y |
| 73 | 386F6E | no decision | buyer | Y |
| 74 | 9FCD0D | competitor | buyer | Y |
| 75 | 55867E | timing | buyer | Y |
| 76 | DAFB82 | timing | buyer | **N** — tag "Lost-Budget/Price"; reason = budget needed elsewhere, not budgeted till 2028, loves Bonusly, will loop back (timing, not price objection) |
| 77 | 2FEDDB | timing | buyer | **N** — tag "Doing nothing/Not a priority/Cost"; reason = "Unsure on timing" |
| 78 | 64B19A | competitor | buyer | Y |
| 79 | 3F86A0 | no decision | buyer | Y |
| 80 | 096750 | no decision | buyer | Y |
| 81 | F325A5 | champion left | buyer | **N** — tag "Lost DM"; reason = layoffs + change in leadership (organizational disruption, not a single DM departure) |
| 82 | ABD14C | no decision | buyer | Y |
| 83 | 79E61A | no decision | buyer | Y |
| 84 | 8A119B | pricing | buyer | Y |
| 85 | AE7C4E | no decision | buyer | Y |
| 86 | DAB4F1 | no decision | buyer | Y |
| 87 | B4B50F | no decision | buyer | Y |
| 88 | 981AD4 | product gap | Bonusly | Y |
| 89 | DC77FE | product gap | Bonusly | **N** — tag "Competitor"; reason = needed customization (label points as dollars), and explicitly says price was NOT the factor |
| 90 | 5885B9 | no decision | buyer | Y |

---

### CATEGORY COUNTS

| Category | Deals | % of 90 |
|----------|------:|--------:|
| no decision | 30 | 33.3% |
| timing | 24 | 26.7% |
| competitor | 22 | 24.4% |
| product gap | 7 | 7.8% |
| pricing | 5 | 5.6% |
| champion left | 1 | 1.1% |
| other | 1 | 1.1% |

Arithmetic: 30 + 24 + 22 + 7 + 5 + 1 + 1 = 90.

---

### SIDE SPLIT

| Side | Deals | % of 90 |
|------|------:|--------:|
| buyer | 65 | 72.2% |
| Bonusly | 13 | 14.4% |
| unknown | 12 | 13.3% |

Arithmetic: 65 + 13 + 12 = 90.

**Bonusly-side deals** (13): F97C37 (product breadth), 422BA6 (ADP integration), DDAB52 (FX/value), 8E27DA (swag-only need), 1BCA50 (gift-card pricing), 242273 (internal points/onsite spend), 7B2236 (too expensive + complex), 9048EB (feature gaps), 3618CC (surveys), 5AD03E (budget access controls), 286F9C (product-fit), 981AD4 (UI + UK localization), DC77FE (customization/point-labeling).

**Unknown-side deals** (12): all competitor-tagged deals where the free text gives no specific Bonusly deficiency or buyer circumstance (e.g., "went another direction," "selected another vendor," "nothing specific provided").

---

### TAG-REASON DISAGREEMENTS

**19 deals** where the structured closed_lost_tag clearly disagrees with the free-text reason:

| Deal | Tag | What the text actually says |
|------|-----|---------------------------|
| 13E9CF | Doing nothing/Not a priority/**Cost** | "Not a budget issue — R&R deprioritized" |
| ED9AE7 | Lost DM | "Timing, budget, authority" (no decision, not a lost person) |
| 381C8C | Competitor | "not going to be moving forward" (no competitor named) |
| F1E8A6 | Competitor | "not going to be moving forward" (no competitor named) |
| 70F704 | Lost DM | "only looking to automate anniversary awards + MIA" |
| 1BCA50 | Competitor | "mostly about the budget and gift card details" |
| FAC17C | Lost DM | "couldn't get final approval from IT Director" (DM still there) |
| 242273 | Competitor | internal points currency / onsite spend (product gap) |
| 7B2236 | Doing nothing/Not a priority/Cost | "simpler and cheaper" (active pricing objection) |
| 9048EB | MIA | "bad fit" + "multiple feature gaps" (product gap) |
| 5E64CE | Doing nothing/Not a priority/Cost | Nectar contract lock-in; plans to switch to Bonusly (timing) |
| 3618CC | Lost DM | "Wanted Surveys" (product gap) |
| 5AD03E | Competitor | "Wanted more defined budget access" (product gap) |
| 413C56 | Doing nothing/Not a priority/Cost | "Back to school is priority, CEO not ready" (timing) |
| FEDBCB | Doing nothing/Not a priority/Cost | "Wants to reconnect closer to end of year" (timing) |
| DAFB82 | Lost-Budget/Price | "budget needed elsewhere, not budgeted until 2028, loves Bonusly, will loop back" (timing) |
| 2FEDDB | Doing nothing/Not a priority/Cost | "Unsure on timing" (timing) |
| F325A5 | Lost DM | "Layoffs and Change in Leadership" (champion left / org disruption) |
| DC77FE | Competitor | needed customization; "price wasn't a factor" (product gap) |

---

### TWO PATTERNS MOST WORTH ACTING ON

**1. Product gaps are silently inflating competitor and MIA counts — and they are specific.**

At least 7 deals (242273, 3618CC, 5AD03E, DC77FE, 981AD4, 9048EB, 8E27DA) were lost to recurring, named product shortcomings: surveys, internal-points-currency + onsite-spend, defined budget-access controls, point-labeling customization, UK localization, and a swag-only path. The current tag system files most of these as "Competitor" or "MIA," making the product-gap problem look smaller than it is. The 19 tag-reason disagreements alone reclassify 5 deals into product gap that the tags hide. Action: build a deduped product-gap register from the free text (not the tags) and rank by cumulative lost ACV to feed the roadmap.

**2. Timing losses with explicit re-engagement dates have no visible follow-up structure.**

24 deals (27% of all losses) are timing-classified, and many cite concrete windows: "early 2027," "Q2 next year," "end of year," "when Nectar contract expires Oct 2027." These represent recoverable pipeline worth an estimated sum in the six figures (individual deals range $2K–$72K). But the tag system also buries at least 5 more timing deals inside "Doing nothing" and "Budget/Price" tags (5E64CE, 413C56, FEDBCB, DAFB82, 2FEDDB). Without a CRM-based re-engagement calendar keyed to the dates prospects actually named, these deals will silently expire. Action: extract every "reconnect [date]" from free text, create dated tasks, and build a quarterly nurture cadence for timing-classified accounts.
pipeline-tieringPASS 1.00

? The Question

Score the open pipeline below on these signals: forecast category, stage, meetings_30d, contact count, and engagement recency. Assign exactly one tier per deal: LOCK, ACTION, BUILD, REVIVE, WATCH, or RISKY (RISKY = the forecast category disagrees with the engagement evidence). Never tier a deal LOCK if it has zero meetings_30d. Do NOT list every deal. Return ONLY a compact JSON object with exactly these keys: - "tier_counts": object mapping each tier to the number of deals in it (must sum to the total deal count), - "tier_examples": object mapping each tier to up to 3 example deal aliases from the data, - "risky_deals": array of the deal aliases you tiered RISKY, - "lock_violations": number of deals tiered LOCK that have zero meetings_30d (should be 0), - "pipeline_shape": one short paragraph on the overall pipeline shape. Note: inbound_emails_30d is 0 for every row due to a data defect, so treat meetings_30d as the inbound signal.
cost $0.1769 · 434s · in 62,289 / out 36,560 tok
✓ json_returned✓ tier_vocabulary✓ tier_counts_sum✓ risky_deals_listed✓ no_lock_violation_claimed

✓ Correct Answer (ground truth)

total: 156
graded checks: json_returned · tier_vocabulary · tier_counts_sum · risky_deals_listed · no_lock_violation_claimed

Model's actual answer

show the model's full answer
```json
{
  "tier_counts": {
    "LOCK": 5,
    "ACTION": 48,
    "BUILD": 50,
    "REVIVE": 19,
    "WATCH": 12,
    "RISKY": 22
  },
  "tier_examples": {
    "LOCK": ["Deal-25F752", "Deal-D348E1", "Deal-C26D20"],
    "ACTION": ["Deal-E53952", "Deal-5408B0", "Deal-9AAE5F"],
    "BUILD": ["Deal-6787C2", "Deal-C9C286", "Deal-332637"],
    "REVIVE": ["Deal-0660B4", "Deal-2D1F1B", "Deal-66D1FC"],
    "WATCH": ["Deal-584EE5", "Deal-C6D97A", "Deal-FD9F4E"],
    "RISKY": ["Deal-B7EBD1", "Deal-523604", "Deal-7BBDFA"]
  },
  "risky_deals": [
    "Deal-B7EBD1", "Deal-BA571A", "Deal-523604", "Deal-01E193",
    "Deal-C1FA6D", "Deal-7BBDFA", "Deal-93C8BF", "Deal-1CCE5C",
    "Deal-927338", "Deal-4A13AD", "Deal-690476", "Deal-635B8E",
    "Deal-DCA846", "Deal-819506", "Deal-BA3DDC", "Deal-7599B8",
    "Deal-03BA69", "Deal-117863", "Deal-F17780", "Deal-8BA24E",
    "Deal-2B39B0", "Deal-8FDCD2"
  ],
  "lock_violations": 0,
  "pipeline_shape": "156 deals total. The pipeline is middle-heavy: LOCK is only 5 deals (3.2%) — late-stage COMMIT/BEST_CASE with active meetings and recent contact. ACTION (48, 30.8%) and BUILD (50, 32.1%) together account for 63%, representing deals with recent engagement but insufficient meeting velocity to lock. REVIVE is 19 deals (12.2%) with last contact older than 14 days. WATCH is 12 (7.7%) — borderline deals with some email activity but no meetings and ambiguous signals. The standout concern is 22 RISKY deals (14.1%): 14 are understated PIPELINE deals with 3+ meetings_30d that should be re-forecast upward, and 8 are overstated COMMIT/BEST_CASE deals with zero meetings, minimal emails, and stale contact (some last touched in July). The 8:14 split between over- and under-statement suggests both sandbagging on hot early-stage deals and wishful forecasting on stalled late-stage ones. With inbound_emails_30d defectively zero across all rows, meetings_30d is the sole inbound signal — deals flagged RISKY on the understated side (high meeting count, low forecast) are the highest-confidence calls."
}
```
call-transcript-extractionPASS 1.00

? The Question

For each transcript, extract CRM write-back fields as JSON: why-buys (prospect statements only), pain points, stakeholders from the speaker list, budget signal (prospect-stated or null), timeline signal, competitor mentioned (only if the prospect raised it), next step (only if explicitly agreed), objections, confidence. Never fill a field from what the rep said or hoped.
cost $0.0125 · 70s · in 1,927 / out 2,561 tok
✓ json_emitted✓ budget_signal✓ null_discipline✓ stakeholders✓ competitor_prospect_only

✓ Correct Answer (ground truth)

rep: Alex Franklin
per_transcript: {"TX-001": {"why_buys": ["automating anniversary and birthday awards"], "pain_points": ["HR team of three cannot keep up with it manually", "track everything in a spreadsheet, and people slip through
rep_sourced_nulls: {"TX-003": ["budget_signal \u2014 only the rep mentioned pricing/budget"], "TX-004": ["next_step \u2014 rep proposed, prospect said 'no promises'"], "TX-006": ["budget_signal \u2014 rep decoy, prospec
allowed_new: TX-001, TX-002, TX-003, TX-004, TX-005, TX-006, Achievers, Bucketlist, Nectar
graded checks: json_emitted · budget_signal · null_discipline · stakeholders · competitor_prospect_only

Model's actual answer

show the model's full answer
```json
{
  "Deal-CFE7F4": {
    "why_buys": [
      "Automating anniversary and birthday awards — HR team of 3 cannot keep up manually",
      "Tracking in spreadsheet causes people to slip through the cracks"
    ],
    "pain_points": [
      "Manual tracking in spreadsheet, people slipping through cracks",
      "HR team too small to manage awards manually"
    ],
    "stakeholders": [
      "Prospect (VP People)",
      "Prospect (HR Admin)"
    ],
    "budget_signal": "$40k earmarked for engagement tools this fiscal year",
    "timeline_signal": "Live before open enrollment in November",
    "competitor_mentioned": "Achievers — looked at last year, too heavy for team their size",
    "next_step": "Security review on September 12",
    "objections": [
      "Need SSO and audit logs for IT to sign off"
    ],
    "confidence": "High"
  },
  "Deal-70BB30": {
    "why_buys": [
      "Tie recognition to retention for hourly workforce",
      "Regretted turnover over 30% for hourly employees"
    ],
    "pain_points": [
      "Regretted turnover over 30% for hourly workforce"
    ],
    "stakeholders": [
      "Prospect (Head of Total Rewards)",
      "Prospect (CFO)"
    ],
    "budget_signal": "$25k pilot budget approved this quarter",
    "timeline_signal": "Decision by end of September",
    "competitor_mentioned": null,
    "next_step": "Send pilot agreement, route to legal this week",
    "objections": [
      "Workday integration must be rock solid (CFO's one condition)"
    ],
    "confidence": "High"
  },
  "Deal-530B50": {
    "why_buys": [
      "Make recognition visible across 12 retail locations"
    ],
    "pain_points": [
      "Store managers have zero budget autonomy for on-the-spot recognition"
    ],
    "stakeholders": [
      "Prospect (People Ops Manager)"
    ],
    "budget_signal": null,
    "timeline_signal": "No rush until Q1",
    "competitor_mentioned": "Bucketlist — CEO used at last company and liked it",
    "next_step": "Schedule call with CEO, prospect to send two times",
    "objections": [
      "CEO has to be sold first, she decides anything people-related"
    ],
    "confidence": "Low"
  },
  "Deal-180D02": {
    "why_buys": [
      "Consolidate three separate recognition tools into one"
    ],
    "pain_points": [
      "Paying for three tools that don't talk to HRIS",
      "Three separate tools creating fragmentation"
    ],
    "stakeholders": [
      "Prospect (VP People)",
      "Prospect (IT Security Lead)"
    ],
    "budget_signal": "Under $15k annually can approve without going to the board",
    "timeline_signal": null,
    "competitor_mentioned": null,
    "next_step": null,
    "objections": [
      "Procurement cycle runs 6-8 weeks minimum",
      "Last vendor security review took 3 months (IT lead hesitation)"
    ],
    "confidence": "Low"
  },
  "Deal-F8767A": {
    "why_buys": [
      "Automate service milestones",
      "Analytics on recognition equity across departments"
    ],
    "pain_points": [
      "Night-shift teams feel invisible, engagement scores 20 points lower",
      "Exec team skeptical after failed rollout 2 years ago"
    ],
    "stakeholders": [
      "Prospect (HR Director)",
      "Prospect (People Ops Coordinator)"
    ],
    "budget_signal": "$12k approved under engagement line",
    "timeline_signal": "Running before January all-hands",
    "competitor_mentioned": "Nectar — mid-pilot, need to beat that experience",
    "next_step": "Present to exec team on October 2",
    "objections": [
      "Mid-pilot with Nectar, need to beat that experience",
      "Exec team skeptical after failed rollout 2 years ago"
    ],
    "confidence": "Medium"
  },
  "Deal-EE195F": {
    "why_buys": [
      "Cut admin time on service awards"
    ],
    "pain_points": [
      "Personally spends 5 hours/month ordering and shipping plaques"
    ],
    "stakeholders": [
      "Prospect (HR Manager)"
    ],
    "budget_signal": null,
    "timeline_signal": "Q1 start realistic, fiscal year resets in February",
    "competitor_mentioned": null,
    "next_step": "Send one-page overview, prospect will forward to COO this week",
    "objections": [
      "COO usually prefers building things in-house"
    ],
    "confidence": "Medium"
  }
}
```

**Confidence rationale (arithmetic):**

| Deal | Budget stated? | Timeline stated? | Next step locked? | Active competitor? | Unresolved objection? | Score (1pt each positive, -1 each negative) | Confidence |
|---|---|---|---|---|---|---|---|
| CFE7F4 | Y | Y | Y | N (evaluated & dismissed) | 1 (SSO/audit — solvable) | 3 | High |
| 70BB30 | Y | Y | Y | N | 1 (Workday — standard) | 3 | High |
| 530B50 | N | Y | Y | Y (Bucketlist, liked by decision-maker) | 1 (CEO gate) | 0 | Low |
| 180D02 | Y | N | N | N | 2 (procurement + security delays) | -1 | Low |
| F8767A | Y | Y | Y | Y (Nectar, active pilot) | 2 (Nectar + exec skepticism) | 1 | Medium |
| EE195F | N | Y | Y | N | 1 (COO prefers in-house) | 1 | Medium |
next-to-close0.75

? The Question

Which 3 deals are closest to signature right now? Use the CRM data and the Slack thread; Slack often has the real status before the CRM does. For each: deal, amount, why it is close, what is left. Under 250 words.
cost $0.0065 · 25s · in 1,265 / out 792 tok
✓ exactly_three✗ top3_named✓ slack_first✓ blocked_excluded

✓ Correct Answer (ground truth)

top3: rank=1; deal_alias=Deal-547B2B; amount=11200; stage=DS5; forecast_category=COMMIT; close_date=2026-09-11 | rank=2; deal_alias=Deal-B7EBD1; amount=9000; stage=DS5; forecast_category=COMMIT; close_date=2026-09-10 | rank=3; deal_alias=Deal-403845; amount=9000; stage=DS5; forecast_category=COMMIT; close_date=2026-09-11
exactly_three: True
blocked_excluded: deal_alias=Deal-2465CE; reason=champion left + procurement freeze (Slack #sales-sparring 2026-09-04)
ranking_rule: Slack signature-imminent deal ranks first; remaining ranked by stage (DS5>DS4), forecast (COMMIT>BEST_CASE>PIPELINE), close_date asc, amount desc
graded checks: exactly_three · top3_named · slack_first · blocked_excluded

Model's actual answer

show the model's full answer
**1. Deal-547B2B — $11,200 (DS5/COMMIT, close Sep 11)**
Closest to signature. Slack: redlines came back clean, signing page is out, VP People confirmed signing tomorrow. Signature-imminent. Left: nothing material — awaiting ink.

**2. Deal-403845 — $9,000 (DS5/COMMIT, close Sep 11)**
Slack: order form is with their finance team. No blockers flagged. Left: finance sign-off on the order form.

**3. Deal-A2B47C — $6,360 (DS5/COMMIT, close Sep 11)**
Slack: "still warm, normal legal-review pace." No obstacles called out. Left: legal review to complete.

**Notable exclusion:** Deal-2465CE ($5,400) still shows DS5/COMMIT in CRM, but Slack confirms it is blocked — champion left, procurement froze new vendors, Dana pulled it from commit. Do not count it.

**Summary:** 547B2B is days (hours?) from signature; 403845 and A2B47C need one internal approval each. Combined pipeline at risk of slipping: $26,560 in the next 48 hours.
product-gap-detectionPASS 1.00

? The Question

Surface true product gaps raised by prospects in active deals. For each candidate: quote the prospect line, classify as real gap, plan gate, or rollout/enablement issue using the product docs excerpt, and name the deal and amount. Only real gaps go in the summary. Rep language does not count as prospect voice.
cost $0.0078 · 40s · in 796 / out 1,387 tok
✓ real_gaps✓ plan_gate_classified✓ enablement_classified✓ gap_summary

✓ Correct Answer (ground truth)

real_gaps: transcript_id=TG-001; deal_alias=Deal-EC3025; amount=62000; feature=SCIM user provisioning; quote=We need SCIM user provisioning — our security team will not approve any tool that requires manual account management.; why_real=not listed in product docs at any tier | transcript_id=TG-002; deal_alias=Deal-D0D6B5; amount=23800; feature=ADP Workforce Now integration; quote=Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us.; why_real=supported HRIS list is Workday/BambooHR/Gusto/Rippling only
plan_gates: transcript_id=TG-003; deal_alias=Deal-CFE7F4; amount=32000; feature=Custom report builder; tier_required=Enterprise; quote=I tried to build a custom report combining recognition frequency with turnover and couldn't — is that possible on our current plan?
enablement_issues: transcript_id=TG-004; deal_alias=Deal-84DBA6; amount=16000; feature=Slack integration recognition adoption; quote=We turned on the Slack integration in July, but half our managers still haven't given a single recognition — nobody ever trained them on it.; why_enablement=Slack integration exists on all plans; managers were never trained
excluded_rep_voice: transcript_id=TG-005; line=Full transparency — we don't have a native mobile app for hourly workers yet, though it's on the roadmap.; reason=rep voice, not prospect-raised
allowed_new: TG-001, TG-002, TG-003, TG-004, TG-005, SCIM user provisioning, ADP Workforce Now integration, Custom report builder
graded checks: real_gaps · plan_gate_classified · enablement_classified · gap_summary

Model's actual answer

show the model's full answer
## Candidate Analysis

**1. TG-001 / Deal-EC3025** — Prospect (IT Security Lead)
> "We need SCIM user provisioning — our security team will not approve any tool that requires manual account management."

- Product docs confirm: "SCIM user provisioning … NOT currently listed as supported capabilities."
- No tier offers SCIM — it simply doesn't exist in the product.
- **Classification: Real gap**
- Deal amount: not provided

---

**2. TG-002 / Deal-D0D6B5** — Prospect (HRIS Manager)
> "Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us."

- Product docs list HRIS integrations as Workday, BambooHR, Gusto, Rippling only. Confirms: "ADP Workforce Now integrations are NOT currently listed as supported capabilities."
- No tier offers ADP — it simply doesn't exist in the product.
- **Classification: Real gap**
- Deal amount: not provided

---

**3. TG-003 / Deal-CFE7F4** — Prospect (VP People)
> "I tried to build a custom report combining recognition frequency with turnover and couldn't — is that possible on our current plan?"

- Product docs: Custom report builder is available on Enterprise tier (not Core, not Pro).
- Feature exists; the prospect is gated by plan tier.
- **Classification: Plan gate** — not a product gap.

---

**4. TG-004 / Deal-84DBA6** — Prospect (People Ops Manager)
> "We turned on the Slack integration in July, but half our managers still haven't given a single recognition — nobody ever trained them on it."

- Slack integration is supported on all plans per docs. The feature works; the problem is no manager training.
- **Classification: Rollout/enablement issue** — not a product gap.

---

**5. TG-005 / Deal-36C33F** — Rep (Alex Franklin) speaks first
> "Full transparency — we don't have a native mobile app for hourly workers yet, though it's on the roadmap."

Prospect responds: "Good to know. The web version should be fine for our office staff for now."

- The mobile-app gap was introduced by the rep, not the prospect. Per the instruction, rep language does not count as prospect voice. The prospect accepted the workaround.
- **Classification: Not a prospect-raised gap** — excluded from summary.

---

## Summary — True Product Gaps

| # | Gap | Prospect Quote | Deal | Amount | Docs Evidence |
|---|---|---|---|---|---|
| 1 | SCIM user provisioning | "our security team will not approve any tool that requires manual account management" | Deal-EC3025 | not provided | Not listed as supported capability on any tier |
| 2 | ADP Workforce Now integration | "ADP anywhere in your integration list — that's a dealbreaker" | Deal-D0D6B5 | not provided | Not listed as supported HRIS; only Workday, BambooHR, Gusto, Rippling |

Two real gaps surface. Three candidates were excluded: one plan gate (custom report builder exists on Enterprise), one enablement issue (Slack works, no training), and one rep-initiated item the prospect didn't block on. Deal amounts are missing from the source data.
rep-performance
stale-pipeline-by-repPASS 1.00

? The Question

The snapshot date is 2026-09-05. Using the open deals and the engagements-by-deal table, list every open deal with no logged email, call, or meeting in the last 7 days. Compute recency from the engagements table's last_email/last_call/last_meeting fields, NOT from any date field on the deal (the deal's last_contacted_field is unreliable). For each stale deal give: deal alias, owner name, stage, amount, days since last contact. Group by owner, order by amount descending within each owner. Then state, per owner, how many stale deals they carry and the total stale amount. (Do not write outreach text.)
cost $0.1312 · 585s · in 40,182 / out 27,451 tok
✓ stale_recall_ge_80✓ grouped_by_owner✓ jake_top_amount✓ kevin_stale_amt✓ jenn_stale_amt✓ stale_count

✓ Correct Answer (ground truth)

stale_count: 65
jake_amt: 626243.0
kevin_amt: 252905.03
jenn_amt: 261645.0
top_alias: Deal-2D1F1B
top_amount: 240000.0
graded checks: stale_recall_ge_80 · grouped_by_owner · jake_top_amount · kevin_stale_amt · jenn_stale_amt · stale_count

Model's actual answer

show the model's full answer
STALE OPEN DEALS (no email/call/meeting in last 7 days as of 2026-09-05)
Recency computed from MAX(last_email, last_call, last_meeting) in the
engagements table. A deal is stale if its most recent contact falls before
2026-08-29 (i.e., 8+ days ago).

=== Bryce Harmon (owner_id 119337721) ===

Deal Alias       Stage     Amount   Days Since   Last Contact
Deal-2D1F1B     DS1       240,000       81       2026-06-16
Deal-66D1FC     DS1        99,000       16       2026-08-20
Deal-950043     DS1        70,000       19       2026-08-17
Deal-B23205     DS1        45,000       16       2026-08-20
Deal-7BBDFA     DS3        37,440       46       2026-07-21
Deal-332637     DS2        36,000        9       2026-08-27
Deal-1BEEBF     DS1        31,500       19       2026-08-17
Deal-C5658B     DS1        23,400       16       2026-08-20
Deal-40522D     DS3        21,000       19       2026-08-17
Deal-F0EBBB     DS3        11,400       24       2026-08-12
Deal-E25A09     DS1         6,000        9       2026-08-27
Deal-C9C286     DS2         5,502        9       2026-08-27
Deal-012CB1     DS1             1       23       2026-08-13

  13 stale deals | $626,243

=== Alex Franklin (owner_id 84342457) ===

Deal Alias       Stage     Amount   Days Since   Last Contact
Deal-CC08D1     DS1        24,000       16       2026-08-20
Deal-E73427     DS3        18,000       10       2026-08-26
Deal-885F45     DS2         9,300       12       2026-08-24
Deal-C2FF3C     DS1         8,316       10       2026-08-26
Deal-3EED2C     DS2         7,200       N/A      NEVER (no engagement row)
Deal-0D2F7A     DS3         5,100       12       2026-08-24
Deal-6C60D4     DS3         4,800       12       2026-08-24
Deal-13FEBD     DS2         4,680       12       2026-08-24
Deal-9D0060     DS3         3,840       12       2026-08-24
Deal-690476     DS2         3,600       18       2026-08-18
Deal-C6D97A     DS4         3,240        8       2026-08-28
Deal-EE195F     DS3         3,120        8       2026-08-28
Deal-278DEC     DS3         2,700        8       2026-08-28
Deal-635B8E     DS3         2,600       18       2026-08-18
Deal-6883F3     DS1         2,400       16       2026-08-20
Deal-4A13AD     DS3         2,160       26       2026-08-10
Deal-F67D31     DS2         1,800        8       2026-08-28
Deal-5FDCE4     DS3         1,600       12       2026-08-24
Deal-BA571A     DS4         1,080       18       2026-08-18

  19 stale deals | $109,536

=== Dana Mercer (owner_id 83155923) ===

Deal Alias       Stage     Amount   Days Since   Last Contact
Deal-44EA29     DS2        60,000       10       2026-08-26
Deal-E51FB7     DS2        43,875       12       2026-08-24
Deal-B42F46     DS1        27,000       19       2026-08-17
Deal-BA3DDC     DS3        23,400       15       2026-08-21
Deal-9DDE86     DS2        20,000       15       2026-08-21
Deal-215CCA     DS3        18,900       17       2026-08-19
Deal-5EED42     DS3        16,250       11       2026-08-25
Deal-57887A     DS2        15,000        8       2026-08-28
Deal-B7EBD1     DS5         9,000       16       2026-08-20
Deal-3974EB     DS4         9,000        8       2026-08-28
Deal-F40F04     DS2         8,100       15       2026-08-21
Deal-87DDD1     DS1         5,000       19       2026-08-17
Deal-F336B6     DS3         4,200       15       2026-08-21
Deal-0660B4     DS4         1,920       16       2026-08-20

  14 stale deals | $261,645

=== Cole Ingram (owner_id 83155924) ===

Deal Alias       Stage     Amount   Days Since   Last Contact
Deal-D04904     DS2        58,529       11       2026-08-25
Deal-B25F40     DS3        40,000        8       2026-08-28
Deal-813836     DS2        32,175       11       2026-08-25
Deal-1BA595     DS2        31,750       11       2026-08-25
Deal-CFE1E8     DS3        18,000       11       2026-08-25
Deal-CD47A6     DS2        12,168       11       2026-08-25
Deal-627646     DS3        11,193       11       2026-08-25
Deal-FF809F     DS2         7,781       11       2026-08-25
Deal-AF932D     DS2         7,225       11       2026-08-25
Deal-A71728     DS2         6,948       11       2026-08-25
Deal-8BC9F5     DS2         5,616       10       2026-08-26
Deal-175395     DS3         4,780       11       2026-08-25
Deal-481E24     DS3         4,140       10       2026-08-26
Deal-C7F9BF     DS2         3,360       11       2026-08-25
Deal-2F3A66     DS3         3,335       11       2026-08-25
Deal-342E96     DS2         2,700       24       2026-08-12
Deal-E568D5     DS3         1,875       11       2026-08-25
Deal-FD9F4E     DS5         1,330       10       2026-08-26

  18 stale deals | $252,905

=== Elena Sinclair (owner_id 701163055) ===

Deal Alias       Stage     Amount   Days Since   Last Contact
Deal-57FF13     DS1         2,100       N/A      NEVER (no engagement row)

  1 stale deal | $2,100

=== Farid Osman (owner_id 716654662) ===

  0 stale deals | $0

=======================================
OWNER SUMMARY (stale deals only)
---------------------------------------
Bryce Harmon .................  13 stale | $626,243
Alex Franklin .............  19 stale | $109,536
Dana Mercer ................  14 stale | $261,645
Cole Ingram ........  18 stale | $252,905
Farid Osman ................   0 stale | $0
Elena Sinclair .............   1 stale | $2,100
---------------------------------------
TOTAL .....................  65 stale | $1,252,429

Notes:
- "Days Since" = 2026-09-05 minus MAX(last_email, last_call, last_meeting).
- A deal is stale if that max date falls before 2026-08-29 (8+ days ago).
- Deal-3EED2C and Deal-57FF13 have no row in the engagements table at all;
  they are treated as never contacted and are unambiguously stale.
- The deal file's last_contacted_field was NOT used (per instruction); several
  deals show a recent last_contacted_field but no corresponding engagement
  activity (e.g., Deal-C6D97A shows last_contacted=2026-08-28 in the deal
  row, and the engagement table confirms last_email=2026-08-28 with no
  call/meeting since -- 8 days stale by engagement data).
activity-mix-vs-outcomePASS 1.00

? The Question

The snapshot date is 2026-09-05. For each sales rep (owner), compute the last-30-day totals of emails, calls, and meetings from the engagements table, and the number of deals that entered DS2 in the last 30 days (t_ds2 within 30 days of the snapshot). Report per rep: the activity mix as percentages (emails/calls/meetings share of that rep's total activities), total activities per DS2 entry, and rank the reps by that efficiency ratio (lowest activities per DS2 entry = most efficient). Name the most efficient rep and the highest-volume rep, and note if they differ. Do not attribute cause.
cost $0.2523 · 537s · in 84,029 / out 47,766 tok
✓ justin_ds2✓ justin_volume_high✓ efficient_named✓ mix_present✓ no_causal_claim

✓ Correct Answer (ground truth)

justin_ds2: 4
graded checks: justin_ds2 · justin_volume_high · efficient_named · mix_present · no_causal_claim

Model's actual answer

show the model's full answer
Here are the results.

SNAPSHOT: 2026-09-05 | 30-day window: 2026-08-06 to 2026-09-05 (inclusive)

DATA GAP: 2 deals have no rows in the engagements table:
- Deal-3EED2C (owner: Alex Franklin) — entered DS2 on 2026-09-03 (counted in DS2 entries, but 0 activities recorded)
- Deal-57FF13 (owner: Elena Sinclair) — DS1 only, no DS2 entry

All activity counts below come from the emails_30d, calls_30d, meetings_30d columns in the engagements table. Archived reps (Gavin Porter, Hugo Lindqvist) are excluded.

PER-REP RESULTS

Alex Franklin (84342457)
  Emails: 307 | Calls: 36 | Meetings: 41 | Total: 384
  Mix: emails 79.9% (307/384) | calls 9.4% (36/384) | meetings 10.7% (41/384)
  DS2 entries (30d): 18 — Deal-403845, Deal-1FC049, Deal-3EED2C, Deal-7FA0C3, Deal-E531A6, Deal-5296C9, Deal-36C33F, Deal-EE195F, Deal-F436DA, Deal-317E6F, Deal-D1E6C2, Deal-D9A72E, Deal-CA5E44, Deal-4F775F, Deal-898FC5, Deal-46988D, Deal-E73427, Deal-92D97D
  Activities per DS2 entry: 384 / 18 = 21.33

Bryce Harmon (119337721)
  Emails: 162 | Calls: 0 | Meetings: 43 | Total: 205
  Mix: emails 79.0% (162/205) | calls 0.0% (0/205) | meetings 21.0% (43/205)
  DS2 entries (30d): 4 — Deal-25F752, Deal-D73B89, Deal-CA7DC0, Deal-1CCE5C
  Activities per DS2 entry: 205 / 4 = 51.25

Cole Ingram (83155924)
  Emails: 96 | Calls: 14 | Meetings: 1 | Total: 111
  Mix: emails 86.5% (96/111) | calls 12.6% (14/111) | meetings 0.9% (1/111)
  DS2 entries (30d): 2 — Deal-42326B, Deal-1BA595
  Activities per DS2 entry: 111 / 2 = 55.50

Farid Osman (716654662)
  Emails: 38 | Calls: 0 | Meetings: 34 | Total: 72
  Mix: emails 52.8% (38/72) | calls 0.0% (0/72) | meetings 47.2% (34/72)
  DS2 entries (30d): 1 — Deal-499BF6
  Activities per DS2 entry: 72 / 1 = 72.00

Dana Mercer (83155923)
  Emails: 84 | Calls: 18 | Meetings: 11 | Total: 113
  Mix: emails 74.3% (84/113) | calls 15.9% (18/113) | meetings 9.7% (11/113)
  DS2 entries (30d): 1 — Deal-57887A
  Activities per DS2 entry: 113 / 1 = 113.00

Elena Sinclair (701163055)
  Emails: 0 | Calls: 0 | Meetings: 0 | Total: 0
  DS2 entries (30d): 0
  Activities per DS2 entry: UNDEFINED (no activities, no DS2 entries)

EFFICIENCY RANKING (lowest activities per DS2 entry = most efficient)

  1. Alex Franklin    21.33  (384 activities / 18 DS2 entries)
  2. Bryce Harmon       51.25  (205 / 4)
  3. Cole Ingram  55.50  (111 / 2)
  4. Farid Osman      72.00  (72 / 1)
  5. Dana Mercer      113.00  (113 / 1)
  --. Elena Sinclair   UNDEFINED (0 DS2 entries)

HIGHEST-VOLUME REP: Alex Franklin (384 activities)

MOST EFFICIENT REP: Alex Franklin (21.33 activities per DS2 entry)

They are the SAME rep. No cause attributed.
ae-qtd-scorecardPASS 1.00

? The Question

Build a QTD scorecard for the named AE as of 2026-09-05: bookings vs quota with attainment percent, new vs expansion split, active pipeline by stage with amount, rolling 90-day DS2-to-won rate, win and loss counts with top loss reason, activity volume by type in the last 30 days. Exclude any closed-won deal dated before the quarter. Close with three coaching observations grounded in the numbers.
cost $0.1348 · 336s · in 34,248 / out 28,872 tok
✓ bookings✓ quota✓ attainment✓ new_expansion_split✓ top_loss_reason✓ coaching_obs

✓ Correct Answer (ground truth)

owner: Alex Franklin
quarter: 2026-Q3
snapshot_date: 2026-09-05
bookings: 150000
quota: 200000
attainment_pct: 75.0
new_bookings: 113500
new_count: 5
expansion_bookings: 36500
expansion_count: 3
pipeline_by_stage: {"DS1": {"count": 20, "amount": 284621}, "DS2": {"count": 28, "amount": 353760}, "DS3": {"count": 67, "amount": 552705}, "DS4": {"count": 5, "amount": 23574}, "DS5": {"count": 5, "amount": 45730}}
open_deals_count: 125
ds2_to_won: window_start=2026-06-07; won_in_window=8; closed_in_window=35; rate=0.2286
wins_qtd: 8
losses_qtd: 27
top_loss_reason: reason=Lost- Timing (1 year or more); count=13
activity_30d_totals: emails_30d=807; calls_30d=112; meetings_30d=128; notes_30d=50
prior_quarter_excluded: deal_alias=Deal-B3E6F1; close_date=2026-06-20; amount=24000
allowed_new: 18 items — e.g. Deal-A1C3E5, Deal-B7D2F4, Deal-C9E1A6, Deal-D4B8C2
graded checks: bookings · quota · attainment · new_expansion_split · top_loss_reason · coaching_obs

Model's actual answer

show the model's full answer
QTD SCORECARD — Alex Franklin
As of 2026-09-05 (Q3: Jul 1 – Sep 30, 2026)

============================================================
1. BOOKINGS VS QUOTA
============================================================

  Excluded (pre-quarter closed-won):
    Deal-B3E6F1 | $24,000 | close 2026-06-20

  QTD closed-won (8 deals):
    Deal-A1C3E5 | $40,000 | new    | 2026-07-15
    Deal-F2C7D8 | $20,000 | expansion | 2026-07-24
    Deal-B7D2F4 | $35,000 | new    | 2026-07-31
    Deal-C9E1A6 | $21,000 | new    | 2026-08-12
    Deal-A8B4D6 | $12,000 | expansion | 2026-08-19
    Deal-D4B8C2 | $11,000 | new    | 2026-08-21
    Deal-E6F3A9 | $6,500  | new    | 2026-09-02
    Deal-C5D9E2 | $4,500  | expansion | 2026-09-03

  Total QTD bookings:  $150,000
  Quota:               $200,000
  Attainment:          75.0%
  Gap to quota:        $50,000

============================================================
2. NEW VS EXPANSION SPLIT
============================================================

  New:        5 deals, $113,500  (75.7%)
  Expansion:  3 deals, $36,500   (24.3%)

============================================================
3. ACTIVE PIPELINE BY STAGE
============================================================

  DS1:  20 deals, $284,621
  DS2:  28 deals, $353,760
  DS3:  67 deals, $552,705
  DS4:   5 deals, $23,574
  DS5:   5 deals, $45,730
  --------------------------------
  TOTAL: 125 deals, $1,260,390
  Pipeline-to-quota ratio: 6.30x

============================================================
4. ROLLING 90-DAY DS2-TO-WON RATE
============================================================

  Window: 2026-06-07 to 2026-09-05
  Deals entering DS2 in window and since closed: 35
    Won:  8
    Lost: 27
    Still open (excluded): 76
  DS2-to-won rate: 8 / 35 = 22.9%

============================================================
5. WIN AND LOSS COUNTS (QTD)
============================================================

  Wins:   8 deals, $150,000
  Losses: 27 deals, $329,272

  Loss reasons (ranked by count):
    Lost- Timing (1 year or more):              13 deals, $184,681
    MIA:                                          5 deals, $45,831
    Competitor:                                   5 deals, $49,020
    Lost DM:                                      2 deals, $17,940
    Feature Request:                              1 deal,  $21,000
    Lost- Does not fit ICP (write in notes):      1 deal,  $10,800

  Top loss reason: "Lost- Timing (1 year or more)"
    13 of 27 losses (48.1%), $184,681

============================================================
6. ACTIVITY VOLUME BY TYPE (LAST 30 DAYS)
============================================================

  Across 161 deals with engagement records:
    Emails:    807
    Calls:     112
    Meetings:  128
    Notes:      50
    Total:    1,097

  By deal status:
    Open  (125 deals): 599 emails, 54 calls, 90 meetings, 1 note
    Won   (  9 deals):  99 emails, 33 calls, 25 meetings, 24 notes
    Lost  ( 27 deals): 109 emails, 25 calls, 13 meetings, 25 notes

============================================================
COACHING OBSERVATIONS
============================================================

1. TIMING OBJECTIONS SIGNAL UPSTREAM QUALIFICATION GAP.
   13 of 27 QTD losses (48%) cite "Lost- Timing (1 year or more)" —
   $184,681 in lost ACV, more than the entire QTD bookings figure
   ($150,000). These are prospects with no near-term buying intent
   entering the pipeline and clogging it. The DS2-to-won rate of
   22.9% (8 wins / 35 closed) confirms weak conversion at the front
   end. Action: tighten DS1-to-DS2 qualification to require verified
   budget cycle and decision timeline under 12 months before
   advancing deals.

2. CALL ACTIVITY ON OPEN PIPELINE IS NEARLY ABSENT.
   54 calls across 125 open deals = 0.43 calls per deal. By
   comparison, won deals received 33 calls across 9 deals = 3.67
   calls per deal (8.5x higher density). 5 MIA losses ($45,831)
   correlate directly — deals go dark without multi-channel
   persistence. The AE is leaning heavily on email (599 of 744
   open-deal activities = 80%). Action: set a minimum of 2 calls
   per open deal per 30-day window; pair every email cadence with
   a call attempt.

3. PIPELINE COVERAGE MASKS A LATE-STAGE SHORTFALL.
   $1,260,390 total pipeline = 6.30x quota, which looks healthy.
   But DS4 + DS5 combined = only $69,304 across 10 deals. The $50K
   gap to quota cannot be covered by late-stage pipeline alone.
   67 of 125 deals (54%) sit in DS3 with no clear acceleration
   path. To close the gap, focus on advancing the 5 DS5 deals
   ($45,730) and pulling 5-8 high-velocity DS3 deals into DS4/DS5
   before quarter end. Raw coverage is not the problem — commit-
   stage coverage is.
multithreading-gapPASS 1.00

? The Question

Find every open deal that is single-threaded (fewer than 2 active contacts) or under-threaded (fewer than 3, or all contacts in one persona). Active means engaged in the last 60 days and not marked former. Personas: economic buyer, champion, HR admin, IT security, finance. For each flagged deal: amount, stage, active contact count, personas present, personas missing, the most valuable persona to add given the stage, and any on-file unengaged contact at that company who fits; otherwise say none on file.
cost $0.0178 · 97s · in 1,954 / out 4,307 tok
✓ flagged_recall✓ persona_language✓ single_thread_term✓ on_file_fit

✓ Correct Answer (ground truth)

active_window_start: 2026-07-07
persona_universe: economic buyer, champion, HR admin, IT security, finance
flag_rule: flag if active_contacts < 2 (single-threaded) or active_contacts < 3 or all active contacts share one persona (under-threaded); active = engaged within 60 days and not former
best_persona_rule: DS2/DS3: economic buyer > IT security > finance > HR admin > champion; DS4/DS5: economic buyer > finance > IT security > HR admin > champion; first persona not present among active contacts
flagged: deal_alias=Deal-36C33F; amount=15000; stage=DS2; active_count=1; reason=single-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-50D386; amount=36000; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-5408B0; amount=14850; stage=DS4; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-5BFE3B; amount=18000; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-885F45; amount=9300; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=IT security | deal_alias=Deal-92D97D; amount=60000; stage=DS2; active_count=1; reason=single-threaded; best_persona_to_add=economic buyer … (+5 more)
healthy_not_flagged: Deal-4B0BEB, Deal-84DBA6, Deal-D348E1
allowed_new: 59 items — e.g. C-077A0E, C-0D15DF, C-1BDFE0, C-2AE3AA
graded checks: flagged_recall · persona_language · single_thread_term · on_file_fit

Model's actual answer

show the model's full answer
**Reference date: 2026-09-08. "Active" = engaged on/after 2026-07-10 AND not marked former.**

**NOTE: Amount and stage are not in the provided data. I flag those as MISSING below.**

---

## Flagged Deals (11 of 14)

### SINGLE-THREADED (<2 active contacts)

| Deal | Company | Active | Personas | Missing | Best Persona to Add | Unengaged on File |
|---|---|---|---|---|---|---|
| Deal-EC3025 | C-FDD0C7 | 1 | champion | economic buyer, HR admin, IT security, finance | economic buyer (former CPO departed) | CT-6827DB (Chief People Officer, economic buyer) |
| Deal-92D97D | C-E23238 | 1 | HR admin | champion, economic buyer, IT security, finance | champion (existing champion CT-A902AE went cold Jun 1) | none on file |
| Deal-36C33F | C-077A0E | 1 | IT security | champion, economic buyer, HR admin, finance | economic buyer (former CPO & champion both departed) | CT-1DB73E (Chief People Officer, economic buyer) |
| Deal-FCBE5B | C-737030 | 1 | champion | economic buyer, HR admin, IT security, finance | economic buyer | none on file |
| Deal-F9A08A | C-0D15DF | 1 | champion | economic buyer (CT-913581 cold since Jun 20), HR admin, IT security, finance | economic buyer (re-engage CT-913581 or replace) | CT-697541 (Chief People Officer, economic buyer) |

### UNDER-THREADED but NOT single-threaded

| Deal | Company | Active | Personas Present | Missing | Flag Reason | Best Persona to Add | Unengaged on File |
|---|---|---|---|---|---|---|---|
| Deal-50D386 | C-EB10E4 | 2 | champion, HR admin | economic buyer, IT security, finance | count < 3 | economic buyer | CT-A1C4B3 (Chief People Officer, economic buyer) |
| Deal-5BFE3B | C-535D36 | 2 | champion, champion | economic buyer, HR admin, IT security, finance | count < 3 AND all one persona | economic buyer | none on file |
| Deal-885F45 | C-5E8EFB | 2 | economic buyer, champion | HR admin, IT security, finance | count < 3 | IT security (common blocker in HR-tech; deal has champion + buyer, needs security clearance) | CT-B3F25D (IT Security Lead, IT security) |
| Deal-5408B0 | C-2AE3AA | 2 | champion, HR admin | economic buyer, IT security, finance | count < 3 | economic buyer | CT-07FA76 (Chief People Officer, economic buyer) |
| Deal-D0D6B5 | C-32918E | 3 | champion, champion, champion | economic buyer, HR admin, IT security, finance | all one persona | economic buyer | CT-1FA4DB (Chief People Officer, economic buyer) |
| Deal-C6D97A | C-5A8FC2 | 3 | champion, champion, champion | economic buyer, HR admin, IT security, finance | all one persona | economic buyer | none on file |

---

## Not Flagged (3 deals — healthy threading)

- **Deal-84DBA6** (C-BE9BE0): 3 active, 3 personas (champion, economic buyer, IT security). Missing HR admin + finance only. Has unengaged finance contact: CT-2A0169 (Controller, finance).
- **Deal-4B0BEB** (C-1BDFE0): 4 active, 4 personas (champion, economic buyer, HR admin, finance). Missing IT security only.
- **Deal-D348E1** (C-804F29): 5 active, all 5 personas covered. No unengaged match needed.

---

## Summary

- 5 deals single-threaded, 6 deals under-threaded only (11 total flagged).
- 7 of 11 flagged deals are missing an **economic buyer** as the #1 gap; 4 of those 7 have a CPO on file who is unengaged.
- 2 champion-heavy deals (Deal-D0D6B5, Deal-C6D97A) have 3 contacts but all champions — no buying authority or implementation owner in the conversation.
- Deal amount and stage: **not provided in the data**. Recommendations above are based on persona-gap logic (economic buyer first, then IT security for deals that already have a champion + buyer). Stage-specific recommendations would require deal amount and pipeline stage.
talk-track-from-transcriptsPASS 1.00

? The Question

From this rep's last 10 calls: what they lead with in the first five minutes, how they handle the three most common objections, how often a concrete next step is agreed (rate), and every competitor a prospect raised. Quote at most one line per finding and only lines that appear in the transcripts. End with two coaching notes.
cost $0.0135 · 69s · in 2,789 / out 2,628 tok
✓ objection_count✓ next_step_rate✓ competitors✓ coaching✓ quote_used

✓ Correct Answer (ground truth)

rep: Alex Franklin
n_transcripts: 10
opener_theme: theme=opens with a customer recognition win story (400-person retailer, 18% turnover reduction); count=8; total=10; quote=Quick one to kick off: a 400-person retailer we work with cut regretted turnover 18% in two quarters after automating milestone awards, and I wanted to walk you through exactly how they did it.; transcript_id=TT-001
objections: objection=budget; count=4; quote=Honestly, budget is locked until next fiscal year — I can't add a new line item right now.; transcript_id=TT-001 | objection=timing; count=3; quote=This is good, but can we revisit it next quarter? Open enrollment starts in October and we're underwater.; transcript_id=TT-002 | objection=in-house; count=3; quote=We already do recognition with a spreadsheet and quarterly gift cards — why would we change?; transcript_id=TT-004
objection_count: 3
next_step: agreed=7; total=10; rate=0.7
competitors_prospect_raised: Awardco, Kudos
competitor_quotes: Awardco=We're also in late talks with Awardco — their rewards catalog looks bigger than yours.; Kudos=How are you different from Kudos? Our CEO used them at her last company.
rep_raised_competitors_excluded: Workhuman
allowed_new: 13 items — e.g. TT-001, TT-002, TT-003, TT-004
graded checks: objection_count · next_step_rate · competitors · coaching · quote_used

Model's actual answer

show the model's full answer
## Alex Franklin — Last 10 Calls Analysis

---

### LEAD-WITH (first 5 min)

8 of 10 calls open with the same social-proof line:

> "Quick one to kick off: a 400-person retailer we work with cut regretted turnover 18% in two quarters after automating milestone awards, and I wanted to walk you through exactly how they did it."

Two calls deviate:
- TT-004: "I put together a short agenda — security review first, then pricing."
- TT-009: "You asked for straight pricing last time, so let's start there."

---

### THREE MOST COMMON OBJECTIONS + HOW HE HANDLES THEM

**1. Budget locked** (4x — TT-001, TT-003, TT-006, TT-010)

> "Totally fair. Most teams fund this out of turnover savings — that retailer saved about $210k in avoided backfills, which is how their finance team signed off."

Same verbatim rebuttal all four times. Repositions spend as self-funding via avoided backfill cost.

**2. Revisit next quarter** (3x — TT-002, TT-005, TT-008)

> "Makes sense. What if we scope a 90-day pilot with one department so you have internal data before next quarter's planning?"

Same verbatim rebuttal all three times. Downscopes to a department pilot to generate internal proof before planning cycle.

**3. Status quo — spreadsheet + gift cards** (3x — TT-004, TT-007, TT-009)

> "Spreadsheets work until they scale — the difference is automation: milestones fire without HR lifting a finger, and you get analytics on who is being recognized."

Same verbatim rebuttal all three times. Frames spreadsheet as fragile at scale; pitches automation + analytics.

---

### NEXT-STEP AGREEMENT RATE

| Call | Next step agreed? |
|------|-------------------|
| TT-001 | Yes |
| TT-002 | Yes |
| TT-003 | Yes |
| TT-004 | No — "I'll leave it with you" |
| TT-005 | Yes |
| TT-006 | Yes |
| TT-007 | No — prospect says "I need to think about it"; rep: "Fair enough" |
| TT-008 | Yes |
| TT-009 | Yes |
| TT-010 | No — prospect says "We'll have to wait for the committee"; rep: "Understood" |

**Rate: 7 / 10 = 70%**

---

### COMPETITORS RAISED BY PROSPECTS

1. **Awardco** (TT-003): "We're also in late talks with Awardco — their rewards catalog looks bigger than yours."
2. **Kudos** (TT-007): "How are you different from Kudos? Our CEO used them at her last company."

(Note: Workhuman is raised by Alex in TT-005, not by the prospect.)

---

### COACHING NOTES

1. **Objection rebuttals are copy-pasted across every call.** The same three verbatim responses recycle regardless of deal context. On the three calls where the next step stalled (TT-004, TT-007, TT-010), Alex never adapted — he either gave up ("I'll leave it with you," "Fair enough," "Understood") or didn't bridge the objection to the 90-day pilot offer he already uses for "revisit next quarter." The pilot framing could work for budget-locked and committee stalls too; he should practice attaching a micro-commitment (pilot, ROI calc, internal champion brief) to every objection branch, not just one.

2. **The social-proof opener is losing effectiveness on repeats and misreads.** In TT-004 the prospect clearly wanted a security/pricing agenda, and in TT-009 the prospect had already requested straight pricing — yet Alex still ran the 400-person retailer script in 8/10 calls. When the prospect's prior signal conflicts with the opener, the script feels tone-deaf. Coach him to read the pre-call context and lead with the frame the prospect already asked for (pricing, security, etc.) before falling back to the default social proof.
reporting-analytics
q3-weighted-forecast0.90

? The Question

Below are the open deals in the default sales pipeline with a close date between 2026-07-01 and 2026-10-15, pulled from the CRM warehouse on 2026-09-05. Stages: DS1 (discovery) through DS5 (contract). Build the Q3 2026 forecast (quarter = 2026-07-01 to 2026-09-30). Weighting rule: 100 percent of COMMIT plus 35 percent of BEST_CASE. PIPELINE deals count zero. Only deals with a close date inside the quarter count. Report: COMMIT total, BEST_CASE total, weighted forecast, count of deals in each category inside the quarter, the deals you excluded for being outside the quarter (count and total amount), and the top 5 BEST_CASE deals by amount inside the quarter. Show arithmetic. Then, in a final '## Data quality' section of no more than four sentences, name the data quality problems in this extract that would make the forecast unreliable if it were run unattended.
cost $0.0323 · 201s · in 3,169 / out 8,700 tok
✓ commit_total_44729✓ best_case_total_203565✓ weighted_115976.75✓ commit_count_7✓ best_case_count_24✓ excluded_count_32✗ excluded_amount_227575✓ top5_aliases✓ flags_blank_owner✓ flags_missing_why_buys

✓ Correct Answer (ground truth)

graded checks: commit_total_44729 · best_case_total_203565 · weighted_115976.75 · commit_count_7 · best_case_count_24 · excluded_count_32 · excluded_amount_227575 · top5_aliases · flags_blank_owner · flags_missing_why_buys

Model's actual answer

show the model's full answer
## Q3 2026 Forecast (2026-07-01 to 2026-09-30)

**Weighting rule:** 100% of COMMIT + 35% of BEST_CASE. PIPELINE = 0%.

---

### COMMIT (inside quarter) — 7 deals

| Deal | Amount | Close Date |
|---|---|---|
| Deal-547B2B | 11,200 | 2026-09-11 |
| Deal-B7EBD1 | 9,000 | 2026-09-10 |
| Deal-403845 | 9,000 | 2026-09-11 |
| Deal-A2B47C | 6,360 | 2026-09-11 |
| Deal-2465CE | 5,400 | 2026-09-10 |
| Deal-1D532E | 2,520 | 2026-09-11 |
| Deal-499BF6 | 1,249 | 2026-09-30 |

COMMIT total = 11,200 + 9,000 + 9,000 + 6,360 + 5,400 + 2,520 + 1,249 = **$44,729**

---

### BEST_CASE (inside quarter) — 24 deals

Total = 38,935 + 24,000 + 19,656 + 16,250 + 11,116 + 10,800 + 10,500 + 9,890 + 9,720 + 9,000 + 7,200 + 3,840 + 3,780 + 3,600 + 3,240 + 3,150 + 3,120 + 3,060 + 2,916 + 2,760 + 2,484 + 2,100 + 1,920 + 528 = **$203,565**

---

### Weighted Forecast

100% x 44,729 + 35% x 203,565
= 44,729 + 0.35 x 203,565
= 44,729 + 71,247.75
= **$115,976.75**

---

### PIPELINE (inside quarter) — 23 deals, $0 contribution per rule

(Pipe total = ~$201,637.40 — not counted.)

---

### Deals excluded (close date outside Q3)

| Category | Count | Total Amount |
|---|---|---|
| COMMIT | 1 (Deal-D348E1, $13,770, 2026-10-15) | $13,770 |
| BEST_CASE | 8 | $23,080 |
| PIPELINE | 20 | $169,365 |
| **Total excluded** | **29** | **$206,215** |

---

### Top 5 BEST_CASE deals by amount (inside quarter)

| Rank | Deal | Amount | Close Date |
|---|---|---|---|
| 1 | Deal-2D7423 | $38,935 | 2026-09-30 |
| 2 | Deal-25F752 | $24,000 | 2026-09-25 |
| 3 | Deal-E53952 | $19,656 | 2026-09-30 |
| 4 | Deal-5EED42 | $16,250 | 2026-09-30 |
| 5 | Deal-FA32A0 | $11,116 | 2026-09-25 |

---

## Data quality

(1) Owner is blank on 79 of 80 deals — no rep-level scrub or attribution is possible. (2) Deal-1D532E sits in DS1 (discovery) but is flagged COMMIT, a stage/forecast-category mismatch that signals stale categorisation. (3) 20 of 24 in-quarter BEST_CASE deals carry why_buys_chars = 0, so the 35% weight has no documented justification per deal. (4) Two PIPELINE deals (Deal-6787C2, Deal-D56743) are in DS4 but still rated PIPELINE rather than BEST_CASE, likely understaging the weighted forecast.
aha-moment-2x20.83

? The Question

Test the activation hypothesis on the company cohort below: companies with 5+ unique givers (m1_users >= 5) AND 1+ successful redemption (m1_redemptions >= 1) in their first calendar month retain better at 24 months than those with one signal or neither. Every company in this extract is already 25+ months old, so current_status = 'active' means retained at 24 months. Report the full 2x2 (cohort size and 24-month retention rate for: both signals, givers-only, redemption-only, neither), state how many companies were excluded from the denominator and why (if none, say so), name the single signal with the largest retention lift, and state what this does and does not prove.
cost $0.0703 · 257s · in 26,763 / out 12,200 tok
✓ both_66✓ givers_47✓ redemp_31✓ neither_40✓ excluded_0✗ largest_lift_givers

✓ Correct Answer (ground truth)

graded checks: both_66 · givers_47 · redemp_31 · neither_40 · excluded_0 · largest_lift_givers

Model's actual answer

show the model's full answer
## Activation Hypothesis Test — 2x2 Results

**Signals tested:** m1_users >= 5 (givers breadth) AND m1_redemptions >= 1 (redemption depth) in first calendar month.
**Retention definition:** current_status = 'active' (all 220 companies are 25+ months old).

### 2x2 Table

| Cell | N | Retained | 24-mo Retention Rate |
|---|---|---|---|
| Both signals (>=5 users AND >=1 red) | 47 | 31 | **66.0%** |
| Givers only (>=5 users, 0 red) | 49 | 23 | 46.9% |
| Redemption only (<5 users, >=1 red) | 29 | 9 | 31.0% |
| Neither (<5 users, 0 red) | 95 | 38 | 40.0% |

### Companies excluded from denominator: **0**
All 220 companies in the extract are 25+ months old. No companies were excluded.

### Single signal with the largest retention lift

**5+ unique givers (m1_users >= 5)** — the givers-only cell retains at 46.9% vs. neither at 40.0%, a lift of **+6.9 pp**.

The redemption-only signal shows *negative* lift: 31.0% vs. neither 40.0% = **-9.0 pp**. Companies with redemptions but fewer than 5 users actually retain *worse* than those with neither signal.

### Arithmetic

- Both: 31/47 = 66.0%
- Givers only: 23/49 = 46.9%
- Redemption only: 9/29 = 31.0%
- Neither: 38/95 = 40.0%
- Givers lift vs neither: 46.9% - 40.0% = +6.9 pp
- Redemption lift vs neither: 31.0% - 40.0% = -9.0 pp

### What this does and does not prove

**Does prove (descriptively):** Within this 2023 cohort, companies hitting both activation thresholds retained at 66.0% — 26 pp above the neither-cell baseline. The givers breadth signal alone is the only single signal that lifts retention above the baseline. The redemption signal *without* user breadth correlates with *lower* retention, not higher.

**Does not prove:** Causation. This is an observational cross-tab on one cohort — it cannot establish that driving more givers or redemptions *causes* better retention. Key confounders are uncontrolled: company size, product tier (tier_three dominates retained companies), HRIS integration, industry, and country all covary with both the activation signals and retention. A company that arrives large enough to have 5+ users in month 1 may retain for structural reasons (budget authority, procurement gravity) unrelated to product engagement. The negative redemption-only result further suggests that redemptions from tiny teams may signal a single power user burning through points, not healthy adoption — but that, too, is interpretation, not proof. A controlled experiment or propensity-matched analysis is needed before claiming these signals are causal levers.
arr-rollforward-reconciliation0.80

? The Question

Reconcile ARR between CRM company records and billing subscriptions as of 2026-09-05. Billing ARR is MRR x 12. Report both totals, the variance, and a decomposition into buckets (status mismatch, rounding, missing records, other) that sums exactly to the variance. List each mismatched account with a suggested owner. Business rule: any subscription with a term other than 12 months must have cf_agreement_end_date populated; list violations.
cost $0.0353 · 135s · in 8,606 / out 5,543 tok
✓ billing_total✓ crm_total✓ variance✗ buckets_sum✓ violations_listed

✓ Correct Answer (ground truth)

as_of: 2026-09-05
billing_arr_total: 604739.28
crm_arr_total: 603581.76
variance: 1157.52
variance_definition: billing_arr_total - crm_arr_total (billing ARR = sum of MRR x 12 over status=active subscriptions)
buckets: {"status_mismatch": {"amount": "-13158.48", "accounts": [{"company_alias": "C-0C8323BF", "billing_arr": "0.00", "hubspot_arr": "4905.24", "diff": "-4905.24", "suggested_owner": "Dana Mercer"}, {"compa
bucket_sum_equals_variance: True
n_mismatched_accounts: 7
rule: term_months != 12 requires cf_agreement_end_date
violations: subscription_id=SUB-0002; company_alias=C-1794A52C; term_months=24 | subscription_id=SUB-0019; company_alias=C-22170CA1; term_months=36
n_companies: 40
graded checks: billing_total · crm_total · variance · buckets_sum · violations_listed

Model's actual answer

show the model's full answer
## ARR Reconciliation Report — As of 2026-09-05

---

### TOTALS

| Source | ARR |
|---|---|
| CRM (HubSpot) | $603,581.76 |
| Billing (Chargebee MRR x 12) | $604,739.28 |
| **Variance (CRM − Billing)** | **−$1,157.52** |

---

### VARIANCE DECOMPOSITION (sums exactly to −$1,157.52)

| Bucket | Amount | Explanation |
|---|---|---|
| Status mismatch | +$13,158.48 | 2 cancelled Chargebee subs still carrying ARR in CRM |
| Rounding | $0.00 | No accounts fell within the ≤$12 rounding threshold |
| Missing billing (CRM-only) | +$16,497.24 | 1 company in CRM with no active Chargebee subscription |
| Missing CRM record (Billing-only) | −$28,449.24 | 1 company in Chargebee with no CRM record |
| Other (active both sides, delta > $12) | −$2,364.00 | 3 accounts with price/MRR drift between systems |
| **Sum** | **−$1,157.52** | ✓ Matches variance |

---

### MISMATCHED ACCOUNTS WITH SUGGESTED OWNER

| Company | CRM ARR | Billing ARR | Delta | Bucket | Suggested Owner |
|---|---|---|---|---|---|
| C-21629AA4 | $0.00 | $28,449.24 | −$28,449.24 | Missing CRM | CS / AM |
| C-0D5BBE3A | $16,497.24 | $0.00 | +$16,497.24 | Missing billing | CS / AM |
| C-0DC4FB8C | $8,253.24 | $0.00 | +$8,253.24 | Status mismatch | RevOps |
| C-0C8323BF | $4,905.24 | $0.00 | +$4,905.24 | Status mismatch | RevOps |
| C-0F7269D7 | $24,396.00 | $26,796.00 | −$2,400.00 | Other | RevOps |
| C-14D70CE0 | $18,200.00 | $18,180.00 | +$20.00 | Other | RevOps |
| C-0D66DF9E | $23,200.00 | $23,184.00 | +$16.00 | Other | RevOps |

**Notes on the "Other" bucket:**
- **C-0F7269D7**: CRM implies MRR of $2,033.00; Chargebee shows $2,233.00. Gap is exactly $200/mo — likely a billing price increase not synced to HubSpot.
- **C-14D70CE0**: CRM $18,200 implies $1,516.67 MRR vs Chargebee $1,515.00. $20/yr drift — possible price-book difference.
- **C-0D66DF9E**: CRM $23,200 implies $1,933.33 MRR vs Chargebee $1,932.00. $16/yr drift — same pattern.

---

### BUSINESS RULE VIOLATIONS

**Rule**: Any subscription with term ≠ 12 months must have `cf_agreement_end_date` populated.

| Subscription | Company | Term | cf_agreement_end_date | Status |
|---|---|---|---|---|
| SUB-0002 | C-1794A52C | 24 mo | EMPTY | VIOLATION |
| SUB-0019 | C-22170CA1 | 36 mo | EMPTY | VIOLATION |

Note: SUB-000C (C-0DB48281, 24 mo) and SUB-001A (C-0FC4DBB8, 36 mo) both have `cf_agreement_end_date` populated and are compliant.

---

### CANCELLED SUBS STILL IN CRM (status mismatch detail)

| Subscription | Company | MRR | Billing ARR (if active) | CRM ARR | Action |
|---|---|---|---|---|---|
| SUB-000E | C-0C8323BF | $408.77 | $4,905.24 | $4,905.24 | Zero-out CRM ARR or reactivate |
| SUB-000F | C-0DC4FB8C | $687.77 | $8,253.24 | $8,253.24 | Zero-out CRM ARR or reactivate |

Combined stale ARR in CRM: $4,905.24 + $8,253.24 = **$13,158.48**
kvm-month-over-monthPASS 1.00

? The Question

Report the four core KVMs for 2026-08 against 2026-07: giving rate, redemptions per user, 1:1 meetings engagement, pulse check engagement. For each: value, prior value, absolute and relative change, direction. Name the KVM with the largest relative move and the segment (plan_tier or size_band) driving it if the data supports that; otherwise say the data does not. Under 200 words plus one table.
cost $0.0409 · 180s · in 10,052 / out 6,946 tok
✓ four_kvms✓ biggest_mover_named✓ direction_correct✓ segment_named✓ word_cap

✓ Correct Answer (ground truth)

aggregation: simple mean of company-level values per month
kvms: {"giving_rate": {"2026-07": 0.6023, "2026-08": 0.6027, "abs_change": 0.0004, "rel_change_pct": 0.07, "direction": "up"}, "redemptions_per_user": {"2026-07": 1.73, "2026-08": 1.7302, "abs_change": 0.00
biggest_mover: pulse_engagement
driving_segment: size_band=enterprise; 2026-07=0.55; 2026-08=0.2743; rel_change_pct=-50.13
segment_detail: {"smb": {"2026-07": 0.6588, "2026-08": 0.6573, "rel_change_pct": -0.23}, "mid_market": {"2026-07": 0.593, "2026-08": 0.5942, "rel_change_pct": 0.2}, "enterprise": {"2026-07": 0.55, "2026-08": 0.2743,
n_company_rows_per_month: 30
graded checks: four_kvms · biggest_mover_named · direction_correct · segment_named · word_cap

Model's actual answer

show the model's full answer
| KVM | Aug-26 | Jul-26 | Abs Chg | Rel Chg | Dir |
|---|---|---|---|---|---|
| Giving rate | 0.6027 | 0.6023 | +0.0004 | +0.07% | up |
| Redemptions/user | 1.7302 | 1.7300 | +0.0002 | +0.01% | up |
| 1:1 engagement | 0.4472 | 0.4469 | +0.0003 | +0.06% | up |
| Pulse engagement | 0.5086 | 0.6006 | -0.0920 | -15.31% | down |

Three KVMs were essentially flat (<0.2% relative). Pulse engagement collapsed: -15.3% overall, driven entirely by the **enterprise** size_band, which fell from 0.5500 to 0.2743 (-50.1% relative). SMB and mid_market pulse engagement were stable (-0.22% and +0.21% respectively). Every enterprise account shows the same cliff -- values that were 0.53-0.57 in July dropped to 0.26-0.29 in August, suggesting a systematic measurement or configuration change in that segment, not organic disengagement.
rewards-ytd-sectionPASS 1.00

? The Question

Produce the redemption section of the weekly rewards report through the last completed month (2026-08): redemption count, spend, unique redeemers, redemptions per redeemer, provider mix as percent of spend, top 5 countries by redemptions. Say which month is the last completed month. Provider shares must sum to 100.
cost $0.0936 · 335s · in 33,691 / out 17,255 tok
✓ count✓ spend✓ unique_redeemers✓ month_named✓ provider_mix✓ top_country

✓ Correct Answer (ground truth)

redemption_count: 378
total_spend_usd: 27846.00
unique_redeemers: 235
redemptions_per_redeemer: 1.61
provider_mix_pct_of_spend: TangoCard=11.6; Tremendous=30.54; custom=39.05; Snappy=18.81
top5_countries_by_count: country=US; count=244 | country=CA; count=24 | country=AU; count=21 | country=GB; count=17 | country=NL; count=17
graded checks: count · spend · unique_redeemers · month_named · provider_mix · top_country

Model's actual answer

show the model's full answer
**REDEMPTIONS -- WEEKLY REWARDS REPORT (through last completed month)**

Last completed month: **2026-08**

| Metric | Value |
|---|---|
| Redemption count | 378 |
| Total spend | $27,846.00 |
| Unique redeemers | 235 |
| Redemptions per redeemer | 378 / 235 = **1.61** |

**Provider mix (% of spend)**

| Provider | Spend share |
|---|---|
| custom | 39.0% |
| Tremendous | 30.5% |
| Snappy | 18.8% |
| TangoCard | 11.6% |
| **Sum** | **100.0%** |

**Top 5 countries by redemption count**

| Country | Redemptions |
|---|---|
| US | 243 |
| CA | 24 |
| AU | 21 |
| NL | 18 |
| GB | 17 |
customer-success
churn-save-eligibilityPASS 1.00

? The Question

Which at-risk accounts qualify for a churn-save offer under the documented eligibility rules, what amount is at stake per account and in total, and which play fits each (usage revival, executive touch, commercial concession)? Cite the signal that justifies each play. List accounts that look at risk but do not qualify and why.
cost $0.0168 · 95s · in 1,340 / out 4,160 tok
✓ eligible_set✓ total_at_stake✓ plays_cited✓ noneligible_named✓ rules_applied

✓ Correct Answer (ground truth)

snapshot_date: 2026-09-05
rules: health_score < 60, churn_save_eligible_amount > 0, renewal within 120 days of snapshot
eligible: account_alias=C-0F6C0F34; amount_at_stake=49707.00; play=executive touch; justifying_signal=champion_active is false - no executive sponsor engaged | account_alias=C-0B827671; amount_at_stake=25365.00; play=usage revival; justifying_signal=usage_trend_3m=declining over the last 3 months | account_alias=C-0B360C78; amount_at_stake=35748.00; play=commercial concession; justifying_signal=usage stable/growing with seat utilization 75% - risk is commercial, not adoption | account_alias=C-0B0F1BAB; amount_at_stake=5494.00; play=executive touch; justifying_signal=champion_active is false - no executive sponsor engaged | account_alias=C-0CA21961; amount_at_stake=16829.00; play=usage revival; justifying_signal=seat utilization 26% is below 50% | account_alias=C-0E9C27D1; amount_at_stake=41235.00; play=commercial concession; justifying_signal=usage stable/growing with seat utilization 85% - risk is commercial, not adoption … (+2 more)
total_amount_at_stake: 224601.00
non_eligible_at_risk: account_alias=C-0BC71BDD; health_score=55 | account_alias=C-0BA71F12; health_score=52 | account_alias=C-0F6694C3; health_score=43 | account_alias=C-0BE96399; health_score=54 | account_alias=C-0F876796; health_score=47 | account_alias=C-0FCCD2DF; health_score=43 … (+1 more)
n_accounts: 30
graded checks: eligible_set · total_at_stake · plays_cited · noneligible_named · rules_applied

Model's actual answer

show the model's full answer
I need to apply the three eligibility rules (all must be met) and then assign plays based on the risk signals in each qualifying account.

**Eligibility rules applied:**
- R1: health_score < 60
- R2: churn_save_eligible_amount > 0
- R3: renewal_date on or before 2027-01-03 (2026-09-05 + 120 days)

---

## QUALIFYING ACCOUNTS (R1 + R2 + R3 all pass)

| Account | Health | Eligible Amount | Renewal | Play | Signal |
|---|---|---|---|---|---|
| C-0E9C27D1 | 39 | $41,235 | 2026-09-24 | Commercial Concession | High seat utilization (134/157 = 85.4%) yet flat usage + low health = value-perception gap |
| C-0F6C0F34 | 51 | $49,707 | 2026-10-03 | Executive Touch | champion_active = false despite growing usage = relationship gap |
| C-0B0F1BAB | 38 | $5,494 | 2026-09-23 | Executive Touch | champion_active = false + health = 38 + flat usage = no internal sponsor |
| C-0B360C78 | 57 | $35,748 | 2026-10-28 | Commercial Concession | Growing usage, active champion, yet health below threshold = pricing/value mismatch |
| C-0BC71BDD excluded — see below | — | — | — | — | — |
| C-0CA21961 | 58 | $16,829 | 2026-12-28 | Usage Revival | Flat usage + 84/325 seats used = 25.8% utilization |
| C-0CEF69FD | 53 | $32,621 | 2026-11-21 | Executive Touch | champion_active = false despite growing usage = advocacy gap |
| C-0B827671 | 56 | $25,365 | 2026-11-14 | Usage Revival | Declining usage + 113/202 seats used = 55.9% utilization |
| C-0D3278C7 | 54 | $17,602 | 2026-11-12 | Usage Revival | Declining usage + 126/380 seats used = 33.2% utilization |

**Total at stake:** $49,707 + $25,365 + $35,748 + $5,494 + $16,829 + $41,235 + $32,621 + $17,602 = **$224,601**

---

## AT-RISK ACCOUNTS THAT DO NOT QUALIFY (health < 60 but fail R2 or R3)

| Account | Health | Eligible Amt | Renewal | Failed Rule | Reason |
|---|---|---|---|---|---|
| C-0BC71BDD | 55 | $0 | 2026-10-27 | R2 | No churn-save eligible amount allocated |
| C-0BE96399 | 54 | $0 | 2026-10-29 | R2 | No churn-save eligible amount allocated |
| C-10A56B0F | 54 | $0 | 2026-12-12 | R2 | No churn-save eligible amount allocated |
| C-0F6694C3 | 43 | $0 | 2027-03-21 | R2 | No churn-save eligible amount allocated |
| C-0FCCD2DF | 43 | $0 | 2027-04-23 | R2 | No churn-save eligible amount allocated |
| C-0BA71F12 | 52 | $6,824 | 2027-04-11 | R3 | Renewal >120 days out (218 days from snapshot) |
| C-0F876796 | 47 | $19,958 | 2027-02-06 | R3 | Renewal >120 days out (154 days from snapshot) |

**ARR left uncovered by rule gaps:** C-0BC71BDD ($54,515) + C-0BE96399 ($52,319) + C-0BA71F12 ($17,061) + C-0F6694C3 ($11,243) + C-0F876796 ($49,895) + C-0FCCD2DF ($65,957) + C-10A56B0F ($25,717) = **$276,707 ARR** at risk but not reachable under current eligibility.

---

**Play logic summary:**
- **Usage Revival** assigned when declining/flat usage + low seat utilization (< 60%) signals an adoption problem.
- **Executive Touch** assigned when champion_active = false, regardless of usage trend — no internal sponsor means engagement efforts have no foothold.
- **Commercial Concession** assigned when the champion is active and utilization is high but health is still low — the account is using the product but perceives insufficient value relative to cost.
expansion-kitPASS 1.00

? The Question

A customer replied to outreach (reply included). Build the expansion kit: seat coverage (licensed over headcount), two lines on usage health, headroom in seats and ARR at the current per-seat rate, who replied and whether they can buy, the right buyer from our contacts, and a reply email under 150 words that moves toward the right person without being pushy and cites one real usage fact.
cost $0.0058 · 28s · in 443 / out 816 tok
✓ coverage_pct✓ headroom_arr✓ right_buyer✓ replier_not_buyer✓ usage_fact✓ email_under_150

✓ Correct Answer (ground truth)

company_alias: C-0DDFC9A7
coverage_pct: 37.5
headroom_seats: 250
per_seat_annual_rate: 60.00
headroom_arr: 15000.00
replier: Maria S.
replier_can_buy: False
right_buyer: name=Dana R.; title=VP People
usage_facts: active_users_2026_08=126; active_users_2026_03=88; growth_pct_6m=43.2; licensed_seat_utilization_pct=84.0
usage_health_lines: Monthly active users grew from 88 (2026-03) to 126 (2026-08), +43.2% over 6 months., Last month 126 of 150 licensed seats were active (84.0% utilization), approaching the seat cap.
graded checks: coverage_pct · headroom_arr · right_buyer · replier_not_buyer · usage_fact · email_under_150

Model's actual answer

show the model's full answer
## Expansion Kit — C-0DDFC9A7

---

### Seat Coverage

- Licensed: 150 | Headcount: 400 | Coverage: **37.5%** (150/400)
- Under-licensed by 250 seats

### Usage Health (2 lines)

- MAU climbing every month: 88 → 126 (Mar–Aug 2026), +38 users in 6 months (~7/month avg).
- 84% of licensed seats already active (126/150) — approaching the ceiling with ~3 months of headroom left at current growth.

### Headroom

| Metric | Value |
|---|---|
| Seats remaining before cap | 150 − 126 = **24 seats** |
| Current per-seat ARR | $9,000 / 150 = **$60/seat** |
| ARR at full headcount (400 × $60) | **$24,000** |
| ARR headroom (uplift from current) | $24,000 − $9,000 = **$15,000** |

### Who Replied & Can They Buy?

**Maria S.**, People Operations Coordinator — replied 2026-09-02. She **cannot buy**; she explicitly states budget and seat expansion sit with Dana R. Maria offered to introduce.

### Right Buyer

**Dana R.**, VP People — owns budget and seat expansion. Last engaged 2026-05-18 (~4 months stale). She's been asking about usage numbers per Maria's note.

---

### Reply Email (under 150 words)

> Hi Maria,
>
> Thanks for the kind words — glad the team is engaged! One thing that stands out: your MAU has climbed from 88 to 126 since March, which means 84% of your licensed seats are now active and you have roughly 24 seats of runway left at the current pace.
>
> An intro to Dana R. would be great whenever it's convenient. We'd love to share the full usage trend and walk through what an expansion to match your actual headcount would look like — no pressure, just good data for her decision.
>
> Thanks again,
> Cole
onboarding-statusPASS 1.00

? The Question

Prep the CSM for the mid-onboarding call: what is complete (integrations connected, allowance set, admins added, first recognition given, first redemption), what is not, early engagement signals, and the three things to cover on the call. Mark nothing complete without a data field that shows it.
cost $0.0069 · 34s · in 450 / out 1,208 tok
✓ complete_items✓ incomplete_items✓ three_items✓ grounded

✓ Correct Answer (ground truth)

company_alias: C-0D284E42
days_live: 25
checklist: {"slack_integration_connected": {"complete": true, "field": "integration_slack", "value": "2026-08-12"}, "hris_integration_connected": {"complete": false, "field": "integration_hris", "value": ""}, "a
engagement_signals: avg_active_givers_week1=4.3; avg_active_givers_last_week=13.0; latest_active_givers=15; days_with_activity=25
call_items: Connect the HRIS integration (integration_hris is empty) to automate user provisioning and offboarding., Get to first redemption (first_redemption_at is empty): walk admins through the rewards catalog and confirm allowance funding., Build the giving habit: daily active givers averaged 4.3 in week 1 vs 13.0 last week - agree on manager nudges and a recognition cadence.
graded checks: complete_items · incomplete_items · three_items · grounded

Model's actual answer

show the model's full answer
## CSM Call Prep — C-0D284E42

### Onboarding Checklist

| Milestone | Status | Evidence |
|---|---|---|
| Integrations connected | PARTIAL | Slack connected 2026-08-12; HRIS field blank — not connected |
| Allowance set | COMPLETE | 2026-08-13 |
| Admins added | COMPLETE | 2 admins |
| First recognition given | COMPLETE | 2026-08-15 14:22 |
| First redemption | NOT COMPLETE | Field blank — no redemption recorded |

Score: 3.5 of 5 milestones met. Slack integration and first recognition landed within 5 days of signup — fast start.

---

### Early Engagement Signals

**Active-giver trend (daily, from onboarding_usage.csv):**

- Week 1 (8/11-8/17): 3, 3, 4, 4, 5, 4, 7 — avg 4.3 givers/day
- Week 2 (8/18-8/24): 5, 7, 6, 9, 8, 9, 9 — avg 7.6 givers/day
- Week 3 (8/25-8/31): 9, 11, 10, 10, 11, 13, 11 — avg 10.7 givers/day
- Week 4 (8/1-9/4): 13, 13, 15, 15 — avg 14.0 givers/day

Week-over-week growth: +77%, +41%, +31%. Trend is positive and decelerating slightly but still strong — 5x growth from day-1 baseline (3) to latest (15) over 24 days. No zero-day dips or stalls.

**Read-through:** The giving flywheel is spinning. Admins and early adopters are pulling peers in. But zero redemptions means the value loop isn't closing — people are giving but not yet experiencing the reward side of the product.

---

### Three Things to Cover on the Call

1. **HRIS integration** — it's the single incomplete integration and the likely blocker for auto-syncing the full employee directory. Without it, adoption stays limited to manually-added users. Get a timeline and owner.

2. **First redemption** — recognition is flowing but no one has redeemed yet. Possible causes: catalog not configured, points haven't accrued enough, or employees don't know how. This is the missing half of the product value; unblock it immediately.

3. **Sustain the giving momentum into the long tail** — active givers are growing fast, but 15/day is still likely a fraction of total headcount (2 admins suggests a small-to-mid team). Ask how they plan to reach employees who haven't given yet, and whether the current pace feels sustainable to the champion.
renewal-risk-conflicting-datesPASS 1.00

? The Question

Produce a 90-day renewal risk brief. Two systems hold renewal dates and disagree on some accounts; decide which to trust per account and say why (multi-year contracts are known to be wrong in ChurnZero). For every renewal: company, CSM, ARR, date used, seat utilization, 3-month usage trend, risk rating with one sentence of evidence. Flag every disagreement. Close with total ARR renewing and ARR at risk.
cost $0.0659 · 328s · in 14,137 / out 15,035 tok
✓ total_renewing✓ arr_at_risk✓ disagreements_flagged✓ trust_rule

✓ Correct Answer (ground truth)

snapshot_date: 2026-09-05
window: 2026-09-05 to 2026-12-04
trust_rule: multi-year contracts: Chargebee is authoritative (ChurnZero known wrong); otherwise systems agree or Chargebee wins
accounts: 20 items — e.g. account_alias=C-0B144C78; csm=Cole Ingram; arr=30899.00; trusted_renewal_date=2026-11-02; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=75.4; usage_3m_ratio=1.03; risk=low; evidence=3-month usage ratio 1.03 (last3 avg 103 vs prior3 100), seat utilization 75% | account_alias=C-0B20DB64; csm=Dana Mercer; arr=21770.00; trusted_renewal_date=2026-10-07; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=56.6; usage_3m_ratio=1.0; risk=medium; evidence=3-month usage ratio 1.00 (last3 avg 295 vs prior3 295), seat utilization 57% | account_alias=C-0B344485; csm=Elena Sinclair; arr=64384.00; trusted_renewal_date=2026-11-16; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=78.0; usage_3m_ratio=1.04; risk=low; evidence=3-month usage ratio 1.04 (last3 avg 241 vs prior3 231), seat utilization 78% | account_alias=C-0B369871; csm=Dana Mercer; arr=85128.00; trusted_renewal_date=2026-10-29; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=75.1; usage_3m_ratio=1.04; risk=low; evidence=3-month usage ratio 1.04 (last3 avg 330 vs prior3 316), seat utilization 75%
disagreements: account_alias=C-0B7D2C30; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-15; is_multi_year=True; trusted=2026-09-15 | account_alias=C-0BBE3E60; cz_renewal_date=2027-09-26; cb_renewal_date=2026-09-26; is_multi_year=True; trusted=2026-09-26 | account_alias=C-0BCDB8C2; cz_renewal_date=2027-09-18; cb_renewal_date=2026-09-18; is_multi_year=True; trusted=2026-09-18 | account_alias=C-0D2AB865; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-22; is_multi_year=True; trusted=2026-09-22 | account_alias=C-0F5D2323; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-29; is_multi_year=True; trusted=2026-09-29
n_disagreements: 5
total_arr_renewing_90d: 1048715.00
arr_at_risk_90d: 359409.00
n_accounts: 20
graded checks: total_renewing · arr_at_risk · disagreements_flagged · trust_rule

Model's actual answer

show the model's full answer
90-DAY RENEWAL RISK BRIEF
As of: 2026-09-08 | Window: Sep 8 - Dec 8, 2026

DATE-SOURCE DECISION RULE
Multi-year contracts: use Chargebee (CZ known wrong for multi-year).
Single-year contracts: use ChurnZero (both systems agree in all single-year cases).

DATE DISAGREEMENTS: 5 accounts (all multi-year)

  C-0B7D2C30  CZ=2026-09-10  CB=2026-09-15  [5 days apart]
    -> Using CB 2026-09-15 (multi-year; CZ unreliable)
  C-0BCDB8C2  CZ=2027-09-18  CB=2026-09-18  [YEAR MISMATCH: CZ=2027, CB=2026]
    -> Using CB 2026-09-18 (multi-year; CZ unreliable; CZ off by a full year)
  C-0D2AB865  CZ=2026-09-10  CB=2026-09-22  [12 days apart]
    -> Using CB 2026-09-22 (multi-year; CZ unreliable)
  C-0BBE3E60  CZ=2027-09-26  CB=2026-09-26  [YEAR MISMATCH: CZ=2027, CB=2026]
    -> Using CB 2026-09-26 (multi-year; CZ unreliable; CZ off by a full year)
  C-0F5D2323  CZ=2026-09-10  CB=2026-09-29  [19 days apart]
    -> Using CB 2026-09-29 (multi-year; CZ unreliable)

All 15 single-year accounts: CZ and CB dates match exactly.

RENEWAL SCHEDULE (sorted by date used)

Account         CSM                    ARR      Renewal     Seat Util  3mo Trend  Risk   Evidence
C-0B7D2C30     Dana Mercer           $65,901  2026-09-15       58%      -18.2%     HIGH   seat util 58% (low); 3mo usage -18.2% declining; 12mo attrition -46%
C-0BCDB8C2     Cole Ingram   $54,427  2026-09-18       55%      -17.6%     HIGH   seat util 55% (low); 3mo usage -17.6% declining; 12mo attrition -45%
C-0D2AB865     Elena Sinclair        $38,022  2026-09-22       61%      -18.9%     HIGH   seat util 61%; 3mo usage -18.9% declining; 12mo attrition -45%
C-0BBE3E60     Dana Mercer           $30,993  2026-09-26       65%      -19.5%     HIGH   seat util 65%; 3mo usage -19.5% declining; 12mo attrition -48%
C-0F5D2323     Cole Ingram   $90,647  2026-09-29       28%       +3.5%     HIGH   seat util 28% (critical); 3mo usage +3.5% growing
C-0EC6999D     Elena Sinclair        $79,419  2026-10-03       28%       +6.7%     HIGH   seat util 28% (critical); 3mo usage +6.7% growing
C-0B20DB64     Dana Mercer           $21,770  2026-10-07       57%       +0.1%   MEDIUM   seat util 57% (low); 3mo usage +0.1% stable
C-0BBC4E7A     Cole Ingram   $56,374  2026-10-10       68%       -0.9%   MEDIUM   seat util 68%; 3mo usage -0.9% stable
C-0FD551AB     Elena Sinclair        $48,815  2026-10-14       56%       -1.6%   MEDIUM   seat util 56% (low); 3mo usage -1.6% stable
C-0F9F8F13     Dana Mercer           $46,230  2026-10-18       57%       +0.2%   MEDIUM   seat util 57% (low); 3mo usage +0.2% stable
C-0BC34584     Cole Ingram   $16,740  2026-10-22       66%       +1.0%   MEDIUM   seat util 66%; 3mo usage +1.0% stable
C-0B7A7546     Elena Sinclair        $35,062  2026-10-25       89%       +4.3%    LOW    seat util 89% (healthy); 3mo usage +4.3% growing
C-0B369871     Dana Mercer           $85,128  2026-10-29       75%       +4.3%    LOW    seat util 75% (healthy); 3mo usage +4.3% growing
C-0B144C78     Cole Ingram   $30,899  2026-11-02       75%       +3.0%    LOW    seat util 75% (healthy); 3mo usage +3.0% growing
C-0FC4DBB8     Elena Sinclair        $94,732  2026-11-05       77%       +4.2%    LOW    seat util 77% (healthy); 3mo usage +4.2% growing
C-0D5BBE3A     Dana Mercer           $39,740  2026-11-09       83%       +3.9%    LOW    seat util 83% (healthy); 3mo usage +3.9% growing
C-0FB9D5AF     Cole Ingram   $63,158  2026-11-13       72%       +4.6%    LOW    seat util 72% (healthy); 3mo usage +4.6% growing
C-0B344485     Elena Sinclair        $64,384  2026-11-16       78%       +4.2%    LOW    seat util 78% (healthy); 3mo usage +4.2% growing
C-0CB2C1B4     Dana Mercer           $40,628  2026-11-20       82%       -2.0%   MEDIUM   seat util 82% (healthy); 3mo usage -2.0% declining
C-22170CA1     Cole Ingram   $45,646  2026-11-24       85%       +2.6%    LOW    seat util 85% (healthy); 3mo usage +2.6% growing

RISK SUMMARY

  HIGH:   6 accounts  |  $359,409 ARR
  MEDIUM: 6 accounts  |  $230,557 ARR
  LOW:    8 accounts  |  $458,749 ARR

  Total ARR renewing in 90 days:  $1,048,715
  ARR at risk (HIGH):             $359,409  (34% of total)
  ARR at risk (HIGH + MEDIUM):    $589,966  (56% of total)

Key callouts:

The four declining multi-year accounts (C-0B7D2C30, C-0BCDB8C2, C-0D2AB865, C-0BBE3E60) show ~45-48% active-user attrition over 12 months and consistent 17-20% quarterly decay. Seat utilization sits 55-65%, meaning roughly 1 in 3 licensed seats is idle. These are early-downgrade or non-renew signals -- the decay rate is accelerating, not stabilizing.

C-0F5D2323 and C-0EC6999D are the highest-ARR accounts in the HIGH bucket ($90,647 and $79,419) but their risk is driven purely by seat utilization (28% each, meaning only 111 and 31 of hundreds of licensed seats are in use). Their usage trends are actually slightly positive. These may be accounts that over-provisioned and could renew at a lower seat count, but the margin compression is severe: at 28% utilization, a 60% seat reduction at renewal would cut ARR by ~$54k and ~$48k respectively.

C-0CB2C1B4 is the only account with healthy seat utilization (82%) but a declining 3-month trend (-2.0%). Not urgent but worth watching -- if the downtick continues it could become a Q4 concern.
ticket-theme-synthesis0.80

? The Question

Synthesize the quarter's support tickets into 4 to 7 themes. Do not trust existing tags; read the text. Per theme: name, count, share, distinct accounts, ARR affected, two ticket ids, one-line recommendation. Separate broad patterns from single-account noise and rank by ARR exposure, not volume.
cost $0.0556 · 275s · in 10,030 / out 12,998 tok
✗ theme_count_range✓ counts_verifiable✓ single_account_flagged✓ arr_ranked✓ ticket_ids

✓ Correct Answer (ground truth)

quarter_window: 2026-06-01 through 2026-08-31
total_tickets: 80
themes: {"points_not_posting": {"count": 20, "share_pct": 25.0, "distinct_accounts": 9, "arr_affected": "31100.00", "example_ticket_ids": ["IC-460001", "IC-460002"], "single_account": false}, "slack_sync": {"
ranking_by_arr_exposure: hris_provisioning, redemption_failures, billing_dispute, points_not_posting, slack_sync
single_account_noise_theme: billing_dispute
note: existing tags are noisy/unreliable; classify by text
sample_ticket_ids: 80 items — e.g. IC-460004, IC-460016, IC-460001, IC-460041
graded checks: theme_count_range · counts_verifiable · single_account_flagged · arr_ranked · ticket_ids

Model's actual answer

show the model's full answer
SUPPORT TICKET THEMES — Q3 2026 (80 tickets total)
Ranked by ARR exposure

─────────────────────────────────────────────────────

1. HRIS PROVISIONING FAILURE
   Count: 12  |  Share: 15.0%  |  Accounts: 3  |  ARR: $114,000
   IDs: IC-460059, IC-460062
   Rec: Audit the HRIS sync webhook — silent failures with no log errors suggest the connector is receiving payloads but not processing them.

2. REDEMPTION / CHECKOUT FAILURES
   Count: 18  |  Share: 22.5%  |  Accounts: 7  |  ARR: $68,800
   IDs: IC-460025, IC-460024
   Rec: Add point-debit rollback on redemption errors; investigate the gift-card vendor API timeout causing the "spins forever" pattern.

3. BILLING / INVOICE ERRORS [SINGLE-ACCOUNT NOISE]
   Count: 16  |  Share: 20.0%  |  Accounts: 1  |  ARR: $52,000
   IDs: IC-460078, IC-460071
   Rec: Escalate C-0E9C27D1 to a named CSM — 16 tickets from one $52K account across three sub-issues (wrong tier, seat-count discrepancy, recurring seat error) signals a systemic billing-config error, not a volume trend.

4. POINTS NOT POSTING
   Count: 20  |  Share: 25.0%  |  Accounts: 9  |  ARR: $31,100
   IDs: IC-460004, IC-460016
   Rec: Trace the points ledger write-path for the weekend batch window — the "after the weekend" clustering and the "delivered but never arrived" pattern both point to an async posting job silently failing.

5. SLACK INTEGRATION BREAKAGE
   Count: 14  |  Share: 17.5%  |  Accounts: 4  |  ARR: $18,900
   IDs: IC-460047, IC-460046
   Rec: Investigate OAuth token refresh — the "re-auth does not stick" and "toggle resets itself" patterns indicate a credential-persistence bug, not a user error.

─────────────────────────────────────────────────────

ARITHMETIC CHECK

  Theme counts: 12 + 18 + 16 + 20 + 14 = 80 (matches total)
  Shares: 15.0 + 22.5 + 20.0 + 25.0 + 17.5 = 100.0%

  HRIS ARR:  $36,000 + $48,000 + $30,000 = $114,000
  Redemp ARR: $10,700 + $8,900 + $8,700 + $11,000 + $9,600 + $10,300 + $9,600 = $68,800
  Billing ARR: $52,000 (C-0E9C27D1 only)
  Points ARR: $2,900 + $2,700 + $3,400 + $3,500 + $2,500 + $4,500 + $4,500 + $4,200 + $2,900 = $31,100
  Slack ARR:  $4,400 + $3,900 + $5,400 + $5,200 = $18,900

BROAD vs. SINGLE-ACCOUNT NOTE

  Billing/Invoice Errors is entirely C-0E9C27D1 — 16 tickets, 1 account, 3
  distinct sub-complaints (wrong-tier pricing, seat-count discrepancy,
  recurring seat error). This is not a platform-wide billing bug; it is one
  $52K account hitting support repeatedly for the same unresolved issue.
  Treat as an escalation gap, not a product theme.

  The remaining four themes are broad (3-9 accounts each) and represent
  genuine product issues.
marketing
case-study-matchPASS 1.00

? The Question

Given the prospect, rank the three most similar existing customers with public case studies for use as social proof. Similarity on industry, size band, use case, region. Cite the fields that drove each match. Do not name any customer without a case study and never include billing data or contact names.
cost $0.0085 · 40s · in 721 / out 1,645 tok
✓ top3✓ decoys_excluded✓ cites_fields✓ no_billing

✓ Correct Answer (ground truth)

top3: company_alias=C-11C31562; similarity_score=3 | company_alias=C-64171065; similarity_score=3 | company_alias=C-A13C193D; similarity_score=3
decoys_excluded: company_alias=C-D6217CAA; similarity_score=4; reason=no_case_study | company_alias=C-C153868F; similarity_score=4; reason=no_case_study
constraints: no_billing_data, no_contact_names, only_customers_with_has_case_study_true_named
prospect_fields_used: industry, size_band, use_case, region
prospect_alias: C-82AF3719
graded checks: top3 · decoys_excluded · cites_fields · no_billing

Model's actual answer

show the model's full answer
## Social-Proof Ranking for Prospect C-82AF3719

**Prospect profile:** Technology | Mid-Market | employee_recognition | NA-West

Only customers with `has_case_study = true` are eligible. Scoring: each matching dimension (industry, size_band, use_case, region) = 1 point; max 4.

---

### 1. C-64171065 — 3/4 match

| Dimension   | Prospect       | Customer       | Match |
|-------------|----------------|----------------|-------|
| industry    | Technology     | Technology     | Yes   |
| size_band   | Mid-Market     | Mid-Market     | Yes   |
| use_case    | employee_recognition | employee_recognition | Yes   |
| region      | NA-West        | NA-East        | No    |

Same industry, same size band, same use case — only region differs. Strongest social-proof fit: the prospect sees a customer that solved the exact same problem at the same scale, in the same sector.

---

### 2. C-11C31562 — 3/4 match

| Dimension   | Prospect       | Customer          | Match |
|-------------|----------------|-------------------|-------|
| industry    | Technology     | Manufacturing     | No    |
| size_band   | Mid-Market     | Mid-Market        | Yes   |
| use_case    | employee_recognition | employee_recognition | Yes   |
| region      | NA-West        | NA-West           | Yes   |

Same use case, same size band, same region — only industry differs. Compelling because the operational context (region + use case) is identical; cross-industry references still resonate when the problem and market are the same.

---

### 3. C-A13C193D — 3/4 match

| Dimension   | Prospect       | Customer          | Match |
|-------------|----------------|-------------------|-------|
| industry    | Technology     | Technology        | Yes   |
| size_band   | Mid-Market     | Mid-Market        | Yes   |
| use_case    | employee_recognition | retention   | No    |
| region      | NA-West        | NA-West           | Yes   |

Same industry, same size band, same region — only use case differs. Still relevant as a sector/geo peer, but the different use case (retention vs. employee_recognition) makes it the weakest of the three for direct social proof.

---

**Tiebreak rationale:** All three scored 3/4. Rank order prioritized use_case alignment (most relevant for social-proof credibility) > industry alignment > region alignment, because a prospect evaluating a specific product capability responds best to customers who deployed the same capability.

**No other case-study customer scored above 2/4.** Five customers with case studies matched on 0–2 dimensions only and were excluded from the top three.
channel-attribution-mathPASS 1.00

? The Question

Compute channel performance for the trailing 6 months. Paid: spend, SQMs, SQOs, cost per SQM, cost per SQO, SQM-to-SQO rate, pipeline amount, pipeline per dollar. Organic: volume, SQO rate, pipeline. Report a channel with spend and zero SQMs as undefined, not zero. Flag rows where SQO date precedes SQM date. Finish with a reallocation recommendation and your confidence given sample sizes.
cost $0.0336 · 80s · in 8,029 / out 6,463 tok
✓ undefined_channel✓ undefined_not_zero✓ flagged_rows✓ best_channel✓ reallocation

✓ Correct Answer (ground truth)

months: 2026-03, 2026-04, 2026-05, 2026-06, 2026-07, 2026-08
per_channel: {"linkedin_ads": {"spend_usd": 24000, "sqms": 25, "sqos": 8, "pipeline_amount": 96000, "cost_per_sqm": 960.0, "cost_per_sqo": 3000.0, "sqm_to_sqo_rate": 0.32, "pipeline_per_dollar": 4.0}, "organic_sea
undefined_channels: paid_social
flagged_rows_sqo_before_sqm: CT-000041, CT-000044
best_pipeline_per_dollar_channel: paid_search
reallocation_recommendation: from=paid_social; to=paid_search; rationale=paid_social has spend but zero SQMs (cost per SQM undefined, not zero); paid_search has the highest pipeline per dollar.
confidence_note: webinars and referral have small SQM samples (<20); treat rates as directional.
graded checks: undefined_channel · undefined_not_zero · flagged_rows · best_channel · reallocation

Model's actual answer

show the model's full answer
CHANNEL PERFORMANCE — TRAILING 6 MONTHS (2026-03 to 2026-08)

============================================================
PAID CHANNELS
============================================================
Channel          Spend    Vol  SQMs  SQOs   $/SQM      $/SQO      SQM→SQO    Pipeline    Pipe/$
------------------------------------------------------------------------------------------
paid_search      $36,000   40    40    18    $900       $2,000     45.0%      $720,000    $20.00
linkedin_ads     $24,000   25    25     8    $960       $3,000     32.0%      $96,000     $4.00
paid_social      $18,000    0     0     0    undefined  undefined  undefined  $0          $0.00
webinars         $9,000    12    12     5    $750       $1,800     41.7%      $60,000     $6.67

Arithmetic shown:
  paid_search:  $36,000 / 40 SQMs = $900/SQM;  $36,000 / 18 SQOs = $2,000/SQO;  18/40 = 45.0%;  $720,000 / $36,000 = $20.00 pipe/$
  linkedin_ads: $24,000 / 25 SQMs = $960/SQM;  $24,000 / 8 SQOs = $3,000/SQO;   8/25 = 32.0%;   $96,000 / $24,000 = $4.00 pipe/$
  paid_social:  0 SQMs and 0 SQOs — $/SQM, $/SQO, and SQM→SQO all reported as undefined (not zero) per your rule
  webinars:     $9,000 / 12 SQMs = $750/SQM;   $9,000 / 5 SQOs = $1,800/SQO;    5/12 = 41.7%;   $60,000 / $9,000 = $6.67 pipe/$

============================================================
ORGANIC CHANNELS (zero spend)
============================================================
Channel          Volume   SQMs  SQOs   SQO Rate    Pipeline
----------------------------------------------------------
organic_search     30      30    10    33.3%       $90,000
referral           15      15     6    40.0%       $48,000

Arithmetic: organic_search 10/30 = 33.3%; referral 6/15 = 40.0%.
Organic channels deliver $138,000 combined pipeline at zero spend.

============================================================
DATA QUALITY FLAGS: SQO DATE PRECEDES SQM DATE
============================================================
  linkedin_ads: CT-000044 — SQM 2026-07-23, SQO 2026-07-18 (SQO 5 days before SQM)
  linkedin_ads: CT-000041 — SQM 2026-06-14, SQO 2026-06-09 (SQO 5 days before SQM)

Both flags are in linkedin_ads. No other channel has this issue.

============================================================
REALLOCATION RECOMMENDATION
============================================================
1. PAID_SOCIAL ($18,000/6mo): Zero attributed contacts, zero SQMs, zero SQOs across all 6 months. This is almost certainly an attribution/tracking failure, not true zero performance. Before reallocating, fix the tracking — if paid_social genuinely drives no attributable pipeline after tracking is repaired, pull the full $18K.

2. IF tracking confirms paid_social is truly dead, reallocate its $18K to paid_search:
   - paid_search returns $20 pipeline per $1 — the highest ROI channel by 5x over linkedin_ads.
   - Additional $18K at $20 pipe/$ = ~$360,000 incremental pipeline (theoretical, before diminishing returns).

3. LINKEDIN_ADS ($24,000): $4 pipe/$ is 5x worse than paid_search, and it carries both data-quality flags (SQO-before-SQM on CT-000044 and CT-000041). Recommend trimming linkedin_ads by ~50% ($12K) and shifting to paid_search, pending investigation of the date-inversion issue. Keep a reduced test budget only if the targeting thesis is still being validated.

4. WEBINARS ($9,000): $6.67 pipe/$ with 41.7% SQM→SQO — second-best paid channel. Maintain current spend; sample is too small to justify an increase.

Proposed reallocation (assuming paid_social tracking confirms zero):
  paid_search:  $36K → $66K  (+$18K from paid_social + $12K from linkedin_ads)
  linkedin_ads: $24K → $12K  (-$12K)
  paid_social:  $18K → $0    (-$18K, pending tracking fix)
  webinars:     $9K  → $9K   (unchanged)

============================================================
CONFIDENCE: MEDIUM
============================================================
Reasons:
  - paid_search: 40 contacts, 18 SQOs — adequate sample; the $20 pipe/$ and 45% conversion are reliable directional signals.
  - linkedin_ads: 25 contacts, 8 SQOs — moderate sample; $4 pipe/$ is directionally trustworthy but the 2 SQO-before-SQM flags introduce measurement uncertainty (pipeline attribution may be slightly off).
  - webinars: 12 contacts, 5 SQOs — small sample; $6.67 pipe/$ could swing meaningfully with a few more data points.
  - paid_social: 0 contacts — no sample at all; cannot evaluate true performance. This is the single biggest risk to the recommendation. If tracking is broken, paid_social may actually be contributing and the reallocation math changes.
  - Organic channels (30 and 15 contacts) confirm they are meaningful pipeline contributors at zero cost, reinforcing that they should be nourished but not the focus of a spend reallocation.
competitor-battlecard-updatePASS 1.00

? The Question

Update the battlecard for the competitor. Sections: one-line positioning, pricing with source and date (newer source wins, note the conflict), where they win, where we win, objections and responses, recent changes, our 12-month win/loss record against them. Cite a snippet id for every factual claim. Rep opinion on a call is not a fact about the competitor. Anything from the old card you cannot re-source gets marked unverified.
cost $0.0162 · 104s · in 1,370 / out 3,952 tok
✓ competitor_named✓ pricing✓ win_loss✓ unverified_marked✓ citations

✓ Correct Answer (ground truth)

competitor: Rivally
pricing: value_per_user_per_month=7; source_snippet=S17; source_date=2026-08-12
win_loss_12mo: wins=13; losses=7
unverified_old_claims: Rivally lacks a Slack integration., Rivally was acquired by WorkHuman in 2025.
rep_opinion_snippet_ids_not_facts: S09, S21
citation_ids_available: 25 items — e.g. S01, S02, S03, S04
graded checks: competitor_named · pricing · win_loss · unverified_marked · citations

Model's actual answer

show the model's full answer
# Battlecard: Rivally (updated 2026-09-08)

---

## One-line positioning

Points-based recognition vendor pivoting from mid-market into EU enterprise via data-residency positioning and add-on survey product.

[S02] [S11] [S15]

---

## Pricing

| Tier | Price | Source | Date | Notes |
|---|---|---|---|---|
| Recognition Starter | $7/user/mo, annual billing required | S17 (pricing_page) | 2026-08-12 | **Current list price** |
| Recognition Starter | $5/user/mo, annual billing required | S03 (pricing_page) | 2026-01-20 | Superseded by S17 |
| Recognition Starter | $5/user/mo | S08 (pricing_page) | 2026-04-01 | Superseded by S17 |
| Quoted to 500-seat prospect | $6.50/user/mo, annual term | S13 (call_notes) | 2026-06-02 | Interim quote — price was moving from $5 toward $7 |
| Quoted list with discount | $7/user/mo list, 15% off for 3-year term | S18 (call_notes) | 2026-08-14 | $7 minus 15% = $5.95/user/mo on 3-year |
| Rivally Pulse | Priced as add-on, not bundled | S23 (press) | 2026-09-01 | Separate line item |

**Conflict note:** S03/S08 show $5/user/mo through April 2026. S17 (Aug 2026) shows $7/user/mo. Newer source wins — current list is $7. The $6.50 quote in June [S13] sits between the two published prices, suggesting a mid-year increase. S18 indicates willingness to discount 15% on multi-year terms ($5.95 effective).

---

## Where they win

- **EU/distributed teams needing data residency.** EU data residency GA [S15], Dublin office [S15], ex-Workday VP EMEA hire [S11], multi-language support praised [S12].
- **Mid-market speed-to-value.** Setup under a week, Slack integration out of the box [S04].
- **Recognition feed UX.** Points-based feed consistently praised [S02] [S16].
- **Support responsiveness.** Response time under 4 hours [S22].
- **Incumbent familiarity for Workday ecosystem buyers.** Ex-Workday VP signals enterprise push [S11].
- **Teams-native orgs.** Microsoft Teams app v2 in public preview [S19].

---

## Where we win

- **Analytics depth.** Rivally's analytics called "limited" [S02], "basic compared to enterprise tools" [S07], exports CSV-only [S20]. 800-seat prospect picked Bonusly explicitly citing analytics depth [S25].
- **Enterprise provisioning.** No SCIM; manual user management called "painful" [S10].
- **Admin tooling.** "Admin tooling lags peers" [S16]; no bulk recognition editing [S24].
- **Data portability / lock-in risk.** CSV-only analytics exports make migration off Rivally hard [S20] — which also means their customers struggle to leave, but it signals weak integration posture.
- **EMEA rewards catalog.** Thinner than US catalog [S14].
- **Pricing transparency / value.** Price rose 40% ($5→$7) in under 8 months [S03→S17]; Pulse is an extra-cost add-on [S23].

---

## Objections and responses

| Objection | Response | Source |
|---|---|---|
| "Rivally has EU data residency" | Acknowledge it's GA. Ask what specific regulation requires it; many orgs satisfy GDPR with US-hosted SOC 2 + DPA. If hard requirement, quantify how often it actually drives deal vs. table-stakes checkbox. | S15 |
| "Rivally is cheaper" | Their list price rose from $5 to $7/user/mo in under 8 months [S03→S17]. Even with 15% multi-year discount they land at ~$5.95 [S18] — comparable to our value with deeper analytics included, not add-on'd. Pulse is an extra charge [S23]. | S03, S17, S18, S23 |
| "Rivally has Pulse for surveys" | Pulse is a paid add-on, not included [S23]. Our engagement analytics are built in, not an extra line item. | S23 |
| "Rivally's Slack integration works out of the box" | True for basic recognition. Probe whether they need SCIM provisioning, bulk admin, or export-grade analytics — those are gaps. | S04, S10, S24, S20 |

---

## Recent changes (last 12 months)

| Date | Change | Source |
|---|---|---|
| 2025-11 | Series C $40M (Northgate Ventures) | S01 |
| 2026-01 → 2026-08 | Price increase: $5 → $7/user/mo on pricing page | S03, S17 |
| 2026-03 | Rivally Pulse survey add-on launched (beta) | S06 |
| 2026-05 | Hired ex-Workday VP EMEA | S11 |
| 2026-07 | Dublin office opened; EU data residency GA | S15 |
| 2026-08 | Microsoft Teams app v2 public preview | S19 |
| 2026-09 | Rivally Pulse exits beta; priced as paid add-on | S23 |

---

## 12-month win/loss record against Rivally (Sep 2025 – Aug 2026)

| Month | Wins | Losses | Deal aliases |
|---|---|---|---|
| 2025-09 | 1 | 1 | W: 072E31 / L: 7767F5 |
| 2025-10 | 2 | 0 | W: A9FD43, F65C8F |
| 2025-11 | 1 | 1 | W: 7AA785 / L: D263E0 |
| 2025-12 | 1 | 1 | W: 44C524 / L: 935746 |
| 2026-01 | 2 | 0 | W: 0D0CD6, E46EAB |
| 2026-02 | 2 | 0 | W: D5B790, 1D2392 |
| 2026-03 | 1 | 1 | W: 5C636E / L: 9066A6 |
| 2026-04 | 0 | 2 | L: 5645A5, 72A02F |
| 2026-05 | 0 | 1 | L: C6FFAA |
| 2026-06 | 1 | 0 | W: 67BE14 |
| 2026-07 | 1 | 0 | W: 1B6969 |
| 2026-08 | 1 | 0 | W: F03E7B |

**Total: 13 wins, 7 losses — 65% win rate (20 deals)**

Trend note: 3 losses in Apr–May 2026 followed their EU push and price still at $5 [S08]. Win rate recovered Jun–Aug after their price increase to $7 [S17].

---

## Claims from old card — verification

| Old claim | Status | Detail |
|---|---|---|
| "Points-based recognition for mid-market" | **Updated** | Still points-based [S02], but repositioning toward EU enterprise [S11, S15]. |
| "Starts at $5/user/month" | **Superseded** | $5 was correct Jan–Apr 2026 [S03, S08]; current list is $7 [S17]. |
| "Rivally lacks a Slack integration" | **Incorrect** | S04 confirms Slack integration works out of the box. |
| "Rivally was acquired by WorkHuman in 2025" | **Unverified** | No snippet sources an acquisition. S01 reports a $40M Series C, not an acquisition. This claim cannot be re-sourced — marked unverified. |
| "Strong in EU enterprise with multi-language support" | **Confirmed** | S12, S15. |
nurture-sequence-diagnosisPASS 1.00

? The Question

Review the sequences. Per sequence: sent, open rate, reply rate, meeting rate, weakest step. Identify tracking errors (opened above sent), audience overlap between sequences, and the failure mode for anything under 2 percent reply. One change per weak sequence and which to fix first. Under 350 words.
cost $0.0227 · 79s · in 11,606 / out 2,839 tok
✓ tracking_error✓ overlap_pair✓ weakest_step✓ fix_priority

✓ Correct Answer (ground truth)

per_sequence: {"New Logo Nurture": {"sent": 1386, "opened": 490, "replied": 90, "meetings": 27, "open_rate": 0.3535, "reply_rate": 0.0649, "meeting_rate": 0.0195, "weakest_step": 3}, "Expansion Nurture": {"sent": 8
tracking_error: sequence=Expansion Nurture; step=2; sent=300; opened=340; issue=opened exceeds sent
audience_overlap_pair: Cold Outbound - HR Leaders, Cold Outbound - People Ops
all_pairwise_overlaps: ['Cold Outbound - HR Leaders', 'Cold Outbound - People Ops'], ['Expansion Nurture', 'New Logo Nurture']
under_2pct_reply: Cold Outbound - HR Leaders
failure_mode: Cold Outbound - HR Leaders=opens healthy (~37%) but replies near zero from step 1 -> body/CTA not landing, not a deliverability or subject-line problem
fix_priority_first: Cold Outbound - HR Leaders
graded checks: tracking_error · overlap_pair · weakest_step · fix_priority

Model's actual answer

show the model's full answer
**Per-Sequence Breakdown**

| Sequence | Step | Sent | Open Rate | Reply Rate | Meeting Rate | Weakest Step |
|---|---|---|---|---|---|---|
| New Logo Nurture | 1 | 500 | 42.0% | 20.0% | 28.6% | Step 3 |
| | 2 | 458 | 34.9% | 18.8% | 30.0% | |
| | 3 | 428 | 28.0% | 15.0% | 33.3% | |
| Expansion Nurture | 1 | 300 | 43.3% | 16.9% | 22.7% | Step 2 (tracking) |
| | 2 | 300 | 113.3%* | 7.4% | 16.0% | |
| | 3 | 275 | 34.5% | 12.6% | 25.0% | |
| Cold Outbound - HR Leaders | 1 | 600 | 40.0% | 2.1% | 0% | Step 3 (but all fail) |
| | 2 | 595 | 29.4% | 1.1% | 0% | |
| | 3 | 590 | 22.0% | 0.8% | 0% | |
| Cold Outbound - People Ops | 1 | 400 | 37.5% | 9.3% | 21.4% | Step 3 |
| | 2 | 386 | 28.5% | 8.2% | 22.2% | |
| | 3 | 377 | 21.2% | 7.5% | 16.7% | |

*Reply rate = replies/opened. Meeting rate = meetings/replied. Reply/sent shown below for the <2% test.*

**Tracking Error**

Expansion Nurture Step 2: opened=340 on sent=300 (113% open rate). Either a double-send fired or the tracking pixel is firing on forwards/forwards to the same contact. This inflates open counts and corrupts downstream rates.

**Audience Overlap**

Nineteen contact_keys appear in both Cold Outbound - HR Leaders and Cold Outbound - People Ops (CT-000849, CT-000890, CT-001033, CT-001105, CT-001109/113/130/153/217/227/255/258/277/285/311/345). Three contacts appear in both New Logo Nurture and Expansion Nurture (CT-000301, CT-000624, CT-001285 — note CT-000301 is in both). The HR Leaders/People Ops overlap means the same person may be getting two parallel cold sequences, depressing reply rates in both.

**Under-2% Reply (reply/sent)**

- Cold Outbound - HR Leaders: 0.83% / 0.34% / 0.17% — all three steps below 2%. Zero meetings total.
- Cold Outbound - People Ops Step 3: 1.6% (6/377).

Failure mode for HR Leaders: opens are healthy (40% at step 1) but reply collapses — message content doesn't resonate with the audience. This is a relevance/value problem, not deliverability. Failure mode for People Ops Step 3: sequence fatigue (21% open, 1.6% reply/sent) after two prior touches.

**One Change Per Weak Sequence**

1. Cold Outbound - HR Leaders — rewrite step 1 with a role-specific pain hook (e.g., turnover-cost dollar figure or HR-admin burden) instead of generic pitch. 40% open → 0.83% reply means the subject earns the click but the body fails.
2. Cold Outbound - People Ops — replace step 3 email with a different channel (LinkedIn interaction or short video email) to break fatigue.
3. Expansion Nurture — audit step 2 send logic; deduplicate any double-fire and exclude forwards from open pixel counts.

**Fix first: Cold Outbound - HR Leaders.** Highest volume (600 contacts) and worst outcome (0 meetings, 0.83% step-1 reply/sent). The opportunity cost of 600 zero-converting contacts exceeds the smaller volume on the other two.
weekly-marketing-goals-rollupPASS 1.00

? The Question

Produce the weekly marketing goals update for the active quarter: SQMs, SQOs, DS2s, closed-lost MIA rate, same-quarter close count, active pipeline coverage against target. For each: QTD actual, target, delta, pace (on/behind/ahead given days elapsed). One paragraph on what moved this week.
cost $0.0064 · 33s · in 319 / out 1,075 tok
✓ metrics_present✓ deltas✓ coverage_ratio✓ pace✓ paragraph

✓ Correct Answer (ground truth)

days_elapsed: 66
days_in_quarter: 92
elapsed_fraction: 0.7174
metrics: {"SQMs": {"qtd_actual": 230, "target": 300, "delta": -70, "expected_at_day_elapsed": 215.22, "pace": "ahead"}, "SQOs": {"qtd_actual": 84, "target": 120, "delta": -36, "expected_at_day_elapsed": 86.09,
coverage_ratio_active_pipeline: 0.75
pace_rule: expected = target * days_elapsed/days_in_quarter; within +/-5% of expected = on, above = ahead, below = behind; lower_better metrics invert.
this_week_paragraph_required: True
graded checks: metrics_present · deltas · coverage_ratio · pace · paragraph

Model's actual answer

show the model's full answer
## Q3-2026 Weekly Marketing Goals Update (Day 66 of 92)

| Metric | QTD Actual | Target | Delta | Pace vs 71.7% Elapsed | Status |
|---|---|---|---|---|---|
| SQMs | 230 | 300 | -70 | 76.7% (230/300) | **Ahead** |
| SQOs | 84 | 120 | -36 | 70.0% (84/120) | **Behind** |
| DS2s | 40 | 75 | -35 | 53.3% (40/75) | **Behind** |
| CL MIA Rate | 5/25 = 20.0% | 10.0% (lower better) | +10.0pp above target | 2.0x target rate | **Behind** |
| Same-Qtr Closes | 10 | 20 | -10 | 50.0% (10/20) | **Behind** |
| Active Pipeline | $3.0M | $4.0M | -$1.0M | 75.0% (3.0/4.0) | **Ahead** |

Pace logic: elapsed = 66/92 = 71.7%. If actual/target ≥ 71.7%, status = Ahead (for higher_better); if < 71.7%, Behind. For lower_better (MIA rate), being at 2x the target at ~72% elapsed is Behind.

**What moved this week:** SQMs and active pipeline are the only metrics tracking ahead of pace, but both are modestly so — pipeline at 75% of target vs 71.7% elapsed leaves only $1M to build in the remaining 26 days. The real pressure is in the mid-funnel: SQOs are pacing 2 points behind elapsed time (70.0% vs 71.7%), and DS2s are critically behind at 53.3% of target — implying a conversion gap between SQM and SQO as well as between SQO and DS2. The closed-lost MIA rate of 20% is double the 10% target, meaning 1 in 5 losses this quarter has no documented next step or re-engagement plan, which compounds the same-quarter close shortfall (10 of 20, pacing at half). With 26 days remaining, DS2s need 35 more (1.35/day vs the 0.61/day run rate to date) and same-quarter closes need 10 more (0.38/day vs 0.15/day run rate) — both requiring acceleration well beyond current trend to hit target.
communication
ceo-slack-compression0.80

? The Question

Compress the forecast analysis below into a single Slack message for the CEO. Rules: at most three sentences; under 80 words; lead with the decision or the so-what; keep every distinct number that matters and the one caveat that changes the conclusion; cut any sentence that does not carry a claim, a number, or a directive. No em dashes, no throat-clearing phrases.
cost $0.0037 · 11s · in 618 / out 85 tok
✓ max_3_sentences✓ under_80_words✓ weighted_present✗ caveat_present✓ no_em_dash

✓ Correct Answer (ground truth)

graded checks: max_3_sentences · under_80_words · weighted_present · caveat_present · no_em_dash

Model's actual answer

show the model's full answer
Q3 weighted forecast is $115,977 (100% of $44,729 COMMIT + 35% of $203,565 BEST_CASE), but all 7 COMMIT deals lack documented why-buys and 85 of 86 deals have no owner, so the committed number is unvalidated. Push the 32 out-of-quarter deals back to reps for re-qualification before board reporting.
stalled-deal-followup-emailPASS 1.00

? The Question

Write a follow-up email for this stalled deal. Under 120 words. Reference one specific prior touch from the engagement history, make one clear ask, and match the tone of the prior thread. No invented details.
cost $0.0047 · 17s · in 428 / out 468 tok
✓ word_cap✓ references_touch✓ single_ask

✓ Correct Answer (ground truth)

deal_alias: Deal-0D2F7A
amount: 5100
stage: DS3
owner: Alex Franklin
days_since_last_engagement: 31
word_cap: 120
must_reference: engagement_id=ENG-00002; date=2026-07-28; type=meeting; why=most substantive prior touch; concrete prospect reactions to milestone awards and points catalog
single_cta: True
tone: warm, concise, professional (matches prior thread)
allowed_facts: 14 items — e.g. Deal-0D2F7A, $5,100, DS3, Alex Franklin
allowed_new: ENG-00001, ENG-00002, ENG-00003, ENG-00004, ENG-00005
graded checks: word_cap · references_touch · single_ask

Model's actual answer

show the model's full answer
Subject: Pricing recap for 150 seats — next steps?

Hi [VP People],

I sent over the pricing recap from our July 28 demo on August 5 and wanted to circle back. The team's interest in the automated milestone awards and points catalog was clear — I'd hate to let that momentum sit.

Would you have 15 minutes this week or next to walk through any questions on the proposal? Happy to adjust the seat count or scope if the budget conversation has shifted.

Best,
Alex

---

**References cited:**
- Prior touch: ENG-00001 (Aug 5 outbound — recap of July 28 demo + pricing for 150 seats, no reply)
- Specific ask: 15-minute call to discuss questions on the proposal
- Tone match: direct, professional, low-pressure outbound (consistent with ENG-00001/ENG-00003 pattern)
- No invented details: seat count (150), demo date (July 28), features (milestone awards, points catalog) all sourced from engagement snippets
funnel-digest-paragraphPASS 1.00

? The Question

Write the weekly GTM digest: one paragraph per team (Marketing, Sales, CS, PLG) in Bonusly brand voice. Every number must trace to the extracts. No customer contact names, no employee performance commentary, no compensation. Four paragraphs, under 400 words total.
cost $0.0048 · 20s · in 291 / out 548 tok
✓ four_paragraphs✓ word_cap✓ teams

✓ Correct Answer (ground truth)

required_paragraphs: Marketing, Sales, CS, PLG
allowed_numbers: 16 items — e.g. 2, 3, 9, 08
word_cap: 400
banned: customer contact names, employee performance commentary, compensation
graded checks: four_paragraphs · word_cap · teams

Model's actual answer

show the model's full answer
**Marketing** -- We landed 46 SQMs this week against a 52 target -- a gap we'll rally on next week. The bright spot: our webinar channel delivered 18 of those SQMs, proving the power of showing up live. Paid search cost per SQM held steady at $150, keeping our acquisition economics consistent.

**Sales** -- The team converted 14 SQOs and booked 9 DS2 meetings, building real momentum in the pipeline. New pipeline created hit $310,000, and we closed 3 same-quarter deals -- solid execution when it counted.

**CS** -- Two renewals saved this week, keeping revenue protected where it matters. Team NPS moved to 61, reflecting the recognition-first experience we deliver. Heading into next week, we're actively managing 3 open red-flag accounts and staying close to those customers.

**PLG** -- Self-serve momentum continues: 412 new signups joined the platform, with activation at 31 percent. The standout signal -- 38 companies reached the aha moment of 10 recognition sends, crossing the threshold where engagement compounds.
partner-digest-sectionPASS 1.00

? The Question

Write the partner digest for the period. Every partner in the roster appears once: active plays get status with numbers, quiet partners get one line. Partner-sourced pipeline count and amount must match the deal data. Nothing omitted, nothing added.
cost $0.0058 · 25s · in 483 / out 818 tok
✓ all_partners✓ pipeline_amount✓ quiet_one_line

✓ Correct Answer (ground truth)

roster: partner_name=Apex Rewards Co; status=active | partner_name=HRCloud Partners; status=active | partner_name=CultureBridge; status=active | partner_name=WorkWell Group; status=active | partner_name=Recogniq; status=quiet | partner_name=KudosWave; status=quiet … (+2 more)
per_partner_pipeline: {"Apex Rewards Co": {"count": 2, "amount": 275000}, "HRCloud Partners": {"count": 1, "amount": 140000}, "CultureBridge": {"count": 2, "amount": 135000}}
partner_sourced_pipeline: count=5; amount=550000
quiet_partners: Recogniq, KudosWave, PeopleFirst Advisors, TotalPerk
active_no_deals: WorkWell Group
rule: every roster partner named exactly once; quiet partners get one line; no partners outside roster.
graded checks: all_partners · pipeline_amount · quiet_one_line

Model's actual answer

show the model's full answer
# PARTNER DIGEST — Period Ending 2026-09-04

---

## Active Partners

**Apex Rewards Co**
Co-webinar locked for 09-15. Brought 2 sourced opps this period, both in DS1; both logged UTM Source = Partner.
Sourced pipeline: 2 deals / $275,000 (Deal-DDAAF2 $180,000 + Deal-2CF33E $95,000)

**HRCloud Partners**
Integration referral cleared security review; opp advanced to DS2. 1 sourced opp this period.
Sourced pipeline: 1 deal / $140,000 (Deal-F1CDA5)

**CultureBridge**
Lunch-and-learn produced 2 sourced opps, both early stage.
Sourced pipeline: 2 deals / $135,000 (Deal-096E1D $60,000 + Deal-067213 $75,000)

**WorkWell Group**
Seeking to restart joint playbook in Q4; no sourced deals this period. Planning call booked for 09-09.
Sourced pipeline: 0 deals / $0

---

## Quiet Partners

**Recogniq** — No activity this period.
**KudosWave** — No activity this period.
**PeopleFirst Advisors** — No activity this period.
**TotalPerk** — No activity this period.

---

## Summary

| Partner | Sourced Deals | Sourced Amount |
|---|---|---|
| Apex Rewards Co | 2 | $275,000 |
| HRCloud Partners | 1 | $140,000 |
| CultureBridge | 2 | $135,000 |
| WorkWell Group | 0 | $0 |
| **Active total** | **5** | **$550,000** |
| Quiet (4 partners) | 0 | $0 |
| **All partners** | **5** | **$550,000** |

Arithmetic: $180,000 + $95,000 + $140,000 + $60,000 + $75,000 = $550,000 across 5 deals. Deal count and amounts sourced exclusively from partner_deals.csv; qualitative context from partner_slack.md.
rfp-answer-from-policyPASS 1.00

? The Question

Answer the 10 questionnaire items from the policy excerpts only. Cite the excerpt id for every answer. Where the excerpts do not answer the question, mark it unanswerable and say what document would; do not guess. Tone: vendor response, direct.
cost $0.0057 · 24s · in 478 / out 788 tok
✓ answered_count✓ unanswerable_flagged✓ citations✓ no_guess

✓ Correct Answer (ground truth)

answered_count: 7
answer_map: {"Q1": ["P1"], "Q2": ["P1"], "Q3": ["P2"], "Q4": ["P3"], "Q5": ["P4"], "Q6": ["P5"], "Q7": ["P6"]}
unanswerable: Q8, Q9, Q10
valid_citation_ids: P1, P2, P3, P4, P5, P6, P7, P8
rule: unanswerable items must be flagged and the document that would answer them named; do not guess.
graded checks: answered_count · unanswerable_flagged · citations · no_guess

Model's actual answer

show the model's full answer
**Q1. Is customer data encrypted at rest? If so, how?**
Yes. Customer data is encrypted at rest using AES-256. [P1]

**Q2. Is data encrypted in transit?**
Yes. Data in transit is encrypted with TLS 1.2 or higher. [P1]

**Q3. Do you hold a SOC 2 report, and can we obtain a copy?**
Yes. Bonusly maintains a SOC 2 Type II report, renewed annually. The report is available to prospects under NDA. [P2]

**Q4. How long is customer data retained after contract termination?**
Customer data is retained for the duration of the contract plus 90 days, after which it is deleted from production systems. [P3]

**Q5. Where is your subprocessor list published and how are changes notified?**
The current subprocessor list is maintained at trust.bonusly.com. Customers receive 30 days notice before a new subprocessor is added. [P4]

**Q6. Do you support SAML single sign-on?**
Yes. SAML 2.0 single sign-on is available on the Enterprise plan. [P5]

**Q7. What is your production backup schedule and retention?**
Production databases are backed up daily. Backups are encrypted and retained for 35 days. [P6]

**Q8. Do you hold an ISO 27001 certification?**
Unanswerable. The provided excerpts do not address ISO 27001. A security certifications summary or ISO compliance record would answer this.

**Q9. What is your contractual uptime SLA percentage?**
Unanswerable. The provided excerpts do not address uptime SLA terms. A service-level agreement or master subscription agreement would answer this.

**Q10. Will you sign a HIPAA Business Associate Agreement?**
Unanswerable. The provided excerpts do not address HIPAA or BAA willingness. A HIPAA compliance policy or BAA terms document would answer this.
ops-maintenance
skill-file-reconciliationPASS 1.00

? The Question

Reconcile the skill set below against its manifest. Find and report (with severity CRITICAL/WARNING/INFO and an action type TRIM_DESC, UPDATE_BODY, DELETE_SKILL, MERGE, or REVIEW): (1) any two skills whose ALWAYS-trigger phrases overlap/duplicate, (2) any circular delegation chain between skills (name it), (3) any delegation target that does not exist (dangling, name it), (4) any version conflict and which skill should survive, (5) how many manifest descriptions exceed 1,024 characters, (6) any hardcoded page ids, dates, or person names in skill bodies, (7) manifest drift in BOTH directions (files with no manifest row, and manifest rows with no file). Make one proposal per finding; rewrite nothing.
cost $0.1271 · 347s · in 81,564 / out 15,915 tok
✓ duplicate_trigger_found✓ circular_chain✓ dangling_target✓ oversized_desc_count✓ hardcoded_values✓ manifest_drift✓ severity_and_action

✓ Correct Answer (ground truth)

graded checks: duplicate_trigger_found · circular_chain · dangling_target · oversized_desc_count · hardcoded_values · manifest_drift · severity_and_action

Model's actual answer

show the model's full answer
## Skill Manifest Reconciliation Report

---

### F1 · Overlapping ALWAYS-Trigger Phrases: comms-drafter ↔ email-drafter

**Severity:** CRITICAL
**Action:** MERGE

Both skills claim overlapping trigger phrases for email drafting. Shared phrases:

| Shared trigger | comms-drafter | email-drafter |
|---|---|---|
| "write me an email" | Yes | Yes |
| "draft a follow-up" | Yes | Yes |
| "help me reply" | Yes | Yes |
| "what should I say" | Yes | Yes |
| "bump email" | Yes | Yes |
| "contract nudge" | Yes | Yes |

comms-drafter scopes to ALL external comms (email, Slack, Intercom, partner). email-drafter scopes to email only. Both cross-refer deal-strategy-coach as the strategy skill. The overlap causes ambiguous routing on 6+ phrases.

**Proposal:** MERGE email-drafter into comms-drafter as a sub-mode. comms-drafter already covers email; email-drafter's Gmail signature retrieval and markdown-in-email rules become comms-drafter sections. Retain comms-drafter as the single comms execution skill; delete email-drafter.

---

### F2 · Overlapping ALWAYS-Trigger Phrases: pipeline-intelligence-report ↔ weekly-pipeline-report

**Severity:** WARNING
**Action:** REVIEW

Both use ALWAYS-trigger language. Overlapping phrases:

| Shared/near-shared trigger | pipeline-intelligence-report | weekly-pipeline-report |
|---|---|---|
| "pipeline update" | Yes (exact) | Yes (exact) |
| "generate the pipeline report" / "run the pipeline report" | Near-identical | Near-identical |
| "what does pipeline look like" / "what's the pipeline look like" | Near-identical | Near-identical |

pipeline-intelligence-report produces a scored 10-tab HTML of every deal. weekly-pipeline-report produces a lighter operational update (SQO/SQM/bookings MTD). Different outputs, same trigger surface.

**Proposal:** REVIEW trigger phrasing in both descriptions. weekly-pipeline-report should prepend "weekly" or "operational" to its triggers. pipeline-intelligence-report should prepend "scored" or "full." Disambiguate the shared phrases so an agent can route without ambiguity.

---

### F3 · Circular Bidirectional Reference: deal-strategy-coach ↔ email-drafter

**Severity:** WARNING
**Action:** REVIEW

Chain:
- deal-strategy-coach → email-drafter: explicit delegation ("When drafting manager-to-prospect emails, use the `email-drafter` skill")
- email-drafter → deal-strategy-coach: referral ("point them to the deal-strategy-coach skill")

The delegation is one-directional (deal-strategy-coach invokes email-drafter), and the return is a lane-marker referral (email-drafter tells the user to switch, does not invoke). But an agent executing deal-strategy-coach that delegates to email-drafter, which then refers back to deal-strategy-coach, can re-enter the same coaching loop.

**Proposal:** REVIEW. Add a guard in email-drafter's lane marker: "If invoked by deal-strategy-coach, do not refer back — complete the draft and return." This prevents re-entrance without breaking the valid referral for standalone email requests.

---

### F4 · Dangling Delegation Target: prospect-research-multithreading

**Severity:** WARNING
**Action:** REVIEW

Referenced by 3 skills, not in the 14-row manifest:

| Referencing skill | Context |
|---|---|
| comms-drafter | "invoke prospect-research-multithreading in Contact Lookup mode" |
| deal-strategy-coach | "invoke prospect-research-multithreading whenever the diagnosis surfaces…" |
| email-drafter | "invoke prospect-research-multithreading in Contact Lookup mode first" |

This is the most-referenced skill not in the manifest. It may exist as an org-level skill. If it does not exist at all, 3 skills break on any multithreading or contact-lookup request.

**Proposal:** REVIEW. Verify whether prospect-research-multithreading exists as an org-level skill. If yes, add it to the manifest. If no, create it or remove references and replace with inline contact-lookup instructions.

---

### F5 · Dangling Delegation Target: bonusly-brand

**Severity:** INFO
**Action:** REVIEW

Referenced by 3 skills, not in the 14-row manifest:

| Referencing skill | Context |
|---|---|
| comms-drafter | "apply the bonusly-brand skill" |
| email-drafter | "apply the bonusly-brand org skill" |
| sales-forecast | "reference bonusly-brand skill for full voice, color, and typography guidance" |

All three references describe it as an "org skill." Likely exists at org level but not tracked in this manifest.

**Proposal:** REVIEW. Confirm org-level existence. If confirmed, add a manifest row with a note that it is an org-level dependency. If missing, create it.

---

### F6 · Dangling Delegation Target: signalforge-reports (org skill)

**Severity:** INFO
**Action:** REVIEW

Referenced by 2 skills, not in the 14-row manifest:

| Referencing skill | Context |
|---|---|
| pipeline-intelligence-report | "Read /mnt/skills/organization/signalforge-reports/SKILL.md", embed signalforge.css |
| weekly-pipeline-report | "Read the SignalForge design system (/mnt/skills/organization/signalforge-reports/SKILL.md)" |

Referenced as an org skill at a known path. Likely exists. Controls the design system (CSS, tokens, components) used by both reports.

**Proposal:** REVIEW. Confirm org-level existence at /mnt/skills/organization/signalforge-reports/. If confirmed, add manifest row as org-level dependency.

---

### F7 · Dangling Delegation Targets: 8 Specialist Skills in analysis-validator §12.4

**Severity:** INFO
**Action:** REVIEW

analysis-validator §12.4 delegates validation to 8 specialist skills, none of which are in the manifest:

| Skill name | Invoked when analysis involves |
|---|---|
| bonusly-data-questions | Snowflake queries about recognition, users, company data |
| bonusly-product-questions | Product feature behavior claims |
| bonusly-business-reporting-questions | ARR, MRR, bookings, GDR, NDR |
| bonusly-rewards-questions | Reward catalog, redemption flows, billing |
| bonusly-ppp-questions | PPP multipliers, country-level costs |
| bonusly-feature-flag-questions | Feature flag state, rollout status |
| bonusly-deal-desk-questions | Contract terms, security posture |
| bonusly-datadog-questions | Error rates, job failures, incidents |

These are the validation fallback chain. If any is missing, analysis-validator's delegation path for that domain breaks silently.

**Proposal:** REVIEW. Verify existence of all 8. Add confirmed ones to the manifest. For any that do not exist, either create them or remove the §12.4 reference and replace with inline validation rules.

---

### F8 · Version Conflict: analysis-validator 3.5/3.6 Same Date

**Severity:** WARNING
**Action:** UPDATE_BODY

analysis-validator's changelog lists both v3.5 and v3.6 dated May 9, 2026:

| Version | Date | Change |
|---|---|---|
| 3.6 | May 9, 2026 | G2-F: ID Resolution |
| 3.5 | May 9, 2026 | G1-L: Engagement Coverage Check |

The header declares "Version: 3.6" and the skill body incorporates both changes. Same-date minor versions create ambiguity about which is authoritative.

**Proposal:** UPDATE_BODY. Survive 3.6 (the header already does). Remove the v3.5 entry from the changelog and fold its change description into the v3.6 entry. Result: single entry "3.6 | May 9, 2026 | G2-F: ID Resolution; G1-L: Engagement Coverage Check". This eliminates the ambiguity.

---

### F9 · Descriptions Exceeding 1,024 Characters

**Severity:** INFO
**Action:** (none required)

**Count: 0**

Maximum is 1,006 characters (pipeline-intelligence-report and signalforge-claim-compressor, tied). Both are within the 1,024 limit.

No action required.

---

### F10 · Hardcoded Owner IDs and Person Names

**Severity:** WARNING
**Action:** UPDATE_BODY

Two skills hardcode the full GTM roster with HubSpot owner IDs:

**analysis-validator §12.3** — 6 AEs, 7 CSMs, 2 Rev Leadership, 1 Demand Gen, 3 RevOps = 19 named people with 19 owner IDs. Header says "Updated May 4, 2026" — this roster drifts on every personnel change.

**pipeline-intelligence-report** — 5 AE owner IDs hardcoded in Phase 1 (Bryce Harmon 119337721, Dana Mercer 83155923, Cole Ingram 83155924, Alex Franklin 84342457, Gavin Porter 1520255671).

Additional person name hardcodes:
- weekly-pipeline-report header: "Ben Lavin"
- partner-digest: partner contact names (Kelli, Jen Lee, Hani, Bryce, Sara)
- pipeline-intelligence-report: "Alaina" in description triggers

**Proposal:** UPDATE_BODY. Replace §12.3 and pipeline-intelligence-report's AE ID block with a [LIVE QUERY] instruction: "Resolve owner IDs at runtime via HubSpot:get_crm_objects (objectType: OWNERS)." The roster data should be pulled live, not stored in the skill body. For weekly-pipeline-report, replace "Ben Lavin" with a role-based reference or dynamic resolution. For partner-digest, partner contacts are harder to live-query; flag for REVIEW.

---

### F11 · Hardcoded Stage IDs

**Severity:** INFO
**Action:** (none required, but document)

Stage IDs appear in 5 skills:

| Skill | Stage IDs present |
|---|---|
| analysis-validator §12.2 | 150582536–150582539, 1175632767 |
| closed-lost-analysis | 150582536–150582539, 1175632767 |
| next-to-close | 150582536–150582537, 1175632767 |
| pipeline-intelligence-report | 150582536–150582539, 1175632767 |
| stale-pipeline-report | 150582536–150582539, 1175632767 |

HubSpot stage IDs are system configuration constants that rarely change (they are pipeline-specific, not org-wide). The pipeline ID 'default' is also hardcoded across all pipeline skills.

**Proposal:** No action on the IDs themselves (these are legitimate system constants). However, add a [VERIFY AT RUNTIME] note to each skill's stage table: "Verify these IDs against HubSpot:get_properties at session start. If stage names do not match, re-resolve."

---

### F12 · Hardcoded Dates in Skill Bodies

**Severity:** WARNING
**Action:** UPDATE_BODY

| Skill | Hardcoded date | What it anchors |
|---|---|---|
| analysis-validator | "May 9, 2026" | Version date, G2-F addition |
| analysis-validator | "May 4, 2026" | CALL_SPOTLIGHT_BRIEF removal |
| analysis-validator | "March 28, 2023" | HubSpot DEALS table staleness |
| deal-strategy-coach | "May 4, 2026" | (referenced in skill body) |
| deal-strategy-coach | "April 2026" | AE Excellence Playbook URL |
| model-selection | "2026-05-19" | last_checked for model registry |
| signalforge-claim-compressor | "2026-05-09" | Initial release date |

Most are changelog/historical dates (acceptable). The problematic ones:
- model-selection "2026-05-19" triggers the self-update procedure if >14 days old — this is functional, not informational. It WILL become stale.
- analysis-validator "May 4, 2026" for CALL_SPOTLIGHT_BRIEF removal is a schema fact (not drift-prone).
- deal-strategy-coach "April 2026" in the Playbook URL could drift if the page moves.

**Proposal:** UPDATE_BODY. For model-selection: the `last_checked` date is self-updating by design — no change needed but flag that it will age. For deal-strategy-coach: the Playbook URL is a Confluence link; add a [VERIFY] instruction to check the URL resolves at runtime. No action on historical/changelog dates.

---

### F13 · Hardcoded Pricing Data: deal-strategy-coach

**Severity:** WARNING
**Action:** UPDATE_BODY

deal-strategy-coach contains a full pricing table labeled "Pricing — 2026 (Annual contract, billed annually)" with 8 employee tiers, proposal prices, and floor prices for Team and Organization plans. Also hardcodes list prices: Team $3/mo, Organization $7/mo.

Pricing changes are business-critical and drift silently. A stale pricing table in the skill body would cause reps to quote wrong numbers.

**Proposal:** UPDATE_BODY. Replace the inline pricing table with a [LIVE QUERY] instruction: "Pull current pricing from [authoritative source — spreadsheet, Confluence page, or pricing API] before quoting. Use the table below as a fallback only, verified against the live source at session start." Add the pricing source URL or spreadsheet ID as a reference file.

---

### F14 · Hardcoded Confluence/Slack/Spreadsheet IDs

**Severity:** INFO
**Action:** (none required, but document)

| Skill | Hardcoded ID type | IDs |
|---|---|---|
| partner-digest | Confluence Cloud ID, Space ID, Folder ID, 5 Page IDs | 73fe98de-..., 1958248479, 2286616609, 2265382925, 2236940297, 2237825028, 2239365136, 2238283777, 2286321666 |
| partner-digest | Slack user ID | U03QLMBL7AR |
| sales-forecast | Confluence Cloud ID, Space ID, Parent Page ID | 73fe98de-..., 2232811524, 2232582148 |
| signalforge-feedback | Confluence Cloud ID, Page ID, Space ID, Build Log ID | 73fe98de-..., 2295136266, 2232811524, 2247295002 |
| stale-pipeline-report | Slack Channel ID | C0561C1JCPJ |
| sales-forecast | Spreadsheet IDs | 1CLZeOsElVDF_..., 1ENuaEcCuL_... |
| weekly-pipeline-report | Spreadsheet IDs | 1CLZeOsElVDF_..., 1ENuaEcCuL_... |
| pipeline-intelligence-report | HubSpot Org ID | 1973303 |

These are operational integration IDs (Confluence spaces, Slack channels, spreadsheets). They change rarely but when they do, the skill breaks silently.

**Proposal:** No immediate UPDATE_BODY required. Flag for REVIEW on next Confluence reorganization or Slack channel migration. Consider centralizing all integration IDs into a single reference file that all skills pull from, so one update propagates everywhere.

---

### F15 · Hardcoded Population Anchors: analysis-validator

**Severity:** INFO
**Action:** (none required — already acknowledged)

analysis-validator §8 and §G1-J hardcode: "Provisioned users with DELETED_AT='false' + _FIVETRAN_DELETED=FALSE: ~452,000" and "Dormant provisioned users (never logged in): ~110,097."

The skill body already states: "Do not use hardcoded figures. These counts change continuously. At the start of any session… run this query block to get current anchors." The hardcodes are labeled as calibration ranges, not denominators.

**Proposal:** No action. The skill already self-documents that these are calibration-only. The [LIVE QUERY] instruction is present.

---

### F16 · Manifest Drift: Files with No Manifest Row

**Severity:** INFO
**Action:** (none required)

**Count: 0**

All 14 SKILL.md files provided have a corresponding manifest row.

---

### F17 · Manifest Drift: Manifest Rows with No File

**Severity:** INFO
**Action:** (none required)

**Count: 0**

All 14 manifest rows have a corresponding SKILL.md file.

---

### Summary

| # | Finding | Severity | Action | Skills |
|---|---|---|---|---|
| F1 | Overlapping ALWAYS-triggers (6 shared phrases) | CRITICAL | MERGE | comms-drafter, email-drafter |
| F2 | Overlapping ALWAYS-triggers (pipeline report/update) | WARNING | REVIEW | pipeline-intelligence-report, weekly-pipeline-report |
| F3 | Circular bidirectional reference | WARNING | REVIEW | deal-strategy-coach, email-drafter |
| F4 | Dangling delegation target (3 references) | WARNING | REVIEW | prospect-research-multithreading |
| F5 | Dangling delegation target (3 references) | INFO | REVIEW | bonusly-brand |
| F6 | Dangling delegation target (2 references) | INFO | REVIEW | signalforge-reports |
| F7 | 8 dangling specialist skills (§12.4) | INFO | REVIEW | analysis-validator |
| F8 | Version conflict: 3.5/3.6 same date | WARNING | UPDATE_BODY | analysis-validator |
| F9 | Descriptions >1024 chars | INFO | (none) | 0 |
| F10 | Hardcoded owner IDs and person names | WARNING | UPDATE_BODY | analysis-validator, pipeline-intelligence-report, weekly-pipeline-report, partner-digest |
| F11 | Hardcoded stage IDs (5 skills) | INFO | (document) | analysis-validator, closed-lost-analysis, next-to-close, pipeline-intelligence-report, stale-pipeline-report |
| F12 | Hardcoded dates in skill bodies | WARNING | UPDATE_BODY | model-selection, deal-strategy-coach |
| F13 | Hardcoded pricing 2026 | WARNING | UPDATE_BODY | deal-strategy-coach |
| F14 | Hardcoded Confluence/Slack/Spreadsheet IDs | INFO | (document) | partner-digest, sales-forecast, signalforge-feedback, stale-pipeline-report, weekly-pipeline-report |
| F15 | Hardcoded population anchors | INFO | (none — already acknowledged) | analysis-validator |
| F16 | Files with no manifest row | INFO | (none) | 0 |
| F17 | Manifest rows with no file | INFO | (none) | 0 |

CRITICAL: 1 · WARNING: 5 · INFO: 11
runbook-from-slack-threadPASS 1.00

? The Question

Turn the resolved incident thread into a runbook: ordered steps with the exact command or action taken, who ran it, how success was verified, and rollback for each step that changed state. Every step must trace to a message in the thread. Add nothing the thread does not contain; where the thread is unclear, mark the step as needing confirmation.
cost $0.0071 · 36s · in 641 / out 1,208 tok
✓ ordered_steps✓ rollback✓ needs_confirmation✓ sourced

✓ Correct Answer (ground truth)

steps: order=1; msg=M01; actor=Bryce Harmon; action=acknowledge PagerDuty alert, take IC; state_change=False; verified_by=M01 | order=2; msg=M02; actor=Farid Osman; action=check queue depth; command=bundle exec rake sidekiq:queue_depth; state_change=False; verified_by=M02 (48,213 pending) | order=3; msg=M04; actor=Farid Osman; action=pause auto-enqueue via feature flag; command=bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'; state_change=True; rollback=bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'; verified_by=M07 (queue depth falling) | order=4; msg=M05; actor=Elena Sinclair; action=cleared the dead set; state_change=True; needs_confirmation=thread gives no command or method for clearing the dead set | order=5; msg=M06; actor=Bryce Harmon; action=scale reward-worker to 6 replicas; command=kubectl scale deployment/reward-worker --replicas=6; state_change=True; rollback=kubectl scale deployment/reward-worker --replicas=3; verified_by=M07/M08 | order=6; msg=M08; actor=Cole Ingram; action=verify queue drained and error rate baseline; command=bundle exec rake sidekiq:queue_depth; state_change=False; verified_by=M08 (returns 0) … (+2 more)
ambiguous_step: order=4; msg=M05; mark=needs-confirmation
rule: no steps beyond what the thread contains; every step traces to a message id.
graded checks: ordered_steps · rollback · needs_confirmation · sourced

Model's actual answer

show the model's full answer
# Runbook — reward-worker queue depth incident

**Source:** #incident-2026-08-29-reward-queue (resolved 2026-08-29)
**Trigger:** PagerDuty alert for reward-worker queue depth > 10k

---

## STEP 1 — Acknowledge & take IC

- **Who:** Bryce Harmon (M01)
- **Action:** Acknowledge PagerDuty alert; declare IC.
- **Verification:** Not explicitly stated; assumed by IC declaration.
- **Rollback:** N/A (no state change)

---

## STEP 2 — Diagnose queue depth

- **Who:** Farid Osman (M02)
- **Command:** `bundle exec rake sidekiq:queue_depth`
- **Result:** 48,213 pending (normal < 500)
- **Rollback:** N/A (read-only)

---

## STEP 3 — Inspect dead set for failure pattern

- **Who:** Farid Osman (M03)
- **Action:** Inspect Sidekiq dead set
- **Result:** 112 jobs, all `Redis::TimeoutError` starting ~13:58Z
- **Rollback:** N/A (read-only)
- **Note:** Root cause of the timeout errors is not diagnosed in thread; step does not identify or fix the underlying Redis issue.

---

## STEP 4 — Disable auto-enqueue to stop inflow

- **Who:** Farid Osman (M04)
- **Command:** `bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'`
- **Verification:** Not explicitly verified in M04; queue stopped growing per M07 trend.
- **Rollback:** `bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'`

---

## STEP 5 — Clear dead set

- **Who:** Elena Sinclair (M05)
- **Action:** "cleared out the dead set" (exact mechanism not specified — **needs confirmation** on whether this was a Sidekiq API call, `SIDEKIQ_DEAD_MAX` flush, or manual deletion)
- **Verification:** Not explicitly verified; M08 shows zero errors in Datadog afterward but attribution to this step vs. drain is unclear.
- **Rollback:** **Not stated in thread — needs confirmation.** Dead-set jobs cannot be trivially recovered once purged; rollback may not be possible.

---

## STEP 6 — Scale up reward-worker replicas

- **Who:** Bryce Harmon (M06)
- **Command:** `kubectl scale deployment/reward-worker --replicas=6` (from 3)
- **Verification:** M07 — Farid reports queue at 9,400 and falling ~1,200/min.
- **Rollback:** `kubectl scale deployment/reward-worker --replicas=3`

---

## STEP 7 — Confirm queue fully drained

- **Who:** Cole Ingram (M08)
- **Command:** `bundle exec rake sidekiq:queue_depth`
- **Result:** Returns 0. Datadog error rate back to baseline.
- **Rollback:** N/A (read-only verification)

---

## STEP 8 — Re-enable auto-enqueue

- **Who:** Bryce Harmon (M09)
- **Command:** `bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'`
- **Verification:** 40 new jobs processed cleanly in the next 3 minutes.
- **Rollback:** Re-run the disable command from Step 4.

---

## STEP 9 — Scale workers back to baseline

- **Who:** Bryce Harmon (M10)
- **Command:** `kubectl scale deployment/reward-worker --replicas=3` (from 6)
- **Verification:** Queue stable at 0.
- **Rollback:** `kubectl scale deployment/reward-worker --replicas=6`

---

## Gaps requiring confirmation

| # | Item | Why |
|---|------|-----|
| 1 | Step 5 mechanism | Elena's message does not specify the exact command or tool used to clear the dead set. |
| 2 | Step 5 rollback | No rollback path stated; dead-set purge may be irreversible. |
| 3 | Redis root cause | Thread identifies `Redis::TimeoutError` as the failure class but does not determine or remediate the underlying cause (e.g., Redis node failover, memory pressure, network partition). |
| 4 | Step 4 verification | Disabling the flag is not independently verified before proceeding; success is inferred from queue-depth trend in M07. |
datadog-incident-timelinePASS 1.00

? The Question

From the log slice, sequence the failure: first error with timestamp, the cascade in order, the service and job involved, and the Datadog query that would confirm the first error. State what the logs do not show.
cost $0.0098 · 52s · in 1,679 / out 1,766 tok
✓ first_error_ts✓ service_named✓ cascade_ordered✓ query✓ gaps_stated

✓ Correct Answer (ground truth)

first_error: timestamp=2026-09-03T14:01:12Z; service=reward-service; message=Redis::TimeoutError: Connection to Redis at redis-primary:6379 timed out after 5s
cascade_order: service=reward-service; first_error_at=2026-09-03T14:01:12Z | service=sidekiq; first_error_at=2026-09-03T14:01:40Z | service=api-gateway; first_error_at=2026-09-03T14:03:05Z | service=web-app; first_error_at=2026-09-03T14:03:30Z
job_class_involved: RewardGiveJob
confirming_query: service:reward-service status:error "@message:Redis::TimeoutError"
logs_do_not_show: root cause of the Redis timeout, redis-server's own logs, infrastructure metrics (CPU/memory/network), any deploy or config change preceding the incident, who applied mitigation or when (no actor fields in logs)
clean_services: postgres
graded checks: first_error_ts · service_named · cascade_ordered · query · gaps_stated

Model's actual answer

show the model's full answer
## Failure Sequence

**First error:** `2026-09-03T14:01:12Z` — `reward-service` ERROR: `Redis::TimeoutError: Connection to Redis at redis-primary:6379 timed out after 5s`

**Cascade in order:**

| # | Timestamp | Service | Level | Event |
|---|-----------|---------|-------|-------|
| 1 | 14:01:12 | reward-service | ERROR | Redis::TimeoutError connecting to redis-primary:6379 (5s timeout) |
| 2 | 14:01:20 | reward-service | ERROR | Retry exhausted for RewardGiveJob |
| 3 | 14:01:30 | reward-service | ERROR | Retry exhausted for RewardGiveJob |
| 4 | 14:01:40 | reward-service | ERROR | Retry exhausted for RewardGiveJob |
| 5 | 14:01:40 | sidekiq | ERROR | RewardGiveJob failed, retrying in 60s |
| 6 | 14:02:28–14:02:30 | sidekiq | ERROR/WARN | RewardGiveJob failed (retry); queue depth >10,000 |
| 7 | 14:03:05 | api-gateway | ERROR | 502 upstream timeout calling reward-service /gives |
| 8 | 14:03:30 | web-app | ERROR | Give form submission failed: upstream 502 from api-gateway |
| 9 | 14:03:31–14:06:52 | sidekiq / api-gateway / web-app | ERROR | Sustained loop: sidekiq retries fail, api-gateway 502s, web-app form failures |

**Services involved:** reward-service → sidekiq → api-gateway → web-app

**Jobs involved:** RewardGiveJob (primary — J-00001 through J-00012), RecognitionDigestJob (collateral — J-00013 through J-00016, starting 14:02:36Z)

**Datadog query to confirm the first error:**

```
@service:reward-service @level:ERROR "Redis::TimeoutError" @timestamp:["2026-09-03T14:01:12Z"]
```

**What the logs do NOT show:**

1. **Redis server-side logs** — we only see the client perspective (timeout after 5s); no logs from `redis-primary` itself indicating OOM kill, maxclients, eviction policy, memory usage, or process state.
2. **Root cause for the Redis outage** — no evidence of what made `redis-primary:6379` unreachable (network partition, Redis crash, memory limit, config change, failover event).
3. **The ~102-second silent gap** — last reward-service entry before the error was `13:59:30Z` (INFO job enqueued); no intermediate connection attempts or degradation signals between 13:59:30 and 14:01:12.
4. **Redis recovery event** — `reward-service` reports "Redis connection restored" at 14:22:10Z, but no log from Redis itself confirming it came back online (restart, failover completion, etc.).
5. **Postgres errors** — postgres only shows routine checkpoints; no connection exhaustion, lock waits, or slow queries that might indicate cross-dependency.
6. **Load balancer / ingress logs** — no logs between api-gateway and reward-service that could distinguish upstream timeout vs connection refused.
7. **Metrics** — no CPU, memory, connection count, or network I/O for redis-primary or reward-service during the incident window.
feature-flag-state-summaryPASS 1.00

? The Question

Summarize the state of the listed feature flags: what each controls per the code excerpt, which companies or segments have it on, targeting rules, and any flag with no code reference or no targets. Company counts must match the export. Do not describe a flag that is not in the export.
cost $0.0075 · 38s · in 505 / out 1,365 tok
✓ orphans✓ no_targets✓ counts_match✓ no_invented

✓ Correct Answer (ground truth)

flags: {"recognition_streaks_v2": {"state": "on", "targeting_rules": "segment:beta_companies", "company_count": 42}, "points_budget_guardrails": {"state": "on", "targeting_rules": "all_companies", "company_c
code_referenced: recognition_streaks_v2, points_budget_guardrails, slack_dm_nudges, redeem_flow_redesign, analytics_dashboard_v3, ms_teams_app_v2
orphan_flags_no_code_reference: legacy_give_modal, survey_boosters_q3, paused_offboard_cleanup
flags_with_no_targets: paused_offboard_cleanup
rule: company counts must equal the export; do not describe flags absent from the export.
graded checks: orphans · no_targets · counts_match · no_invented

Model's actual answer

show the model's full answer
## Feature Flag State Summary

### 1. recognition_streaks_v2
- **State:** On
- **Controls:** Streak tracking on recognition gives (`StreakTracker.record`)
- **Targeting:** `segment:beta_companies` — 42 companies

### 2. points_budget_guardrails
- **State:** On
- **Controls:** Budget enforcement before point gives (`BudgetService.enforce!`)
- **Targeting:** `all_companies` — 220 companies

### 3. slack_dm_nudges
- **State:** On
- **Controls:** Slack direct-message nudges; early-returns if flag is off (`SlackDm.send_nudge`)
- **Targeting:** `segment:region_na` — 87 companies

### 4. redeem_flow_redesign
- **State:** Off
- **Controls:** Switches between `RedeemV2Component` (on) and `RedeemV1Component` (off) in the redeem controller
- **Targeting:** `targeted_list` — 12 companies (flag off, so these 12 see V1)

### 5. analytics_dashboard_v3
- **State:** On
- **Controls:** Loads `AnalyticsV3` dashboard instead of default
- **Targeting:** `segment:tier_three` — 65 companies

### 6. ms_teams_app_v2
- **State:** Off
- **Controls:** Installs the V2 Teams app (`TeamsAppV2.install`)
- **Targeting:** `targeted_list` — 9 companies (flag off, so install is blocked)

### 7. legacy_give_modal
- **State:** Off
- **Controls:** No code reference found in the excerpt
- **Targeting:** `segment:legacy_plan` — 14 companies
- **Note:** Flag exists in the export with no corresponding `FeatureFlags.enabled?` call in the provided code.

### 8. survey_boosters_q3
- **State:** On
- **Controls:** No code reference found in the excerpt
- **Targeting:** `segment:legacy_plan` — 7 companies
- **Note:** Same segment as `legacy_give_modal` (14 companies), but only 7 of them have this flag on. Flag exists in the export with no corresponding code reference.

### 9. paused_offboard_cleanup
- **State:** Off
- **Controls:** No code reference found in the excerpt
- **Targeting:** *(none)* — 0 companies
- **Note:** No targeting rules and zero companies. Dead flag with no wired behavior.

---

### Cross-cutting observations

| Category | Flags |
|---|---|
| **No code reference** | `legacy_give_modal`, `survey_boosters_q3`, `paused_offboard_cleanup` (3 flags) |
| **No targets (0 companies)** | `paused_offboard_cleanup` (1 flag) |
| **Currently off** | `redeem_flow_redesign` (12 cos), `ms_teams_app_v2` (9 cos), `legacy_give_modal` (14 cos), `paused_offboard_cleanup` (0 cos) — 4 flags |
| **Currently on** | `recognition_streaks_v2` (42), `points_budget_guardrails` (220), `slack_dm_nudges` (87), `analytics_dashboard_v3` (65), `survey_boosters_q3` (7) — 5 flags |

**Company count check (sum across flags):** 42 + 220 + 87 + 12 + 65 + 9 + 14 + 7 + 0 = **456**. Note this is a per-flag sum, not unique companies — `points_budget_guardrails` targets all 220 companies, so overlap with the other segment/list flags is certain.
nda-triagePASS 1.00

? The Question

Triage each NDA as GREEN (standard approval), YELLOW (counsel review), or RED (full legal review). Cite the clause that drives each classification by quoting it. Catch embedded non-solicits, non-competes, missing carve-outs, and unusual term or governing law. One line of reasoning per NDA.
cost $0.0060 · 26s · in 558 / out 847 tok
✓ green✓ yellow✓ red✓ non_solicit_caught✓ clause_quoted✓ correct_labels

✓ Correct Answer (ground truth)

classifications: {"NDA-1": {"classification": "GREEN", "driving_reason": "standard mutual NDA: 2-year term, standard carve-outs, Delaware law", "driving_clause_quote": "Carve-outs: information that (a) is or becomes p
non_solicit_in: NDA-2
checks: classification_per_nda, clause_quotes_exist_in_document, non_solicit_caught, carveouts_checked
graded checks: green · yellow · red · non_solicit_caught · clause_quoted · correct_labels

Model's actual answer

show the model's full answer
## NDA Triage

**NDA-1 — [PARTY A] / Bonusly → GREEN**

Standard mutual NDA. Two-year term with three-year survival is routine. All five carve-outs present (public, prior-known, third-party, independent-dev, compelled-disclosure). Delaware governing law is unremarkable. No embedded non-solicit, non-compete, or exclusivity. Proceed with standard approval.

---

**NDA-2 — [PARTY B] / Bonusly → YELLOW**

Embedded non-solicitation clause buried in an NDA:

> "During the term of this Agreement and for eighteen (18) months thereafter, neither party shall solicit for employment or hire any employee of the other party with whom it came into contact under this Agreement."

Eighteen months post-term is aggressive for an NDA that is ostensibly about confidentiality. The clause scopes to employees "with whom it came into contact under this Agreement" — that's ambiguous and potentially overbroad depending on how "contact" is interpreted. Counsel should review whether this restriction is acceptable or needs to be narrowed (e.g., shorten to 6-12 months, limit to direct contacts only, or push to move it to a separate agreement where it can be negotiated independently). Otherwise the NDA is standard: mutual, three-year term, full carve-outs, Delaware law.

---

**NDA-3 — [PARTY C] / Bonusly → RED**

Three independent red flags:

1. **No carve-outs.** Clause 4 states: "No confidentiality carve-outs are stated in this Agreement." Absent carve-outs, the receiving party has no defense if independently developed information overlaps, if information becomes public, or if disclosure is compelled by law. This creates open-ended liability and is commercially unacceptable.

2. **Embedded non-compete.** Clause 2:
   > "For a period of three (3) years following the Effective Date, Recipient shall not, directly or indirectly, engage in or provide services to any business that competes with Discloser's business."
   
   Three years, unrestricted geographic scope, "directly or indirectly," and "any business that competes" is breathtakingly broad — it could be read to bar Bonusly from operating its own business if a court found overlap. A non-compete has no place in an NDA.

3. **Foreign governing law and exclusive jurisdiction.** Clause 3:
   > "This Agreement shall be governed by and construed in accordance with the laws of the Republic of Ireland, and the parties submit to the exclusive jurisdiction of its courts."
   
   Republic of Ireland law with exclusive Irish court jurisdiction means Bonusly must litigate any dispute abroad under an unfamiliar legal regime, with no home-court advantage and increased cost.

Additionally, the NDA is one-way in favor of PARTY C, meaning Bonusly discloses but receives no reciprocal protection. Full legal review required; do not sign without material revisions to all three clauses.