home · ← back to all models
inclusionai

inclusionai/ling-3.0-flash

Mean 0.919 · 28/40 perfect tests · $0.06 total · each card: the question → the correct answer (gold) → this model's actual answer

data-crm
stage-hygiene-auditPASS 1.00

? The Question

Audit stage hygiene on the open pipeline below. Rules: a deal at DS2 or later must have a substantive why-buys (15+ characters, not a placeholder such as tbd, n/a, see notes). A why_buys value of the form LONG(n) means the real text was present but truncated for display; treat it as present and substantive. Every deal should have a DS1 entry timestamp. A deal whose stage entry timestamps are out of order regressed (an earlier-stage timestamp is later than a later-stage timestamp). Report: percent of DS2+ deals non-compliant on why-buys overall and by owner (use the owners table for names), count of deals with no DS1 timestamp, count that regressed, and three process fixes. State definitions before numbers.
cost $0.0015 · 34s · in 45,876 / out 6,156 tok
✓ noncomp_pct✓ no_ds1_0✓ regressed_0✓ per_owner_breakdown✓ definitions_first✓ three_fixes

✓ Correct Answer (ground truth)

noncomp_pct: 81.5
ds2plus: 124
justin_noncomp: 15
no_ds1: 0
regressed: 0
n_deals: 156
graded checks: noncomp_pct · no_ds1_0 · regressed_0 · per_owner_breakdown · definitions_first · three_fixes

Model's actual answer

show the model's full answer
Definitions:
- DS2+ deal: stage DS2 or later (DS2, DS3, DS4, DS5)
- Substantive why_buys: field length ≥ 15 characters, not a placeholder (tbd, n/a, see notes). LONG(n) format is treated as present/substantive (truncated for display).
- DS1 timestamp present: t_ds1 field is non-empty
- Regressed: an earlier-stage entry timestamp is later than a later-stage entry timestamp

Results:

Why-buys non-compliance (DS2+ deals): 101 / 124 = 81.5% overall

By owner:
- Bryce Harmon: 15 / 18 (83.3%)
- Dana Mercer: 18 / 22 (81.8%)
- Alex Franklin: 48 / 61 (78.7%)
- Cole Ingram: 19 / 22 (86.4%)
- Farid Osman: 1 / 1 (100.0%)

Deals with no DS1 timestamp: 0
Deals with regressed timestamps: 0

Three process fixes:
1. Enforce mandatory why-buys entry at DS2 stage advance — validate 15+ chars and reject placeholders (tbd, n/a, see notes) before allowing the move.
2. Automate DS1 timestamp capture at deal creation — eliminate manual entry to ensure 100% coverage.
3. Add stage-order validation in the CRM pipeline flow — block stage advances if the new stage's entry timestamp precedes any earlier stage's timestamp.
owner-resolutionPASS 1.00

? The Question

The deal extract below has an owner name field that is blank on nearly every row and an owner id field that is populated. Using the owners table, resolve each open deal to an owner name. Report: number of deals resolved, deals whose owner id has no match in the owners table (list them), the owner ids that map to archived/deactivated owners (list their names), and the total pipeline amount per resolved owner. Do not guess a name for any unmatched id.
cost $0.0010 · 17s · in 36,887 / out 1,127 tok
✓ resolved_count✓ unmatched_none✓ deactivated_flagged✓ jake_amount✓ justin_amount✓ no_invented

✓ Correct Answer (ground truth)

resolved: 156
jake_amount: 1054144.0
justin_amount: 624310.0
per_owner: Bryce Harmon=1054144.0; Alex Franklin=624310.0; Dana Mercer=341195.0; Cole Ingram=288161.43; Farid Osman=4134.0; Elena Sinclair=2100.0
deactivated:
graded checks: resolved_count · unmatched_none · deactivated_flagged · jake_amount · justin_amount · no_invented

Model's actual answer

show the model's full answer
All 156 deals resolved. Every owner_id in the deals file has a match in the owners table.

**Deals resolved:** 156

**Deals with no owner match:** 0 (none)

**Archived/deactivated owners mapped to deals:** none. The two archived owners (Gavin Porter, id 1520255671; Hugo Lindqvist, id 77260721) do not appear in any deal row.

**Total pipeline per resolved owner:**

| Owner | Pipeline |
|---|---|
| Bryce Harmon | $1,054,144.00 |
| Alex Franklin | $624,310.00 |
| Dana Mercer | $341,195.00 |
| Cole Ingram | $288,161.43 |
| Farid Osman | $4,134.00 |
| Elena Sinclair | $2,100.00 |
gong-hubspot-join-integrityPASS 1.00

? The Question

Check the integrity of the call-to-deal mapping below. The gong table lists calls in the last 90 days keyed to HubSpot deal ids; deals_open lists the current open default-pipeline deals. Report: (a) count of gong rows whose deal id has NO match in deals_open (orphans; list a few sample aliases), (b) duplicate conversation keys (rows where calls_90d exceeds distinct_conversation_keys), (c) among the open deals that ARE at stage DS3 or later, the share that have at least one logged call. Give sample aliases for the orphan finding, all taken from the data. Note: gong_owner_id is a Gong user id, not a HubSpot owner id.
cost $0.0016 · 15s · in 66,936 / out 2,436 tok
✓ orphan_count✓ dup_keys_0✓ ds3_coverage✓ sample_ids_present

✓ Correct Answer (ground truth)

orphans: 30
orphan_aliases: Deal-8FA85D, Deal-8FC3F9, Deal-3B7945, Deal-42B265, Deal-9CCC42, Deal-36EA09
dup: 0
ds3_cov_pct: 29.4
graded checks: orphan_count · dup_keys_0 · ds3_coverage · sample_ids_present

Model's actual answer

show the model's full answer
(a) Orphans (gong rows with no match in deals_open): 30 out of 67. Sample aliases: Deal-8FA85D, Deal-8FC3F9, Deal-3B7945, Deal-42B265, Deal-9CCC42.

(b) Duplicate conversation keys: 0. Every gong row has calls_90d == distinct_conversation_keys (no row has calls_90d exceeding distinct_conversation_keys).

(c) DS3+ open deals with at least one logged call: 25 / 85 = 29.41%. The remaining 60 DS3+ deals have zero logged calls in the gong table.
snowflake-sql-trapped-catalogPASS 1.00

? The Question

Using ONLY the catalog provided, write one Snowflake SQL query returning, per customer company, for its first calendar month as a customer: unique givers, recognition count, successful redemption count. Read the catalog notes carefully: some tables are stale or unpopulated and must be avoided, and one documented filter (excluding deleted givers) must NOT be applied to historical giving counts. Put a comment above each FROM or JOIN explaining the table choice. Return only fenced SQL.
cost $0.0009 · 45s · in 1,243 / out 12,934 tok
✓ avoids_stale_deals✓ avoids_unpopulated_live✓ uses_cohort_summary✓ succeeded_filter✓ no_giver_deletion_filter✓ returns_sql

✓ Correct Answer (ground truth)

graded checks: avoids_stale_deals · avoids_unpopulated_live · uses_cohort_summary · succeeded_filter · no_giver_deletion_filter · returns_sql

Model's actual answer

show the model's full answer
```sql
SELECT
    ccs.COMPANY_ID,
    DATE_TRUNC('month', ccs.SIGNUP_DATE) AS first_calendar_month,
    COUNT(DISTINCT rrc.GIVER_ID) AS unique_givers,
    COUNT(rrc.RECOGNITION_ID) AS recognition_count,
    SUM(CASE WHEN rrc.STATE = 'succeeded' THEN 1 ELSE 0 END) AS successful_redemption_count
FROM PRODUCTION.PLG.COMPANY_COHORT_SUMMARY ccs
-- COMPANY_COHORT_SUMMARY: one row per self-serve company; SIGNUP_DATE determines the
-- first calendar month as a customer; provides the COMPANY_ID join key to the
-- granular recognition/redemption records table.
LEFT JOIN PRODUCTION.DEPRECATED_RECOGNITION.REDEMPTION_RECORDS_V2 rrc
    -- REDEMPTION_RECORDS_V2: documented source for recognition/giving events and
    -- redemption records; per business rule, do NOT exclude deleted givers (no
    -- IS_DELETED or similar filter applied to historical giving counts, as it
    -- understates history); STATE = 'succeeded' filter is applied only in the
    -- CASE expression for successful_redemption_count, not for unique_givers or
    -- recognition_count.
    ON rrc.COMPANY_ID = ccs.COMPANY_ID
    AND DATE_TRUNC('month', rrc.CREATED_AT) = DATE_TRUNC('month', ccs.SIGNUP_DATE)
GROUP BY ccs.COMPANY_ID, DATE_TRUNC('month', ccs.SIGNUP_DATE)
```
crm-completeness-enrichmentPASS 1.00

? The Question

Audit the CRM extract for completeness and propose fixes. Deals need owner, stage, amount, close date, why-buys; companies need industry, employee count, HQ country; contacts need email, title, persona. Report completeness percent per field. Identify duplicate company clusters (name variants, shared domain) with a survivor per cluster. List invalid emails and domain mismatches. Fill missing company fields only where the enrichment export has a matching row; where CRM and enrichment disagree list both and recommend a source. Never invent a value. End with the 10 fixes with the most pipeline amount at stake.
cost $0.0027 · 44s · in 49,733 / out 10,823 tok
✓ dup_clusters✓ invalid_emails✓ survivor_language✓ completeness_pct✓ no_invent

✓ Correct Answer (ground truth)

n_companies: 34
n_contacts: 52
company_missing: industry=0; employee_count=9; hq_country=6
invalid_email_count: 4
invalid_emails: CT-0010, CT-0080, CT-0081, CT-0192
domain_mismatch_count: 1
domain_mismatches: CT-0011
duplicate_clusters: ['C-0A092931', 'C-0A092932'], ['C-0A092933', 'C-0A092934']
n_dup_clusters: 2
enrichment_matches: 25
graded checks: dup_clusters · invalid_emails · survivor_language · completeness_pct · no_invent

Model's actual answer

show the model's full answer
Now I have all the data needed. Let me compile the final audit report.

---

CRM COMPLETENESS AUDIT
======================

1. DEAL COMPLETENESS (86 deals)
   owner:          1/86 = 1.2%  (85 missing)
   stage:          86/86 = 100.0%
   amount:         86/86 = 100.0%
   close_date:     86/86 = 100.0%
   why_buys_chars: 15/86 = 17.4%  (71 missing)

2. COMPANY COMPLETENESS (34 companies)
   industry:       34/34 = 100.0%
   employee_count: 25/34 = 73.5%  (9 missing)
   hq_country:     28/34 = 82.4%  (6 missing)

3. CONTACT COMPLETENESS (52 contacts)
   email:  52/52 = 100.0%
   title:  40/52 = 76.9%  (12 missing)
   persona: 37/52 = 71.2%  (15 missing)

---

DUPLICATE COMPANY CLUSTERS
===========================

Cluster 1: acme-corp.com
  C-0A092931 — industry=Technology, employees=500, hq=US
  C-0A092932 — industry=tech, employees=510, hq=USA
  Survivor: C-0A092931 (most complete; 3/3 fields populated; lowest alias)

Cluster 2: globex.io
  C-0A092933 — industry=SaaS, employees=200, hq=US
  C-0A092934 — industry=Technology, employees=200, hq=US
  Survivor: C-0A092933 (both equally complete; first alias)

---

INVALID EMAILS
==============

CT-0010: email='user0@' — NO_DOMAIN (trailing @, no domain)
CT-0011: email='user1@other-domain.com' — DOMAIN_MISMATCH (email domain other-domain.com != contact domain 66d1fc.com)
CT-0080: email='user0@' — NO_DOMAIN
CT-0081: email='user1@' — NO_DOMAIN
CT-0192: email='user2@' — NO_DOMAIN

Total: 5 invalid emails across 5 contacts.

---

CRM vs ENRICHMENT DISAGREEMENTS (only where enrichment has a matching row)
============================================================================

The following CRM fields disagree with ZoomInfo enrichment and enrichment is recommended as the authoritative source in every case:

Industry (format-only disagreements — CRM shorthand vs ZoomInfo canonical):
  C-66D1FC: CRM='tech' vs ZoomInfo='Computer Software' -> recommend ZoomInfo
  C-EC3025: CRM='Technology' vs ZoomInfo='Computer Software' -> recommend ZoomInfo
  C-44EA29: CRM='tech' vs ZoomInfo='Computer Software' -> recommend ZoomInfo
  C-92D97D: CRM='Technology' vs ZoomInfo='Computer Software' -> recommend ZoomInfo
  C-D04904: CRM='Technology' vs ZoomInfo='Computer Software' -> recommend ZoomInfo
  C-77A95A: CRM='Technology' vs ZoomInfo='Computer Software' -> recommend ZoomInfo
  C-AA8DDA: CRM='Technology' vs ZoomInfo='Computer Software' -> recommend ZoomInfo
  C-B25F40: CRM='Tech' vs ZoomInfo='Computer Software' -> recommend ZoomInfo
  C-60C75F: CRM='tech' vs ZoomInfo='Computer Software' -> recommend ZoomInfo
  C-425E2A: CRM='Tech' vs ZoomInfo='Computer Software' -> recommend ZoomInfo

Country (format-only: US/USA vs United States):
  C-66D1FC: CRM='US' vs ZoomInfo='United States' -> recommend ZoomInfo
  C-950043: CRM='US' vs ZoomInfo='United States' -> recommend ZoomInfo
  C-EC3025: CRM='USA' vs ZoomInfo='United States' -> recommend ZoomInfo
  C-96039F: CRM='USA' vs ZoomInfo='United States' -> recommend ZoomInfo
  C-77A95A: CRM='US' vs ZoomInfo='United States' -> recommend ZoomInfo
  C-B23205: CRM='US' vs ZoomInfo='United States' -> recommend ZoomInfo
  C-E51FB7: CRM='USA' vs ZoomInfo='United States' -> recommend ZoomInfo
  C-D0662E: CRM='US' vs ZoomInfo='United States' -> recommend ZoomInfo
  C-B25F40: CRM='Canada' vs ZoomInfo='Canada' -> AGREE (no conflict)
  C-425E2A: CRM='USA' vs ZoomInfo='United States' -> recommend ZoomInfo
  C-2D7423: CRM='USA' vs ZoomInfo='United States' -> recommend ZoomInfo

Employee count (true numerical disagreement):
  C-B25F40: CRM=50 vs ZoomInfo=120 -> recommend ZoomInfo

Missing fields fillable from enrichment (CRM empty, enrichment has value):
  C-EC3025: employee_count CRM='' -> ZoomInfo='400'
  C-96039F: employee_count CRM='' -> ZoomInfo='400'
  C-44EA29: employee_count CRM='' -> ZoomInfo='400'
  C-D04904: employee_count CRM='' -> ZoomInfo='400'
  C-B23205: employee_count CRM='' -> ZoomInfo='400'
  C-60C75F: employee_count CRM='' -> ZoomInfo='400'
  C-7BBDFA: employee_count CRM='' -> ZoomInfo='400'
  C-50D386: employee_count CRM='' -> ZoomInfo='400'

Note: 5 companies have no enrichment row (332637.com, 93c8bf.com, ba969b.com, c9bb20.com, ee9ffb.com). Their missing fields cannot be filled from the provided enrichment export.

---

TOP 10 FIXES BY PIPELINE AMOUNT AT STAKE
==========================================

1. Deal-E51FB7 (56896838550): $43,875.00 — missing owner, why_buys
2. Deal-2D7423 (63189452296): $38,935.00 — missing owner, why_buys
3. Deal-EE9FFB (63361715607): $35,940.00 — missing owner, why_buys
4. Deal-25F752 (63223160241): $24,000.00 — missing owner, why_buys
5. Deal-D56743 (59963612314): $24,000.00 — missing owner
6. Deal-E53952 (60524834645): $19,656.00 — missing owner, why_buys
7. Deal-B936FE (60039273647): $18,000.00 — missing owner, why_buys
8. Deal-CFE1E8 (63083909749): $18,000.00 — missing owner, why_buys
9. Deal-4F775F (63434077517): $18,000.00 — missing owner, why_buys
10. Deal-D9A12F (55922600992): $17,000.00 — missing owner, why_buys

Aggregate pipeline at stake: $672,004.40 for missing owner; $597,566.40 for missing why_buys. The single highest-leverage fix is populating the owner field on Deal-E51FB7 ($43,875).
deal-intelligence
closed-lost-classification0.80

? The Question

Classify every deal lost in the last 6 months below into one primary category: pricing, competitor, no decision, timing, product gap, champion left, or other. Also assign a side (Bonusly, buyer, unknown). Then summarize: category counts, the side split, the number of deals where the structured closed_lost_tag clearly disagrees with the free-text reason, and the two patterns most worth acting on. Base everything only on the tag and free-text provided.
cost $0.0080 · 232s · in 89,050 / out 75,678 tok
✓ total_90✗ competitor_top✓ category_counts_present✓ side_split✓ two_patterns

✓ Correct Answer (ground truth)

graded checks: total_90 · competitor_top · category_counts_present · side_split · two_patterns

Model's actual answer

show the model's full answer
Here is the full classification and summary based solely on the data in `closed_lost_6mo.csv`.

---

## Deal Classifications (all 90 deals)

| Deal Alias | Tag | Primary Category | Side |
|---|---|---|---|
| Deal-DB0AAC | Lost- Timing (1 year or more) | timing | buyer |
| Deal-F7F635 | Competitor | competitor | Bonusly |
| Deal-AC944F | MIA | no decision | buyer |
| Deal-214060 | MIA | no decision | buyer |
| Deal-91A056 | Lost- Timing (1 year or more) | timing | buyer |
| Deal-29326C | Lost- Timing (1 year or more) | timing | buyer |
| Deal-5DB9B0 | Lost- Does not fit ICP (write in notes) | product gap | Bonusly |
| Deal-831B7B | Lost- Timing (1 year or more) | timing | buyer |
| Deal-F97C37 | Competitor | competitor | Bonusly |
| Deal-13E9CF | Doing nothing/Not a priority/Cost | pricing | buyer |
| Deal-39E25C | Lost- Timing (1 year or more) | timing | buyer |
| Deal-7ED004 | Lost- Budget/Price | pricing | buyer |
| Deal-21B045 | MIA | no decision | buyer |
| Deal-B3ABED | Lost- Timing (1 year or more) | timing | buyer |
| Deal-422BA6 | Competitor | competitor | Bonusly |
| Deal-ED9AE7 | Lost DM | timing | buyer |
| Deal-988493 | MIA | no decision | buyer |
| Deal-381C8C | Competitor | competitor | Bonusly |
| Deal-F308CA | MIA | no decision | buyer |
| Deal-F1E8A6 | Competitor | competitor | Bonusly |
| Deal-B6AC09 | Lost- Timing (1 year or more) | timing | buyer |
| Deal-70F704 | Lost DM | product gap | Bonusly |
| Deal-E6E80A | Lost- Timing (1 year or more) | timing | buyer |
| Deal-B038F0 | Lost- Timing (1 year or more) | timing | buyer |
| Deal-4664E1 | MIA | no decision | buyer |
| Deal-175756 | Lost- Timing (1 year or more) | timing | buyer |
| Deal-E74A73 | Doing nothing/Not a priority/Cost | pricing | buyer |
| Deal-DDAB52 | Competitor | competitor | Bonusly |
| Deal-ACE061 | Competitor | competitor | Bonusly |
| Deal-BB78F3 | Lost- Timing (1 year or more) | timing | buyer |
| Deal-D48E0B | MIA | no decision | buyer |
| Deal-15DA99 | Lost- Timing (1 year or more) | timing | buyer |
| Deal-F4AF5D | Lost- Timing (1 year or more) | timing | buyer |
| Deal-79B7A1 | Lost- Timing (1 year or more) | timing | buyer |
| Deal-583ADB | MIA | no decision | buyer |
| Deal-8E27DA | Feature Request | product gap | Bonusly |
| Deal-2D2F8D | Competitor | competitor | Bonusly |
| Deal-E0441F | MIA | champion left | buyer |
| Deal-7CB44D | MIA | no decision | buyer |
| Deal-0F96AA | Competitor | competitor | Bonusly |
| Deal-1BCA50 | Competitor | competitor | Bonusly |
| Deal-7CC678 | Competitor | other | unknown |
| Deal-FAC17C | Lost DM | champion left | buyer |
| Deal-242273 | Competitor | competitor | Bonusly |
| Deal-50E5D8 | Doing nothing/Not a priority/Cost | pricing | buyer |
| Deal-A2C349 | Competitor | competitor | Bonusly |
| Deal-9F176A | Lost- Timing (1 year or more) | timing | buyer |
| Deal-7B2236 | Doing nothing/Not a priority/Cost | pricing | buyer |
| Deal-AFA56C | MIA | no decision | buyer |
| Deal-C7156E | Competitor | competitor | Bonusly |
| Deal-C33D91 | Lost- Budget/Price | pricing | buyer |
| Deal-9048EB | MIA | product gap | Bonusly |
| Deal-5E64CE | Doing nothing/Not a priority/Cost | pricing | buyer |
| Deal-8A0992 | Competitor | competitor | Bonusly |
| Deal-D0C698 | Competitor | competitor | Bonusly |
| Deal-69CF3D | Lost- Timing (1 year or more) | timing | buyer |
| Deal-ECBF89 | Lost- Timing (1 year or more) | timing | buyer |
| Deal-3618CC | Lost DM | product gap | Bonusly |
| Deal-EECC02 | Competitor | competitor | Bonusly |
| Deal-5AD03E | Competitor | pricing | buyer |
| Deal-D1A623 | Lost- Timing (1 year or more) | timing | buyer |
| Deal-413C56 | Doing nothing/Not a priority/Cost | pricing | buyer |
| Deal-47F1A1 | Competitor | competitor | Bonusly |
| Deal-BF2A98 | Competitor | competitor | Bonusly |
| Deal-2A292B | Doing nothing/Not a priority/Cost | pricing | buyer |
| Deal-D1AABF | MIA | no decision | buyer |
| Deal-FEDBCB | Doing nothing/Not a priority/Cost | pricing | buyer |
| Deal-1E7DA9 | Competitor | competitor | Bonusly |
| Deal-2BBA21 | MIA | no decision | buyer |
| Deal-286F9C | Competitor | competitor | Bonusly |
| Deal-7FBAC6 | Doing nothing/Not a priority/Cost | pricing | buyer |
| Deal-369281 | Competitor | competitor | Bonusly |
| Deal-386F6E | MIA | no decision | buyer |
| Deal-9FCD0D | Competitor | competitor | Bonusly |
| Deal-55867E | Lost- Timing (1 year or more) | competitor | Bonusly |
| Deal-DAFB82 | Lost- Budget/Price | pricing | buyer |
| Deal-2FEDDB | Doing nothing/Not a priority/Cost | pricing | buyer |
| Deal-64B19A | Competitor | competitor | Bonusly |
| Deal-3F86A0 | MIA | no decision | buyer |
| Deal-096750 | MIA | no decision | buyer |
| Deal-F325A5 | Lost DM | champion left | buyer |
| Deal-ABD14C | Doing nothing/Not a priority/Cost | product gap | Bonusly |
| Deal-79E61A | MIA | no decision | buyer |
| Deal-8A119B | Lost- Budget/Price | pricing | buyer |
| Deal-AE7C4E | MIA | no decision | buyer |
| Deal-DAB4F1 | MIA | no decision | buyer |
| Deal-B4B50F | MIA | no decision | buyer |
| Deal-981AD4 | Feature Request | product gap | Bonusly |
| Deal-DC77FE | Competitor | competitor | Bonusly |
| Deal-5885B9 | MIA | no decision | buyer |

---

## Category Counts

| Category | Count |
|---|---|
| competitor | 24 |
| no decision | 20 |
| timing | 17 |
| pricing | 15 |
| product gap | 7 |
| champion left | 3 |
| other | 4 |
| **TOTAL** | **90** |

## Side Split

| Side | Count |
|---|---|
| buyer | 55 |
| Bonusly | 31 |
| unknown | 4 |
| **TOTAL** | **90** |

## Tag-vs-Text Disagreements

7 deals where the structured `closed_lost_tag` clearly disagrees with the free-text `closed_lost_reason_text`:

1. **Deal-7CC678** — tag=Competitor, text="Nothing specific provided." (no competitor evidence)
2. **Deal-9048EB** — tag=MIA, text reveals "bad fit" + "multiple feature gaps" (product gap)
3. **Deal-5AD03E** — tag=Competitor, text="Wanted more defined budget access" (pricing)
4. **Deal-BF2A98** — tag=Competitor, text="Recently deployed HiThrive" (competitor — tag is correct but the classifier flagged it; this is actually consistent)
5. **Deal-55867E** — tag=Lost- Timing, text="After careful consideration, I don't think we'll be moving forward" (competitor — they decided against Bonusly, not just paused on timing)
6. **Deal-DC77FE** — tag=Competitor, text confirms a competitor was chosen (consistent; not a real disagreement)
7. **Deal-7B2236** — tag=Doing nothing/Not a priority/Cost, text is about budget/cost (consistent; not a real disagreement)

After removing false positives (deals where the tag and text are actually consistent), the **true disagreements = 4**:

- Deal-7CC678 (Competitor tag, no competitor in text)
- Deal-9048EB (MIA tag, text reveals product gap)
- Deal-5AD03E (Competitor tag, text is about budget access)
- Deal-55867E (Timing tag, text is a firm decision not to proceed)

## Two Patterns Most Worth Acting On

**1. Timing (17 deals, 19% of total)**
All 17 are buyer-side. Free-text consistently says "revisiting in 2027," "push to early 2027," "on hold," "reconnect next year." These are not dead deals — they need a Q1 2027 re-engagement sequence with updated ROI data.

**2. Competitor (24 deals, 27% of total)**
All 24 are Bonusly-side — the competitor won. Named competitors in the free-text: HiThrive (Deal-BF2A98), Rippl (Deal-DDAB52), HeyTaco (Deal-ACE061), Awardco (Deal-A2C349), WorkTango (Deal-47F1A1), Kudos (Deal-D0C698, Deal-242273), Motivosity (Deal-64B19A), Paylocity (Deal-369281), ADP TotalSource PEO partner (Deal-422BA6), a Canadian provider (Deal-8A0992, Deal-9FCD0D). Action: build competitive battle cards for each named competitor; address the ADP preferred-partner advantage (Deal-422BA6) and the "more diversified offerings" objection (Deal-F97C37).
pipeline-tieringPASS 1.00

? The Question

Score the open pipeline below on these signals: forecast category, stage, meetings_30d, contact count, and engagement recency. Assign exactly one tier per deal: LOCK, ACTION, BUILD, REVIVE, WATCH, or RISKY (RISKY = the forecast category disagrees with the engagement evidence). Never tier a deal LOCK if it has zero meetings_30d. Do NOT list every deal. Return ONLY a compact JSON object with exactly these keys: - "tier_counts": object mapping each tier to the number of deals in it (must sum to the total deal count), - "tier_examples": object mapping each tier to up to 3 example deal aliases from the data, - "risky_deals": array of the deal aliases you tiered RISKY, - "lock_violations": number of deals tiered LOCK that have zero meetings_30d (should be 0), - "pipeline_shape": one short paragraph on the overall pipeline shape. Note: inbound_emails_30d is 0 for every row due to a data defect, so treat meetings_30d as the inbound signal.
cost $0.0021 · 40s · in 74,743 / out 7,265 tok
✓ json_returned✓ tier_vocabulary✓ tier_counts_sum✓ risky_deals_listed✓ no_lock_violation_claimed

✓ Correct Answer (ground truth)

total: 156
graded checks: json_returned · tier_vocabulary · tier_counts_sum · risky_deals_listed · no_lock_violation_claimed

Model's actual answer

show the model's full answer
```json
{
  "tier_counts": {
    "LOCK": 9,
    "ACTION": 46,
    "BUILD": 63,
    "REVIVE": 1,
    "WATCH": 2,
    "RISKY": 35
  },
  "tier_examples": {
    "LOCK": ["Deal-25F752", "Deal-C26D20", "Deal-C6FE92"],
    "RISKY": ["Deal-E53952", "Deal-5408B0", "Deal-9AAE5F"],
    "ACTION": ["Deal-D348E1", "Deal-944310", "Deal-403845"],
    "BUILD": ["Deal-6787C2", "Deal-66D1FC", "Deal-950043"],
    "REVIVE": ["Deal-2D1F1B"],
    "WATCH": ["Deal-3EED2C", "Deal-57FF13"]
  },
  "risky_deals": [
    "Deal-E53952", "Deal-5408B0", "Deal-9AAE5F", "Deal-547B2B", "Deal-B7EBD1", "Deal-A2B47C", "Deal-2465CE", "Deal-C61CF7", "Deal-62D607", "Deal-584EE5", "Deal-C6D97A", "Deal-7B3B0F", "Deal-F9A08A", "Deal-0660B4", "Deal-FD9F4E", "Deal-BA571A", "Deal-FC22A3", "Deal-7BBDFA", "Deal-60C2C2", "Deal-4A13AD", "Deal-8AD4A5", "Deal-15D24F", "Deal-9D0060", "Deal-690476", "Deal-635B8E", "Deal-ED725A", "Deal-55164C", "Deal-3BA5EA", "Deal-5FDCE4", "Deal-F336B6", "Deal-5EED42", "Deal-BA3DDC", "Deal-7599B8", "Deal-F9A3C1", "Deal-FA32A0"
  ],
  "lock_violations": 0,
  "pipeline_shape": "Pipeline spans 156 deals totaling $2,314,044 across 5 stages (DS1: 32, DS2: 39, DS3: 61, DS4: 14, DS5: 10). Forecast split is heavily skewed toward PIPELINE (105 deals, $1,055,202) vs. BEST_CASE (40) and COMMIT (11). Tier distribution is top-heavy at BUILD (63) and ACTION (46), with 35 RISKY deals — all BEST_CASE or COMMIT deals whose engagement evidence (zero or minimal meetings_30d, stale contacts) contradicts the optimistic forecast. No LOCK violations. The pipeline is a speculative early-stage backlog with a thin layer of late-stage committed deals, and the RISKY count indicates forecast/engagement misalignment concentrated at closure."
}
```
call-transcript-extractionPASS 1.00

? The Question

For each transcript, extract CRM write-back fields as JSON: why-buys (prospect statements only), pain points, stakeholders from the speaker list, budget signal (prospect-stated or null), timeline signal, competitor mentioned (only if the prospect raised it), next step (only if explicitly agreed), objections, confidence. Never fill a field from what the rep said or hoped.
cost $0.0004 · 13s · in 2,175 / out 4,012 tok
✓ json_emitted✓ budget_signal✓ null_discipline✓ stakeholders✓ competitor_prospect_only

✓ Correct Answer (ground truth)

rep: Alex Franklin
per_transcript: {"TX-001": {"why_buys": ["automating anniversary and birthday awards"], "pain_points": ["HR team of three cannot keep up with it manually", "track everything in a spreadsheet, and people slip through
rep_sourced_nulls: {"TX-003": ["budget_signal \u2014 only the rep mentioned pricing/budget"], "TX-004": ["next_step \u2014 rep proposed, prospect said 'no promises'"], "TX-006": ["budget_signal \u2014 rep decoy, prospec
allowed_new: TX-001, TX-002, TX-003, TX-004, TX-005, TX-006, Achievers, Bucketlist, Nectar
graded checks: json_emitted · budget_signal · null_discipline · stakeholders · competitor_prospect_only

Model's actual answer

show the model's full answer
```json
{
  "TX-001": {
    "deal_alias": "Deal-CFE7F4",
    "why_buys": [
      "automating anniversary and birthday awards — our HR team of three cannot keep up with it manually"
    ],
    "pain_points": [
      "HR team of three cannot keep up with manual processes",
      "track everything in a spreadsheet and people slip through the cracks",
      "need SSO and audit logs for IT to sign off"
    ],
    "stakeholders": ["VP People", "HR Admin"],
    "budget_signal": "$40k earmarked for engagement tools this fiscal year",
    "timeline_signal": "live before open enrollment in November",
    "competitor_mentioned": "Achievers",
    "next_step": "security review on September 12",
    "objections": ["need SSO and audit logs for IT to sign off"],
    "confidence": null
  },
  "TX-002": {
    "deal_alias": "Deal-70BB30",
    "why_buys": [
      "tie recognition to retention for our hourly workforce — regretted turnover there is over 30%"
    ],
    "pain_points": [
      "regretted turnover over 30% for hourly workforce",
      "Integration with Workday has to be rock solid — that's my one condition"
    ],
    "stakeholders": ["Head of Total Rewards", "CFO"],
    "budget_signal": "$25k pilot budget for this quarter",
    "timeline_signal": "decision by end of September",
    "competitor_mentioned": null,
    "next_step": "send pilot agreement and route to legal this week",
    "objections": [],
    "confidence": null
  },
  "TX-003": {
    "deal_alias": "Deal-530B50",
    "why_buys": [
      "make recognition visible across our 12 retail locations",
      "store managers need budget autonomy for on-the-spot recognition"
    ],
    "pain_points": [
      "store managers have zero budget autonomy for on-the-spot recognition today"
    ],
    "stakeholders": ["People Ops Manager"],
    "budget_signal": null,
    "timeline_signal": "no rush until Q1",
    "competitor_mentioned": "Bucketlist",
    "next_step": "schedule a call with our CEO — prospect will send two times",
    "objections": ["the CEO has to be sold first — she decides anything people-related"],
    "confidence": null
  },
  "TX-004": {
    "deal_alias": "Deal-180D02",
    "why_buys": [
      "consolidate three separate recognition tools into one",
      "three tools are not talking to our HRIS"
    ],
    "pain_points": [
      "paying for three tools and none of them talk to our HRIS",
      "the security review took three months for our last vendor"
    ],
    "stakeholders": ["VP People", "IT Security Lead"],
    "budget_signal": "under $15k annually can be approved without going to the board",
    "timeline_signal": "procurement cycle runs six to eight weeks minimum",
    "competitor_mentioned": null,
    "next_step": null,
    "objections": ["the security review took three months for our last vendor — that's my hesitation"],
    "confidence": null
  },
  "TX-005": {
    "deal_alias": "Deal-F8767A",
    "why_buys": [
      "automate service milestones",
      "give us analytics on recognition equity across departments"
    ],
    "pain_points": [
      "night-shift teams feel invisible — engagement scores run 20 points lower",
      "exec team is skeptical after a failed rollout two years ago"
    ],
    "stakeholders": ["HR Director", "People Ops Coordinator"],
    "budget_signal": "$12k approved under our engagement line",
    "timeline_signal": "running before our January all-hands",
    "competitor_mentioned": "Nectar",
    "next_step": "present to exec team on October 2",
    "objections": ["exec team is skeptical after a failed rollout two years ago"],
    "confidence": null
  },
  "TX-006": {
    "deal_alias": "Deal-EE195F",
    "why_buys": [
      "cut the admin time on service awards"
    ],
    "pain_points": [
      "spending five hours a month ordering and shipping plaques"
    ],
    "stakeholders": ["HR Manager"],
    "budget_signal": null,
    "timeline_signal": "Q1 start is realistic (fiscal year resets in February)",
    "competitor_mentioned": null,
    "next_step": "send one-page overview and forward to COO this week",
    "objections": ["COO usually prefers building things in-house"],
    "confidence": null
  }
}
```
next-to-close0.75

? The Question

Which 3 deals are closest to signature right now? Use the CRM data and the Slack thread; Slack often has the real status before the CRM does. For each: deal, amount, why it is close, what is left. Under 250 words.
cost $0.0002 · 7s · in 1,614 / out 1,259 tok
✓ exactly_three✗ top3_named✓ slack_first✓ blocked_excluded

✓ Correct Answer (ground truth)

top3: rank=1; deal_alias=Deal-547B2B; amount=11200; stage=DS5; forecast_category=COMMIT; close_date=2026-09-11 | rank=2; deal_alias=Deal-B7EBD1; amount=9000; stage=DS5; forecast_category=COMMIT; close_date=2026-09-10 | rank=3; deal_alias=Deal-403845; amount=9000; stage=DS5; forecast_category=COMMIT; close_date=2026-09-11
exactly_three: True
blocked_excluded: deal_alias=Deal-2465CE; reason=champion left + procurement freeze (Slack #sales-sparring 2026-09-04)
ranking_rule: Slack signature-imminent deal ranks first; remaining ranked by stage (DS5>DS4), forecast (COMMIT>BEST_CASE>PIPELINE), close_date asc, amount desc
graded checks: exactly_three · top3_named · slack_first · blocked_excluded

Model's actual answer

show the model's full answer
1. Deal-547B2B — $11,200 — Alex Franklin, DS5/COMMIT. VP People told Alex they are signing tomorrow; the signing page is already out. What's left: signature.

2. Deal-403845 — $9,000 — Dana Mercer, DS5/COMMIT. Order form is with their finance team per Slack (2026-09-04 10:02). What's left: finance approval, then signature.

3. Deal-A2B47C — $6,360 — Alex Franklin, DS5/COMMIT. Still warm, normal legal-review pace per Slack (2026-09-04 09:44). What's left: legal review, then signature.

Note: Deal-2465CE ($5,400) was pulled out of commit — champion left and procurement froze. Excluded. Deal-D348E1 is also warm but has a later close date (Oct 15) than A2B47C (Sep 11).
product-gap-detectionPASS 1.00

? The Question

Surface true product gaps raised by prospects in active deals. For each candidate: quote the prospect line, classify as real gap, plan gate, or rollout/enablement issue using the product docs excerpt, and name the deal and amount. Only real gaps go in the summary. Rep language does not count as prospect voice.
cost $0.0002 · 6s · in 858 / out 1,340 tok
✓ real_gaps✓ plan_gate_classified✓ enablement_classified✓ gap_summary

✓ Correct Answer (ground truth)

real_gaps: transcript_id=TG-001; deal_alias=Deal-EC3025; amount=62000; feature=SCIM user provisioning; quote=We need SCIM user provisioning — our security team will not approve any tool that requires manual account management.; why_real=not listed in product docs at any tier | transcript_id=TG-002; deal_alias=Deal-D0D6B5; amount=23800; feature=ADP Workforce Now integration; quote=Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us.; why_real=supported HRIS list is Workday/BambooHR/Gusto/Rippling only
plan_gates: transcript_id=TG-003; deal_alias=Deal-CFE7F4; amount=32000; feature=Custom report builder; tier_required=Enterprise; quote=I tried to build a custom report combining recognition frequency with turnover and couldn't — is that possible on our current plan?
enablement_issues: transcript_id=TG-004; deal_alias=Deal-84DBA6; amount=16000; feature=Slack integration recognition adoption; quote=We turned on the Slack integration in July, but half our managers still haven't given a single recognition — nobody ever trained them on it.; why_enablement=Slack integration exists on all plans; managers were never trained
excluded_rep_voice: transcript_id=TG-005; line=Full transparency — we don't have a native mobile app for hourly workers yet, though it's on the roadmap.; reason=rep voice, not prospect-raised
allowed_new: TG-001, TG-002, TG-003, TG-004, TG-005, SCIM user provisioning, ADP Workforce Now integration, Custom report builder
graded checks: real_gaps · plan_gate_classified · enablement_classified · gap_summary

Model's actual answer

show the model's full answer
**Prospect-raised product gaps (real gaps only)**

1. **Deal-EC3025** — SCIM user provisioning
   - Prospect line: "We need SCIM user provisioning — our security team will not approve any tool that requires manual account management."
   - Classification: **Real gap** — SCIM user provisioning is not listed as a supported capability in the product docs, and is not mentioned as on the roadmap.
   - Amount: not provided in the data.

2. **Deal-D0D6B5** — ADP Workforce Now integration
   - Prospect line: "Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us."
   - Classification: **Real gap** — ADP Workforce Now is explicitly stated as not currently listed as a supported capability, and is not mentioned as on the roadmap.
   - Amount: not provided in the data.

**Excluded candidates (not real gaps)**

- **Deal-CFE7F4** — Custom report builder: **Plan gate** (available on Enterprise tier; not a missing capability).
- **Deal-84DBA6** — Slack recognition adoption: **Rollout/enablement issue** (Slack integration is supported on all plans; the problem is lack of manager training, not product capability).
- **Deal-36C33F** — Mobile app for hourly workers: raised by rep (Alex Franklin), not by the prospect. Prospect's response ("The web version should be fine") accepts the current offering. Excluded per rep-language rule.
rep-performance
stale-pipeline-by-repPASS 1.00

? The Question

The snapshot date is 2026-09-05. Using the open deals and the engagements-by-deal table, list every open deal with no logged email, call, or meeting in the last 7 days. Compute recency from the engagements table's last_email/last_call/last_meeting fields, NOT from any date field on the deal (the deal's last_contacted_field is unreliable). For each stale deal give: deal alias, owner name, stage, amount, days since last contact. Group by owner, order by amount descending within each owner. Then state, per owner, how many stale deals they carry and the total stale amount. (Do not write outreach text.)
cost $0.0015 · 20s · in 46,917 / out 5,377 tok
✓ stale_recall_ge_80✓ grouped_by_owner✓ jake_top_amount✓ kevin_stale_amt✓ jenn_stale_amt✓ stale_count

✓ Correct Answer (ground truth)

stale_count: 65
jake_amt: 626243.0
kevin_amt: 252905.03
jenn_amt: 261645.0
top_alias: Deal-2D1F1B
top_amount: 240000.0
graded checks: stale_recall_ge_80 · grouped_by_owner · jake_top_amount · kevin_stale_amt · jenn_stale_amt · stale_count

Model's actual answer

show the model's full answer
Owner: Bryce Harmon (13 stale deal(s), $626,243.00 total stale amount)

  Deal-2D1F1B | Bryce Harmon | DS1 | $240,000.00 | 81 days since last contact
  Deal-66D1FC | Bryce Harmon | DS1 | $99,000.00 | 16 days since last contact
  Deal-950043 | Bryce Harmon | DS1 | $70,000.00 | 19 days since last contact
  Deal-B23205 | Bryce Harmon | DS1 | $45,000.00 | 16 days since last contact
  Deal-7BBDFA | Bryce Harmon | DS3 | $37,440.00 | 46 days since last contact
  Deal-332637 | Bryce Harmon | DS2 | $36,000.00 | 9 days since last contact
  Deal-1BEEBF | Bryce Harmon | DS1 | $31,500.00 | 19 days since last contact
  Deal-C5658B | Bryce Harmon | DS1 | $23,400.00 | 16 days since last contact
  Deal-40522D | Bryce Harmon | DS3 | $21,000.00 | 19 days since last contact
  Deal-F0EBBB | Bryce Harmon | DS3 | $11,400.00 | 24 days since last contact
  Deal-E25A09 | Bryce Harmon | DS1 | $6,000.00 | 9 days since last contact
  Deal-C9C286 | Bryce Harmon | DS2 | $5,502.00 | 9 days since last contact
  Deal-012CB1 | Bryce Harmon | DS1 | $1.00 | 23 days since last contact

Owner: Dana Mercer (14 stale deal(s), $261,645.00 total stale amount)

  Deal-44EA29 | Dana Mercer | DS2 | $60,000.00 | 10 days since last contact
  Deal-E51FB7 | Dana Mercer | DS2 | $43,875.00 | 12 days since last contact
  Deal-B42F46 | Dana Mercer | DS1 | $27,000.00 | 19 days since last contact
  Deal-BA3DDC | Dana Mercer | DS3 | $23,400.00 | 15 days since last contact
  Deal-9DDE86 | Dana Mercer | DS2 | $20,000.00 | 15 days since last contact
  Deal-215CCA | Dana Mercer | DS3 | $18,900.00 | 17 days since last contact
  Deal-5EED42 | Dana Mercer | DS3 | $16,250.00 | 11 days since last contact
  Deal-57887A | Dana Mercer | DS2 | $15,000.00 | 8 days since last contact
  Deal-B7EBD1 | Dana Mercer | DS5 | $9,000.00 | 16 days since last contact
  Deal-3974EB | Dana Mercer | DS4 | $9,000.00 | 8 days since last contact
  Deal-F40F04 | Dana Mercer | DS2 | $8,100.00 | 15 days since last contact
  Deal-87DDD1 | Dana Mercer | DS1 | $5,000.00 | 19 days since last contact
  Deal-F336B6 | Dana Mercer | DS3 | $4,200.00 | 15 days since last contact
  Deal-0660B4 | Dana Mercer | DS4 | $1,920.00 | 16 days since last contact

Owner: Alex Franklin (19 stale deal(s), $109,536.00 total stale amount)

  Deal-CC08D1 | Alex Franklin | DS1 | $24,000.00 | 16 days since last contact
  Deal-E73427 | Alex Franklin | DS3 | $18,000.00 | 10 days since last contact
  Deal-885F45 | Alex Franklin | DS2 | $9,300.00 | 12 days since last contact
  Deal-C2FF3C | Alex Franklin | DS1 | $8,316.00 | 10 days since last contact
  Deal-3EED2C | Alex Franklin | DS2 | $7,200.00 | N/A (no contact logged)
  Deal-0D2F7A | Alex Franklin | DS3 | $5,100.00 | 12 days since last contact
  Deal-6C60D4 | Alex Franklin | DS3 | $4,800.00 | 12 days since last contact
  Deal-13FEBD | Alex Franklin | DS2 | $4,680.00 | 12 days since last contact
  Deal-9D0060 | Alex Franklin | DS3 | $3,840.00 | 12 days since last contact
  Deal-690476 | Alex Franklin | DS2 | $3,600.00 | 18 days since last contact
  Deal-C6D97A | Alex Franklin | DS4 | $3,240.00 | 8 days since last contact
  Deal-EE195F | Alex Franklin | DS3 | $3,120.00 | 8 days since last contact
  Deal-278DEC | Alex Franklin | DS3 | $2,700.00 | 8 days since last contact
  Deal-635B8E | Alex Franklin | DS3 | $2,600.00 | 18 days since last contact
  Deal-6883F3 | Alex Franklin | DS1 | $2,400.00 | 16 days since last contact
  Deal-4A13AD | Alex Franklin | DS3 | $2,160.00 | 26 days since last contact
  Deal-F67D31 | Alex Franklin | DS2 | $1,800.00 | 8 days since last contact
  Deal-5FDCE4 | Alex Franklin | DS3 | $1,600.00 | 12 days since last contact
  Deal-BA571A | Alex Franklin | DS4 | $1,080.00 | 18 days since last contact

Owner: Cole Ingram (18 stale deal(s), $252,905.03 total stale amount)

  Deal-D04904 | Cole Ingram | DS2 | $58,529.25 | 11 days since last contact
  Deal-B25F40 | Cole Ingram | DS3 | $40,000.00 | 8 days since last contact
  Deal-813836 | Cole Ingram | DS2 | $32,175.00 | 11 days since last contact
  Deal-1BA595 | Cole Ingram | DS2 | $31,750.00 | 11 days since last contact
  Deal-CFE1E8 | Cole Ingram | DS3 | $18,000.00 | 11 days since last contact
  Deal-CD47A6 | Cole Ingram | DS2 | $12,168.00 | 11 days since last contact
  Deal-627646 | Cole Ingram | DS3 | $11,193.00 | 11 days since last contact
  Deal-FF809F | Cole Ingram | DS2 | $7,781.20 | 11 days since last contact
  Deal-AF932D | Cole Ingram | DS2 | $7,225.40 | 11 days since last contact
  Deal-A71728 | Cole Ingram | DS2 | $6,947.50 | 11 days since last contact
  Deal-8BC9F5 | Cole Ingram | DS2 | $5,616.00 | 10 days since last contact
  Deal-175395 | Cole Ingram | DS3 | $4,779.88 | 11 days since last contact
  Deal-481E24 | Cole Ingram | DS3 | $4,140.00 | 10 days since last contact
  Deal-C7F9BF | Cole Ingram | DS2 | $3,360.00 | 11 days since last contact
  Deal-2F3A66 | Cole Ingram | DS3 | $3,334.80 | 11 days since last contact
  Deal-342E96 | Cole Ingram | DS2 | $2,700.00 | 24 days since last contact
  Deal-E568D5 | Cole Ingram | DS3 | $1,875.00 | 11 days since last contact
  Deal-FD9F4E | Cole Ingram | DS5 | $1,330.00 | 10 days since last contact

Owner: Elena Sinclair (1 stale deal(s), $2,100.00 total stale amount)

  Deal-57FF13 | Elena Sinclair | DS1 | $2,100.00 | N/A (no contact logged)

================================================================================
TOTAL: 65 stale deals across 5 owners, $1,252,429.03 total stale amount

Methodology: Snapshot date 2026-09-05, cutoff 2026-08-29 (7 days). Recency computed from engagements_by_deal_90d.csv fields last_email, last_call, last_meeting (not deal.last_contacted_field). A deal is stale when none of those three fields has a date >= cutoff. Days since last contact = (2026-09-05 minus max of the three fields). Deals with all three fields empty get N/A. Owners sourced from owners.csv. Deals grouped by owner_name, sorted by amount descending within group.
activity-mix-vs-outcomePASS 1.00

? The Question

The snapshot date is 2026-09-05. For each sales rep (owner), compute the last-30-day totals of emails, calls, and meetings from the engagements table, and the number of deals that entered DS2 in the last 30 days (t_ds2 within 30 days of the snapshot). Report per rep: the activity mix as percentages (emails/calls/meetings share of that rep's total activities), total activities per DS2 entry, and rank the reps by that efficiency ratio (lowest activities per DS2 entry = most efficient). Name the most efficient rep and the highest-volume rep, and note if they differ. Do not attribute cause.
cost $0.0012 · 20s · in 30,872 / out 4,316 tok
✓ justin_ds2✓ justin_volume_high✓ efficient_named✓ mix_present✓ no_causal_claim

✓ Correct Answer (ground truth)

justin_ds2: 4
graded checks: justin_ds2 · justin_volume_high · efficient_named · mix_present · no_causal_claim

Model's actual answer

show the model's full answer
Per-rep report (snapshot 2026-09-05, window 2026-08-06 to 2026-09-05):

| Rank | Rep | Emails | Calls | Meetings | Total | DS2 entries (30d) | Activities/DS2 |
|------|-----|--------|-------|----------|-------|-------------------|----------------|
| 1 | Alex Franklin | 307 (79.9%) | 36 (9.4%) | 41 (10.7%) | 384 | 18 | 21.33 |
| 2 | Bryce Harmon | 162 (79.0%) | 0 (0.0%) | 43 (21.0%) | 205 | 4 | 51.25 |
| 3 | Cole Ingram | 96 (86.5%) | 14 (12.6%) | 1 (0.9%) | 111 | 2 | 55.50 |
| 4 | Farid Osman | 38 (52.8%) | 0 (0.0%) | 34 (47.2%) | 72 | 1 | 72.00 |
| 5 | Dana Mercer | 84 (74.3%) | 18 (15.9%) | 11 (9.7%) | 113 | 1 | 113.00 |

Most efficient rep (lowest activities per DS2 entry): Alex Franklin (21.33 activities/DS2)
Highest-volume rep (most total activities): Alex Franklin (384 activities)

They are the same person.
ae-qtd-scorecardPASS 1.00

? The Question

Build a QTD scorecard for the named AE as of 2026-09-05: bookings vs quota with attainment percent, new vs expansion split, active pipeline by stage with amount, rolling 90-day DS2-to-won rate, win and loss counts with top loss reason, activity volume by type in the last 30 days. Exclude any closed-won deal dated before the quarter. Close with three coaching observations grounded in the numbers.
cost $0.0025 · 82s · in 33,605 / out 25,395 tok
✓ bookings✓ quota✓ attainment✓ new_expansion_split✓ top_loss_reason✓ coaching_obs

✓ Correct Answer (ground truth)

owner: Alex Franklin
quarter: 2026-Q3
snapshot_date: 2026-09-05
bookings: 150000
quota: 200000
attainment_pct: 75.0
new_bookings: 113500
new_count: 5
expansion_bookings: 36500
expansion_count: 3
pipeline_by_stage: {"DS1": {"count": 20, "amount": 284621}, "DS2": {"count": 28, "amount": 353760}, "DS3": {"count": 67, "amount": 552705}, "DS4": {"count": 5, "amount": 23574}, "DS5": {"count": 5, "amount": 45730}}
open_deals_count: 125
ds2_to_won: window_start=2026-06-07; won_in_window=8; closed_in_window=35; rate=0.2286
wins_qtd: 8
losses_qtd: 27
top_loss_reason: reason=Lost- Timing (1 year or more); count=13
activity_30d_totals: emails_30d=807; calls_30d=112; meetings_30d=128; notes_30d=50
prior_quarter_excluded: deal_alias=Deal-B3E6F1; close_date=2026-06-20; amount=24000
allowed_new: 18 items — e.g. Deal-A1C3E5, Deal-B7D2F4, Deal-C9E1A6, Deal-D4B8C2
graded checks: bookings · quota · attainment · new_expansion_split · top_loss_reason · coaching_obs

Model's actual answer

show the model's full answer
QTD SCORECARD — Alex Franklin (as of 2026-09-05)
========================================================

1. BOOKINGS vs QUOTA
  Bookings (QTD): $150,000
  Quota (2026-Q3): $200,000
  Attainment: 75.0%
  Won deals (Q3, on/before 2026-09-05): 8
    Deal-A1C3E5 ($40,000, 2026-07-15, new)
    Deal-F2C7D8 ($20,000, 2026-07-24, expansion)
    Deal-B7D2F4 ($35,000, 2026-07-31, new)
    Deal-C9E1A6 ($21,000, 2026-08-12, new)
    Deal-A8B4D6 ($12,000, 2026-08-19, expansion)
    Deal-D4B8C2 ($11,000, 2026-08-21, new)
    Deal-E6F3A9 ($6,500, 2026-09-02, new)
    Deal-C5D9E2 ($4,500, 2026-09-03, expansion)

2. NEW vs EXPANSION SPLIT
  New: 5 deals, $113,500 (75.7% of bookings)
  Expansion: 3 deals, $36,500 (24.3% of bookings)
  Arithmetic: 113,500 + 36,500 = 150,000. 113,500 / 150,000 = 75.7%. 36,500 / 150,000 = 24.3%.

3. ACTIVE PIPELINE BY STAGE (open deals only)
  DS1: 20 deals, $284,621
  DS2: 28 deals, $353,760
  DS3: 67 deals, $552,705
  DS4: 5 deals, $23,574
  DS5: 5 deals, $45,730
  Total: 125 open deals, $1,260,390 pipeline

4. ROLLING 90-DAY DS2-TO-WON RATE (window: 2026-06-07 to 2026-09-05)
  Deals entered DS2 in window: 111
  Of those, closed-won: 8
  DS2-to-won rate: 7.2% (8 / 111 = 7.2%)

5. WIN / LOSS (Q3, on/before 2026-09-05)
  Wins: 8
  Losses: 27
  Top loss reason: "Lost- Timing (1 year or more)" — 13 of 27 losses (48.1%)
  All loss reasons: MIA (5), Competitor (5), Lost- Timing (1 year or more) (13), Lost DM (2), Feature Request (1), Lost- Does not fit ICP (write in notes) (1)

6. ACTIVITY VOLUME (last 30 days from 2026-09-05, across all Alex Franklin deals)
  Emails: 807
  Calls: 112
  Meetings: 128
  Notes: 50
  Total activities: 1,097
  Arithmetic: 807 + 112 + 128 + 50 = 1,097

COACHING OBSERVATIONS
---------------------
1. DS2-to-won rate is 7.2% (8 wins out of 111 deals that entered DS2 in the last 90 days). The AE is generating high volume at DS2 but converting at a very low rate — the bottleneck is advancing deals from early-stage engagement to late-stage closure. Focusing on fewer, higher-quality DS2-to-DS4 pushes would improve both conversion and attainment.

2. "Lost- Timing (1 year or more)" is the #1 loss reason at 13 of 27 losses (48.1%). Nearly half the losses are cycle-length issues, not product or competitive gaps. The AE should qualify deal timing earlier in the funnel — if a prospect cannot commit within 6 months, the deal should be disqualified or parked rather than consuming weekly cadence and stage advancement effort.

3. The pipeline is heavily top-heavy at DS3 (67 deals, $552,705) with minimal DS5 coverage (5 deals, $45,730) and 28 DS2 deals still sitting idle. At 75.0% attainment with $150K of bookings against a $200K quota, the AE needs more deals reaching decision stage. Prioritizing advancement of the 28 DS2 deals into DS3/DS4 — and accelerating the 67 DS3 deals toward DS5 — would directly close the $50K gap to quota.
multithreading-gapPASS 1.00

? The Question

Find every open deal that is single-threaded (fewer than 2 active contacts) or under-threaded (fewer than 3, or all contacts in one persona). Active means engaged in the last 60 days and not marked former. Personas: economic buyer, champion, HR admin, IT security, finance. For each flagged deal: amount, stage, active contact count, personas present, personas missing, the most valuable persona to add given the stage, and any on-file unengaged contact at that company who fits; otherwise say none on file.
cost $0.0025 · 79s · in 42,239 / out 21,194 tok
✓ flagged_recall✓ persona_language✓ single_thread_term✓ on_file_fit

✓ Correct Answer (ground truth)

active_window_start: 2026-07-07
persona_universe: economic buyer, champion, HR admin, IT security, finance
flag_rule: flag if active_contacts < 2 (single-threaded) or active_contacts < 3 or all active contacts share one persona (under-threaded); active = engaged within 60 days and not former
best_persona_rule: DS2/DS3: economic buyer > IT security > finance > HR admin > champion; DS4/DS5: economic buyer > finance > IT security > HR admin > champion; first persona not present among active contacts
flagged: deal_alias=Deal-36C33F; amount=15000; stage=DS2; active_count=1; reason=single-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-50D386; amount=36000; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-5408B0; amount=14850; stage=DS4; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-5BFE3B; amount=18000; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-885F45; amount=9300; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=IT security | deal_alias=Deal-92D97D; amount=60000; stage=DS2; active_count=1; reason=single-threaded; best_persona_to_add=economic buyer … (+5 more)
healthy_not_flagged: Deal-4B0BEB, Deal-84DBA6, Deal-D348E1
allowed_new: 59 items — e.g. C-077A0E, C-0D15DF, C-1BDFE0, C-2AE3AA
graded checks: flagged_recall · persona_language · single_thread_term · on_file_fit

Model's actual answer

show the model's full answer
Here are the results. Amount and stage are not present in the input data — marked as NOT PROVIDED for every deal.

---

11 of 14 deals flagged. 3 deals pass (Deal-84DBA6: 3 contacts / 3 personas; Deal-4B0BEB: 4 contacts / 4 personas; Deal-D348E1: 5 contacts / 5 personas).

---

Deal-F9A08A (deal_id=49757401138)
  company: C-0D15DF
  amount: NOT PROVIDED in data
  stage: NOT PROVIDED in data
  active contact count: 1
  flag reason: single-threaded (1 active contact)
  personas present: champion
  personas missing: economic buyer, finance, HR admin, IT security
  most valuable persona to add: economic buyer
  on-file unengaged contact: CT-697541 / Chief People Officer / economic buyer

Deal-5BFE3B (deal_id=51674270311)
  company: C-535D36
  amount: NOT PROVIDED in data
  stage: NOT PROVIDED in data
  active contact count: 2
  flag reason: under-threaded (2 active contacts, fewer than 3); under-threaded (2 contacts all in champion persona)
  personas present: champion
  personas missing: economic buyer, finance, HR admin, IT security
  most valuable persona to add: economic buyer
  on-file unengaged contacts fitting missing personas: none on file

Deal-92D97D (deal_id=59728118877)
  company: C-E23238
  amount: NOT PROVIDED in data
  stage: NOT PROVIDED in data
  active contact count: 1
  flag reason: single-threaded (1 active contact)
  personas present: HR admin
  personas missing: champion, economic buyer, finance, IT security
  most valuable persona to add: economic buyer
  on-file unengaged contacts fitting missing personas: none on file

Deal-D0D6B5 (deal_id=60081655042)
  company: C-32918E
  amount: NOT PROVIDED in data
  stage: NOT PROVIDED in data
  active contact count: 3
  flag reason: under-threaded (3 contacts all in champion persona)
  personas present: champion
  personas missing: economic buyer, finance, HR admin, IT security
  most valuable persona to add: economic buyer
  on-file unengaged contact: CT-1FA4DB / Chief People Officer / economic buyer

Deal-5408B0 (deal_id=60182332309)
  company: C-2AE3AA
  amount: NOT PROVIDED in data
  stage: NOT PROVIDED in data
  active contact count: 2
  flag reason: under-threaded (2 active contacts, fewer than 3)
  personas present: champion, HR admin
  personas missing: economic buyer, finance, IT security
  most valuable persona to add: economic buyer
  on-file unengaged contact: CT-07FA76 / Chief People Officer / economic buyer

Deal-885F45 (deal_id=60686135564)
  company: C-5E8EFB
  amount: NOT PROVIDED in data
  stage: NOT PROVIDED in data
  active contact count: 2
  flag reason: under-threaded (2 active contacts, fewer than 3)
  personas present: champion, economic buyer
  personas missing: finance, HR admin, IT security
  most valuable persona to add: IT security
  on-file unengaged contact: CT-B3F25D / IT Security Lead / IT security

Deal-EC3025 (deal_id=61032318100)
  company: C-FDD0C7
  amount: NOT PROVIDED in data
  stage: NOT PROVIDED in data
  active contact count: 1
  flag reason: single-threaded (1 active contact)
  personas present: champion
  personas missing: economic buyer, finance, HR admin, IT security
  most valuable persona to add: economic buyer
  on-file unengaged contact: CT-6827DB / Chief People Officer / economic buyer

Deal-50D386 (deal_id=61055128146)
  company: C-EB10E4
  amount: NOT PROVIDED in data
  stage: NOT PROVIDED in data
  active contact count: 2
  flag reason: under-threaded (2 active contacts, fewer than 3)
  personas present: champion, HR admin
  personas missing: economic buyer, finance, IT security
  most valuable persona to add: economic buyer
  on-file unengaged contact: CT-A1C4B3 / Chief People Officer / economic buyer

Deal-C6D97A (deal_id=62121783047)
  company: C-5A8FC2
  amount: NOT PROVIDED in data
  stage: NOT PROVIDED in data
  active contact count: 3
  flag reason: under-threaded (3 contacts all in champion persona)
  personas present: champion
  personas missing: economic buyer, finance, HR admin, IT security
  most valuable persona to add: economic buyer
  on-file unengaged contacts fitting missing personas: none on file

Deal-FCBE5B (deal_id=62639586615)
  company: C-737030
  amount: NOT PROVIDED in data
  stage: NOT PROVIDED in data
  active contact count: 1
  flag reason: single-threaded (1 active contact)
  personas present: champion
  personas missing: economic buyer, finance, HR admin, IT security
  most valuable persona to add: economic buyer
  on-file unengaged contacts fitting missing personas: none on file

Deal-36C33F (deal_id=63739413805)
  company: C-077A0E
  amount: NOT PROVIDED in data
  stage: NOT PROVIDED in data
  active contact count: 1
  flag reason: single-threaded (1 active contact)
  personas present: IT security
  personas missing: champion, economic buyer, finance, HR admin
  most valuable persona to add: economic buyer
  on-file unengaged contact: CT-1DB73E / Chief People Officer / economic buyer

---

Notes on methodology:
- Reference date: 2026-09-06. Active = last_engaged_date >= 2026-07-08 AND is_former=false.
- Single-threaded = fewer than 2 active contacts. Under-threaded = fewer than 3 active contacts OR all active contacts share one persona.
- Most valuable persona to add uses priority order: economic buyer > champion > finance > HR admin > IT security, constrained to missing personas with an on-file unengaged contact where available.
- Amount and stage are not columns in deal_contacts.csv; marked NOT PROVIDED for all deals.
talk-track-from-transcriptsPASS 1.00

? The Question

From this rep's last 10 calls: what they lead with in the first five minutes, how they handle the three most common objections, how often a concrete next step is agreed (rate), and every competitor a prospect raised. Quote at most one line per finding and only lines that appear in the transcripts. End with two coaching notes.
cost $0.0003 · 9s · in 3,268 / out 1,839 tok
✓ objection_count✓ next_step_rate✓ competitors✓ coaching✓ quote_used

✓ Correct Answer (ground truth)

rep: Alex Franklin
n_transcripts: 10
opener_theme: theme=opens with a customer recognition win story (400-person retailer, 18% turnover reduction); count=8; total=10; quote=Quick one to kick off: a 400-person retailer we work with cut regretted turnover 18% in two quarters after automating milestone awards, and I wanted to walk you through exactly how they did it.; transcript_id=TT-001
objections: objection=budget; count=4; quote=Honestly, budget is locked until next fiscal year — I can't add a new line item right now.; transcript_id=TT-001 | objection=timing; count=3; quote=This is good, but can we revisit it next quarter? Open enrollment starts in October and we're underwater.; transcript_id=TT-002 | objection=in-house; count=3; quote=We already do recognition with a spreadsheet and quarterly gift cards — why would we change?; transcript_id=TT-004
objection_count: 3
next_step: agreed=7; total=10; rate=0.7
competitors_prospect_raised: Awardco, Kudos
competitor_quotes: Awardco=We're also in late talks with Awardco — their rewards catalog looks bigger than yours.; Kudos=How are you different from Kudos? Our CEO used them at her last company.
rep_raised_competitors_excluded: Workhuman
allowed_new: 13 items — e.g. TT-001, TT-002, TT-003, TT-004
graded checks: objection_count · next_step_rate · competitors · coaching · quote_used

Model's actual answer

show the model's full answer
**First-five-minutes opener (8 of 10 calls):**
"Quick one to kick off: a 400-person retailer we work with cut regretted turnover 18% in two quarters after automating milestone awards, and I wanted to walk you through exactly how they did it."
Exceptions: Deal-403845 led with "I put together a short agenda — security review first, then pricing."; Deal-1E2498 led with "You asked for straight pricing last time, so let's start there."

**Three most common objections and response:**
1. Budget locked (4x, Deals D348E1/547B2B/60C2C2/84DBA6): "Totally fair. Most teams fund this out of turnover savings — that retailer saved about $210k in avoided backfills, which is how their finance team signed off."
2. Revisit next quarter (3x, Deals 5408B0/C61CF7/D9A12F): "Makes sense. What if we scope a 90-day pilot with one department so you have internal data before next quarter's planning?"
3. Already doing recognition with spreadsheet/gift cards (3x, Deals 403845/EDC141/1E2498): "Spreadsheets work until they scale — the difference is automation: milestones fire without HR lifting a finger, and you get analytics on who is being recognized."

**Next-step agreed rate: 7/10 calls (70%).**
Agreed: Deals D348E1, 5408B0, 547B2B, C61CF7, 60C2C2, D9A12F, 1E2498. No next step: Deals 403845 ("I'll leave it with you"), EDC141 ("I need to think about it"), 84DBA6 ("I can't commit to anything today").

**Competitors raised by prospects:**
Awardco (Deal-547B2B): "We're also in late talks with Awardco — their rewards catalog looks bigger than yours."
Kudos (Deal-EDC141): "How are you different from Kudos? Our CEO used them at her last company."

**Coaching notes:**
1. Lead with social proof (the 400-person retailer case, 18% turnover reduction) in every call — it appears in 8 of 10 openers and directly ties to the $210k turnover-savings rebuttal for the budget objection.
2. Tie the "revisit next quarter" objection to a concrete low-risk action (90-day single-department pilot) so the prospect has internal data before their next planning cycle — this converted 3 of 3 occurrences into a next-step agreement.
reporting-analytics
q3-weighted-forecastPASS 1.00

? The Question

Below are the open deals in the default sales pipeline with a close date between 2026-07-01 and 2026-10-15, pulled from the CRM warehouse on 2026-09-05. Stages: DS1 (discovery) through DS5 (contract). Build the Q3 2026 forecast (quarter = 2026-07-01 to 2026-09-30). Weighting rule: 100 percent of COMMIT plus 35 percent of BEST_CASE. PIPELINE deals count zero. Only deals with a close date inside the quarter count. Report: COMMIT total, BEST_CASE total, weighted forecast, count of deals in each category inside the quarter, the deals you excluded for being outside the quarter (count and total amount), and the top 5 BEST_CASE deals by amount inside the quarter. Show arithmetic. Then, in a final '## Data quality' section of no more than four sentences, name the data quality problems in this extract that would make the forecast unreliable if it were run unattended.
cost $0.0016 · 43s · in 50,663 / out 7,938 tok
✓ commit_total_44729✓ best_case_total_203565✓ weighted_115976.75✓ commit_count_7✓ best_case_count_24✓ excluded_count_32✓ excluded_amount_227575✓ top5_aliases✓ flags_blank_owner✓ flags_missing_why_buys

✓ Correct Answer (ground truth)

graded checks: commit_total_44729 · best_case_total_203565 · weighted_115976.75 · commit_count_7 · best_case_count_24 · excluded_count_32 · excluded_amount_227575 · top5_aliases · flags_blank_owner · flags_missing_why_buys

Model's actual answer

show the model's full answer
## Q3 2026 Forecast (2026-07-01 to 2026-09-30)

### Deal counts inside quarter by category
- COMMIT: 7 deals
- BEST_CASE: 24 deals
- PIPELINE: 23 deals (counted as zero per weighting rule)

### Totals
- COMMIT total (100%): $44,729.00
- BEST_CASE total (100%): $203,565.00
- Weighted forecast = $44,729.00 + ($203,565.00 × 0.35) = $44,729.00 + $71,247.75 = **$115,976.75**

### Excluded deals (outside Q3 close date)
- 32 deals totaling $227,575.00

### Top 5 BEST_CASE deals inside Q3 by amount
1. Deal-2D7423 (63189452296): $38,935.00 closed 2026-09-30
2. Deal-25F752 (63223160241): $24,000.00 closed 2026-09-25
3. Deal-E53952 (60524834645): $19,656.00 closed 2026-09-30
4. Deal-5EED42 (61129535583): $16,250.00 closed 2026-09-30
5. Deal-FA32A0 (61390497109): $11,116.00 closed 2026-09-25

---

## Data quality

1. Owner is missing on 85 of 86 deals; the extract provides no accountability assignment, making it impossible to route follow-ups or assess capacity without rejoining to a CRM user table.
2. The `why_buys_chars` field reads `0` on 71 of 86 deals — this is almost certainly null/missing rather than genuinely zero characters, and is indistinguishable from actual zero values, so the field cannot be used for deal-quality scoring.
3. Deal-42326B carries a decimal amount ($2,480.4) while every other amount in the extract is a whole number, suggesting a data entry or currency-rounding error that will skew any sum or average if not caught.
4. The source extract spans close dates from 2026-09-02 through 2026-10-15; if this pull is re-run unattended, deals added after the initial extract or with late close-date updates would silently shift the quarter boundary arithmetic.
aha-moment-2x2PASS 1.00

? The Question

Test the activation hypothesis on the company cohort below: companies with 5+ unique givers (m1_users >= 5) AND 1+ successful redemption (m1_redemptions >= 1) in their first calendar month retain better at 24 months than those with one signal or neither. Every company in this extract is already 25+ months old, so current_status = 'active' means retained at 24 months. Report the full 2x2 (cohort size and 24-month retention rate for: both signals, givers-only, redemption-only, neither), state how many companies were excluded from the denominator and why (if none, say so), name the single signal with the largest retention lift, and state what this does and does not prove.
cost $0.0015 · 58s · in 24,048 / out 13,514 tok
✓ both_66✓ givers_47✓ redemp_31✓ neither_40✓ excluded_0✓ largest_lift_givers

✓ Correct Answer (ground truth)

graded checks: both_66 · givers_47 · redemp_31 · neither_40 · excluded_0 · largest_lift_givers

Model's actual answer

show the model's full answer
ACTIVATION HYPOTHESIS TEST — 2x2 RESULTS

Cohort sizes and 24-month retention rates (current_status = 'active' = retained):

| Cohort | n | Retained | Retention Rate |
|---|---|---|---|
| Both signals (m1_users >= 5 AND m1_redemptions >= 1) | 47 | 31 | 66.0% |
| Givers-only (m1_users >= 5, m1_redemptions = 0) | 49 | 23 | 46.9% |
| Redemption-only (m1_users < 5, m1_redemptions >= 1) | 29 | 9 | 31.0% |
| Neither signal | 95 | 38 | 40.0% |

Excluded from denominator: 0 companies. Every company in the 220-row extract falls into exactly one of the four cells; the classification is exhaustive (m1_users >= 5 or < 5) x (m1_redemptions >= 1 or 0).

Single signal with the largest retention lift: m1_users >= 5 (givers), at +6.9 percentage points vs. the neither-signal baseline (46.9% - 40.0%). The redemption-only signal actually underperforms the baseline (-9.0pp).

What this does prove: Companies exhibiting both activation signals in month 1 have a materially higher 24-month retention rate (66.0%) than companies with neither signal (40.0%), confirming a positive association between early activation and later retention in this cohort.

What this does not prove: Causation. The +26.0pp gap between the both-signals cohort and the neither cohort may be confounded by unobserved factors (company size, industry mix, product tier, seasonality, or engagement quality not captured by these two binary signals). The data is also limited to companies with signup months in the extract (2023-01 through 2023-07), and 'active' status at 25+ months is a proxy for 24-month retention, not a direct measurement of it.
arr-rollforward-reconciliation0.60

? The Question

Reconcile ARR between CRM company records and billing subscriptions as of 2026-09-05. Billing ARR is MRR x 12. Report both totals, the variance, and a decomposition into buckets (status mismatch, rounding, missing records, other) that sums exactly to the variance. List each mismatched account with a suggested owner. Business rule: any subscription with a term other than 12 months must have cf_agreement_end_date populated; list violations.
cost $0.0011 · 35s · in 9,976 / out 9,831 tok
✓ billing_total✓ crm_total✗ variance✗ buckets_sum✓ violations_listed

✓ Correct Answer (ground truth)

as_of: 2026-09-05
billing_arr_total: 604739.28
crm_arr_total: 603581.76
variance: 1157.52
variance_definition: billing_arr_total - crm_arr_total (billing ARR = sum of MRR x 12 over status=active subscriptions)
buckets: {"status_mismatch": {"amount": "-13158.48", "accounts": [{"company_alias": "C-0C8323BF", "billing_arr": "0.00", "hubspot_arr": "4905.24", "diff": "-4905.24", "suggested_owner": "Dana Mercer"}, {"compa
bucket_sum_equals_variance: True
n_mismatched_accounts: 7
rule: term_months != 12 requires cf_agreement_end_date
violations: subscription_id=SUB-0002; company_alias=C-1794A52C; term_months=24 | subscription_id=SUB-0019; company_alias=C-22170CA1; term_months=36
n_companies: 40
graded checks: billing_total · crm_total · variance · buckets_sum · violations_listed

Model's actual answer

show the model's full answer
Here is the reconciliation.

---

ARR RECONCILIATION as of 2026-09-05

Total Billing ARR (sum of MRR x 12): 617,897.76
Total CRM ARR (sum of hubspot_arr):  603,581.76
Variance (CRM - Billing):            -14,316.00

MISMATCHED ACCOUNTS

[CRM Missing] — billing subscription exists but no CRM company record:
  C-21629AA4: billing ARR = 28,449.24 | diff = -28,449.24 | suggested_owner = RevOps

[Billing Missing] — CRM company record exists but no billing subscription:
  C-0D5BBE3A: CRM ARR = 16,497.24 | diff = +16,497.24 | suggested_owner = RevOps

[Rounding] — small diffs (<= $20):
  C-0D66DF9E: billing ARR = 23,184.00, CRM ARR = 23,200.00 | diff = +16.00 | suggested_owner = RevOps
  C-14D70CE0: billing ARR = 18,180.00, CRM ARR = 18,200.00 | diff = +20.00 | suggested_owner = RevOps

[Other] — unexplained larger diffs:
  C-0F7269D7: billing ARR = 26,796.00, CRM ARR = 24,396.00 | diff = -2,400.00 | suggested_owner = RevOps

Note on status mismatches: C-0C8323BF (SUB-000E, cancelled) and C-0DC4FB8C (SUB-000F, cancelled) are cancelled in billing but still carry CRM ARR. Their amounts match exactly (4,905.24 and 8,253.24 respectively), so they contribute $0.00 to variance but should be reviewed for CRM status updates.

VARIANCE DECOMPOSITION

Bucket                      Sum
-------------------------  ---------------
Status mismatch             0.00
Rounding                   36.00
Missing CRM record      -28,449.24
Missing billing record   16,497.24
Other                    -2,400.00
-------------------------  ---------------
Total decomposition       -14,316.00
Variance (CRM - Billing)  -14,316.00
Match: yes

BUSINESS RULE VIOLATIONS

Rule: any subscription with term_months != 12 must have cf_agreement_end_date populated.

  VIOLATION: subscription_id=SUB-0002  company_alias=C-1794A52C  term_months=24  cf_agreement_end_date=EMPTY
  VIOLATION: subscription_id=SUB-0019  company_alias=C-22170CA1  term_months=36  cf_agreement_end_date=EMPTY

All non-12-month subscriptions (for reference):
  SUB-0002  company_alias=C-1794A52C  term=24mo  status=active  cf_agreement_end_date=EMPTY  (VIOLATION)
  SUB-000C  company_alias=C-0DB48281  term=24mo  status=active  cf_agreement_end_date=2027-11-30
  SUB-0019  company_alias=C-22170CA1  term=36mo  status=active  cf_agreement_end_date=EMPTY  (VIOLATION)
  SUB-001A  company_alias=C-0FC4DBB8  term=36mo  status=active  cf_agreement_end_date=2027-11-30
kvm-month-over-month0.80

? The Question

Report the four core KVMs for 2026-08 against 2026-07: giving rate, redemptions per user, 1:1 meetings engagement, pulse check engagement. For each: value, prior value, absolute and relative change, direction. Name the KVM with the largest relative move and the segment (plan_tier or size_band) driving it if the data supports that; otherwise say the data does not. Under 200 words plus one table.
cost $0.0011 · 205s · in 10,062 / out 5,878 tok
✗ four_kvms✓ biggest_mover_named✓ direction_correct✓ segment_named✓ word_cap

✓ Correct Answer (ground truth)

aggregation: simple mean of company-level values per month
kvms: {"giving_rate": {"2026-07": 0.6023, "2026-08": 0.6027, "abs_change": 0.0004, "rel_change_pct": 0.07, "direction": "up"}, "redemptions_per_user": {"2026-07": 1.73, "2026-08": 1.7302, "abs_change": 0.00
biggest_mover: pulse_engagement
driving_segment: size_band=enterprise; 2026-07=0.55; 2026-08=0.2743; rel_change_pct=-50.13
segment_detail: {"smb": {"2026-07": 0.6588, "2026-08": 0.6573, "rel_change_pct": -0.23}, "mid_market": {"2026-07": 0.593, "2026-08": 0.5942, "rel_change_pct": 0.2}, "enterprise": {"2026-07": 0.55, "2026-08": 0.2743,
n_company_rows_per_month: 30
graded checks: four_kvms · biggest_mover_named · direction_correct · segment_named · word_cap

Model's actual answer

show the model's full answer
Pulse engagement is the KVM with the largest relative move (−15.31%). The data supports size_band as the driver: enterprise accounts for the entire drop (0.5500 → 0.2743, −50.1%), while smb and mid_market are flat. Plan_tier does not differentiate (all tier_three).

| KVM | 2026-08 | 2026-07 | Δ Abs | Δ Rel | Dir |
|---|---|---|---|---|---|
| giving_rate | 0.6027 | 0.6023 | +0.0004 | +0.07% | ↑ |
| redemptions_per_user | 1.7302 | 1.7300 | +0.0002 | +0.01% | ↑ |
| 1:1 meetings engagement | 0.4472 | 0.4469 | +0.0003 | +0.06% | ↑ |
| pulse check engagement | 0.5086 | 0.6006 | −0.0920 | −15.31% | ↓ |

Largest relative move: **pulse_engagement** (−15.31%), driven by **size_band = enterprise** (the only segment with a material shift; plan_tier does not differentiate).
rewards-ytd-sectionPASS 1.00

? The Question

Produce the redemption section of the weekly rewards report through the last completed month (2026-08): redemption count, spend, unique redeemers, redemptions per redeemer, provider mix as percent of spend, top 5 countries by redemptions. Say which month is the last completed month. Provider shares must sum to 100.
cost $0.0029 · 81s · in 44,784 / out 24,028 tok
✓ count✓ spend✓ unique_redeemers✓ month_named✓ provider_mix✓ top_country

✓ Correct Answer (ground truth)

redemption_count: 378
total_spend_usd: 27846.00
unique_redeemers: 235
redemptions_per_redeemer: 1.61
provider_mix_pct_of_spend: TangoCard=11.6; Tremendous=30.54; custom=39.05; Snappy=18.81
top5_countries_by_count: country=US; count=244 | country=CA; count=24 | country=AU; count=21 | country=GB; count=17 | country=NL; count=17
graded checks: count · spend · unique_redeemers · month_named · provider_mix · top_country

Model's actual answer

show the model's full answer
REDEMPTION SECTION — WEEKLY REWARDS REPORT
Period: through 2026-08 (last completed month)

REDEMPTION COUNT: 378
SPEND: $27,846.00
UNIQUE REDEEMERS: 235
REDICTIONS PER REDEEMER: 1.61 (378 ÷ 235)

PROVIDER MIX (% of spend, sums to 100.0%):
  custom: 39.0% ($10,873.00)
  Tremendous: 30.5% ($8,505.00)
  Snappy: 18.8% ($5,238.00)
  TangoCard: 11.6% ($3,230.00)

TOP 5 COUNTRIES BY REDEMPTIONS:
  1. US — 244
  2. CA — 24
  3. AU — 21
  4. GB — 17
  5. NL — 17
customer-success
churn-save-eligibility0.80

? The Question

Which at-risk accounts qualify for a churn-save offer under the documented eligibility rules, what amount is at stake per account and in total, and which play fits each (usage revival, executive touch, commercial concession)? Cite the signal that justifies each play. List accounts that look at risk but do not qualify and why.
cost $0.0005 · 21s · in 1,794 / out 5,598 tok
✓ eligible_set✓ total_at_stake✗ plays_cited✓ noneligible_named✓ rules_applied

✓ Correct Answer (ground truth)

snapshot_date: 2026-09-05
rules: health_score < 60, churn_save_eligible_amount > 0, renewal within 120 days of snapshot
eligible: account_alias=C-0F6C0F34; amount_at_stake=49707.00; play=executive touch; justifying_signal=champion_active is false - no executive sponsor engaged | account_alias=C-0B827671; amount_at_stake=25365.00; play=usage revival; justifying_signal=usage_trend_3m=declining over the last 3 months | account_alias=C-0B360C78; amount_at_stake=35748.00; play=commercial concession; justifying_signal=usage stable/growing with seat utilization 75% - risk is commercial, not adoption | account_alias=C-0B0F1BAB; amount_at_stake=5494.00; play=executive touch; justifying_signal=champion_active is false - no executive sponsor engaged | account_alias=C-0CA21961; amount_at_stake=16829.00; play=usage revival; justifying_signal=seat utilization 26% is below 50% | account_alias=C-0E9C27D1; amount_at_stake=41235.00; play=commercial concession; justifying_signal=usage stable/growing with seat utilization 85% - risk is commercial, not adoption … (+2 more)
total_amount_at_stake: 224601.00
non_eligible_at_risk: account_alias=C-0BC71BDD; health_score=55 | account_alias=C-0BA71F12; health_score=52 | account_alias=C-0F6694C3; health_score=43 | account_alias=C-0BE96399; health_score=54 | account_alias=C-0F876796; health_score=47 | account_alias=C-0FCCD2DF; health_score=43 … (+1 more)
n_accounts: 30
graded checks: eligible_set · total_at_stake · plays_cited · noneligible_named · rules_applied

Model's actual answer

show the model's full answer
Qualifying accounts (all 3 rules: health_score < 60, churn_save_eligible_amount > 0, renewal within 120 days of 2026-09-05):

| Account | ARR | At Stake (eligible) | Play | Signal |
|---|---|---|---|---|
| C-0F6C0F34 | $86,741.00 | $49,707.00 | executive_touch | champion_active=false — no internal advocate to drive renewal |
| C-0B827671 | $72,088.00 | $25,365.00 | usage_revival | usage_trend_3m=declining — disengagement risk |
| C-0B360C78 | $60,427.00 | $35,748.00 | commercial_concession | champion_active=true + usage_trend_3m=growing — has advocacy and usage, needs commercial reason to stay |
| C-0B0F1BAB | $15,391.00 | $5,494.00 | executive_touch | champion_active=false — no internal champion |
| C-0CA21961 | $31,501.00 | $16,829.00 | commercial_concession | champion_active=true + usage_trend_3m=flat — stable usage, needs incentive |
| C-0E9C27D1 | $75,093.00 | $41,235.00 | commercial_concession | champion_active=true + usage_trend_3m=flat — stable usage, needs incentive |
| C-0CEF69FD | $79,324.00 | $32,621.00 | executive_touch | champion_active=false — no internal advocate |
| C-0D3278C7 | $33,815.00 | $17,602.00 | usage_revival | usage_trend_3m=declining — disengagement risk |

**Total at stake: $224,601.00** (sum of churn_save_eligible_amount across 8 qualifying accounts).

Arithmetic: 49707 + 25365 + 35748 + 5494 + 16829 + 41235 + 32621 + 17602 = 224601.

---

Accounts that look at risk (health_score < 60) but do NOT qualify:

| Account | Rule Failed | Reason |
|---|---|---|
| C-0BC71BDD | R2 | churn_save_eligible_amount=0.00 (not > 0) |
| C-0BA71F12 | R3 | renewal_date=2027-04-11 is 218 days from snapshot (exceeds 120) |
| C-0F6694C3 | R2 | churn_save_eligible_amount=0.00 (not > 0) |
| C-0BE96399 | R2 | churn_save_eligible_amount=0.00 (not > 0) |
| C-0F876796 | R3 | renewal_date=2027-02-06 is 154 days from snapshot (exceeds 120) |
| C-0FCCD2DF | R2 | churn_save_eligible_amount=0.00 (not > 0) |
| C-10A56B0F | R2 | churn_save_eligible_amount=0.00 (not > 0) |

All 7 fail either R2 (eligible amount = 0) or R3 (renewal outside 120-day window). No qualifying account fails R1.
expansion-kit0.83

? The Question

A customer replied to outreach (reply included). Build the expansion kit: seat coverage (licensed over headcount), two lines on usage health, headroom in seats and ARR at the current per-seat rate, who replied and whether they can buy, the right buyer from our contacts, and a reply email under 150 words that moves toward the right person without being pushy and cites one real usage fact.
cost $0.0002 · 9s · in 530 / out 1,661 tok
✓ coverage_pct✗ headroom_arr✓ right_buyer✓ replier_not_buyer✓ usage_fact✓ email_under_150

✓ Correct Answer (ground truth)

company_alias: C-0DDFC9A7
coverage_pct: 37.5
headroom_seats: 250
per_seat_annual_rate: 60.00
headroom_arr: 15000.00
replier: Maria S.
replier_can_buy: False
right_buyer: name=Dana R.; title=VP People
usage_facts: active_users_2026_08=126; active_users_2026_03=88; growth_pct_6m=43.2; licensed_seat_utilization_pct=84.0
usage_health_lines: Monthly active users grew from 88 (2026-03) to 126 (2026-08), +43.2% over 6 months., Last month 126 of 150 licensed seats were active (84.0% utilization), approaching the seat cap.
graded checks: coverage_pct · headroom_arr · right_buyer · replier_not_buyer · usage_fact · email_under_150

Model's actual answer

show the model's full answer
Expansion Kit for C-0DDFC9A7

---

SEAT COVERAGE
Licensed seats: 150 / Headcount: 400 → 150 ÷ 400 = 37.5% coverage

USAGE HEALTH
1. MAU grew from 88 (Mar) to 126 (Aug) — a net +38 users (+43.2%) over six months, with a consistent +7-8 users/month cadence.
2. All months show sequential growth with no dips; the trend is linear and accelerating (Mar→Apr +7, Apr→May +7, May→Jun +8, Jun→Jul +8, Jul→Aug +8).

HEADROOM
Current users (Aug): 126
Seat headroom: 150 − 126 = 24 seats
Per-seat rate: $9,000.00 ÷ 150 = $60.00/seat
ARR headroom at current rate: 24 × $60.00 = $1,440.00

WHO REPLIED & CAN THEY BUY?
Maria S. (People Operations Coordinator) replied on 2026-09-02. She explicitly states she is NOT the right person for purchasing decisions — budget and seat expansion sit with Dana R.

RIGHT BUYER
Dana R., VP People (last engaged 2026-05-18)

---

REPLY EMAIL (to Maria, moving toward Dana, under 150 words)

Hi Maria,

Thanks for the update — glad the team is loving Bonusly. I appreciate the introduction to Dana.

One thing worth noting: monthly active users have grown from 88 in March to 126 in August, a 43% increase over six months. That kind of momentum makes a strong case for expanding seats now rather than waiting.

I'd love to connect with Dana when she's ready to discuss options. No rush — just want to make sure we're aligned with where usage is heading.

Best,
Cole
onboarding-statusPASS 1.00

? The Question

Prep the CSM for the mid-onboarding call: what is complete (integrations connected, allowance set, admins added, first recognition given, first redemption), what is not, early engagement signals, and the three things to cover on the call. Mark nothing complete without a data field that shows it.
cost $0.0002 · 14s · in 724 / out 2,046 tok
✓ complete_items✓ incomplete_items✓ three_items✓ grounded

✓ Correct Answer (ground truth)

company_alias: C-0D284E42
days_live: 25
checklist: {"slack_integration_connected": {"complete": true, "field": "integration_slack", "value": "2026-08-12"}, "hris_integration_connected": {"complete": false, "field": "integration_hris", "value": ""}, "a
engagement_signals: avg_active_givers_week1=4.3; avg_active_givers_last_week=13.0; latest_active_givers=15; days_with_activity=25
call_items: Connect the HRIS integration (integration_hris is empty) to automate user provisioning and offboarding., Get to first redemption (first_redemption_at is empty): walk admins through the rewards catalog and confirm allowance funding., Build the giving habit: daily active givers averaged 4.3 in week 1 vs 13.0 last week - agree on manager nudges and a recognition cadence.
graded checks: complete_items · incomplete_items · three_items · grounded

Model's actual answer

show the model's full answer
**Company:** C-0D284E42 | **Signup:** 2026-08-11 | **Days in program:** 25 (Aug 11 – Sep 4, 2026)

---

### What Is Complete (marked with data fields only)

| Milestone | Status | Data field |
|---|---|---|
| Integration Slack connected | Complete | 2026-08-12 |
| Allowance set | Complete | 2026-08-13 |
| Admins added (2) | Complete | admins_added = 2 |
| First recognition given | Complete | 2026-08-15 14:22 |

### What Is NOT Complete

| Milestone | Status | Data field |
|---|---|---|
| Integration HRIS connected | **Missing** | integration_hris = (empty) |
| First redemption | **Missing** | first_redemption_at = (empty) |

---

### Early Engagement Signals

- Active givers grew from 3 (signup day) to 15 (Sep 4) — a **5x increase over 25 days**.
- Four usage spikes worth noting:
  - **Aug 17:** 3 → 7 (+4) — one day after first recognition (Aug 15). Correlation likely.
  - **Aug 21:** 6 → 9 (+3)
  - **Aug 26:** 9 → 11 (+2)
  - **Aug 30:** 11 → 13 (+2)
- Plateau at 9 active givers Aug 23–25 (3 days), then resumed growth.
- 12 of the last 13 days show 9+ active givers — usage has stabilized in the upper range since Aug 23.
- **No redemption has occurred** (first_redemption_at is empty). The reward loop has not yet closed.

---

### Three Things to Cover on the Call

1. **HRIS integration is not connected.** Ask: is the IT/security review pending? This blocks any HR-synced workflows and is the single biggest gap in the onboarding stack.
2. **No redemption yet despite 25 days and 15 active givers with allowance set since Aug 13.** Is the allowance value clear to users? Is the redemption path surfaced in Slack? Need to diagnose why the loop hasn't closed.
3. **Recognize and leverage the growth trajectory.** 3 → 15 in 25 days with the post-recognition spike is a strong signal. The CSM should ask what the team has done to sustain it (e.g., channel naming, launch cadence) and propose tying the next milestone (first 20 active givers) to a second recognition wave to compound the momentum.
renewal-risk-conflicting-datesPASS 1.00

? The Question

Produce a 90-day renewal risk brief. Two systems hold renewal dates and disagree on some accounts; decide which to trust per account and say why (multi-year contracts are known to be wrong in ChurnZero). For every renewal: company, CSM, ARR, date used, seat utilization, 3-month usage trend, risk rating with one sentence of evidence. Flag every disagreement. Close with total ARR renewing and ARR at risk.
cost $0.0165 · 569s · in 219,249 / out 72,146 tok
✓ total_renewing✓ arr_at_risk✓ disagreements_flagged✓ trust_rule

✓ Correct Answer (ground truth)

snapshot_date: 2026-09-05
window: 2026-09-05 to 2026-12-04
trust_rule: multi-year contracts: Chargebee is authoritative (ChurnZero known wrong); otherwise systems agree or Chargebee wins
accounts: 20 items — e.g. account_alias=C-0B144C78; csm=Cole Ingram; arr=30899.00; trusted_renewal_date=2026-11-02; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=75.4; usage_3m_ratio=1.03; risk=low; evidence=3-month usage ratio 1.03 (last3 avg 103 vs prior3 100), seat utilization 75% | account_alias=C-0B20DB64; csm=Dana Mercer; arr=21770.00; trusted_renewal_date=2026-10-07; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=56.6; usage_3m_ratio=1.0; risk=medium; evidence=3-month usage ratio 1.00 (last3 avg 295 vs prior3 295), seat utilization 57% | account_alias=C-0B344485; csm=Elena Sinclair; arr=64384.00; trusted_renewal_date=2026-11-16; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=78.0; usage_3m_ratio=1.04; risk=low; evidence=3-month usage ratio 1.04 (last3 avg 241 vs prior3 231), seat utilization 78% | account_alias=C-0B369871; csm=Dana Mercer; arr=85128.00; trusted_renewal_date=2026-10-29; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=75.1; usage_3m_ratio=1.04; risk=low; evidence=3-month usage ratio 1.04 (last3 avg 330 vs prior3 316), seat utilization 75%
disagreements: account_alias=C-0B7D2C30; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-15; is_multi_year=True; trusted=2026-09-15 | account_alias=C-0BBE3E60; cz_renewal_date=2027-09-26; cb_renewal_date=2026-09-26; is_multi_year=True; trusted=2026-09-26 | account_alias=C-0BCDB8C2; cz_renewal_date=2027-09-18; cb_renewal_date=2026-09-18; is_multi_year=True; trusted=2026-09-18 | account_alias=C-0D2AB865; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-22; is_multi_year=True; trusted=2026-09-22 | account_alias=C-0F5D2323; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-29; is_multi_year=True; trusted=2026-09-29
n_disagreements: 5
total_arr_renewing_90d: 1048715.00
arr_at_risk_90d: 359409.00
n_accounts: 20
graded checks: total_renewing · arr_at_risk · disagreements_flagged · trust_rule

Model's actual answer

show the model's full answer
Here is the 90-day renewal risk brief, written to `/Users/amaniphipps/90_day_renewal_risk_brief.txt`.

---

# 90-Day Renewal Risk Brief

**Generated from session data (2026-09-06). No facts invented.**

## Methodology

- Seat utilization = seats_used / seats (churnzero_renewals.csv)
- 3-month usage trend = Jun/Jul/Aug avg vs Mar/Apr/May avg active_users (usage_12m.csv)
- Date trust rule: Chargebee for multi-year accounts (is_multi_year=true), ChurnZero for non-multi-year. Rationale: multi-year contracts have known-wrong RenewalDate in ChurnZero.
- Risk rating: HIGH = utilization under 40%; MEDIUM = utilization 40-60% with flat/declining trend; LOW = utilization above 60% with stable or growing usage.

## Renewal Details (sorted by renewal date)

| Account | CSM | ARR | Date Used | Utilization | 3-Mo Trend | Risk | Source & Flag |
|---|---|---|---|---|---|---|---|
| C-0B7D2C30 | Dana Mercer | $65,901.00 | 2026-09-15 | 274/476 = 57.6% | -18.2% (Jun avg 92 vs Mar avg 112) | MEDIUM | Chargebee (multi-year; CZ RenewalDate wrong) [DISAGREE] |
| C-0BCDB8C2 | Cole Ingram | $54,427.00 | 2026-09-18 | 232/424 = 54.7% | -17.6% (Jun avg 118 vs Mar avg 144) | MEDIUM | Chargebee (multi-year; CZ RenewalDate wrong) [DISAGREE] |
| C-0D2AB865 | Elena Sinclair | $38,022.00 | 2026-09-22 | 250/407 = 61.4% | -18.9% (Jun avg 117 vs Mar avg 144) | MEDIUM | Chargebee (multi-year; CZ RenewalDate wrong) [DISAGREE] |
| C-0BBE3E60 | Dana Mercer | $30,993.00 | 2026-09-26 | 74/114 = 64.9% | -19.5% (Jun avg 36 vs Mar avg 44) | MEDIUM | Chargebee (multi-year; CZ RenewalDate wrong) [DISAGREE] |
| C-0F5D2323 | Cole Ingram | $90,647.00 | 2026-09-29 | 111/390 = 28.5% | +3.5% (Jun avg 20 vs Mar avg 19) | HIGH | Chargebee (multi-year; CZ RenewalDate wrong) [DISAGREE] |
| C-0EC6999D | Elena Sinclair | $79,419.00 | 2026-10-03 | 31/112 = 27.7% | +6.7% (Jun avg 16 vs Mar avg 15) | HIGH | ChurnZero (non-multi-year) |
| C-0B20DB64 | Dana Mercer | $21,770.00 | 2026-10-07 | 214/378 = 56.6% | +0.1% (Jun avg 295 vs Mar avg 295) | MEDIUM | ChurnZero (non-multi-year) |
| C-0BBC4E7A | Cole Ingram | $56,374.00 | 2026-10-10 | 228/337 = 67.7% | -0.9% (Jun avg 141 vs Mar avg 142) | LOW | ChurnZero (non-multi-year) |
| C-0FD551AB | Elena Sinclair | $48,815.00 | 2026-10-14 | 210/376 = 55.9% | -1.6% (Jun avg 124 vs Mar avg 126) | MEDIUM | ChurnZero (non-multi-year) |
| C-0F9F8F13 | Dana Mercer | $46,230.00 | 2026-10-18 | 199/352 = 56.5% | +0.2% (Jun avg 184 vs Mar avg 184) | MEDIUM | ChurnZero (non-multi-year) |
| C-0BC34584 | Cole Ingram | $16,740.00 | 2026-10-22 | 327/494 = 66.2% | +1.0% (Jun avg 105 vs Mar avg 104) | LOW | ChurnZero (non-multi-year) |
| C-0B7A7546 | Elena Sinclair | $35,062.00 | 2026-10-25 | 182/205 = 88.8% | +4.3% (Jun avg 64 vs Mar avg 61) | LOW | ChurnZero (non-multi-year) |
| C-0B369871 | Dana Mercer | $85,128.00 | 2026-10-29 | 317/422 = 75.1% | +4.3% (Jun avg 330 vs Mar avg 316) | LOW | ChurnZero (non-multi-year) |
| C-0B144C78 | Cole Ingram | $30,899.00 | 2026-11-02 | 169/224 = 75.4% | +3.0% (Jun avg 103 vs Mar avg 100) | LOW | ChurnZero (non-multi-year) |
| C-0FC4DBB8 | Elena Sinclair | $94,732.00 | 2026-11-05 | 356/464 = 76.7% | +4.2% (Jun avg 191 vs Mar avg 183) | LOW | ChurnZero (non-multi-year) |
| C-0D5BBE3A | Dana Mercer | $39,740.00 | 2026-11-09 | 85/102 = 83.3% | +3.9% (Jun avg 90 vs Mar avg 86) | LOW | ChurnZero (non-multi-year) |
| C-0FB9D5AF | Cole Ingram | $63,158.00 | 2026-11-13 | 144/199 = 72.4% | +4.6% (Jun avg 174 vs Mar avg 166) | LOW | ChurnZero (non-multi-year) |
| C-0B344485 | Elena Sinclair | $64,384.00 | 2026-11-16 | 224/287 = 78.0% | +4.2% (Jun avg 241 vs Mar avg 231) | LOW | ChurnZero (non-multi-year) |
| C-0CB2C1B4 | Dana Mercer | $40,628.00 | 2026-11-20 | 386/473 = 81.6% | -2.0% (Jun avg 48 vs Mar avg 49) | LOW | ChurnZero (non-multi-year) |
| C-22170CA1 | Cole Ingram | $45,646.00 | 2026-11-24 | 251/294 = 85.4% | +2.6% (Jun avg 146 vs Mar avg 142) | LOW | ChurnZero (non-multi-year) |

## Date Disagreements (all 20 accounts)

Every account has a CZ RenewalDate that differs from the Chargebee cb_renewal_date. Five are multi-year contracts where CZ dates are known-wrong:

| Account | CZ RenewalDate | Chargebee RenewalDate | Trusted Date | Reason |
|---|---|---|---|---|
| C-0B7D2C30 | 2026-09-10 | 2026-09-15 | 2026-09-15 | Multi-year (36mo); CZ RenewalDate wrong |
| C-0BCDB8C2 | 2027-09-18 | 2026-09-18 | 2026-09-18 | Multi-year (36mo); CZ has 2027 date (1 year off) |
| C-0D2AB865 | 2026-09-10 | 2026-09-22 | 2026-09-22 | Multi-year (24mo); CZ RenewalDate wrong |
| C-0BBE3E60 | 2027-09-26 | 2026-09-26 | 2026-09-26 | Multi-year (24mo); CZ has 2027 date (1 year off) |
| C-0F5D2323 | 2026-09-10 | 2026-09-29 | 2026-09-29 | Multi-year (24mo); CZ RenewalDate wrong |

The remaining 15 non-multi-year accounts all have CZ RenewalDate != CB RenewalDate (they differ from CB but CZ is trusted for non-multi-year). For the 5 multi-year accounts, the CB date is used.

## Totals

- Total ARR renewing in 90-day window: $1,048,715.00
- ARR at risk (HIGH): $170,066.00 (2 accounts: C-0F5D2323 at $90,647.00, C-0EC6999D at $79,419.00)
- ARR at medium risk: $306,158.00 (7 accounts)
- ARR at low risk: $572,491.00 (11 accounts)

## High-Risk Detail

- C-0F5D2323 (Cole Ingram, $90,647.00): 28.5% seat utilization (111/390), +3.5% trend. Multi-year contract, Chargebee date 2026-09-29 used. Risk driven by critically low utilization despite modest growth.
- C-0EC6999D (Elena Sinclair, $79,419.00): 27.7% seat utilization (31/112), +6.7% trend. Non-multi-year, ChurnZero date 2026-10-03 used. Risk driven by critically low utilization despite growth.

## Medium-Risk Accounts (declining utilization)

- C-0B7D2C30 (57.6%, -18.2%), C-0BCDB8C2 (54.7%, -17.6%), C-0D2AB865 (61.4%, -18.9%), C-0BBE3E60 (64.9%, -19.5%) -- all multi-year, all showing steep 3-month declines.
- C-0B20DB64 (56.6%, flat), C-0F9F8F13 (56.5%, flat), C-0FD551AB (55.9%, -1.6%) -- non-multi-year with moderate utilization.
ticket-theme-synthesis0.80

? The Question

Synthesize the quarter's support tickets into 4 to 7 themes. Do not trust existing tags; read the text. Per theme: name, count, share, distinct accounts, ARR affected, two ticket ids, one-line recommendation. Separate broad patterns from single-account noise and rank by ARR exposure, not volume.
cost $0.0017 · 46s · in 24,945 / out 12,358 tok
✗ theme_count_range✓ counts_verifiable✓ single_account_flagged✓ arr_ranked✓ ticket_ids

✓ Correct Answer (ground truth)

quarter_window: 2026-06-01 through 2026-08-31
total_tickets: 80
themes: {"points_not_posting": {"count": 20, "share_pct": 25.0, "distinct_accounts": 9, "arr_affected": "31100.00", "example_ticket_ids": ["IC-460001", "IC-460002"], "single_account": false}, "slack_sync": {"
ranking_by_arr_exposure: hris_provisioning, redemption_failures, billing_dispute, points_not_posting, slack_sync
single_account_noise_theme: billing_dispute
note: existing tags are noisy/unreliable; classify by text
sample_ticket_ids: 80 items — e.g. IC-460004, IC-460016, IC-460001, IC-460041
graded checks: theme_count_range · counts_verifiable · single_account_flagged · arr_ranked · ticket_ids

Model's actual answer

show the model's full answer
1. HRIS Provisioning Failures
   Count: 12 (12/79 = 15.2%)
   Distinct accounts: 3 — C-0F6C0F34, C-0B2213A9, C-0DDFC9A7
   ARR affected: $114,000.00 (30,000 + 36,000 + 48,000)
   Tickets: IC-460059, IC-460055
   Recommendation: Escalate HRIS sync reliability — three accounts report provisioning gaps affecting new-hire onboarding; the "log shows no errors" pattern (IC-460062) suggests a silent API or permission failure, not user error.

2. Redemption & Gift Card Failures
   Count: 18 (18/79 = 22.8%)
   Distinct accounts: 7 — C-0F876796, C-0B827671, C-14264ABD, C-0CEF69FD, C-0D9CA315, C-0FCCD2DF, C-0B0F1BAB
   ARR affected: $68,800.00 (8,700 + 10,700 + 11,000 + 8,900 + 9,600 + 9,600 + 10,300)
   Tickets: IC-460025, IC-460030
   Recommendation: Audit the redemption pipeline end-to-end — gift-card deduction without delivery and checkout timeouts indicate a race condition or payment-provider timeout eroding trust across 7 accounts.

3. Billing & Invoice Errors [single-account noise]
   Count: 15 (15/79 = 19.0%)
   Distinct accounts: 1 — C-0E9C27D1
   ARR affected: $52,000.00
   Tickets: IC-460071, IC-460069
   Recommendation: Isolate C-0E9C27D1 for a dedicated billing review — the recurring seat-count discrepancy and wrong-tier renewal point to configuration drift, not a systemic billing bug.

4. Points Not Posting / Missing Points
   Count: 20 (20/79 = 25.3%)
   Distinct accounts: 9 — C-0D3278C7, C-0BE96399, C-0BF20542, C-0D0B047C, C-0D284E42, C-0B2895EF, C-0D6CC8E3, C-21FEBCBB, C-0DD0626C
   ARR affected: $31,100.00 (3,500 + 2,700 + 4,500 + 4,500 + 3,400 + 2,900 + 4,200 + 2,900 + 2,500)
   Tickets: IC-460004, IC-460016
   Recommendation: Investigate the recognition-to-points event pipeline — delivered-but-unrecorded points across 9 accounts suggests a message-queue or event-processing failure, not individual user error.

5. Slack Integration Failures
   Count: 14 (14/79 = 17.7%)
   Distinct accounts: 4 — C-0B843542, C-10A56B0F, C-8C2E8F00, C-0BA71F12
   ARR affected: $18,900.00 (4,400 + 5,400 + 5,200 + 3,900)
   Tickets: IC-460041, IC-460047
   Recommendation: Debug the Slack sync toggle and slash-command reliability — the toggle-reset and auth-not-sticking patterns point to a token-refresh or webhook-registration bug affecting 4 accounts.

---

Arithmetic notes:
- 79 total tickets across 24 distinct accounts. Total sum of all ticket ARR: $1,546,900.00.
- "ARR affected" above is unique-account ARR (each account counted once per theme), not the sum of all ticket rows.
- Themes ranked by unique-account ARR exposure, not ticket volume. The highest-volume theme (Points Not Posting, 25.3%) ranks 4th by ARR.
- Billing & Invoice Errors flagged as single-account noise (C-0E9C27D1 only). All other themes are broad patterns (2+ accounts).
- 2 tickets (IC-460063, IC-460057) had empty body_text in the CSV but their tag and account (C-0B2213A9) confirmed them as HRIS Provisioning Failures; they are included in theme 1.
marketing
case-study-matchPASS 1.00

? The Question

Given the prospect, rank the three most similar existing customers with public case studies for use as social proof. Similarity on industry, size band, use case, region. Cite the fields that drove each match. Do not name any customer without a case study and never include billing data or contact names.
cost $0.0002 · 6s · in 791 / out 1,248 tok
✓ top3✓ decoys_excluded✓ cites_fields✓ no_billing

✓ Correct Answer (ground truth)

top3: company_alias=C-11C31562; similarity_score=3 | company_alias=C-64171065; similarity_score=3 | company_alias=C-A13C193D; similarity_score=3
decoys_excluded: company_alias=C-D6217CAA; similarity_score=4; reason=no_case_study | company_alias=C-C153868F; similarity_score=4; reason=no_case_study
constraints: no_billing_data, no_contact_names, only_customers_with_has_case_study_true_named
prospect_fields_used: industry, size_band, use_case, region
prospect_alias: C-82AF3719
graded checks: top3 · decoys_excluded · cites_fields · no_billing

Model's actual answer

show the model's full answer
1. C-64171065 (Technology, Mid-Market, employee_recognition, NA-East) — 3/4 fields match: industry, size_band, use_case. Region differs (NA-East vs NA-West).
2. C-11C31562 (Manufacturing, Mid-Market, employee_recognition, NA-West) — 3/4 fields match: size_band, use_case, region. Industry differs (Manufacturing vs Technology).
3. C-A13C193D (Technology, Mid-Market, retention, NA-West) — 3/4 fields match: industry, size_band, region. Use_case differs (retention vs employee_recognition).
channel-attribution-mathPASS 1.00

? The Question

Compute channel performance for the trailing 6 months. Paid: spend, SQMs, SQOs, cost per SQM, cost per SQO, SQM-to-SQO rate, pipeline amount, pipeline per dollar. Organic: volume, SQO rate, pipeline. Report a channel with spend and zero SQMs as undefined, not zero. Flag rows where SQO date precedes SQM date. Finish with a reallocation recommendation and your confidence given sample sizes.
cost $0.0006 · 14s · in 7,231 / out 4,020 tok
✓ undefined_channel✓ undefined_not_zero✓ flagged_rows✓ best_channel✓ reallocation

✓ Correct Answer (ground truth)

months: 2026-03, 2026-04, 2026-05, 2026-06, 2026-07, 2026-08
per_channel: {"linkedin_ads": {"spend_usd": 24000, "sqms": 25, "sqos": 8, "pipeline_amount": 96000, "cost_per_sqm": 960.0, "cost_per_sqo": 3000.0, "sqm_to_sqo_rate": 0.32, "pipeline_per_dollar": 4.0}, "organic_sea
undefined_channels: paid_social
flagged_rows_sqo_before_sqm: CT-000041, CT-000044
best_pipeline_per_dollar_channel: paid_search
reallocation_recommendation: from=paid_social; to=paid_search; rationale=paid_social has spend but zero SQMs (cost per SQM undefined, not zero); paid_search has the highest pipeline per dollar.
confidence_note: webinars and referral have small SQM samples (<20); treat rates as directional.
graded checks: undefined_channel · undefined_not_zero · flagged_rows · best_channel · reallocation

Model's actual answer

show the model's full answer
CHANNEL PERFORMANCE — TRAILING 6 MONTHS (MAR–AUG 2026)
======================================================

PAID CHANNELS
-------------

channel       | spend    | SQMs | SQOs | cost/SQM  | cost/SQO  | SQM→SQO | pipeline    | pipeline/$
paid_search   | $36,000  | 40   | 18   | $900.00   | $2,000.00 | 45.00%  | $720,000    | $20.00
linkedin_ads  | $24,000  | 25   | 8    | $960.00   | $3,000.00 | 32.00%  | $96,000     | $4.00
paid_social   | $18,000  | 0    | 0    | UNDEFINED | UNDEFINED | N/A     | $0          | $0.00
webinars      | $9,000   | 12   | 5    | $750.00   | $1,800.00 | 41.67%  | $60,000     | $6.67

FLAGGED ROWS (SQO date precedes SQM date):
- CT-000044: sqm=2026-07-23, sqo=2026-07-18 (linkedin_ads)
- CT-000041: sqm=2026-06-14, sqo=2026-06-09 (linkedin_ads)

ORGANIC CHANNELS
----------------

channel        | volume | SQOs | SQO rate | pipeline
organic_search | 30     | 10   | 33.33%   | $90,000
referral       | 15     | 6    | 40.00%   | $48,000

NOTES
- paid_social has $18,000 in spend but zero contacts in first-touch data. Cost per SQM and cost per SQO are UNDEFINED (not zero) because the denominator is 0. Pipeline per dollar is $0.00.
- paid_search has no contacts with SQO preceding SQM.
- webinars has no SQO-before-SQM flags.
- organic channels have no spend, so cost-per metrics and pipeline-per-dollar are N/A.

REALLOCATION RECOMMENDATION
---------------------------
Shift paid_social's $18,000/month ($108,000 over 6 months) to paid_search, which delivers the highest pipeline-per-dollar at $20.00 and the best SQM-to-SQO rate at 45%. If paid_social's zero SQMs reflect a tracking gap (no UTM capture or misattributed first-touch), fix attribution before reallocating — otherwise the spend is invisible. Also audit the two linkedin_ads rows flagged with SQO preceding SQM (CT-000044, CT-000041) — these may indicate data-entry errors or reverse attribution that inflates linkedin_ads' apparent SQO count.

CONFIDENCE: MEDIUM. The paid_social finding is high-confidence (spend is real, contacts are absent). However, the sample sizes for SQO counts are small (8–18 SQOs per paid channel), and the 2 flagged rows represent 25% of linkedin_ads' SQOs — enough to potentially distort its 32% SQM-to-SQO rate. Organic channel volumes are based on 122 total contacts with no spend baseline, so pipeline-per-dollar comparisons between paid and organic are not apples-to-apples.
competitor-battlecard-update0.40

? The Question

Update the battlecard for the competitor. Sections: one-line positioning, pricing with source and date (newer source wins, note the conflict), where they win, where we win, objections and responses, recent changes, our 12-month win/loss record against them. Cite a snippet id for every factual claim. Rep opinion on a call is not a fact about the competitor. Anything from the old card you cannot re-source gets marked unverified.
cost $0.0004 · 66s · in 1,718 / out 4,340 tok
✓ competitor_named✓ pricing✗ win_loss✗ unverified_marked✗ citations

✓ Correct Answer (ground truth)

competitor: Rivally
pricing: value_per_user_per_month=7; source_snippet=S17; source_date=2026-08-12
win_loss_12mo: wins=13; losses=7
unverified_old_claims: Rivally lacks a Slack integration., Rivally was acquired by WorkHuman in 2025.
rep_opinion_snippet_ids_not_facts: S09, S21
citation_ids_available: 25 items — e.g. S01, S02, S03, S04
graded checks: competitor_named · pricing · win_loss · unverified_marked · citations

Model's actual answer

show the model's full answer
# Updated Battlecard: Rivally

**Positioning:** Points-based recognition platform targeting mid-market [S02]. Notes limited analytics. Cause/Snap brand awareness ~93.

**Pricing:**
- Starts at $7 per user/month, annual billing (Recognition Starter tier, as of 2026-08-12) [S17]. — Conflict exists: two older pages (S03, 2026-01-20 and S08,  Sims could be wrong. S03 $5 and S08 $5 are older price. S17 the $7 price wins. S13 also shows $6.50 quote — likely a discounted deal. The list price is @7 per user/mo now.

**Where Rivally Wins:**
- Setup speed: setup under a week, SLACK integration OOB [S04]
- EU distributed teams, multi-language support praised by EU enterprise reviewer [S10, &(S12). Or the 2 valid and 3 in circl). Wait. Wait S — gets this. Pricing and all — Decide: per-user/min

**Where competing where男主中文 wants pagination off. Want%%%%%%%
nurture-sequence-diagnosisPASS 1.00

? The Question

Review the sequences. Per sequence: sent, open rate, reply rate, meeting rate, weakest step. Identify tracking errors (opened above sent), audience overlap between sequences, and the failure mode for anything under 2 percent reply. One change per weak sequence and which to fix first. Under 350 words.
cost $0.0008 · 15s · in 17,351 / out 4,040 tok
✓ tracking_error✓ overlap_pair✓ weakest_step✓ fix_priority

✓ Correct Answer (ground truth)

per_sequence: {"New Logo Nurture": {"sent": 1386, "opened": 490, "replied": 90, "meetings": 27, "open_rate": 0.3535, "reply_rate": 0.0649, "meeting_rate": 0.0195, "weakest_step": 3}, "Expansion Nurture": {"sent": 8
tracking_error: sequence=Expansion Nurture; step=2; sent=300; opened=340; issue=opened exceeds sent
audience_overlap_pair: Cold Outbound - HR Leaders, Cold Outbound - People Ops
all_pairwise_overlaps: ['Cold Outbound - HR Leaders', 'Cold Outbound - People Ops'], ['Expansion Nurture', 'New Logo Nurture']
under_2pct_reply: Cold Outbound - HR Leaders
failure_mode: Cold Outbound - HR Leaders=opens healthy (~37%) but replies near zero from step 1 -> body/CTA not landing, not a deliverability or subject-line problem
fix_priority_first: Cold Outbound - HR Leaders
graded checks: tracking_error · overlap_pair · weakest_step · fix_priority

Model's actual answer

show the model's full answer
**Sequences Review**

**Per-sequence metrics (step-level):**

| Sequence | Step | Sent | Open Rate | Reply Rate | Meeting Rate |
|---|---|---|---|---|---|
| New Logo Nurture | 1 | 500 | 42.0% | 8.4% | 2.4% |
| New Logo Nurture | 2 | 458 | 34.9% | 6.6% | 2.0% |
| New Logo Nurture | 3 | 428 | 28.0% | 4.2% | 1.4% |
| Expansion Nurture | 1 | 300 | 43.3% | 7.3% | 1.7% |
| Expansion Nurture | 2 | 300 | 113.3% | 8.3% | 1.3% |
| Expansion Nurture | 3 | 275 | 34.5% | 4.4% | 1.1% |
| Cold Outbound - HR Leaders | 1 | 600 | 40.0% | 0.8% | 0.0% |
| Cold Outbound - HR Leaders | 2 | 595 | 29.4% | 0.3% | 0.0% |
| Cold Outbound - HR Leaders | 3 | 590 | 22.0% | 0.2% | 0.0% |
| Cold Outbound - People Ops | 1 | 400 | 37.5% | 3.5% | 0.8% |
| Cold Outbound - People Ops | 2 | 386 | 28.5% | 2.3% | 0.5% |
| Cold Outbound - People Ops | 3 | 377 | 21.2% | 1.6% | 0.3% |

**Weakest step per sequence:** New Logo Nurture Step 3 (4.2% reply), Expansion Nurture Step 3 (4.4% reply), Cold Outbound - HR Leaders Step 3 (0.2% reply), Cold Outbound - People Ops Step 3 (1.6% reply).

**Tracking error:** Expansion Nurture Step 2 — opened (340) exceeds sent (300). Data integrity issue; likely a misconfigured UTM or duplicate-counting bug.

**Audience overlap:** CT-000034 appears twice in audiences.csv, but both rows map to New Logo Nurture only. No cross-sequence overlap found.

**Under 2% reply failure modes:**
- Cold Outbound - HR Leaders: all three steps (0.8%, 0.3%, 0.2%). Messaging does not resonate with HR Leader persona — no meetings booked.
- Cold Outbound - People Ops Step 3: 1.6% reply, declining from Step 2 (2.3%).

**One change per weak sequence and fix priority:**
1. Cold Outbound - HR Leaders (fix first): rewrite subject line and opening hook — the 0% meeting rate across all steps signals complete persona mismatch or value-proposition failure.
2. Expansion Nurture: fix the Step 2 tracking error (opened > sent) before Step 3 metrics can be trusted.
3. Cold Outbound - People Ops Step 3: shorten the email cadence or change the CTA — the drop from 2.3% to 1.6% reply suggests fatigue.
4. New Logo Nurture Step 3: introduce a content upgrade or social proof element to arrest the decline from 8.4% to 4.2% reply.
weekly-marketing-goals-rollupPASS 1.00

? The Question

Produce the weekly marketing goals update for the active quarter: SQMs, SQOs, DS2s, closed-lost MIA rate, same-quarter close count, active pipeline coverage against target. For each: QTD actual, target, delta, pace (on/behind/ahead given days elapsed). One paragraph on what moved this week.
cost $0.0002 · 10s · in 373 / out 2,273 tok
✓ metrics_present✓ deltas✓ coverage_ratio✓ pace✓ paragraph

✓ Correct Answer (ground truth)

days_elapsed: 66
days_in_quarter: 92
elapsed_fraction: 0.7174
metrics: {"SQMs": {"qtd_actual": 230, "target": 300, "delta": -70, "expected_at_day_elapsed": 215.22, "pace": "ahead"}, "SQOs": {"qtd_actual": 84, "target": 120, "delta": -36, "expected_at_day_elapsed": 86.09,
coverage_ratio_active_pipeline: 0.75
pace_rule: expected = target * days_elapsed/days_in_quarter; within +/-5% of expected = on, above = ahead, below = behind; lower_better metrics invert.
this_week_paragraph_required: True
graded checks: metrics_present · deltas · coverage_ratio · pace · paragraph

Model's actual answer

show the model's full answer
Weekly Marketing Goals Update — Q3-2026 (66 of 92 days elapsed)

---

METRIC | QTD ACTUAL | TARGET | DELTA | PACING
SQMs | 230 | 300 | -70 (76.7%) | AHEAD — running 3.49/day vs 3.26/day required; on track to finish ~321
SQOs | 84 | 120 | -36 (70.0%) | BEHIND — running 1.27/day vs 1.30/day required; projected ~117
DS2s | 40 | 75 | -35 (53.3%) | BEHIND — running 0.61/day vs 0.82/day required; projected ~56
closed_lost_mia_rate | 20% (5/25) | 10% | +10pp (200% of target) | BEHIND — MIA rate is 2x the acceptable threshold
same_quarter_closes | 10 | 20 | -10 (50.0%) | BEHIND — running 0.15/day vs 0.22/day required; projected ~14
active_pipeline | $3,000,000 | $4,000,000 | -$1,000,000 (75.0%) | AHEAD — running $45,455/day vs $43,478/day required; projected ~$4.18M

---

What moved this week: Weekly breakdown is not available in the provided data (only QTD totals are supplied), so week-over-week movement cannot be isolated. From the QTD picture, SQMs and active_pipeline are the only two metrics pacing ahead of target; SQOs, DS2s, closed_lost_mia_rate, and same_quarter_closes are all behind. The MIA rate (20%) is the most acute gap — 2x the 10% target — and the DS2 shortfall (40 vs 75, 53.3%) is the largest absolute gap in pipeline generation. Same-quarter closes at 10 vs 20 target means the quarter is halfway through and only half the closes needed have materialized.
communication
ceo-slack-compressionPASS 1.00

? The Question

Compress the forecast analysis below into a single Slack message for the CEO. Rules: at most three sentences; under 80 words; lead with the decision or the so-what; keep every distinct number that matters and the one caveat that changes the conclusion; cut any sentence that does not carry a claim, a number, or a directive. No em dashes, no throat-clearing phrases.
cost $0.0002 · 8s · in 746 / out 1,787 tok
✓ max_3_sentences✓ under_80_words✓ weighted_present✓ caveat_present✓ no_em_dash

✓ Correct Answer (ground truth)

graded checks: max_3_sentences · under_80_words · weighted_present · caveat_present · no_em_dash

Model's actual answer

show the model's full answer
Q3 weighted forecast is $115,976.75 (7 COMMIT deals at 44,729 + 35% of 24 BEST_CASE at 203,565). 32 deals totaling 227,575 are excluded for post-quarter close dates, including one COMMIT (Deal-D348E1, 13,770) and 9 BEST_CASE (28,240). Caveat: owner is blank on 85 of 86 deals and why-buys is empty on all 7 COMMIT deals, so the forecast cannot be rolled up by rep or validated against documented buyer rationale.
stalled-deal-followup-emailPASS 1.00

? The Question

Write a follow-up email for this stalled deal. Under 120 words. Reference one specific prior touch from the engagement history, make one clear ask, and match the tone of the prior thread. No invented details.
cost $0.0001 · 4s · in 513 / out 340 tok
✓ word_cap✓ references_touch✓ single_ask

✓ Correct Answer (ground truth)

deal_alias: Deal-0D2F7A
amount: 5100
stage: DS3
owner: Alex Franklin
days_since_last_engagement: 31
word_cap: 120
must_reference: engagement_id=ENG-00002; date=2026-07-28; type=meeting; why=most substantive prior touch; concrete prospect reactions to milestone awards and points catalog
single_cta: True
tone: warm, concise, professional (matches prior thread)
allowed_facts: 14 items — e.g. Deal-0D2F7A, $5,100, DS3, Alex Franklin
allowed_new: ENG-00001, ENG-00002, ENG-00003, ENG-00004, ENG-00005
graded checks: word_cap · references_touch · single_ask

Model's actual answer

show the model's full answer
Subject: Pricing for 150 Seats — Deal-0D2F7A Follow-Up

Hi Alex,

Following up on the email I sent on August 5 with the pricing recap for 150 seats (ENG-00001). I wanted to check in and see if you had a chance to review it.

Could you let me know if the pricing works for your team, or if there are any adjustments you'd like to discuss?

Thanks,
Alex Franklin
funnel-digest-paragraph0.67

? The Question

Write the weekly GTM digest: one paragraph per team (Marketing, Sales, CS, PLG) in Bonusly brand voice. Every number must trace to the extracts. No customer contact names, no employee performance commentary, no compensation. Four paragraphs, under 400 words total.
cost $0.0001 · 6s · in 333 / out 993 tok
✓ four_paragraphs✓ word_cap✗ teams

✓ Correct Answer (ground truth)

required_paragraphs: Marketing, Sales, CS, PLG
allowed_numbers: 16 items — e.g. 2, 3, 9, 08
word_cap: 400
banned: customer contact names, employee performance commentary, compensation
graded checks: four_paragraphs · word_cap · teams

Model's actual answer

show the model's full answer
SQMs hit 46 against a 52-target week, yielding 88% attainment. The webinar channel contributed 18 of those SQMs. Paid search cost per SQM held at $150.

The team converted 14 SQOs and set 9 DS2 meetings. New pipeline created was $310,000. Same-quarter close count for the week was 3.

Renewals saved this week came to 2. Team NPS moved to 61. Three open red-flag accounts head into next week.

PLG added 412 new signups with activation at 31%. 38 companies reached the aha moment of 10 recognition gives.
partner-digest-section0.67

? The Question

Write the partner digest for the period. Every partner in the roster appears once: active plays get status with numbers, quiet partners get one line. Partner-sourced pipeline count and amount must match the deal data. Nothing omitted, nothing added.
cost $0.0001 · 5s · in 555 / out 710 tok
✓ all_partners✗ pipeline_amount✓ quiet_one_line

✓ Correct Answer (ground truth)

roster: partner_name=Apex Rewards Co; status=active | partner_name=HRCloud Partners; status=active | partner_name=CultureBridge; status=active | partner_name=WorkWell Group; status=active | partner_name=Recogniq; status=quiet | partner_name=KudosWave; status=quiet … (+2 more)
per_partner_pipeline: {"Apex Rewards Co": {"count": 2, "amount": 275000}, "HRCloud Partners": {"count": 1, "amount": 140000}, "CultureBridge": {"count": 2, "amount": 135000}}
partner_sourced_pipeline: count=5; amount=550000
quiet_partners: Recogniq, KudosWave, PeopleFirst Advisors, TotalPerk
active_no_deals: WorkWell Group
rule: every roster partner named exactly once; quiet partners get one line; no partners outside roster.
graded checks: all_partners · pipeline_amount · quiet_one_line

Model's actual answer

show the model's full answer
Partner Digest — 2026-08-24 to 2026-09-04

Active Plays:

Apex Rewards Co — 2 sourced opps, $275,000 pipeline (Deal-DDAAF2: $180,000; Deal-2CF33E: $95,000). Co-webinar locked for 09-15. Both opps in DS1, UTM Source = Partner.

HRCloud Partners — 1 sourced opp, $140,000 pipeline (Deal-F1CDA5). Integration referral closed security review; opp moved to DS2.

CultureBridge — 2 sourced opps, $135,000 pipeline (Deal-096E1D: $60,000; Deal-067213: $75,000). Lunch-and-learn produced both opps; early stage.

WorkWell Group — 0 sourced opps, $0 pipeline. Planning call booked for 09-09; joint playbook restart targeted for Q4.

Quiet Partners:

Recogniq — no activity this period.
KudosWave — no activity this period.
PeopleFirst Advisors — no activity this period.
TotalPerk — no activity this period.
rfp-answer-from-policyPASS 1.00

? The Question

Answer the 10 questionnaire items from the policy excerpts only. Cite the excerpt id for every answer. Where the excerpts do not answer the question, mark it unanswerable and say what document would; do not guess. Tone: vendor response, direct.
cost $0.0001 · 4s · in 529 / out 745 tok
✓ answered_count✓ unanswerable_flagged✓ citations✓ no_guess

✓ Correct Answer (ground truth)

answered_count: 7
answer_map: {"Q1": ["P1"], "Q2": ["P1"], "Q3": ["P2"], "Q4": ["P3"], "Q5": ["P4"], "Q6": ["P5"], "Q7": ["P6"]}
unanswerable: Q8, Q9, Q10
valid_citation_ids: P1, P2, P3, P4, P5, P6, P7, P8
rule: unanswerable items must be flagged and the document that would answer them named; do not guess.
graded checks: answered_count · unanswerable_flagged · citations · no_guess

Model's actual answer

show the model's full answer
Q1: Yes. Customer data is encrypted at rest using AES-256. [P1]

Q2: Yes. Data in transit is encrypted with TLS 1.2 or higher. [P1]

Q3: Yes. Bonusly maintains a SOC 2 Type II report, renewed annually. The report is available to prospects under NDA. [P2]

Q4: Customer data is retained for the duration of the contract plus 90 days, after which it is deleted from production systems. [P3]

Q5: The current subprocessor list is published at trust.bonusly.com. Customers receive 30 days notice before a new subprocessor is added. [P4]

Q6: Yes, SAML 2.0 single sign-on is available on the Enterprise plan. [P5]

Q7: Production databases are backed up daily. Backups are encrypted and retained for 35 days. [P6]

Q8: Unanswerable. The policy excerpts do not mention ISO 27001. A certification evidence document or the full trust center would be needed to answer this.

Q9: Unanswerable. The policy excerpts do not contain a contractual uptime SLA percentage. The service-level agreement (SLA) document or contract template would be needed to answer this.

Q10: Unanswerable. The policy excerpts do not address HIPAA Business Associate Agreements. A BAA template or the legal/compliance team would be needed to answer this.
ops-maintenance
skill-file-reconciliation0.86

? The Question

Reconcile the skill set below against its manifest. Find and report (with severity CRITICAL/WARNING/INFO and an action type TRIM_DESC, UPDATE_BODY, DELETE_SKILL, MERGE, or REVIEW): (1) any two skills whose ALWAYS-trigger phrases overlap/duplicate, (2) any circular delegation chain between skills (name it), (3) any delegation target that does not exist (dangling, name it), (4) any version conflict and which skill should survive, (5) how many manifest descriptions exceed 1,024 characters, (6) any hardcoded page ids, dates, or person names in skill bodies, (7) manifest drift in BOTH directions (files with no manifest row, and manifest rows with no file). Make one proposal per finding; rewrite nothing.
cost $0.0020 · 36s · in 71,183 / out 7,434 tok
✗ duplicate_trigger_found✓ circular_chain✓ dangling_target✓ oversized_desc_count✓ hardcoded_values✓ manifest_drift✓ severity_and_action

✓ Correct Answer (ground truth)

graded checks: duplicate_trigger_found · circular_chain · dangling_target · oversized_desc_count · hardcoded_values · manifest_drift · severity_and_action

Model's actual answer

show the model's full answer
1. [WARNING] [TRIM_DESC] pipeline-intelligence-report and weekly-pipeline-report both use "pipeline update" as an ALWAYS-trigger phrase. pipeline-intelligence-report lists "pipeline update" in its trigger set; weekly-pipeline-report lists "run the pipeline update". Trim the overlapping phrase from one to prevent ambiguous routing.

2. [CRITICAL] [REVIEW] Circular delegation chain: deal-strategy-coach → email-drafter → deal-strategy-coach. deal-strategy-coach delegates manager-to-prospect email drafting to email-drafter; email-drafter delegates deal strategy/coaching back to deal-strategy-coach. No exit condition — a user asking for deal coaching that involves email drafting would loop indefinitely between the two skills.

3. [CRITICAL] [REVIEW] Dangling delegation target: prospect-research-multithreading. Referenced in deal-strategy-coach (§Cross-skill handoff), comms-drafter (Step 2 — Partner communications, If the Recipient Is Unknown), and email-drafter (If the recipient is unknown). No manifest row and no corresponding SKILL.md file exists. Three skills delegate to a target that does not exist.

4. [WARNING] [UPDATE_BODY] Version conflict in analysis-validator: changelog lists both v3.5 (May 9, 2026) and v3.6 (May 9, 2026) with the same date. Per the reconciliation rule, v3.6 survives. Remove or supersede the v3.5 changelog entry so the body reflects a single unambiguous version.

5. [INFO] [TRIM_DESC] Zero manifest descriptions exceed 1,024 characters. The maximum is 1,006 (pipeline-intelligence-report and signalforge-claim-compressor). No trim required, but both are within 2% of the threshold — monitor on next reconciliation.

6. [WARNING] [UPDATE_BODY] Hardcoded anchors in skill bodies: (a) Page IDs — pipeline-intelligence-report embeds Confluence page 2257879045 (AE Excellence Playbook), partner-digest embeds folder 2286616609 and page IDs 2286321666/2236940297/2237825028/2239365136/2238283777, signalforge-feedback embeds page 2295136266 and space ID 2232811524, sales-forecast embeds space ID 2232811524 and page 2232582148; (b) Person names — analysis-validator lists the full GTM roster (16 names with HubSpot IDs), deal-strategy-coach names Alaina Loori/Manish/Amani Phipps, weekly-pipeline-report names Ben Lavin; (c) Dates — multiple skills hardcode dates like "May 9, 2026", "May 4, 2026", "April 26, 2026", "March 28, 2023", "2026-05-19". Enforce [LIVE QUERY] tags on all updateable values.

7. [WARNING] [MERGE] Manifest drift — 3 referenced skills have no manifest row: bonusly-brand (referenced in comms-drafter Step 0 and email-drafter Step 0), prospect-research-multithreading (referenced in deal-strategy-coach, comms-drafter, email-drafter), and signalforge-reports org skill (referenced in pipeline-intelligence-report Phase 5 pre-build steps). All 14 manifest rows have corresponding files — drift is only in the "files without manifest row" direction.
runbook-from-slack-threadPASS 1.00

? The Question

Turn the resolved incident thread into a runbook: ordered steps with the exact command or action taken, who ran it, how success was verified, and rollback for each step that changed state. Every step must trace to a message in the thread. Add nothing the thread does not contain; where the thread is unclear, mark the step as needing confirmation.
cost $0.0002 · 6s · in 789 / out 1,244 tok
✓ ordered_steps✓ rollback✓ needs_confirmation✓ sourced

✓ Correct Answer (ground truth)

steps: order=1; msg=M01; actor=Bryce Harmon; action=acknowledge PagerDuty alert, take IC; state_change=False; verified_by=M01 | order=2; msg=M02; actor=Farid Osman; action=check queue depth; command=bundle exec rake sidekiq:queue_depth; state_change=False; verified_by=M02 (48,213 pending) | order=3; msg=M04; actor=Farid Osman; action=pause auto-enqueue via feature flag; command=bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'; state_change=True; rollback=bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'; verified_by=M07 (queue depth falling) | order=4; msg=M05; actor=Elena Sinclair; action=cleared the dead set; state_change=True; needs_confirmation=thread gives no command or method for clearing the dead set | order=5; msg=M06; actor=Bryce Harmon; action=scale reward-worker to 6 replicas; command=kubectl scale deployment/reward-worker --replicas=6; state_change=True; rollback=kubectl scale deployment/reward-worker --replicas=3; verified_by=M07/M08 | order=6; msg=M08; actor=Cole Ingram; action=verify queue drained and error rate baseline; command=bundle exec rake sidekiq:queue_depth; state_change=False; verified_by=M08 (returns 0) … (+2 more)
ambiguous_step: order=4; msg=M05; mark=needs-confirmation
rule: no steps beyond what the thread contains; every step traces to a message id.
graded checks: ordered_steps · rollback · needs_confirmation · sourced

Model's actual answer

show the model's full answer
# Runbook: #incident-2026-08-29-reward-queue

**Trigger:** PagerDuty alert — reward-worker queue depth > 10k
**Resolution:** Queue drained, workers scaled back to baseline, enqueue re-enabled, all metrics at baseline.

---

## Step 1: Acknowledge alert and take IC
- **Message:** M01 (Bryce Harmon, 14:02:10Z)
- **Action:** Acknowledge PagerDuty alert; take incident command.
- **Verification:** Not specified in thread.
- **Rollback:** N/A (no state change).

---

## Step 2: Check queue depth and dead set
- **Message:** M02 (Farid Osman, 14:04:33Z), M03 (Farid Osman, 14:06:02Z)
- **Action:** Run `bundle exec rake sidekiq:queue_depth` and inspect dead set.
- **Command:** `bundle exec rake sidekiq:queue_depth`
- **Expected normal:** < 500 pending jobs
- **Observed:** 48,213 pending jobs; dead set has 112 jobs, all `Redis::TimeoutError` from ~13:58.
- **Verification:** Command output (M02, M03).
- **Rollback:** N/A (read-only).

---

## Step 3: Pause enqueue via feature flag
- **Message:** M04 (Farid Osman, 14:08:45Z)
- **Action:** Disable auto-recognition enqueue to stop the bleed.
- **Command:** `bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'`
- **Verification:** Not explicitly stated in thread; assumed successful per M04.
- **Rollback:** `bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'`

---

## Step 4: Clear the dead set
- **Message:** M05 (Elena Sinclair, 14:15:20Z)
- **Action:** Clear out the dead set from the console.
- **Command:** (exact command not specified in thread — Elena states "I cleared out the dead set" while in the console)
- **Verification:** Not explicitly stated in thread.
- **Rollback:** Needs confirmation — no rollback command provided in thread for dead-set clearance.

---

## Step 5: Scale workers up
- **Message:** M06 (Bryce Harmon, 14:21:07Z)
- **Action:** Increase worker replicas from 3 to 6 to drain the queue faster.
- **Command:** `kubectl scale deployment/reward-worker --replicas=6`
- **Verification:** Not explicitly stated in thread (M07 shows queue dropping, implying scaling took effect).
- **Rollback:** `kubectl scale deployment/reward-worker --replicas=3`

---

## Step 6: Monitor queue drain
- **Message:** M07 (Farid Osman, 14:33:41Z), M08 (Cole Ingram, 14:47:55Z)
- **Action:** Watch queue depth and error rate until both reach baseline.
- **Command:** `bundle exec rake sidekiq:queue_depth`
- **Expected:** 0 pending jobs, error rate at baseline in Datadog.
- **Observed (M07):** 9,400 and falling ~1,200/min.
- **Observed (M08):** 0 pending jobs, error rate back to baseline.
- **Verification:** M08 — Cole Ingram confirms queue depth 0 and Datadog error rate at baseline.
- **Rollback:** N/A (monitoring step).

---

## Step 7: Re-enable enqueue
- **Message:** M09 (Bryce Harmon, 14:49:10Z)
- **Action:** Re-enable auto-recognition enqueue.
- **Command:** `bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'`
- **Verification:** 40 new jobs processed cleanly in the next 3 minutes (M09).
- **Rollback:** `bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'`

---

## Step 8: Scale workers back down
- **Message:** M10 (Bryce Harmon, 14:55:00Z)
- **Action:** Return worker replicas to baseline.
- **Command:** `kubectl scale deployment/reward-worker --replicas=3`
- **Verification:** Queue stable at 0 (M10).
- **Rollback:** `kubectl scale deployment/reward-worker --replicas=6`

---

## Steps needing confirmation
- **Step 4 (dead set clearance):** Exact command used by Elena Sinclair is not in the thread.
- **Step 4 rollback:** No rollback procedure provided in the thread for dead-set clearance.
- **Step 3 verification:** M04 does not explicitly confirm the feature flag disable succeeded; assumed based on subsequent queue drain.
datadog-incident-timeline0.80

? The Question

From the log slice, sequence the failure: first error with timestamp, the cascade in order, the service and job involved, and the Datadog query that would confirm the first error. State what the logs do not show.
cost $0.0002 · 10s · in 2,294 / out 1,497 tok
✗ first_error_ts✓ service_named✓ cascade_ordered✓ query✓ gaps_stated

✓ Correct Answer (ground truth)

first_error: timestamp=2026-09-03T14:01:12Z; service=reward-service; message=Redis::TimeoutError: Connection to Redis at redis-primary:6379 timed out after 5s
cascade_order: service=reward-service; first_error_at=2026-09-03T14:01:12Z | service=sidekiq; first_error_at=2026-09-03T14:01:40Z | service=api-gateway; first_error_at=2026-09-03T14:03:05Z | service=web-app; first_error_at=2026-09-03T14:03:30Z
job_class_involved: RewardGiveJob
confirming_query: service:reward-service status:error "@message:Redis::TimeoutError"
logs_do_not_show: root cause of the Redis timeout, redis-server's own logs, infrastructure metrics (CPU/memory/network), any deploy or config change preceding the incident, who applied mitigation or when (no actor fields in logs)
clean_services: postgres
graded checks: first_error_ts · service_named · cascade_ordered · query · gaps_stated

Model's actual answer

show the model's full answer
Failure sequence from datadog_logs.csv:

1. **First error (14:01:12Z)** — `reward-service` ERROR: Redis::TimeoutError (Connection to Redis at redis-primary:6379 timed out after 5s)
2. **14:01:20Z** — `reward-service` ERROR: retry exhausted for RewardGiveJob
3. **14:01:30Z** — `reward-service` ERROR: retry exhausted for RewardGiveJob
4. **14:01:40Z** — `reward-service` ERROR: retry exhausted for RewardGiveJob
5. **14:01:40Z** — `sidekiq` ERROR: RewardGiveJob failed: Redis::TimeoutError; retrying in 60s
6. **14:02:28Z** — `sidekiq` ERROR: RewardGiveJob failed: Redis::TimeoutError; retrying
7. **14:02:30Z** — `sidekiq` WARN: Queue reward depth above 10,000
8. **14:03:05Z** — `api-gateway` ERROR: 502 upstream timeout calling reward-service /gives
9. **14:03:30Z** — `web-app` ERROR: Give form submission failed: upstream 502 from api-gateway
10. **14:03:31Z–14:06:52Z** — Continued cascading failures across sidekiq, api-gateway, web-app as retries and upstream timeouts propagate
11. **Recovery** — 14:22:10Z `reward-service` reports Redis connection restored; 14:24:45Z `sidekiq` reports queue depth below 500.

**Service and job:** `reward-service` is the origin; the job is `RewardGiveJob` (sidekiq_jobs.csv confirms all RewardGiveJob entries J-00001 through J-00012 failed with Redis::TimeoutError, with `failed_at` timestamps between 14:01:46Z and 14:02:58Z). RecognitionDigestJob entries (J-00013 through J-00016) also failed with Redis::TimeoutError starting at 14:02:36Z, indicating the Redis outage affected multiple job classes.

**Datadog query to confirm the first error:**
`service:reward-service level:error "Redis::TimeoutError" "Connection to Redis at redis-primary:6379" @timestamp:>2026-09-03T14:01:00Z`

**What the logs do not show:** root cause of the Redis connection timeout (e.g., Redis server crash, network partition, Redis-primary deployment or scaling event), Redis server health metrics, any infrastructure or deployment changes in the preceding hour, and whether Redis-primary was under load or misconfigured.
feature-flag-state-summaryPASS 1.00

? The Question

Summarize the state of the listed feature flags: what each controls per the code excerpt, which companies or segments have it on, targeting rules, and any flag with no code reference or no targets. Company counts must match the export. Do not describe a flag that is not in the export.
cost $0.0002 · 6s · in 578 / out 1,145 tok
✓ orphans✓ no_targets✓ counts_match✓ no_invented

✓ Correct Answer (ground truth)

flags: {"recognition_streaks_v2": {"state": "on", "targeting_rules": "segment:beta_companies", "company_count": 42}, "points_budget_guardrails": {"state": "on", "targeting_rules": "all_companies", "company_c
code_referenced: recognition_streaks_v2, points_budget_guardrails, slack_dm_nudges, redeem_flow_redesign, analytics_dashboard_v3, ms_teams_app_v2
orphan_flags_no_code_reference: legacy_give_modal, survey_boosters_q3, paused_offboard_cleanup
flags_with_no_targets: paused_offboard_cleanup
rule: company counts must equal the export; do not describe flags absent from the export.
graded checks: orphans · no_targets · counts_match · no_invented

Model's actual answer

show the model's full answer
## Feature Flag Summary

### Flags with code references

| Flag | State | Controls (per code) | Targeting | Company Count |
|---|---|---|---|---|
| recognition_streaks_v2 | on | `StreakTracker.record(give)` in `app/models/recognition.rb` | segment:beta_companies | 42 |
| points_budget_guardrails | on | `BudgetService.new(company).enforce!(giver, points)` in `app/services/budget_service.rb` | all_companies | 220 |
| slack_dm_nudges | on | `SlackDm.send_nudge(user)` in `app/jobs/nudge_job.rb` | segment:region_na | 87 |
| redeem_flow_redesign | off | `RedeemV2Component` (on) vs `RedeemV1Component` (off) in `app/controllers/redeem_controller.rb` | targeted_list | 12 |
| analytics_dashboard_v3 | on | `AnalyticsV3.new(company)` in `app/controllers/analytics_controller.rb` | segment:tier_three | 65 |
| ms_teams_app_v2 | off | `TeamsAppV2.install(company)` in `app/services/teams_installer.rb` | targeted_list | 9 |

### Flags with no code reference in the excerpt

- **legacy_give_modal** — off, segment:legacy_plan, count 14. No matching `FeatureFlags.enabled?` call in `flag_code.md`.
- **survey_boosters_q3** — on, segment:legacy_plan, count 7. No matching `FeatureFlags.enabled?` call in `flag_code.md`.
- **paused_offboard_cleanup** — off, targeting rules: (empty), count 0. No matching `FeatureFlags.enabled?` call in `flag_code.md`. This flag has no targets and zero companies.

### Targeting rules not covered by segment targeting

- `redeem_flow_redesign` and `ms_teams_app_v2` both use `targeted_list` — the specific list is not enumerated in the export.
- `paused_offboard_cleanup` has no targeting rules at all (empty field) and count 0.
nda-triagePASS 1.00

? The Question

Triage each NDA as GREEN (standard approval), YELLOW (counsel review), or RED (full legal review). Cite the clause that drives each classification by quoting it. Catch embedded non-solicits, non-competes, missing carve-outs, and unusual term or governing law. One line of reasoning per NDA.
cost $0.0001 · 4s · in 589 / out 478 tok
✓ green✓ yellow✓ red✓ non_solicit_caught✓ clause_quoted✓ correct_labels

✓ Correct Answer (ground truth)

classifications: {"NDA-1": {"classification": "GREEN", "driving_reason": "standard mutual NDA: 2-year term, standard carve-outs, Delaware law", "driving_clause_quote": "Carve-outs: information that (a) is or becomes p
non_solicit_in: NDA-2
checks: classification_per_nda, clause_quotes_exist_in_document, non_solicit_caught, carveouts_checked
graded checks: green · yellow · red · non_solicit_caught · clause_quoted · correct_labels

Model's actual answer

show the model's full answer
NDA-1 — GREEN. Standard mutual confidentiality with full carve-outs, reasonable 2-year term, Delaware law, no restrictive covenants; no clause requires escalation.

NDA-2 — YELLOW. Contains an embedded non-solicitation clause ("During the term of this Agreement and for eighteen (18) months thereafter, neither party shall solicit for employment or hire any employee of the other party with whom it came into contact under this Agreement") requiring counsel review before signing.

NDA-3 — RED. Multiple escalators: (1) non-compete ("For a period of three (3) years following the Effective Date, Recipient shall not, directly or indirectly, engage in or provide services to any business that competes with Discloser's business"), (2) unusual governing law (Republic of Ireland, exclusive jurisdiction), (3) one-way obligation favoring [PARTY C], (4) zero confidentiality carve-outs stated.