home · ← back to all models
openai

openai/gpt-5.5

Mean 0.941 · 31/40 perfect tests · $9.06 total · each card: the question → the correct answer (gold) → this model's actual answer

data-crm
stage-hygiene-audit0.67

? The Question

Audit stage hygiene on the open pipeline below. Rules: a deal at DS2 or later must have a substantive why-buys (15+ characters, not a placeholder such as tbd, n/a, see notes). A why_buys value of the form LONG(n) means the real text was present but truncated for display; treat it as present and substantive. Every deal should have a DS1 entry timestamp. A deal whose stage entry timestamps are out of order regressed (an earlier-stage timestamp is later than a later-stage timestamp). Report: percent of DS2+ deals non-compliant on why-buys overall and by owner (use the owners table for names), count of deals with no DS1 timestamp, count that regressed, and three process fixes. State definitions before numbers.
cost $0.3206 · 45s · in 20,096 / out 6,690 tok
✓ noncomp_pct✗ no_ds1_0✗ regressed_0✓ per_owner_breakdown✓ definitions_first✓ three_fixes

✓ Correct Answer (ground truth)

noncomp_pct: 81.5
ds2plus: 124
justin_noncomp: 15
no_ds1: 0
regressed: 0
n_deals: 156
graded checks: noncomp_pct · no_ds1_0 · regressed_0 · per_owner_breakdown · definitions_first · three_fixes

Model's actual answer

show the model's full answer
Definitions:
- DS2+ deals = deals with stage DS2, DS3, DS4, or DS5.
- Why-buys compliant = why_buys has 15+ substantive characters, is not blank, and is not a placeholder such as tbd, n/a, or see notes. LONG(n) counts as present and substantive.
- No DS1 timestamp = t_ds1 is blank.
- Regressed = any earlier-stage timestamp is later than a later-stage timestamp.

Results:
- Overall DS2+ why-buys non-compliance:
  - 101 non-compliant / 124 DS2+ deals = 81.5%

By owner:
- Bryce Harmon:
  - 16 / 19 = 84.2%
- Dana Mercer:
  - 18 / 22 = 81.8%
- Alex Franklin:
  - 47 / 60 = 78.3%
- Cole Ingram:
  - 19 / 22 = 86.4%
- Farid Osman:
  - 1 / 1 = 100.0%

DS1 timestamp hygiene:
- Deals with no DS1 timestamp:
  - 0

Stage-order hygiene:
- Deals that regressed:
  - 0

Three process fixes:
1. Make why_buys required before a deal can enter DS2 or later.
2. Add validation to reject blank or placeholder why_buys values, while allowing LONG(n) as compliant.
3. Add a daily pipeline hygiene report by owner showing DS2+ deals missing substantive why_buys before forecast review.
owner-resolution0.67

? The Question

The deal extract below has an owner name field that is blank on nearly every row and an owner id field that is populated. Using the owners table, resolve each open deal to an owner name. Report: number of deals resolved, deals whose owner id has no match in the owners table (list them), the owner ids that map to archived/deactivated owners (list their names), and the total pipeline amount per resolved owner. Do not guess a name for any unmatched id.
cost $0.2206 · 30s · in 16,400 / out 3,953 tok
✓ resolved_count✗ unmatched_none✗ deactivated_flagged✓ jake_amount✓ justin_amount✓ no_invented

✓ Correct Answer (ground truth)

resolved: 156
jake_amount: 1054144.0
justin_amount: 624310.0
per_owner: Bryce Harmon=1054144.0; Alex Franklin=624310.0; Dana Mercer=341195.0; Cole Ingram=288161.43; Farid Osman=4134.0; Elena Sinclair=2100.0
deactivated:
graded checks: resolved_count · unmatched_none · deactivated_flagged · jake_amount · justin_amount · no_invented

Model's actual answer

show the model's full answer
Number of deals resolved: 156

Deals whose owner id has no match in owners table: none

Owner ids that map to archived/deactivated owners: none

Total pipeline amount per resolved owner:

Bryce Harmon (owner_id 119337721): 35 deals, total = 1,054,144
Arithmetic: 24,000 + 19,656 + 13,500 + 7,000 + 2,520 + 240,000 + 99,000 + 72,000 + 70,000 + 63,600 + 45,000 + 1 + 21,000 + 23,400 + 13,680 + 5,502 + 8,160 + 1 + 11,400 + 1 + 36,000 + 31,500 + 6,000 + 10,800 + 30,275 + 17,400 + 12,600 + 18,000 + 37,440 + 18,828 + 2,880 + 36,000 + 20,880 + 10,920 + 25,200 = 1,054,144

Dana Mercer (owner_id 83155923): 24 deals, total = 341,195
Arithmetic: 11,250 + 10,500 + 9,000 + 9,000 + 5,400 + 4,800 + 4,600 + 1,920 + 15,000 + 4,200 + 18,900 + 27,000 + 43,875 + 20,000 + 60,000 + 8,100 + 16,250 + 3,150 + 5,000 + 2,100 + 23,400 + 5,400 + 7,350 + 25,000 = 341,195

Alex Franklin (owner_id 84342457): 67 deals, total = 624,310
Arithmetic: 14,850 + 13,770 + 11,200 + 9,000 + 6,360 + 5,400 + 3,240 + 2,484 + 1,920 + 1,080 + 7,200 + 19,000 + 2,880 + 1,400 + 4,800 + 1,632 + 10,000 + 9,300 + 2,700 + 2,160 + 1,800 + 3,600 + 3,840 + 15,000 + 1,968 + 4,000 + 3,600 + 4,800 + 3,120 + 2,520 + 9,000 + 2,400 + 62,000 + 5,400 + 5,100 + 16,700 + 4,400 + 1,620 + 2,600 + 7,200 + 18,000 + 17,000 + 8,316 + 8,100 + 18,000 + 12,600 + 24,000 + 15,000 + 9,000 + 7,200 + 3,780 + 16,200 + 7,200 + 4,680 + 1,800 + 18,000 + 2,730 + 2,400 + 3,060 + 18,000 + 12,000 + 1,800 + 4,400 + 31,200 + 7,200 + 1,600 + 60,000 = 624,310

Cole Ingram (owner_id 83155924): 22 deals, total = 288,161.43
Arithmetic: 2,760 + 1,330 + 7,225.4 + 3,360 + 5,616 + 2,700 + 7,781.2 + 6,947.5 + 40,000 + 12,168 + 2,480.4 + 11,116 + 11,193 + 1,875 + 31,750 + 32,175 + 4,779.88 + 3,334.8 + 58,529.25 + 4,140 + 18,000 + 18,900 = 288,161.43

Farid Osman (owner_id 716654662): 7 deals, total = 4,134
Arithmetic: 1,249 + 1 + 1 + 1 + 2,880 + 1 + 1 = 4,134

Elena Sinclair (owner_id 701163055): 1 deal, total = 2,100
Arithmetic: 2,100 = 2,100
gong-hubspot-join-integrityPASS 1.00

? The Question

Check the integrity of the call-to-deal mapping below. The gong table lists calls in the last 90 days keyed to HubSpot deal ids; deals_open lists the current open default-pipeline deals. Report: (a) count of gong rows whose deal id has NO match in deals_open (orphans; list a few sample aliases), (b) duplicate conversation keys (rows where calls_90d exceeds distinct_conversation_keys), (c) among the open deals that ARE at stage DS3 or later, the share that have at least one logged call. Give sample aliases for the orphan finding, all taken from the data. Note: gong_owner_id is a Gong user id, not a HubSpot owner id.
cost $0.3056 · 38s · in 21,376 / out 5,925 tok
✓ orphan_count✓ dup_keys_0✓ ds3_coverage✓ sample_ids_present

✓ Correct Answer (ground truth)

orphans: 30
orphan_aliases: Deal-8FA85D, Deal-8FC3F9, Deal-3B7945, Deal-42B265, Deal-9CCC42, Deal-36EA09
dup: 0
ds3_cov_pct: 29.4
graded checks: orphan_count · dup_keys_0 · ds3_coverage · sample_ids_present

Model's actual answer

show the model's full answer
(a) Orphan gong rows with no matching deal_id in deals_open: 30

Arithmetic:
67 gong rows - 37 gong rows matched to deals_open = 30 orphan rows

Sample orphan aliases from gong_calls_by_deal_90d.csv:
Deal-8FA85D
Deal-8FC3F9
Deal-3B7945
Deal-42B265
Deal-9CCC42
Deal-36EA09
Deal-9A43B4
Deal-605F3C
Deal-E2D34B
Deal-76821A

(b) Duplicate conversation keys: 0 rows

Arithmetic:
Rows where calls_90d > distinct_conversation_keys = 0
All provided gong rows have calls_90d equal to distinct_conversation_keys.

(c) Open deals at DS3 or later with at least one logged call: 25 of 85 = 29.4%

Arithmetic:
Open DS3+ deals = 85
Open DS3+ deals with deal_id present in gong_calls_by_deal_90d.csv = 25
Share = 25 / 85 = 0.294117647 = 29.4%
snowflake-sql-trapped-catalog0.67

? The Question

Using ONLY the catalog provided, write one Snowflake SQL query returning, per customer company, for its first calendar month as a customer: unique givers, recognition count, successful redemption count. Read the catalog notes carefully: some tables are stale or unpopulated and must be avoided, and one documented filter (excluding deleted givers) must NOT be applied to historical giving counts. Put a comment above each FROM or JOIN explaining the table choice. Return only fenced SQL.
cost $0.0383 · 9s · in 2,396 / out 663 tok
✓ avoids_stale_deals✓ avoids_unpopulated_live✗ uses_cohort_summary✗ succeeded_filter✓ no_giver_deletion_filter✓ returns_sql

✓ Correct Answer (ground truth)

graded checks: avoids_stale_deals · avoids_unpopulated_live · uses_cohort_summary · succeeded_filter · no_giver_deletion_filter · returns_sql

Model's actual answer

show the model's full answer
```sql
-- Cannot produce the requested per-customer-company query using only this catalog:
-- PRODUCTION.PLG.COMPANY_COHORT_SUMMARY is documented as one row per self-serve company
-- and contains M1_USERS, M1_GIVES, and M1_REDEMPTIONS, but the catalog does not provide
-- any customer company identifier/name column to return "per customer company".
-- No documented recognition event table is provided for raw giver/recognition counts,
-- and the redemption table's company/date columns are not documented here.
SELECT
  'MISSING DATA: catalog does not provide a customer company identifier column or raw recognition event table columns' AS error_message;
```
crm-completeness-enrichment0.80

? The Question

Audit the CRM extract for completeness and propose fixes. Deals need owner, stage, amount, close date, why-buys; companies need industry, employee count, HQ country; contacts need email, title, persona. Report completeness percent per field. Identify duplicate company clusters (name variants, shared domain) with a survivor per cluster. List invalid emails and domain mismatches. Fill missing company fields only where the enrichment export has a matching row; where CRM and enrichment disagree list both and recommend a source. Never invent a value. End with the 10 fixes with the most pipeline amount at stake.
cost $0.2493 · 40s · in 10,639 / out 6,042 tok
✓ dup_clusters✗ invalid_emails✓ survivor_language✓ completeness_pct✓ no_invent

✓ Correct Answer (ground truth)

n_companies: 34
n_contacts: 52
company_missing: industry=0; employee_count=9; hq_country=6
invalid_email_count: 4
invalid_emails: CT-0010, CT-0080, CT-0081, CT-0192
domain_mismatch_count: 1
domain_mismatches: CT-0011
duplicate_clusters: ['C-0A092931', 'C-0A092932'], ['C-0A092933', 'C-0A092934']
n_dup_clusters: 2
enrichment_matches: 25
graded checks: dup_clusters · invalid_emails · survivor_language · completeness_pct · no_invent

Model's actual answer

show the model's full answer
Deal data missing:
No deals extract was provided. I cannot calculate completeness for owner, stage, amount, close date, or why-buys. I also cannot rank fixes by pipeline amount at stake because no deal amounts or deal-company mapping were provided.

Completeness audit

Companies: 34 rows
- industry: 34 / 34 = 100.0%
- employee_count: 25 / 34 = 73.5%
- hq_country: 28 / 34 = 82.4%

Contacts: 52 rows
- email present: 52 / 52 = 100.0%
- email valid: 48 / 52 = 92.3%
- title: 39 / 52 = 75.0%
- persona: 37 / 52 = 71.2%

Deals:
- owner: not calculable; deals file missing
- stage: not calculable; deals file missing
- amount: not calculable; deals file missing
- close date: not calculable; deals file missing
- why-buys: not calculable; deals file missing

Duplicate company clusters

1. Shared domain: acme-corp.com
- Cluster: C-0A092931, C-0A092932
- Survivor: C-0A092931
- Reason: same completeness as C-0A092932; first CRM row retained.
- Conflicts:
  - C-0A092931: industry=Technology, employee_count=500, hq_country=US
  - C-0A092932: industry=tech, employee_count=510, hq_country=USA
- Fix: merge C-0A092932 into C-0A092931; employee_count conflict cannot be resolved from enrichment because acme-corp.com has no matching enrichment row.

2. Shared domain: globex.io
- Cluster: C-0A092933, C-0A092934
- Survivor: C-0A092933
- Reason: same completeness as C-0A092934; first CRM row retained.
- Conflicts:
  - C-0A092933: industry=SaaS, employee_count=200, hq_country=US
  - C-0A092934: industry=Technology, employee_count=200, hq_country=US
- Fix: merge C-0A092934 into C-0A092933; industry conflict cannot be resolved from enrichment because globex.io has no matching enrichment row.

Invalid emails

- CT-0010, C-66D1FC: email=user0@, expected domain=66d1fc.com
- CT-0080, C-92D97D: email=user0@, expected domain=92d97d.com
- CT-0081, C-92D97D: email=user1@, expected domain=92d97d.com
- CT-0192, C-425E2A: email=user2@, expected domain=425e2a.com

Domain mismatches

- CT-0011, C-66D1FC: email=user1@other-domain.com, contact domain=66d1fc.com, email domain=other-domain.com

Company fields that can be filled from matching enrichment rows

- C-EC3025, ec3025.com: employee_count blank -> 400
- C-96039F, 96039f.com: employee_count blank -> 400
- C-44EA29, 44ea29.com: employee_count blank -> 400
- C-D04904, d04904.com: employee_count blank -> 400
- C-B23205, b23205.com: employee_count blank -> 400
- C-60C75F, 60c75f.com: employee_count blank -> 400
- C-7BBDFA, 7bbdfa.com: employee_count blank -> 400
- C-50D386, 50d386.com: employee_count blank -> 400

Company fields still missing after enrichment

- C-2D1F1B, 2d1f1b.com: hq_country missing; enrichment also blank
- C-D73B89, d73b89.com: hq_country missing; enrichment also blank
- C-44EA29, 44ea29.com: hq_country missing; enrichment also blank
- C-D04904, d04904.com: hq_country missing; enrichment also blank
- C-2C60E5, 2c60e5.com: hq_country missing; enrichment also blank
- C-93C8BF, 93c8bf.com: employee_count missing; no matching enrichment row
- C-EE9FFB, ee9ffb.com: hq_country missing; no matching enrichment row

CRM vs enrichment disagreements

Recommend enrichment export as the standardization source where it supplies normalized country names or more specific software industry values. Do not overwrite employee_count where CRM and enrichment both have a value unless explicitly accepted; no employee_count disagreements appeared among matched rows.

- C-66D1FC, 66d1fc.com, industry: CRM=tech, enrichment=Computer Software; recommend enrichment
- C-66D1FC, 66d1fc.com, hq_country: CRM=US, enrichment=United States; recommend enrichment
- C-950043, 950043.com, hq_country: CRM=US, enrichment=United States; recommend enrichment
- C-EC3025, ec3025.com, industry: CRM=Technology, enrichment=Computer Software; recommend enrichment
- C-EC3025, ec3025.com, hq_country: CRM=USA, enrichment=United States; recommend enrichment
- C-96039F, 96039f.com, hq_country: CRM=USA, enrichment=United States; recommend enrichment
- C-44EA29, 44ea29.com, industry: CRM=tech, enrichment=Computer Software; recommend enrichment
- C-92D97D, 92d97d.com, industry: CRM=Technology, enrichment=Computer Software; recommend enrichment
- C-D04904, d04904.com, industry: CRM=Technology, enrichment=Computer Software; recommend enrichment
- C-77A95A, 77a95a.com, industry: CRM=Technology, enrichment=Computer Software; recommend enrichment
- C-77A95A, 77a95a.com, hq_country: CRM=US, enrichment=United States; recommend enrichment
- C-AA8DDA, aa8dda.com, industry: CRM=Technology, enrichment=Computer Software; recommend enrichment
- C-B23205, b23205.com, hq_country: CRM=US, enrichment=United States; recommend enrichment
- C-E51FB7, e51fb7.com, hq_country: CRM=USA, enrichment=United States; recommend enrichment
- C-D0662E, d0662e.com, hq_country: CRM=US, enrichment=United States; recommend enrichment
- C-B25F40, b25f40.com, industry: CRM=Technology, enrichment=Computer Software; recommend enrichment
- C-60C75F, 60c75f.com, industry: CRM=tech, enrichment=Computer Software; recommend enrichment
- C-425E2A, 425e2a.com, industry: CRM=Tech, enrichment=Computer Software; recommend enrichment
- C-425E2A, 425e2a.com, hq_country: CRM=USA, enrichment=United States; recommend enrichment
- C-2D7423, 2d7423.com, hq_country: CRM=USA, enrichment=United States; recommend enrichment

Contact completeness fixes needed

Missing title and persona:
- CT-0000, C-2D1F1B
- CT-0022, C-C6FE92
- CT-0081, C-92D97D
- CT-0092, C-D04904
- CT-0132, C-B23205
- CT-0162, C-D0662E

Missing title only:
- CT-0072, C-44EA29
- CT-0080, C-92D97D
- CT-0120, C-AA8DDA
- CT-0121, C-AA8DDA
- CT-0122, C-AA8DDA
- CT-0141, C-E51FB7
- CT-0170, C-B25F40

Missing persona only:
- CT-0041, C-D73B89
- CT-0060, C-96039F
- CT-0070, C-44EA29
- CT-0082, C-92D97D
- CT-0110, C-77A95A
- CT-0171, C-B25F40
- CT-0172, C-B25F40
- CT-0180, C-60C75F
- CT-0181, C-60C75F

10 fixes with the most pipeline amount at stake

Cannot provide this ranking. The deals extract is missing, so pipeline amount at stake is unavailable. No deal amounts, deal owners, stages, close dates, why-buys, or deal-company mappings were provided.
deal-intelligence
closed-lost-classification0.80

? The Question

Classify every deal lost in the last 6 months below into one primary category: pricing, competitor, no decision, timing, product gap, champion left, or other. Also assign a side (Bonusly, buyer, unknown). Then summarize: category counts, the side split, the number of deals where the structured closed_lost_tag clearly disagrees with the free-text reason, and the two patterns most worth acting on. Base everything only on the tag and free-text provided.
cost $0.3351 · 82s · in 9,532 / out 8,754 tok
✓ total_90✗ competitor_top✓ category_counts_present✓ side_split✓ two_patterns

✓ Correct Answer (ground truth)

graded checks: total_90 · competitor_top · category_counts_present · side_split · two_patterns

Model's actual answer

show the model's full answer
Classification table:

| Deal | Primary category | Side |
|---|---:|---:|
| Deal-DB0AAC | timing | buyer |
| Deal-F7F635 | competitor | unknown |
| Deal-AC944F | no decision | buyer |
| Deal-214060 | no decision | buyer |
| Deal-91A056 | timing | buyer |
| Deal-29326C | timing | buyer |
| Deal-5DB9B0 | other | unknown |
| Deal-831B7B | timing | buyer |
| Deal-F97C37 | competitor | Bonusly |
| Deal-13E9CF | no decision | buyer |
| Deal-39E25C | timing | buyer |
| Deal-7ED004 | pricing | buyer |
| Deal-21B045 | no decision | buyer |
| Deal-B3ABED | timing | buyer |
| Deal-422BA6 | competitor | Bonusly |
| Deal-ED9AE7 | timing | buyer |
| Deal-988493 | no decision | buyer |
| Deal-381C8C | competitor | unknown |
| Deal-F308CA | no decision | buyer |
| Deal-F1E8A6 | competitor | unknown |
| Deal-B6AC09 | timing | buyer |
| Deal-70F704 | no decision | buyer |
| Deal-E6E80A | timing | buyer |
| Deal-B038F0 | timing | buyer |
| Deal-4664E1 | no decision | buyer |
| Deal-175756 | timing | buyer |
| Deal-E74A73 | no decision | buyer |
| Deal-DDAB52 | competitor | Bonusly |
| Deal-ACE061 | competitor | unknown |
| Deal-BB78F3 | timing | buyer |
| Deal-D48E0B | no decision | buyer |
| Deal-15DA99 | timing | buyer |
| Deal-F4AF5D | timing | buyer |
| Deal-79B7A1 | timing | buyer |
| Deal-583ADB | no decision | buyer |
| Deal-8E27DA | competitor | buyer |
| Deal-2D2F8D | competitor | unknown |
| Deal-E0441F | no decision | buyer |
| Deal-7CB44D | no decision | buyer |
| Deal-0F96AA | competitor | unknown |
| Deal-1BCA50 | competitor | Bonusly |
| Deal-7CC678 | competitor | unknown |
| Deal-FAC17C | no decision | buyer |
| Deal-242273 | product gap | Bonusly |
| Deal-50E5D8 | no decision | buyer |
| Deal-A2C349 | competitor | Bonusly |
| Deal-9F176A | timing | buyer |
| Deal-7B2236 | pricing | Bonusly |
| Deal-AFA56C | no decision | buyer |
| Deal-C7156E | competitor | unknown |
| Deal-C33D91 | pricing | buyer |
| Deal-9048EB | product gap | Bonusly |
| Deal-5E64CE | timing | buyer |
| Deal-8A0992 | competitor | Bonusly |
| Deal-D0C698 | competitor | buyer |
| Deal-69CF3D | timing | buyer |
| Deal-ECBF89 | timing | buyer |
| Deal-3618CC | product gap | Bonusly |
| Deal-EECC02 | competitor | unknown |
| Deal-5AD03E | product gap | Bonusly |
| Deal-D1A623 | timing | buyer |
| Deal-413C56 | no decision | buyer |
| Deal-47F1A1 | competitor | buyer |
| Deal-BF2A98 | competitor | buyer |
| Deal-2A292B | no decision | buyer |
| Deal-D1AABF | no decision | buyer |
| Deal-FEDBCB | timing | buyer |
| Deal-1E7DA9 | competitor | unknown |
| Deal-2BBA21 | no decision | buyer |
| Deal-286F9C | competitor | Bonusly |
| Deal-7FBAC6 | no decision | buyer |
| Deal-369281 | competitor | buyer |
| Deal-386F6E | no decision | buyer |
| Deal-9FCD0D | competitor | Bonusly |
| Deal-55867E | other | unknown |
| Deal-DAFB82 | pricing | buyer |
| Deal-2FEDDB | timing | buyer |
| Deal-64B19A | competitor | unknown |
| Deal-3F86A0 | no decision | buyer |
| Deal-096750 | no decision | buyer |
| Deal-F325A5 | champion left | buyer |
| Deal-ABD14C | no decision | buyer |
| Deal-79E61A | no decision | buyer |
| Deal-8A119B | pricing | buyer |
| Deal-AE7C4E | no decision | buyer |
| Deal-DAB4F1 | no decision | buyer |
| Deal-B4B50F | no decision | buyer |
| Deal-981AD4 | product gap | Bonusly |
| Deal-DC77FE | product gap | Bonusly |
| Deal-5885B9 | no decision | buyer |

Category counts:

| Category | Count |
|---|---:|
| no decision | 30 |
| competitor | 24 |
| timing | 22 |
| product gap | 6 |
| pricing | 5 |
| other | 2 |
| champion left | 1 |
| Total | 90 |

Arithmetic: 30 + 24 + 22 + 6 + 5 + 2 + 1 = 90.

Side split:

| Side | Count |
|---|---:|
| buyer | 62 |
| Bonusly | 15 |
| unknown | 13 |
| Total | 90 |

Arithmetic: 62 + 15 + 13 = 90.

Structured tag clearly disagrees with free-text reason: 5 deals.

Deals:
Deal-70F704, Deal-8E27DA, Deal-3618CC, Deal-5AD03E, Deal-55867E

Arithmetic: 5 / 90 deals.

Two patterns most worth acting on:

1. “No decision” and “timing” dominate the losses.
Arithmetic: no decision 30 + timing 22 = 52 of 90 deals.
This is mostly buyer-side: pauses, deprioritization, MIA/unresponsive, reconnect next year, budget approval not secured, or internal priorities.

2. Competitive/product gaps are concentrated around alternatives with broader fit, integrations, customization, surveys, budget access, local/Canadian alignment, or existing HR/payroll/vendor ecosystems.
Arithmetic: competitor 24 + product gap 6 = 30 of 90 deals.
Bonusly-side examples include Deal-F97C37, Deal-422BA6, Deal-DDAB52, Deal-242273, Deal-9048EB, Deal-3618CC, Deal-5AD03E, Deal-981AD4, and Deal-DC77FE.
pipeline-tieringPASS 1.00

? The Question

Score the open pipeline below on these signals: forecast category, stage, meetings_30d, contact count, and engagement recency. Assign exactly one tier per deal: LOCK, ACTION, BUILD, REVIVE, WATCH, or RISKY (RISKY = the forecast category disagrees with the engagement evidence). Never tier a deal LOCK if it has zero meetings_30d. Do NOT list every deal. Return ONLY a compact JSON object with exactly these keys: - "tier_counts": object mapping each tier to the number of deals in it (must sum to the total deal count), - "tier_examples": object mapping each tier to up to 3 example deal aliases from the data, - "risky_deals": array of the deal aliases you tiered RISKY, - "lock_violations": number of deals tiered LOCK that have zero meetings_30d (should be 0), - "pipeline_shape": one short paragraph on the overall pipeline shape. Note: inbound_emails_30d is 0 for every row due to a data defect, so treat meetings_30d as the inbound signal.
cost $0.7943 · 118s · in 37,752 / out 19,451 tok
✓ json_returned✓ tier_vocabulary✓ tier_counts_sum✓ risky_deals_listed✓ no_lock_violation_claimed

✓ Correct Answer (ground truth)

total: 156
graded checks: json_returned · tier_vocabulary · tier_counts_sum · risky_deals_listed · no_lock_violation_claimed

Model's actual answer

show the model's full answer
{"tier_counts":{"LOCK":3,"ACTION":26,"BUILD":34,"REVIVE":14,"WATCH":58,"RISKY":21},"tier_examples":{"LOCK":["Deal-D348E1","Deal-C26D20","Deal-403845"],"ACTION":["Deal-25F752","Deal-E53952","Deal-5408B0"],"BUILD":["Deal-D73B89","Deal-012CB1","Deal-523604"],"REVIVE":["Deal-2D1F1B","Deal-66D1FC","Deal-950043"],"WATCH":["Deal-6787C2","Deal-C6D97A","Deal-C9C286"],"RISKY":["Deal-547B2B","Deal-B7EBD1","Deal-A2B47C"]},"risky_deals":["Deal-547B2B","Deal-B7EBD1","Deal-A2B47C","Deal-2465CE","Deal-584EE5","Deal-A5E80A","Deal-0660B4","Deal-FD9F4E","Deal-499BF6","Deal-BA571A","Deal-7BBDFA","Deal-1CCE5C","Deal-FA053A","Deal-E531A6","Deal-5296C9","Deal-4A13AD","Deal-690476","Deal-635B8E","Deal-898FC5","Deal-F336B6","Deal-BA3DDC"],"lock_violations":0,"pipeline_shape":"Arithmetic: 3 LOCK + 26 ACTION + 34 BUILD + 14 REVIVE + 58 WATCH + 21 RISKY = 156 total deals; 55 deals have meetings_30d > 0 and 101 have zero meetings_30d, so the pipe is engagement-thin despite 24 late-stage DS4/DS5 deals and 51 forecasted COMMIT/BEST_CASE deals. Engagement rows are missing for Deal-3EED2C and Deal-57FF13, so their meetings_30d evidence is missing from the provided data."}
call-transcript-extractionPASS 1.00

? The Question

For each transcript, extract CRM write-back fields as JSON: why-buys (prospect statements only), pain points, stakeholders from the speaker list, budget signal (prospect-stated or null), timeline signal, competitor mentioned (only if the prospect raised it), next step (only if explicitly agreed), objections, confidence. Never fill a field from what the rep said or hoped.
cost $0.0774 · 15s · in 3,258 / out 1,825 tok
✓ json_emitted✓ budget_signal✓ null_discipline✓ stakeholders✓ competitor_prospect_only

✓ Correct Answer (ground truth)

rep: Alex Franklin
per_transcript: {"TX-001": {"why_buys": ["automating anniversary and birthday awards"], "pain_points": ["HR team of three cannot keep up with it manually", "track everything in a spreadsheet, and people slip through
rep_sourced_nulls: {"TX-003": ["budget_signal \u2014 only the rep mentioned pricing/budget"], "TX-004": ["next_step \u2014 rep proposed, prospect said 'no promises'"], "TX-006": ["budget_signal \u2014 rep decoy, prospec
allowed_new: TX-001, TX-002, TX-003, TX-004, TX-005, TX-006, Achievers, Bucketlist, Nectar
graded checks: json_emitted · budget_signal · null_discipline · stakeholders · competitor_prospect_only

Model's actual answer

show the model's full answer
[
  {
    "transcript_id": "TX-001",
    "deal_alias": "Deal-CFE7F4",
    "why_buys": [
      "The big win for us would be automating anniversary and birthday awards — our HR team of three cannot keep up with it manually."
    ],
    "pain_points": [
      "HR team of three cannot keep up with anniversary and birthday awards manually.",
      "Right now we track everything in a spreadsheet, and people slip through the cracks."
    ],
    "stakeholders_from_speaker_list": [
      "Prospect (VP People)",
      "Prospect (HR Admin)"
    ],
    "budget_signal": "We have about $40k earmarked for engagement tools this fiscal year.",
    "timeline_signal": "Ideally we would have this live before open enrollment in November.",
    "competitor_mentioned": "Achievers",
    "next_step": "Security review on September 12.",
    "objections": [
      "We looked at Achievers last year, but it was too heavy for a team our size.",
      "One concern: we need SSO and audit logs for IT to sign off."
    ],
    "confidence": "high"
  },
  {
    "transcript_id": "TX-002",
    "deal_alias": "Deal-70BB30",
    "why_buys": [
      "We want to tie recognition to retention for our hourly workforce — regretted turnover there is over 30%."
    ],
    "pain_points": [
      "Regretted turnover for hourly workforce is over 30%."
    ],
    "stakeholders_from_speaker_list": [
      "Prospect (Head of Total Rewards)",
      "Prospect (CFO)"
    ],
    "budget_signal": "Finance has approved a $25k pilot budget for this quarter.",
    "timeline_signal": "We want a decision by end of September.",
    "competitor_mentioned": null,
    "next_step": "Send the pilot agreement; prospect will route it to legal this week.",
    "objections": [
      "Integration with Workday has to be rock solid — that's my one condition."
    ],
    "confidence": "high"
  },
  {
    "transcript_id": "TX-003",
    "deal_alias": "Deal-530B50",
    "why_buys": [
      "We need to make recognition visible across our 12 retail locations."
    ],
    "pain_points": [
      "Store managers have zero budget autonomy for on-the-spot recognition today."
    ],
    "stakeholders_from_speaker_list": [
      "Prospect (People Ops Manager)"
    ],
    "budget_signal": null,
    "timeline_signal": "Honestly there's no rush on our side until Q1.",
    "competitor_mentioned": "Bucketlist",
    "next_step": "Schedule a call with CEO; prospect will send two times.",
    "objections": [
      "The CEO has to be sold first — she decides anything people-related."
    ],
    "confidence": "high"
  },
  {
    "transcript_id": "TX-004",
    "deal_alias": "Deal-180D02",
    "why_buys": [
      "We want to consolidate three separate recognition tools into one."
    ],
    "pain_points": [
      "We're paying for three tools and none of them talk to our HRIS."
    ],
    "stakeholders_from_speaker_list": [
      "Prospect (VP People)",
      "Prospect (IT Security Lead)"
    ],
    "budget_signal": "If it's under $15k annually, I can approve it without going to the board.",
    "timeline_signal": "Our procurement cycle runs six to eight weeks minimum.",
    "competitor_mentioned": null,
    "next_step": null,
    "objections": [
      "The security review took three months for our last vendor — that's my hesitation.",
      "Maybe — I need to check her calendar, no promises."
    ],
    "confidence": "high"
  },
  {
    "transcript_id": "TX-005",
    "deal_alias": "Deal-F8767A",
    "why_buys": [
      "Two things: automate service milestones, and give us analytics on recognition equity across departments."
    ],
    "pain_points": [
      "Night-shift teams feel invisible.",
      "Night-shift engagement scores run 20 points lower.",
      "Exec team is skeptical after a failed rollout two years ago."
    ],
    "stakeholders_from_speaker_list": [
      "Prospect (HR Director)",
      "Prospect (People Ops Coordinator)"
    ],
    "budget_signal": "We have $12k approved under our engagement line.",
    "timeline_signal": "We need this running before our January all-hands.",
    "competitor_mentioned": "Nectar",
    "next_step": "Present to exec team on October 2.",
    "objections": [
      "We're mid-pilot with Nectar right now, so you'd need to beat that experience.",
      "Our exec team is skeptical after a failed rollout two years ago."
    ],
    "confidence": "high"
  },
  {
    "transcript_id": "TX-006",
    "deal_alias": "Deal-EE195F",
    "why_buys": [
      "I'd love to cut the admin time on service awards."
    ],
    "pain_points": [
      "I personally spend five hours a month ordering and shipping plaques.",
      "Budget isn't the issue — time is."
    ],
    "stakeholders_from_speaker_list": [
      "Prospect (HR Manager)"
    ],
    "budget_signal": "Budget isn't the issue — time is.",
    "timeline_signal": "Our fiscal year resets in February, so a Q1 start is realistic.",
    "competitor_mentioned": "doing it internally",
    "next_step": "Send the one-page overview; prospect will forward it to COO this week.",
    "objections": [
      "Nobody else — we're comparing this against just doing it internally.",
      "Fair warning, our COO usually prefers building things in-house."
    ],
    "confidence": "high"
  }
]
next-to-closePASS 1.00

? The Question

Which 3 deals are closest to signature right now? Use the CRM data and the Slack thread; Slack often has the real status before the CRM does. For each: deal, amount, why it is close, what is left. Under 250 words.
cost $0.0666 · 14s · in 3,718 / out 1,140 tok
✓ exactly_three✓ top3_named✓ slack_first✓ blocked_excluded

✓ Correct Answer (ground truth)

top3: rank=1; deal_alias=Deal-547B2B; amount=11200; stage=DS5; forecast_category=COMMIT; close_date=2026-09-11 | rank=2; deal_alias=Deal-B7EBD1; amount=9000; stage=DS5; forecast_category=COMMIT; close_date=2026-09-10 | rank=3; deal_alias=Deal-403845; amount=9000; stage=DS5; forecast_category=COMMIT; close_date=2026-09-11
exactly_three: True
blocked_excluded: deal_alias=Deal-2465CE; reason=champion left + procurement freeze (Slack #sales-sparring 2026-09-04)
ranking_rule: Slack signature-imminent deal ranks first; remaining ranked by stage (DS5>DS4), forecast (COMMIT>BEST_CASE>PIPELINE), close_date asc, amount desc
graded checks: exactly_three · top3_named · slack_first · blocked_excluded

Model's actual answer

show the model's full answer
1. Deal-547B2B — $11,200
Why close: CRM says DS5/COMMIT, close date 2026-09-11. Arithmetic: 2026-09-11 - Slack date 2026-09-04 = 7 days. Slack is strongest: “redlines came back clean,” signing page is out, VP People said they are “signing tomorrow,” and it is “signature-imminent.”
What is left: signature.

2. Deal-403845 — $9,000
Why close: CRM says DS5/COMMIT, close date 2026-09-11. Arithmetic: 2026-09-11 - 2026-09-04 = 7 days. Slack says it is “moving fine” and “the order form is with their finance team.”
What is left: finance approval / signed order form.

3. Deal-B7EBD1 — $9,000
Why close: CRM says DS5/COMMIT, close date 2026-09-10. Arithmetic: 2026-09-10 - 2026-09-04 = 6 days. No Slack update provided, so closeness is based only on CRM: latest stage among near-term deals, commit forecast, earliest close date.
What is left: not stated in provided data.

Excluded: Deal-2465CE is DS5/COMMIT in CRM, but Slack says it “went sideways,” procurement froze new vendors, and it is “blocked.”
product-gap-detectionPASS 1.00

? The Question

Surface true product gaps raised by prospects in active deals. For each candidate: quote the prospect line, classify as real gap, plan gate, or rollout/enablement issue using the product docs excerpt, and name the deal and amount. Only real gaps go in the summary. Rep language does not count as prospect voice.
cost $0.0440 · 10s · in 2,182 / out 891 tok
✓ real_gaps✓ plan_gate_classified✓ enablement_classified✓ gap_summary

✓ Correct Answer (ground truth)

real_gaps: transcript_id=TG-001; deal_alias=Deal-EC3025; amount=62000; feature=SCIM user provisioning; quote=We need SCIM user provisioning — our security team will not approve any tool that requires manual account management.; why_real=not listed in product docs at any tier | transcript_id=TG-002; deal_alias=Deal-D0D6B5; amount=23800; feature=ADP Workforce Now integration; quote=Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us.; why_real=supported HRIS list is Workday/BambooHR/Gusto/Rippling only
plan_gates: transcript_id=TG-003; deal_alias=Deal-CFE7F4; amount=32000; feature=Custom report builder; tier_required=Enterprise; quote=I tried to build a custom report combining recognition frequency with turnover and couldn't — is that possible on our current plan?
enablement_issues: transcript_id=TG-004; deal_alias=Deal-84DBA6; amount=16000; feature=Slack integration recognition adoption; quote=We turned on the Slack integration in July, but half our managers still haven't given a single recognition — nobody ever trained them on it.; why_enablement=Slack integration exists on all plans; managers were never trained
excluded_rep_voice: transcript_id=TG-005; line=Full transparency — we don't have a native mobile app for hourly workers yet, though it's on the roadmap.; reason=rep voice, not prospect-raised
allowed_new: TG-001, TG-002, TG-003, TG-004, TG-005, SCIM user provisioning, ADP Workforce Now integration, Custom report builder
graded checks: real_gaps · plan_gate_classified · enablement_classified · gap_summary

Model's actual answer

show the model's full answer
Data limitations: no deal amounts or active/inactive deal status were provided. Amount is therefore missing for every deal below.

Candidates:

1. Deal-EC3025
Amount: missing
Prospect quote: “We need SCIM user provisioning — our security team will not approve any tool that requires manual account management.”
Classification: real gap
Reason: product docs state “SCIM user provisioning” is “NOT currently listed as supported capabilities.”

2. Deal-D0D6B5
Amount: missing
Prospect quote: “Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us.”
Classification: real gap
Reason: product docs list HRIS integrations as Workday, BambooHR, Gusto, Rippling, and state “ADP Workforce Now integrations are NOT currently listed as supported capabilities.”

3. Deal-CFE7F4
Amount: missing
Prospect quote: “I tried to build a custom report combining recognition frequency with turnover and couldn't — is that possible on our current plan?”
Classification: plan gate
Reason: product docs show “Custom report builder” is available on Enterprise only. The prospect’s current plan is not provided.

4. Deal-84DBA6
Amount: missing
Prospect quote: “We turned on the Slack integration in July, but half our managers still haven't given a single recognition — nobody ever trained them on it.”
Classification: rollout/enablement issue
Reason: product docs show Slack integration is supported on all plans; the stated issue is lack of manager training.

Excluded:
Deal-36C33F
Reason: native mobile app limitation was stated by Alex Franklin, not the prospect. Rep language does not count as prospect voice. Prospect said: “Good to know. The web version should be fine for our office staff for now.”

Summary — real gaps only:
Real gaps = 1 SCIM gap + 1 ADP Workforce Now gap = 2 real gaps.

Real-gap deals:
- Deal-EC3025: SCIM user provisioning gap; amount missing.
- Deal-D0D6B5: ADP Workforce Now integration gap; amount missing.

Total real-gap amount: not computable because deal amounts were not provided.
rep-performance
stale-pipeline-by-rep0.67

? The Question

The snapshot date is 2026-09-05. Using the open deals and the engagements-by-deal table, list every open deal with no logged email, call, or meeting in the last 7 days. Compute recency from the engagements table's last_email/last_call/last_meeting fields, NOT from any date field on the deal (the deal's last_contacted_field is unreliable). For each stale deal give: deal alias, owner name, stage, amount, days since last contact. Group by owner, order by amount descending within each owner. Then state, per owner, how many stale deals they carry and the total stale amount. (Do not write outreach text.)
cost $1.0136 · 158s · in 38,000 / out 25,874 tok
✓ stale_recall_ge_80✓ grouped_by_owner✓ jake_top_amount✓ kevin_stale_amt✗ jenn_stale_amt✗ stale_count

✓ Correct Answer (ground truth)

stale_count: 65
jake_amt: 626243.0
kevin_amt: 252905.03
jenn_amt: 261645.0
top_alias: Deal-2D1F1B
top_amount: 240000.0
graded checks: stale_recall_ge_80 · grouped_by_owner · jake_top_amount · kevin_stale_amt · jenn_stale_amt · stale_count

Model's actual answer

show the model's full answer
Rule used: stale = no last_email, last_call, or last_meeting from 2026-08-29 through 2026-09-05. Days since last contact = 2026-09-05 minus the most recent provided contact date on or before 2026-09-05.

Data missing: Deal-3EED2C and Deal-57FF13 have no engagements_by_deal_90d row, so stale status and days since last contact cannot be computed from the provided engagement fields.

Bryce Harmon

Deal-2D1F1B | Bryce Harmon | DS1 | 240000 | 2026-09-05 - 2026-06-16 = 81 days
Deal-66D1FC | Bryce Harmon | DS1 | 99000 | 2026-09-05 - 2026-08-20 = 16 days
Deal-950043 | Bryce Harmon | DS1 | 70000 | 2026-09-05 - 2026-08-17 = 19 days
Deal-B23205 | Bryce Harmon | DS1 | 45000 | 2026-09-05 - 2026-08-20 = 16 days
Deal-7BBDFA | Bryce Harmon | DS3 | 37440 | 2026-09-05 - 2026-07-21 = 46 days
Deal-332637 | Bryce Harmon | DS2 | 36000 | 2026-09-05 - 2026-08-27 = 9 days
Deal-1BEEBF | Bryce Harmon | DS1 | 31500 | 2026-09-05 - 2026-08-17 = 19 days
Deal-A414F6 | Bryce Harmon | DS1 | 25200 | 2026-09-05 - 2026-08-17 = 19 days
Deal-C5658B | Bryce Harmon | DS1 | 23400 | 2026-09-05 - 2026-08-20 = 16 days
Deal-40522D | Bryce Harmon | DS3 | 21000 | 2026-09-05 - 2026-08-17 = 19 days
Deal-C1FA6D | Bryce Harmon | DS1 | 18000 | 2026-09-05 - 2026-08-20 = 16 days
Deal-01E193 | Bryce Harmon | DS1 | 12600 | 2026-09-05 - 2026-08-28 = 8 days
Deal-F0EBBB | Bryce Harmon | DS3 | 11400 | 2026-09-05 - 2026-08-12 = 24 days
Deal-927338 | Bryce Harmon | DS1 | 10920 | 2026-09-05 - 2026-08-18 = 18 days
Deal-E25A09 | Bryce Harmon | DS1 | 6000 | 2026-09-05 - 2026-08-27 = 9 days
Deal-C9C286 | Bryce Harmon | DS2 | 5502 | 2026-09-05 - 2026-08-27 = 9 days
Deal-012CB1 | Bryce Harmon | DS1 | 1 | 2026-09-05 - 2026-08-13 = 23 days
Deal-3795AD | Bryce Harmon | DS2 | 1 | 2026-09-05 - 2026-08-28 = 8 days

Bryce Harmon total: 18 stale deals; stale amount = 240000 + 99000 + 70000 + 45000 + 37440 + 36000 + 31500 + 25200 + 23400 + 21000 + 18000 + 12600 + 11400 + 10920 + 6000 + 5502 + 1 + 1 = 692964

Dana Mercer

Deal-44EA29 | Dana Mercer | DS2 | 60000 | 2026-09-05 - 2026-08-26 = 10 days
Deal-E51FB7 | Dana Mercer | DS2 | 43875 | 2026-09-05 - 2026-08-24 = 12 days
Deal-B42F46 | Dana Mercer | DS1 | 27000 | 2026-09-05 - 2026-08-17 = 19 days
Deal-BA3DDC | Dana Mercer | DS3 | 23400 | 2026-09-05 - 2026-08-21 = 15 days
Deal-9DDE86 | Dana Mercer | DS2 | 20000 | 2026-09-05 - 2026-08-21 = 15 days
Deal-215CCA | Dana Mercer | DS3 | 18900 | 2026-09-05 - 2026-08-19 = 17 days
Deal-5EED42 | Dana Mercer | DS3 | 16250 | 2026-09-05 - 2026-08-25 = 11 days
Deal-57887A | Dana Mercer | DS2 | 15000 | 2026-09-05 - 2026-08-28 = 8 days
Deal-944310 | Dana Mercer | DS4 | 10500 | 2026-09-05 - 2026-08-03 = 33 days
Deal-B7EBD1 | Dana Mercer | DS5 | 9000 | 2026-09-05 - 2026-08-20 = 16 days
Deal-3974EB | Dana Mercer | DS4 | 9000 | 2026-09-05 - 2026-08-28 = 8 days
Deal-F40F04 | Dana Mercer | DS2 | 8100 | 2026-09-05 - 2026-08-21 = 15 days
Deal-7599B8 | Dana Mercer | DS3 | 7350 | 2026-09-05 - 2026-08-18 = 18 days
Deal-87DDD1 | Dana Mercer | DS1 | 5000 | 2026-09-05 - 2026-08-17 = 19 days
Deal-F336B6 | Dana Mercer | DS3 | 4200 | 2026-09-05 - 2026-08-21 = 15 days
Deal-0660B4 | Dana Mercer | DS4 | 1920 | 2026-09-05 - 2026-08-20 = 16 days

Dana Mercer total: 16 stale deals; stale amount = 60000 + 43875 + 27000 + 23400 + 20000 + 18900 + 16250 + 15000 + 10500 + 9000 + 9000 + 8100 + 7350 + 5000 + 4200 + 1920 = 279495

Alex Franklin

Deal-CC08D1 | Alex Franklin | DS1 | 24000 | 2026-09-05 - 2026-08-20 = 16 days
Deal-E73427 | Alex Franklin | DS3 | 18000 | 2026-09-05 - 2026-08-26 = 10 days
Deal-885F45 | Alex Franklin | DS2 | 9300 | 2026-09-05 - 2026-08-24 = 12 days
Deal-C2FF3C | Alex Franklin | DS1 | 8316 | 2026-09-05 - 2026-08-26 = 10 days
Deal-0D2F7A | Alex Franklin | DS3 | 5100 | 2026-09-05 - 2026-08-24 = 12 days
Deal-6C60D4 | Alex Franklin | DS3 | 4800 | 2026-09-05 - 2026-08-24 = 12 days
Deal-13FEBD | Alex Franklin | DS2 | 4680 | 2026-09-05 - 2026-08-24 = 12 days
Deal-819506 | Alex Franklin | DS1 | 4400 | 2026-09-05 - 2026-08-28 = 8 days
Deal-9D0060 | Alex Franklin | DS3 | 3840 | 2026-09-05 - 2026-08-24 = 12 days
Deal-690476 | Alex Franklin | DS2 | 3600 | 2026-09-05 - 2026-08-18 = 18 days
Deal-C6D97A | Alex Franklin | DS4 | 3240 | 2026-09-05 - 2026-08-28 = 8 days
Deal-EE195F | Alex Franklin | DS3 | 3120 | 2026-09-05 - 2026-08-28 = 8 days
Deal-278DEC | Alex Franklin | DS3 | 2700 | 2026-09-05 - 2026-08-28 = 8 days
Deal-635B8E | Alex Franklin | DS3 | 2600 | 2026-09-05 - 2026-08-18 = 18 days
Deal-6883F3 | Alex Franklin | DS1 | 2400 | 2026-09-05 - 2026-08-20 = 16 days
Deal-4A13AD | Alex Franklin | DS3 | 2160 | 2026-09-05 - 2026-08-10 = 26 days
Deal-F67D31 | Alex Franklin | DS2 | 1800 | 2026-09-05 - 2026-08-28 = 8 days
Deal-5FDCE4 | Alex Franklin | DS3 | 1600 | 2026-09-05 - 2026-08-24 = 12 days
Deal-BA571A | Alex Franklin | DS4 | 1080 | 2026-09-05 - 2026-08-18 = 18 days

Alex Franklin total: 19 stale deals; stale amount = 24000 + 18000 + 9300 + 8316 + 5100 + 4800 + 4680 + 4400 + 3840 + 3600 + 3240 + 3120 + 2700 + 2600 + 2400 + 2160 + 1800 + 1600 + 1080 = 106736

Cole Ingram

Deal-D04904 | Cole Ingram | DS2 | 58529.25 | 2026-09-05 - 2026-08-25 = 11 days
Deal-B25F40 | Cole Ingram | DS3 | 40000 | 2026-09-05 - 2026-08-28 = 8 days
Deal-813836 | Cole Ingram | DS2 | 32175 | 2026-09-05 - 2026-08-25 = 11 days
Deal-1BA595 | Cole Ingram | DS2 | 31750 | 2026-09-05 - 2026-08-25 = 11 days
Deal-CFE1E8 | Cole Ingram | DS3 | 18000 | 2026-09-05 - 2026-08-25 = 11 days
Deal-CD47A6 | Cole Ingram | DS2 | 12168 | 2026-09-05 - 2026-08-25 = 11 days
Deal-627646 | Cole Ingram | DS3 | 11193 | 2026-09-05 - 2026-08-25 = 11 days
Deal-FF809F | Cole Ingram | DS2 | 7781.2 | 2026-09-05 - 2026-08-25 = 11 days
Deal-AF932D | Cole Ingram | DS2 | 7225.4 | 2026-09-05 - 2026-08-25 = 11 days
Deal-A71728 | Cole Ingram | DS2 | 6947.5 | 2026-09-05 - 2026-08-25 = 11 days
Deal-8BC9F5 | Cole Ingram | DS2 | 5616 | 2026-09-05 - 2026-08-26 = 10 days
Deal-175395 | Cole Ingram | DS3 | 4779.88 | 2026-09-05 - 2026-08-25 = 11 days
Deal-481E24 | Cole Ingram | DS3 | 4140 | 2026-09-05 - 2026-08-26 = 10 days
Deal-C7F9BF | Cole Ingram | DS2 | 3360 | 2026-09-05 - 2026-08-25 = 11 days
Deal-2F3A66 | Cole Ingram | DS3 | 3334.8 | 2026-09-05 - 2026-08-25 = 11 days
Deal-342E96 | Cole Ingram | DS2 | 2700 | 2026-09-05 - 2026-08-12 = 24 days
Deal-E568D5 | Cole Ingram | DS3 | 1875 | 2026-09-05 - 2026-08-25 = 11 days
Deal-FD9F4E | Cole Ingram | DS5 | 1330 | 2026-09-05 - 2026-08-26 = 10 days

Cole Ingram total: 18 stale deals; stale amount = 58529.25 + 40000 + 32175 + 31750 + 18000 + 12168 + 11193 + 7781.2 + 7225.4 + 6947.5 + 5616 + 4779.88 + 4140 + 3360 + 3334.8 + 2700 + 1875 + 1330 = 252905.03

Farid Osman

Deal-8BA24E | Farid Osman | DS1 | 2880 | 2026-09-05 - 2026-08-28 = 8 days
Deal-8FDCD2 | Farid Osman | DS1 | 1 | 2026-09-05 - 2026-08-21 = 15 days

Farid Osman total: 2 stale deals; stale amount = 2880 + 1 = 2881
activity-mix-vs-outcomePASS 1.00

? The Question

The snapshot date is 2026-09-05. For each sales rep (owner), compute the last-30-day totals of emails, calls, and meetings from the engagements table, and the number of deals that entered DS2 in the last 30 days (t_ds2 within 30 days of the snapshot). Report per rep: the activity mix as percentages (emails/calls/meetings share of that rep's total activities), total activities per DS2 entry, and rank the reps by that efficiency ratio (lowest activities per DS2 entry = most efficient). Name the most efficient rep and the highest-volume rep, and note if they differ. Do not attribute cause.
cost $0.3302 · 44s · in 23,012 / out 6,421 tok
✓ justin_ds2✓ justin_volume_high✓ efficient_named✓ mix_present✓ no_causal_claim

✓ Correct Answer (ground truth)

justin_ds2: 4
graded checks: justin_ds2 · justin_volume_high · efficient_named · mix_present · no_causal_claim

Model's actual answer

show the model's full answer
Window used: 2026-08-06 through 2026-09-05 inclusive.

Per-rep results, ranked by efficiency ratio: lowest activities per DS2 entry = most efficient.

| Rank | Owner | Emails | Calls | Meetings | Total activities | DS2 entries | DS2 deal aliases | Activity mix | Activities / DS2 entry |
|---:|---|---:|---:|---:|---:|---:|---|---|---:|
| 1 | Alex Franklin | 307 | 36 | 41 | 384 | 18 | Deal-403845, Deal-1FC049, Deal-3EED2C, Deal-7FA0C3, Deal-E531A6, Deal-5296C9, Deal-36C33F, Deal-EE195F, Deal-F436DA, Deal-317E6F, Deal-D1E6C2, Deal-D9A72E, Deal-CA5E44, Deal-4F775F, Deal-898FC5, Deal-46988D, Deal-E73427, Deal-92D97D | Emails 307/384 = 79.9%; Calls 36/384 = 9.4%; Meetings 41/384 = 10.7% | 384/18 = 21.33 |
| 2 | Bryce Harmon | 162 | 0 | 43 | 205 | 4 | Deal-25F752, Deal-D73B89, Deal-CA7DC0, Deal-1CCE5C | Emails 162/205 = 79.0%; Calls 0/205 = 0.0%; Meetings 43/205 = 21.0% | 205/4 = 51.25 |
| 3 | Cole Ingram | 96 | 14 | 1 | 111 | 2 | Deal-42326B, Deal-1BA595 | Emails 96/111 = 86.5%; Calls 14/111 = 12.6%; Meetings 1/111 = 0.9% | 111/2 = 55.50 |
| 4 | Farid Osman | 38 | 0 | 34 | 72 | 1 | Deal-499BF6 | Emails 38/72 = 52.8%; Calls 0/72 = 0.0%; Meetings 34/72 = 47.2% | 72/1 = 72.00 |
| 5 | Dana Mercer | 84 | 18 | 11 | 113 | 1 | Deal-57887A | Emails 84/113 = 74.3%; Calls 18/113 = 15.9%; Meetings 11/113 = 9.7% | 113/1 = 113.00 |
| Not ranked | Elena Sinclair | 0 | 0 | 0 | 0 | 0 | None | No activity mix: 0 total activities | Undefined: 0 DS2 entries |

Most efficient rep: Alex Franklin, with 384 total activities / 18 DS2 entries = 21.33 activities per DS2 entry.

Highest-volume rep: Alex Franklin, with 384 total activities.

They do not differ.

Data note: Deal-3EED2C and Deal-57FF13 appear in deals_open.csv but have no matching row in engagements_by_deal_90d.csv, so engagement activity for those deals is missing from the provided engagements table.
ae-qtd-scorecardPASS 1.00

? The Question

Build a QTD scorecard for the named AE as of 2026-09-05: bookings vs quota with attainment percent, new vs expansion split, active pipeline by stage with amount, rolling 90-day DS2-to-won rate, win and loss counts with top loss reason, activity volume by type in the last 30 days. Exclude any closed-won deal dated before the quarter. Close with three coaching observations grounded in the numbers.
cost $0.4790 · 74s · in 21,574 / out 11,791 tok
✓ bookings✓ quota✓ attainment✓ new_expansion_split✓ top_loss_reason✓ coaching_obs

✓ Correct Answer (ground truth)

owner: Alex Franklin
quarter: 2026-Q3
snapshot_date: 2026-09-05
bookings: 150000
quota: 200000
attainment_pct: 75.0
new_bookings: 113500
new_count: 5
expansion_bookings: 36500
expansion_count: 3
pipeline_by_stage: {"DS1": {"count": 20, "amount": 284621}, "DS2": {"count": 28, "amount": 353760}, "DS3": {"count": 67, "amount": 552705}, "DS4": {"count": 5, "amount": 23574}, "DS5": {"count": 5, "amount": 45730}}
open_deals_count: 125
ds2_to_won: window_start=2026-06-07; won_in_window=8; closed_in_window=35; rate=0.2286
wins_qtd: 8
losses_qtd: 27
top_loss_reason: reason=Lost- Timing (1 year or more); count=13
activity_30d_totals: emails_30d=807; calls_30d=112; meetings_30d=128; notes_30d=50
prior_quarter_excluded: deal_alias=Deal-B3E6F1; close_date=2026-06-20; amount=24000
allowed_new: 18 items — e.g. Deal-A1C3E5, Deal-B7D2F4, Deal-C9E1A6, Deal-D4B8C2
graded checks: bookings · quota · attainment · new_expansion_split · top_loss_reason · coaching_obs

Model's actual answer

show the model's full answer
QTD scorecard for Alex Franklin as of 2026-09-05

1. Bookings vs quota

Closed-won deals included QTD only, excluding pre-quarter closed-won Deal-B3E6F1 dated 2026-06-20.

Bookings arithmetic:
Deal-A1C3E5 $40,000
+ Deal-F2C7D8 $20,000
+ Deal-B7D2F4 $35,000
+ Deal-C9E1A6 $21,000
+ Deal-A8B4D6 $12,000
+ Deal-D4B8C2 $11,000
+ Deal-E6F3A9 $6,500
+ Deal-C5D9E2 $4,500
= $150,000 QTD bookings

Quota: $200,000

Attainment arithmetic:
$150,000 / $200,000 = 75.0%

Result: $150,000 booked vs $200,000 quota = 75.0% attainment

2. New vs expansion split

New bookings:
Deal-A1C3E5 $40,000
+ Deal-B7D2F4 $35,000
+ Deal-C9E1A6 $21,000
+ Deal-D4B8C2 $11,000
+ Deal-E6F3A9 $6,500
= $113,500 new

Expansion bookings:
Deal-F2C7D8 $20,000
+ Deal-A8B4D6 $12,000
+ Deal-C5D9E2 $4,500
= $36,500 expansion

Split:
New: $113,500 / $150,000 = 75.7%
Expansion: $36,500 / $150,000 = 24.3%

3. Active pipeline by stage

Active pipeline includes open deals only.

DS1: $284,621 across 20 deals
DS2: $353,760 across 28 deals
DS3: $552,705 across 67 deals
DS4: $23,574 across 5 deals
DS5: $45,730 across 5 deals

Total active pipeline arithmetic:
$284,621 + $353,760 + $552,705 + $23,574 + $45,730 = $1,260,390

4. Rolling 90-day DS2-to-won rate

Rolling 90-day window as of 2026-09-05: deals with entered_ds2 from 2026-06-07 through 2026-09-05.

Deals entered DS2 in window: 111
Won deals from that DS2 cohort: 8

Won aliases:
Deal-A1C3E5, Deal-F2C7D8, Deal-B7D2F4, Deal-C9E1A6, Deal-A8B4D6, Deal-D4B8C2, Deal-E6F3A9, Deal-C5D9E2

Arithmetic:
8 / 111 = 7.2%

Rolling 90-day DS2-to-won rate: 7.2%

5. Win and loss counts with top loss reason

QTD wins: 8
QTD losses: 27

Top loss reason:
Lost- Timing (1 year or more): 13 losses

Loss reason arithmetic:
Lost- Timing (1 year or more): 13
MIA: 5
Competitor: 5
Lost DM: 2
Feature Request: 1
Lost- Does not fit ICP (write in notes): 1

6. Activity volume by type in the last 30 days

Using ae_engagements.csv columns as provided.

Emails: 807
Calls: 112
Meetings: 128
Notes: 50

Total activity arithmetic:
807 + 112 + 128 + 50 = 1,097 total activities

7. Coaching observations grounded in the numbers

1. Alex Franklin is at 75.0% QTD quota attainment with $150,000 booked against a $200,000 quota, leaving $50,000 needed to reach quota.

2. Active pipeline is large at $1,260,390, but it is concentrated in DS3: $552,705 of $1,260,390, or 43.9%. DS5 is only $45,730, which is below the $50,000 remaining quota gap.

3. Loss volume is high relative to wins: 27 losses vs 8 wins QTD. The top loss reason is Lost- Timing (1 year or more) with 13 losses, which is 48.1% of QTD losses.
multithreading-gapPASS 1.00

? The Question

Find every open deal that is single-threaded (fewer than 2 active contacts) or under-threaded (fewer than 3, or all contacts in one persona). Active means engaged in the last 60 days and not marked former. Personas: economic buyer, champion, HR admin, IT security, finance. For each flagged deal: amount, stage, active contact count, personas present, personas missing, the most valuable persona to add given the stage, and any on-file unengaged contact at that company who fits; otherwise say none on file.
cost $0.1936 · 34s · in 6,901 / out 4,825 tok
✓ flagged_recall✓ persona_language✓ single_thread_term✓ on_file_fit

✓ Correct Answer (ground truth)

active_window_start: 2026-07-07
persona_universe: economic buyer, champion, HR admin, IT security, finance
flag_rule: flag if active_contacts < 2 (single-threaded) or active_contacts < 3 or all active contacts share one persona (under-threaded); active = engaged within 60 days and not former
best_persona_rule: DS2/DS3: economic buyer > IT security > finance > HR admin > champion; DS4/DS5: economic buyer > finance > IT security > HR admin > champion; first persona not present among active contacts
flagged: deal_alias=Deal-36C33F; amount=15000; stage=DS2; active_count=1; reason=single-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-50D386; amount=36000; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-5408B0; amount=14850; stage=DS4; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-5BFE3B; amount=18000; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-885F45; amount=9300; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=IT security | deal_alias=Deal-92D97D; amount=60000; stage=DS2; active_count=1; reason=single-threaded; best_persona_to_add=economic buyer … (+5 more)
healthy_not_flagged: Deal-4B0BEB, Deal-84DBA6, Deal-D348E1
allowed_new: 59 items — e.g. C-077A0E, C-0D15DF, C-1BDFE0, C-2AE3AA
graded checks: flagged_recall · persona_language · single_thread_term · on_file_fit

Model's actual answer

show the model's full answer
Data missing: deal amount, stage, and open/closed status are not provided. Therefore, amount = not provided, stage = not provided, and I cannot verify which deals are open. I evaluated all deal aliases present in deal_contacts.csv.

Arithmetic used:
Active contact = last_engaged_date within last 60 days and is_former = false.
Flag if active contact count < 2 OR active contact count < 3 OR all active contacts are in one persona.
Personas checked: economic buyer, champion, HR admin, IT security, finance.

Flagged deals:

1. Deal-EC3025
- Amount: not provided
- Stage: not provided
- Active contact arithmetic: 1 active contact = CT-047C54; CT-F2C1AE excluded because is_former = true
- Active contact count: 1
- Personas present: champion
- Personas missing: economic buyer, HR admin, IT security, finance
- Flag reason: single-threaded because 1 < 2; under-threaded because 1 < 3; all contacts in one persona
- Most valuable persona to add given the stage: cannot determine because stage is not provided
- On-file unengaged contact who fits: CT-6827DB, Chief People Officer, economic buyer

2. Deal-92D97D
- Amount: not provided
- Stage: not provided
- Active contact arithmetic: 1 active contact = CT-01F5B4; CT-A902AE excluded because last_engaged_date = 2026-06-01 is outside last 60 days
- Active contact count: 1
- Personas present: HR admin
- Personas missing: economic buyer, champion, IT security, finance
- Flag reason: single-threaded because 1 < 2; under-threaded because 1 < 3; all contacts in one persona
- Most valuable persona to add given the stage: cannot determine because stage is not provided
- On-file unengaged contact who fits: none on file

3. Deal-50D386
- Amount: not provided
- Stage: not provided
- Active contact arithmetic: 2 active contacts = CT-AA41B2 + CT-B9C35B
- Active contact count: 2
- Personas present: champion, HR admin
- Personas missing: economic buyer, IT security, finance
- Flag reason: under-threaded because 2 < 3
- Most valuable persona to add given the stage: cannot determine because stage is not provided
- On-file unengaged contact who fits: CT-A1C4B3, Chief People Officer, economic buyer

4. Deal-D0D6B5
- Amount: not provided
- Stage: not provided
- Active contact arithmetic: 3 active contacts = CT-87CED4 + CT-DE6D7C + CT-FD70B2
- Active contact count: 3
- Personas present: champion
- Personas missing: economic buyer, HR admin, IT security, finance
- Flag reason: under-threaded because all 3 active contacts are in one persona
- Most valuable persona to add given the stage: cannot determine because stage is not provided
- On-file unengaged contact who fits: CT-1FA4DB, Chief People Officer, economic buyer

5. Deal-5BFE3B
- Amount: not provided
- Stage: not provided
- Active contact arithmetic: 2 active contacts = CT-57123B + CT-5CE757
- Active contact count: 2
- Personas present: champion
- Personas missing: economic buyer, HR admin, IT security, finance
- Flag reason: under-threaded because 2 < 3; all contacts in one persona
- Most valuable persona to add given the stage: cannot determine because stage is not provided
- On-file unengaged contact who fits: none on file

6. Deal-36C33F
- Amount: not provided
- Stage: not provided
- Active contact arithmetic: 1 active contact = CT-4FE556; CT-405B45 and CT-86B22F excluded because is_former = true
- Active contact count: 1
- Personas present: IT security
- Personas missing: economic buyer, champion, HR admin, finance
- Flag reason: single-threaded because 1 < 2; under-threaded because 1 < 3; all contacts in one persona
- Most valuable persona to add given the stage: cannot determine because stage is not provided
- On-file unengaged contact who fits: CT-1DB73E, Chief People Officer, economic buyer

7. Deal-885F45
- Amount: not provided
- Stage: not provided
- Active contact arithmetic: 2 active contacts = CT-51C81E + CT-D9A0E8
- Active contact count: 2
- Personas present: economic buyer, champion
- Personas missing: HR admin, IT security, finance
- Flag reason: under-threaded because 2 < 3
- Most valuable persona to add given the stage: cannot determine because stage is not provided
- On-file unengaged contact who fits: CT-B3F25D, IT Security Lead, IT security

8. Deal-FCBE5B
- Amount: not provided
- Stage: not provided
- Active contact arithmetic: 1 active contact = CT-4A5317
- Active contact count: 1
- Personas present: champion
- Personas missing: economic buyer, HR admin, IT security, finance
- Flag reason: single-threaded because 1 < 2; under-threaded because 1 < 3; all contacts in one persona
- Most valuable persona to add given the stage: cannot determine because stage is not provided
- On-file unengaged contact who fits: none on file

9. Deal-5408B0
- Amount: not provided
- Stage: not provided
- Active contact arithmetic: 2 active contacts = CT-D33AE4 + CT-8742FD
- Active contact count: 2
- Personas present: champion, HR admin
- Personas missing: economic buyer, IT security, finance
- Flag reason: under-threaded because 2 < 3
- Most valuable persona to add given the stage: cannot determine because stage is not provided
- On-file unengaged contact who fits: CT-07FA76, Chief People Officer, economic buyer

10. Deal-C6D97A
- Amount: not provided
- Stage: not provided
- Active contact arithmetic: 3 active contacts = CT-223DDC + CT-B03555 + CT-4E8A2B
- Active contact count: 3
- Personas present: champion
- Personas missing: economic buyer, HR admin, IT security, finance
- Flag reason: under-threaded because all 3 active contacts are in one persona
- Most valuable persona to add given the stage: cannot determine because stage is not provided
- On-file unengaged contact who fits: none on file

11. Deal-F9A08A
- Amount: not provided
- Stage: not provided
- Active contact arithmetic: 1 active contact = CT-931B10; CT-913581 excluded because last_engaged_date = 2026-06-20 is outside last 60 days
- Active contact count: 1
- Personas present: champion
- Personas missing: economic buyer, HR admin, IT security, finance
- Flag reason: single-threaded because 1 < 2; under-threaded because 1 < 3; all contacts in one persona
- Most valuable persona to add given the stage: cannot determine because stage is not provided
- On-file unengaged contact who fits: CT-697541, Chief People Officer, economic buyer
talk-track-from-transcriptsPASS 1.00

? The Question

From this rep's last 10 calls: what they lead with in the first five minutes, how they handle the three most common objections, how often a concrete next step is agreed (rate), and every competitor a prospect raised. Quote at most one line per finding and only lines that appear in the transcripts. End with two coaching notes.
cost $0.1954 · 31s · in 10,081 / out 4,355 tok
✓ objection_count✓ next_step_rate✓ competitors✓ coaching✓ quote_used

✓ Correct Answer (ground truth)

rep: Alex Franklin
n_transcripts: 10
opener_theme: theme=opens with a customer recognition win story (400-person retailer, 18% turnover reduction); count=8; total=10; quote=Quick one to kick off: a 400-person retailer we work with cut regretted turnover 18% in two quarters after automating milestone awards, and I wanted to walk you through exactly how they did it.; transcript_id=TT-001
objections: objection=budget; count=4; quote=Honestly, budget is locked until next fiscal year — I can't add a new line item right now.; transcript_id=TT-001 | objection=timing; count=3; quote=This is good, but can we revisit it next quarter? Open enrollment starts in October and we're underwater.; transcript_id=TT-002 | objection=in-house; count=3; quote=We already do recognition with a spreadsheet and quarterly gift cards — why would we change?; transcript_id=TT-004
objection_count: 3
next_step: agreed=7; total=10; rate=0.7
competitors_prospect_raised: Awardco, Kudos
competitor_quotes: Awardco=We're also in late talks with Awardco — their rewards catalog looks bigger than yours.; Kudos=How are you different from Kudos? Our CEO used them at her last company.
rep_raised_competitors_excluded: Workhuman
allowed_new: 13 items — e.g. TT-001, TT-002, TT-003, TT-004
graded checks: objection_count · next_step_rate · competitors · coaching · quote_used

Model's actual answer

show the model's full answer
First five minutes: what Alex Franklin leads with

- 8 of 10 calls lead with the same customer-outcome story: a 400-person retailer, 18% regretted-turnover reduction, two quarters, automated milestone awards.
  Quote: “Quick one to kick off: a 400-person retailer we work with cut regretted turnover 18% in two quarters after automating milestone awards, and I wanted to walk you through exactly how they did it.”
  Deals: Deal-D348E1, Deal-5408B0, Deal-547B2B, Deal-C61CF7, Deal-60C2C2, Deal-EDC141, Deal-D9A12F, Deal-84DBA6

- 1 of 10 calls leads with agenda/security/pricing.
  Quote: “I put together a short agenda — security review first, then pricing.”
  Deal: Deal-403845

- 1 of 10 calls leads with straight pricing.
  Quote: “You asked for straight pricing last time, so let's start there.”
  Deal: Deal-1E2498

Arithmetic: 8 + 1 + 1 = 10 calls.

Three most common objections and how he handles them

1. Budget locked until next fiscal year
   - Frequency: 4 of 10 calls
   - Deals: Deal-D348E1, Deal-547B2B, Deal-60C2C2, Deal-84DBA6
   - Handling: reframes funding around turnover savings and avoided backfills.
   - Quote: “Totally fair. Most teams fund this out of turnover savings — that retailer saved about $210k in avoided backfills, which is how their finance team signed off.”

2. Revisit next quarter / timing conflict
   - Frequency: 3 of 10 calls
   - Deals: Deal-5408B0, Deal-C61CF7, Deal-D9A12F
   - Handling: proposes a 90-day pilot with one department to create internal data before planning.
   - Quote: “Makes sense. What if we scope a 90-day pilot with one department so you have internal data before next quarter's planning?”

3. Existing spreadsheet and quarterly gift cards
   - Frequency: 3 of 10 calls
   - Deals: Deal-403845, Deal-EDC141, Deal-1E2498
   - Handling: contrasts manual recognition with automation, milestone triggers, and recognition analytics.
   - Quote: “Spreadsheets work until they scale — the difference is automation: milestones fire without HR lifting a finger, and you get analytics on who is being recognized.”

Concrete next-step agreement rate

- Agreed next step: 7 calls
  Deals: Deal-D348E1, Deal-5408B0, Deal-547B2B, Deal-C61CF7, Deal-60C2C2, Deal-D9A12F, Deal-1E2498

- No concrete next step agreed: 3 calls
  Deals: Deal-403845, Deal-EDC141, Deal-84DBA6

Arithmetic: 7 agreed / 10 total calls = 70%.

Every competitor a prospect raised

- Awardco
  Deal: Deal-547B2B
  Quote: “We're also in late talks with Awardco — their rewards catalog looks bigger than yours.”

- Kudos
  Deal: Deal-EDC141
  Quote: “How are you different from Kudos? Our CEO used them at her last company.”

No prospect raised Workhuman. Workhuman appears only in a Alex Franklin line in Deal-C61CF7.

Coaching notes

1. The customer-outcome opener is consistent and appears tied to many of the calls with next-step agreement, but the data provided is not enough to prove causation.

2. When the call reaches the standard next-step ask, prospects often agree; in the stalled calls, Alex Franklin does not consistently re-anchor to a concrete next step after committee/no-urgency responses.
reporting-analytics
q3-weighted-forecastPASS 1.00

? The Question

Below are the open deals in the default sales pipeline with a close date between 2026-07-01 and 2026-10-15, pulled from the CRM warehouse on 2026-09-05. Stages: DS1 (discovery) through DS5 (contract). Build the Q3 2026 forecast (quarter = 2026-07-01 to 2026-09-30). Weighting rule: 100 percent of COMMIT plus 35 percent of BEST_CASE. PIPELINE deals count zero. Only deals with a close date inside the quarter count. Report: COMMIT total, BEST_CASE total, weighted forecast, count of deals in each category inside the quarter, the deals you excluded for being outside the quarter (count and total amount), and the top 5 BEST_CASE deals by amount inside the quarter. Show arithmetic. Then, in a final '## Data quality' section of no more than four sentences, name the data quality problems in this extract that would make the forecast unreliable if it were run unattended.
cost $0.2652 · 31s · in 22,775 / out 4,764 tok
✓ commit_total_44729✓ best_case_total_203565✓ weighted_115976.75✓ commit_count_7✓ best_case_count_24✓ excluded_count_32✓ excluded_amount_227575✓ top5_aliases✓ flags_blank_owner✓ flags_missing_why_buys

✓ Correct Answer (ground truth)

graded checks: commit_total_44729 · best_case_total_203565 · weighted_115976.75 · commit_count_7 · best_case_count_24 · excluded_count_32 · excluded_amount_227575 · top5_aliases · flags_blank_owner · flags_missing_why_buys

Model's actual answer

show the model's full answer
## Q3 2026 forecast

Quarter filter: close_date from 2026-07-01 through 2026-09-30.

COMMIT total: 44,729
Arithmetic: 11,200 + 9,000 + 9,000 + 6,360 + 5,400 + 2,520 + 1,249 = 44,729

BEST_CASE total: 203,565
Arithmetic: sum of 24 BEST_CASE deals inside Q3 = 203,565

Weighted forecast: 115,976.75
Arithmetic: 100% of COMMIT + 35% of BEST_CASE = 44,729 + (203,565 × 0.35) = 44,729 + 71,247.75 = 115,976.75

## Deal counts inside Q3

COMMIT: 7 deals
BEST_CASE: 24 deals
PIPELINE: 23 deals

## Excluded for being outside Q3

Excluded deal count: 32
Excluded amount total: 227,575
Arithmetic: sum of all deals with close_date after 2026-09-30 = 227,575

Excluded deals:
- Deal-E51FB7: 43,875, close_date 2026-10-01
- Deal-B936FE: 18,000, close_date 2026-10-09
- Deal-D9A12F: 17,000, close_date 2026-10-15
- Deal-D348E1: 13,770, close_date 2026-10-15
- Deal-4062CF: 10,800, close_date 2026-10-15
- Deal-293AF3: 9,000, close_date 2026-10-09
- Deal-034D49: 9,000, close_date 2026-10-15
- Deal-E0ADD8: 7,920, close_date 2026-10-15
- Deal-9F2E43: 7,690, close_date 2026-10-08
- Deal-FCBE5B: 7,500, close_date 2026-10-07
- Deal-712010: 7,200, close_date 2026-10-15
- Deal-6691E0: 5,700, close_date 2026-10-15
- Deal-C61CF7: 5,400, close_date 2026-10-09
- Deal-600CD9: 5,400, close_date 2026-10-02
- Deal-A92065: 5,400, close_date 2026-10-15
- Deal-1D532E: 5,400, close_date 2026-10-15
- Deal-48B656: 5,160, close_date 2026-10-15
- Deal-E531A6: 4,800, close_date 2026-10-15
- Deal-D1E6C2: 4,400, close_date 2026-10-09
- Deal-D9E112: 4,300, close_date 2026-10-09
- Deal-5AD94B: 4,000, close_date 2026-10-15
- Deal-901332: 3,600, close_date 2026-10-15
- Deal-47AE31: 3,600, close_date 2026-10-09
- Deal-15D24F: 3,600, close_date 2026-10-09
- Deal-766C74: 3,300, close_date 2026-10-14
- Deal-ED725A: 2,400, close_date 2026-10-08
- Deal-8AD4A5: 1,800, close_date 2026-10-07
- Deal-D7E999: 1,800, close_date 2026-10-15
- Deal-ED13B0: 1,680, close_date 2026-10-09
- Deal-5FDCE4: 1,600, close_date 2026-10-01
- Deal-7FA0C3: 1,400, close_date 2026-10-01
- Deal-F5A622: 1,080, close_date 2026-10-08

## Top 5 BEST_CASE deals inside Q3

1. Deal-2D7423: 38,935, DS3, close_date 2026-09-30
2. Deal-25F752: 24,000, DS4, close_date 2026-09-25
3. Deal-E53952: 19,656, DS4, close_date 2026-09-30
4. Deal-5EED42: 16,250, DS3, close_date 2026-09-30
5. Deal-FA32A0: 11,116, DS3, close_date 2026-09-25

## Data quality

Owner is missing on 85 of 86 deals. why_buys_chars is 0 on 71 of 86 deals, so most deals lack buyer rationale in this extract. Forecast category and stage are inconsistent for some Q3 deals: Deal-A5E80A is COMMIT at DS1 and Deal-499BF6 is COMMIT at DS2. Amount format is inconsistent because Deal-42326B has a decimal amount of 2480.4 while the other amounts are whole numbers.
aha-moment-2x20.83

? The Question

Test the activation hypothesis on the company cohort below: companies with 5+ unique givers (m1_users >= 5) AND 1+ successful redemption (m1_redemptions >= 1) in their first calendar month retain better at 24 months than those with one signal or neither. Every company in this extract is already 25+ months old, so current_status = 'active' means retained at 24 months. Report the full 2x2 (cohort size and 24-month retention rate for: both signals, givers-only, redemption-only, neither), state how many companies were excluded from the denominator and why (if none, say so), name the single signal with the largest retention lift, and state what this does and does not prove.
cost $0.4740 · 67s · in 23,628 / out 11,264 tok
✓ both_66✓ givers_47✓ redemp_31✓ neither_40✓ excluded_0✗ largest_lift_givers

✓ Correct Answer (ground truth)

graded checks: both_66 · givers_47 · redemp_31 · neither_40 · excluded_0 · largest_lift_givers

Model's actual answer

show the model's full answer
2x2 activation cohort:

| m1_users >= 5 | m1_redemptions >= 1 | Cohort | Retained at 24 months | 24-month retention |
|---|---:|---:|---:|---:|
| Yes | Yes | both signals | 47 | 31 / 47 = 65.96% |
| Yes | No | givers-only | 49 | 23 / 49 = 46.94% |
| No | Yes | redemption-only | 29 | 9 / 29 = 31.03% |
| No | No | neither | 95 | 38 / 95 = 40.00% |

Excluded from denominator: 0 companies. No exclusions were needed because all 220 companies had usable `current_status`, `m1_users`, and `m1_redemptions` values.

Single-signal retention lift:

- Giver signal: `m1_users >= 5`
  - With signal: both signals + givers-only = 47 + 49 = 96 companies
  - Retained: 31 + 23 = 54
  - Retention: 54 / 96 = 56.25%
  - Without signal: redemption-only + neither = 29 + 95 = 124 companies
  - Retained: 9 + 38 = 47
  - Retention: 47 / 124 = 37.90%
  - Lift: 56.25% - 37.90% = 18.35 percentage points

- Redemption signal: `m1_redemptions >= 1`
  - With signal: both signals + redemption-only = 47 + 29 = 76 companies
  - Retained: 31 + 9 = 40
  - Retention: 40 / 76 = 52.63%
  - Without signal: givers-only + neither = 49 + 95 = 144 companies
  - Retained: 23 + 38 = 61
  - Retention: 61 / 144 = 42.36%
  - Lift: 52.63% - 42.36% = 10.27 percentage points

Largest single-signal lift: `m1_users >= 5`, with +18.35 percentage points.

What this proves: in this extract, companies with both activation signals retained better at 24 months than givers-only, redemption-only, or neither: 65.96% vs 46.94%, 31.03%, and 40.00%.

What this does not prove: it does not prove the signals caused retention. The extract does not control for company size, tier, industry, geography, integrations, sales motion, customer quality, or other confounders, and no statistical significance test was requested or performed.
arr-rollforward-reconciliationPASS 1.00

? The Question

Reconcile ARR between CRM company records and billing subscriptions as of 2026-09-05. Billing ARR is MRR x 12. Report both totals, the variance, and a decomposition into buckets (status mismatch, rounding, missing records, other) that sums exactly to the variance. List each mismatched account with a suggested owner. Business rule: any subscription with a term other than 12 months must have cf_agreement_end_date populated; list violations.
cost $0.2157 · 31s · in 19,577 / out 3,679 tok
✓ billing_total✓ crm_total✓ variance✓ buckets_sum✓ violations_listed

✓ Correct Answer (ground truth)

as_of: 2026-09-05
billing_arr_total: 604739.28
crm_arr_total: 603581.76
variance: 1157.52
variance_definition: billing_arr_total - crm_arr_total (billing ARR = sum of MRR x 12 over status=active subscriptions)
buckets: {"status_mismatch": {"amount": "-13158.48", "accounts": [{"company_alias": "C-0C8323BF", "billing_arr": "0.00", "hubspot_arr": "4905.24", "diff": "-4905.24", "suggested_owner": "Dana Mercer"}, {"compa
bucket_sum_equals_variance: True
n_mismatched_accounts: 7
rule: term_months != 12 requires cf_agreement_end_date
violations: subscription_id=SUB-0002; company_alias=C-1794A52C; term_months=24 | subscription_id=SUB-0019; company_alias=C-22170CA1; term_months=36
n_companies: 40
graded checks: billing_total · crm_total · variance · buckets_sum · violations_listed

Model's actual answer

show the model's full answer
Billing ARR total (active subscriptions only): $604,739.28
CRM company ARR total: $603,581.76
Variance (CRM - Billing): -$1,157.52

Arithmetic:
Billing ARR = sum(active MRR x 12) = $604,739.28
CRM ARR = sum(hubspot_arr) = $603,581.76
Variance = $603,581.76 - $604,739.28 = -$1,157.52

Variance decomposition, CRM - Billing:
Status mismatch: +$13,158.48
  C-0C8323BF: CRM $4,905.24 - Billing $0.00 because SUB-000E is cancelled = +$4,905.24
  C-0DC4FB8C: CRM $8,253.24 - Billing $0.00 because SUB-000F is cancelled = +$8,253.24

Rounding: +$36.00
  C-0D66DF9E: CRM $23,200.00 - Billing $23,184.00 = +$16.00
  C-14D70CE0: CRM $18,200.00 - Billing $18,180.00 = +$20.00

Missing records: -$11,952.00
  C-21629AA4: CRM $0.00 - Billing $28,449.24 = -$28,449.24
  C-0D5BBE3A: CRM $16,497.24 - Billing $0.00 = +$16,497.24
  Net missing records = -$28,449.24 + $16,497.24 = -$11,952.00

Other: -$2,400.00
  C-0F7269D7: CRM $24,396.00 - Billing $26,796.00 = -$2,400.00

Check:
+$13,158.48 + $36.00 - $11,952.00 - $2,400.00 = -$1,157.52

Mismatched accounts and suggested owner:
C-0C8323BF: status mismatch; suggested owner: RevOps / Billing Ops
C-0DC4FB8C: status mismatch; suggested owner: RevOps / Billing Ops
C-0D66DF9E: rounding; suggested owner: RevOps
C-14D70CE0: rounding; suggested owner: RevOps
C-21629AA4: missing CRM company record; suggested owner: RevOps / CRM Admin
C-0D5BBE3A: missing billing subscription record; suggested owner: Billing Ops
C-0F7269D7: other ARR mismatch; suggested owner: RevOps / Billing Ops

Agreement end date violations:
SUB-0002 / C-1794A52C: term_months = 24, cf_agreement_end_date is missing
SUB-0019 / C-22170CA1: term_months = 36, cf_agreement_end_date is missing
kvm-month-over-monthPASS 1.00

? The Question

Report the four core KVMs for 2026-08 against 2026-07: giving rate, redemptions per user, 1:1 meetings engagement, pulse check engagement. For each: value, prior value, absolute and relative change, direction. Name the KVM with the largest relative move and the segment (plan_tier or size_band) driving it if the data supports that; otherwise say the data does not. Under 200 words plus one table.
cost $0.2204 · 25s · in 21,013 / out 3,581 tok
✓ four_kvms✓ biggest_mover_named✓ direction_correct✓ segment_named✓ word_cap

✓ Correct Answer (ground truth)

aggregation: simple mean of company-level values per month
kvms: {"giving_rate": {"2026-07": 0.6023, "2026-08": 0.6027, "abs_change": 0.0004, "rel_change_pct": 0.07, "direction": "up"}, "redemptions_per_user": {"2026-07": 1.73, "2026-08": 1.7302, "abs_change": 0.00
biggest_mover: pulse_engagement
driving_segment: size_band=enterprise; 2026-07=0.55; 2026-08=0.2743; rel_change_pct=-50.13
segment_detail: {"smb": {"2026-07": 0.6588, "2026-08": 0.6573, "rel_change_pct": -0.23}, "mid_market": {"2026-07": 0.593, "2026-08": 0.5942, "rel_change_pct": 0.2}, "enterprise": {"2026-07": 0.55, "2026-08": 0.2743,
n_company_rows_per_month: 30
graded checks: four_kvms · biggest_mover_named · direction_correct · segment_named · word_cap

Model's actual answer

show the model's full answer
Arithmetic: unweighted company average by month; absolute change = 2026-08 value - 2026-07 value; relative change = absolute change / 2026-07 value.

| KVM | 2026-08 value | 2026-07 prior | Absolute change | Relative change | Direction |
|---|---:|---:|---:|---:|---|
| giving rate | 0.6027 | 0.6023 | 0.6027 - 0.6023 = +0.0004 | +0.0004 / 0.6023 = +0.07% | Up |
| redemptions per user | 1.7302 | 1.7300 | 1.7302 - 1.7300 = +0.0002 | +0.0002 / 1.7300 = +0.01% | Up |
| 1:1 meetings engagement | 0.4472 | 0.4469 | 0.4472 - 0.4469 = +0.0003 | +0.0003 / 0.4469 = +0.06% | Up |
| pulse check engagement | 0.5086 | 0.6006 | 0.5086 - 0.6006 = -0.0920 | -0.0920 / 0.6006 = -15.31% | Down |

Largest relative move: pulse check engagement, down 15.31%. The data supports size_band as the driver: enterprise pulse engagement fell from 0.5500 to 0.2743, or -0.2757 / 0.5500 = -50.13%. plan_tier does not support segmentation because all rows are tier_three.
rewards-ytd-sectionPASS 1.00

? The Question

Produce the redemption section of the weekly rewards report through the last completed month (2026-08): redemption count, spend, unique redeemers, redemptions per redeemer, provider mix as percent of spend, top 5 countries by redemptions. Say which month is the last completed month. Provider shares must sum to 100.
cost $0.7239 · 165s · in 33,855 / out 17,285 tok
✓ count✓ spend✓ unique_redeemers✓ month_named✓ provider_mix✓ top_country

✓ Correct Answer (ground truth)

redemption_count: 378
total_spend_usd: 27846.00
unique_redeemers: 235
redemptions_per_redeemer: 1.61
provider_mix_pct_of_spend: TangoCard=11.6; Tremendous=30.54; custom=39.05; Snappy=18.81
top5_countries_by_count: country=US; count=244 | country=CA; count=24 | country=AU; count=21 | country=GB; count=17 | country=NL; count=17
graded checks: count · spend · unique_redeemers · month_named · provider_mix · top_country

Model's actual answer

show the model's full answer
Redemption section — through last completed month: 2026-08

Arithmetic:
- Redemption count = 378 redemptions
- Spend = sum(amount_usd) = $27,846.00
- Unique redeemers = distinct user_key count = 235
- Redemptions per redeemer = 378 / 235 = 1.61

Provider mix by spend:
- TangoCard: $3,230.00 / $27,846.00 = 11.6%
- Tremendous: $8,505.00 / $27,846.00 = 30.5%
- custom: $10,873.00 / $27,846.00 = 39.0%
- Snappy: $5,238.00 / $27,846.00 = 18.9%

Provider share total = 11.6% + 30.5% + 39.0% + 18.9% = 100.0%

Top 5 countries by redemptions:
1. US: 244
2. CA: 24
3. AU: 21
4. GB: 17
5. NL: 17
customer-success
churn-save-eligibilityPASS 1.00

? The Question

Which at-risk accounts qualify for a churn-save offer under the documented eligibility rules, what amount is at stake per account and in total, and which play fits each (usage revival, executive touch, commercial concession)? Cite the signal that justifies each play. List accounts that look at risk but do not qualify and why.
cost $0.1440 · 29s · in 5,487 / out 3,424 tok
✓ eligible_set✓ total_at_stake✓ plays_cited✓ noneligible_named✓ rules_applied

✓ Correct Answer (ground truth)

snapshot_date: 2026-09-05
rules: health_score < 60, churn_save_eligible_amount > 0, renewal within 120 days of snapshot
eligible: account_alias=C-0F6C0F34; amount_at_stake=49707.00; play=executive touch; justifying_signal=champion_active is false - no executive sponsor engaged | account_alias=C-0B827671; amount_at_stake=25365.00; play=usage revival; justifying_signal=usage_trend_3m=declining over the last 3 months | account_alias=C-0B360C78; amount_at_stake=35748.00; play=commercial concession; justifying_signal=usage stable/growing with seat utilization 75% - risk is commercial, not adoption | account_alias=C-0B0F1BAB; amount_at_stake=5494.00; play=executive touch; justifying_signal=champion_active is false - no executive sponsor engaged | account_alias=C-0CA21961; amount_at_stake=16829.00; play=usage revival; justifying_signal=seat utilization 26% is below 50% | account_alias=C-0E9C27D1; amount_at_stake=41235.00; play=commercial concession; justifying_signal=usage stable/growing with seat utilization 85% - risk is commercial, not adoption … (+2 more)
total_amount_at_stake: 224601.00
non_eligible_at_risk: account_alias=C-0BC71BDD; health_score=55 | account_alias=C-0BA71F12; health_score=52 | account_alias=C-0F6694C3; health_score=43 | account_alias=C-0BE96399; health_score=54 | account_alias=C-0F876796; health_score=47 | account_alias=C-0FCCD2DF; health_score=43 … (+1 more)
n_accounts: 30
graded checks: eligible_set · total_at_stake · plays_cited · noneligible_named · rules_applied

Model's actual answer

show the model's full answer
Eligibility rules applied:
R1 health_score < 60
R2 churn_save_eligible_amount > 0
R3 renewal_date within 120 days of 2026-09-05, so through 2027-01-03

Qualifying at-risk churn-save accounts:

| account_alias | Amount at stake | Renewal timing | Play | Signal justifying play |
|---|---:|---|---|---|
| C-0F6C0F34 | $49,707.00 | 2026-10-03 = 28 days | executive touch | champion_active = false |
| C-0B827671 | $25,365.00 | 2026-11-14 = 70 days | usage revival | usage_trend_3m = declining; seats_used/seats = 113/202 = 55.9% |
| C-0B360C78 | $35,748.00 | 2026-10-28 = 53 days | commercial concession | churn_save_eligible_amount = $35,748.00; usage_trend_3m = growing; champion_active = true |
| C-0B0F1BAB | $5,494.00 | 2026-09-23 = 18 days | executive touch | champion_active = false |
| C-0CA21961 | $16,829.00 | 2026-12-28 = 114 days | usage revival | usage_trend_3m = flat; seats_used/seats = 84/325 = 25.8% |
| C-0E9C27D1 | $41,235.00 | 2026-09-24 = 19 days | commercial concession | churn_save_eligible_amount = $41,235.00; seats_used/seats = 134/157 = 85.4%; champion_active = true |
| C-0CEF69FD | $32,621.00 | 2026-11-21 = 77 days | executive touch | champion_active = false |
| C-0D3278C7 | $17,602.00 | 2026-11-12 = 68 days | usage revival | usage_trend_3m = declining; seats_used/seats = 126/380 = 33.2% |

Arithmetic for total amount at stake:
$49,707.00 + $25,365.00 + $35,748.00 + $5,494.00 + $16,829.00 + $41,235.00 + $32,621.00 + $17,602.00 = $224,601.00

Total qualified churn-save amount at stake: $224,601.00

At-risk accounts that do not qualify:

| account_alias | Why it looks at risk | Why it does not qualify |
|---|---|---|
| C-0BC71BDD | health_score = 55, below 60 | churn_save_eligible_amount = $0.00, fails R2 |
| C-0BA71F12 | health_score = 52, below 60 | renewal_date = 2027-04-11, 218 days from snapshot, fails R3 |
| C-0F6694C3 | health_score = 43, below 60 | churn_save_eligible_amount = $0.00, fails R2; renewal_date = 2027-03-21, 197 days from snapshot, fails R3 |
| C-0BE96399 | health_score = 54, below 60 | churn_save_eligible_amount = $0.00, fails R2 |
| C-0F876796 | health_score = 47, below 60 | renewal_date = 2027-02-06, 154 days from snapshot, fails R3 |
| C-0FCCD2DF | health_score = 43, below 60 | churn_save_eligible_amount = $0.00, fails R2; renewal_date = 2027-04-23, 230 days from snapshot, fails R3 |
| C-10A56B0F | health_score = 54, below 60 | churn_save_eligible_amount = $0.00, fails R2 |
expansion-kitPASS 1.00

? The Question

A customer replied to outreach (reply included). Build the expansion kit: seat coverage (licensed over headcount), two lines on usage health, headroom in seats and ARR at the current per-seat rate, who replied and whether they can buy, the right buyer from our contacts, and a reply email under 150 words that moves toward the right person without being pushy and cites one real usage fact.
cost $0.0440 · 9s · in 3,000 / out 523 tok
✓ coverage_pct✓ headroom_arr✓ right_buyer✓ replier_not_buyer✓ usage_fact✓ email_under_150

✓ Correct Answer (ground truth)

company_alias: C-0DDFC9A7
coverage_pct: 37.5
headroom_seats: 250
per_seat_annual_rate: 60.00
headroom_arr: 15000.00
replier: Maria S.
replier_can_buy: False
right_buyer: name=Dana R.; title=VP People
usage_facts: active_users_2026_08=126; active_users_2026_03=88; growth_pct_6m=43.2; licensed_seat_utilization_pct=84.0
usage_health_lines: Monthly active users grew from 88 (2026-03) to 126 (2026-08), +43.2% over 6 months., Last month 126 of 150 licensed seats were active (84.0% utilization), approaching the seat cap.
graded checks: coverage_pct · headroom_arr · right_buyer · replier_not_buyer · usage_fact · email_under_150

Model's actual answer

show the model's full answer
Seat coverage:
150 licensed seats / 400 headcount = 37.5% coverage.

Usage health:
Monthly active users rose every month from 88 in 2026_03 to 126 in 2026_08.
2026_08 usage is 126 / 150 licensed seats = 84.0% of licensed capacity.

Headroom:
Seat headroom to full headcount = 400 headcount - 150 licensed seats = 250 seats.
Current per-seat ARR = $9,000.00 / 150 seats = $60.00 per seat.
ARR headroom at current rate = 250 seats * $60.00 = $15,000.00.

Who replied / can they buy:
Maria S., People Operations Coordinator, replied.
She explicitly said she is not the right person for purchasing decisions.

Right buyer:
Dana R., VP People.

Reply email:
Hi Maria,

Thanks — really glad to hear the team is loving Bonusly, and I appreciate you pointing me in the right direction.

One useful data point: monthly active users have climbed every month, from 88 in March to 126 in August, which puts the team at 84% of current licensed seats.

Since Dana R. owns budget and seat expansion, would you be open to introducing us? I can keep it lightweight and share the usage snapshot so she has the context she’s already been asking about.

Best,
Cole
onboarding-statusPASS 1.00

? The Question

Prep the CSM for the mid-onboarding call: what is complete (integrations connected, allowance set, admins added, first recognition given, first redemption), what is not, early engagement signals, and the three things to cover on the call. Mark nothing complete without a data field that shows it.
cost $0.0788 · 15s · in 3,842 / out 1,543 tok
✓ complete_items✓ incomplete_items✓ three_items✓ grounded

✓ Correct Answer (ground truth)

company_alias: C-0D284E42
days_live: 25
checklist: {"slack_integration_connected": {"complete": true, "field": "integration_slack", "value": "2026-08-12"}, "hris_integration_connected": {"complete": false, "field": "integration_hris", "value": ""}, "a
engagement_signals: avg_active_givers_week1=4.3; avg_active_givers_last_week=13.0; latest_active_givers=15; days_with_activity=25
call_items: Connect the HRIS integration (integration_hris is empty) to automate user provisioning and offboarding., Get to first redemption (first_redemption_at is empty): walk admins through the rewards catalog and confirm allowance funding., Build the giving habit: daily active givers averaged 4.3 in week 1 vs 13.0 last week - agree on manager nudges and a recognition cadence.
graded checks: complete_items · incomplete_items · three_items · grounded

Model's actual answer

show the model's full answer
CSM mid-onboarding prep for C-0D284E42

Complete — only where a data field shows it:
- Slack integration connected: complete.
  - integration_slack = 2026-08-12
  - Arithmetic: 2026-08-12 - signup_date 2026-08-11 = 1 day after signup
- Allowance set: complete.
  - allowance_set = 2026-08-13
  - Arithmetic: 2026-08-13 - signup_date 2026-08-11 = 2 days after signup
- Admins added: complete.
  - admins_added = 2
- First recognition given: complete.
  - first_recognition_at = 2026-08-15 14:22
  - Arithmetic: 2026-08-15 14:22 - signup_date 2026-08-11 = 4 days, 14:22 after signup

Not complete / not shown complete:
- HRIS integration connected: not complete / missing.
  - integration_hris is blank
- First redemption: not complete / missing.
  - first_redemption_at is blank

Early engagement signals:
- Active givers increased from 3 on 2026-08-11 to 15 on 2026-09-04.
  - Arithmetic: 15 - 3 = +12 active givers
  - Percentage increase: 12 / 3 = 400%
- 25 days of usage data are provided.
- Total active-giver daily count across the period = 214.
  - Arithmetic: sum of daily active_givers = 214
- Average daily active givers across the full period = 8.56.
  - Arithmetic: 214 / 25 = 8.56
- First 7-day average active givers = 4.29.
  - Arithmetic: (3+3+4+4+5+4+7) / 7 = 30 / 7 = 4.29
- Last 7-day average active givers = 13.00.
  - Arithmetic: (11+13+11+13+13+15+15) / 7 = 91 / 7 = 13.00
- Last 7-day average vs. first 7-day average increased by 8.71 active givers.
  - Arithmetic: 13.00 - 4.29 = 8.71
  - Percentage increase: 8.71 / 4.29 = 203.33%
- Peak active givers = 15 on 2026-09-03 and 2026-09-04.
- Engagement trend is upward, but with some day-to-day variability:
  - Increases: 11 day-over-day changes
  - Decreases: 6 day-over-day changes
  - Flat: 7 day-over-day changes

Three things to cover on the call:
1. HRIS integration: confirm why integration_hris is blank and agree on the next step to connect it.
2. First redemption: first_redemption_at is blank, so confirm whether users understand how to redeem and whether enablement or nudges are needed.
3. Expansion of early engagement: active givers grew from 3 to 15, so review what drove adoption and how to broaden participation beyond the current active givers.
renewal-risk-conflicting-datesPASS 1.00

? The Question

Produce a 90-day renewal risk brief. Two systems hold renewal dates and disagree on some accounts; decide which to trust per account and say why (multi-year contracts are known to be wrong in ChurnZero). For every renewal: company, CSM, ARR, date used, seat utilization, 3-month usage trend, risk rating with one sentence of evidence. Flag every disagreement. Close with total ARR renewing and ARR at risk.
cost $0.3541 · 58s · in 13,465 / out 9,029 tok
✓ total_renewing✓ arr_at_risk✓ disagreements_flagged✓ trust_rule

✓ Correct Answer (ground truth)

snapshot_date: 2026-09-05
window: 2026-09-05 to 2026-12-04
trust_rule: multi-year contracts: Chargebee is authoritative (ChurnZero known wrong); otherwise systems agree or Chargebee wins
accounts: 20 items — e.g. account_alias=C-0B144C78; csm=Cole Ingram; arr=30899.00; trusted_renewal_date=2026-11-02; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=75.4; usage_3m_ratio=1.03; risk=low; evidence=3-month usage ratio 1.03 (last3 avg 103 vs prior3 100), seat utilization 75% | account_alias=C-0B20DB64; csm=Dana Mercer; arr=21770.00; trusted_renewal_date=2026-10-07; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=56.6; usage_3m_ratio=1.0; risk=medium; evidence=3-month usage ratio 1.00 (last3 avg 295 vs prior3 295), seat utilization 57% | account_alias=C-0B344485; csm=Elena Sinclair; arr=64384.00; trusted_renewal_date=2026-11-16; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=78.0; usage_3m_ratio=1.04; risk=low; evidence=3-month usage ratio 1.04 (last3 avg 241 vs prior3 231), seat utilization 78% | account_alias=C-0B369871; csm=Dana Mercer; arr=85128.00; trusted_renewal_date=2026-10-29; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=75.1; usage_3m_ratio=1.04; risk=low; evidence=3-month usage ratio 1.04 (last3 avg 330 vs prior3 316), seat utilization 75%
disagreements: account_alias=C-0B7D2C30; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-15; is_multi_year=True; trusted=2026-09-15 | account_alias=C-0BBE3E60; cz_renewal_date=2027-09-26; cb_renewal_date=2026-09-26; is_multi_year=True; trusted=2026-09-26 | account_alias=C-0BCDB8C2; cz_renewal_date=2027-09-18; cb_renewal_date=2026-09-18; is_multi_year=True; trusted=2026-09-18 | account_alias=C-0D2AB865; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-22; is_multi_year=True; trusted=2026-09-22 | account_alias=C-0F5D2323; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-29; is_multi_year=True; trusted=2026-09-29
n_disagreements: 5
total_arr_renewing_90d: 1048715.00
arr_at_risk_90d: 359409.00
n_accounts: 20
graded checks: total_renewing · arr_at_risk · disagreements_flagged · trust_rule

Model's actual answer

show the model's full answer
90-day renewal risk brief

Risk rubric used from provided data only: High = seat utilization <50% OR 3-month usage decline >=10%; Medium = seat utilization <70% OR any usage decline; Low = otherwise.
3-month trend = 2026-06 -> 2026-07 -> 2026-08.

Disagreements flagged:
- C-0B7D2C30: ChurnZero 2026-09-10 vs Chargebee 2026-09-15; used Chargebee because is_multi_year=true and multi-year contracts are known wrong in ChurnZero.
- C-0BCDB8C2: ChurnZero 2027-09-18 vs Chargebee 2026-09-18; used Chargebee because is_multi_year=true and multi-year contracts are known wrong in ChurnZero.
- C-0D2AB865: ChurnZero 2026-09-10 vs Chargebee 2026-09-22; used Chargebee because is_multi_year=true and multi-year contracts are known wrong in ChurnZero.
- C-0BBE3E60: ChurnZero 2027-09-26 vs Chargebee 2026-09-26; used Chargebee because is_multi_year=true and multi-year contracts are known wrong in ChurnZero.
- C-0F5D2323: ChurnZero 2026-09-10 vs Chargebee 2026-09-29; used Chargebee because is_multi_year=true and multi-year contracts are known wrong in ChurnZero.

Renewals

1. C-0B7D2C30 | CSM: Dana Mercer | ARR: $65,901 | Date used: 2026-09-15
   Seat utilization: 274 / 476 = 57.6%
   3-month usage trend: 97 -> 94 -> 84; 84 - 97 = -13 (-13 / 97 = -13.4%)
   Risk: High — Usage declined 13.4% over the last 3 months and utilization is only 57.6%.

2. C-0BCDB8C2 | CSM: Cole Ingram | ARR: $54,427 | Date used: 2026-09-18
   Seat utilization: 232 / 424 = 54.7%
   3-month usage trend: 127 -> 118 -> 110; 110 - 127 = -17 (-17 / 127 = -13.4%)
   Risk: High — Usage declined 13.4% over the last 3 months and utilization is only 54.7%.

3. C-0D2AB865 | CSM: Elena Sinclair | ARR: $38,022 | Date used: 2026-09-22
   Seat utilization: 250 / 407 = 61.4%
   3-month usage trend: 125 -> 117 -> 109; 109 - 125 = -16 (-16 / 125 = -12.8%)
   Risk: High — Usage declined 12.8% over the last 3 months despite 61.4% seat utilization.

4. C-0BBE3E60 | CSM: Dana Mercer | ARR: $30,993 | Date used: 2026-09-26
   Seat utilization: 74 / 114 = 64.9%
   3-month usage trend: 39 -> 35 -> 33; 33 - 39 = -6 (-6 / 39 = -15.4%)
   Risk: High — Usage declined 15.4% over the last 3 months despite 64.9% seat utilization.

5. C-0F5D2323 | CSM: Cole Ingram | ARR: $90,647 | Date used: 2026-09-29
   Seat utilization: 111 / 390 = 28.5%
   3-month usage trend: 20 -> 21 -> 18; 18 - 20 = -2 (-2 / 20 = -10.0%)
   Risk: High — Utilization is very low at 28.5% and usage declined 10.0% over the last 3 months.

6. C-0EC6999D | CSM: Elena Sinclair | ARR: $79,419 | Date used: 2026-10-03
   Seat utilization: 31 / 112 = 27.7%
   3-month usage trend: 17 -> 16 -> 15; 15 - 17 = -2 (-2 / 17 = -11.8%)
   Risk: High — Utilization is very low at 27.7% and usage declined 11.8% over the last 3 months.

7. C-0B20DB64 | CSM: Dana Mercer | ARR: $21,770 | Date used: 2026-10-07
   Seat utilization: 214 / 378 = 56.6%
   3-month usage trend: 294 -> 298 -> 294; 294 - 294 = 0 (0 / 294 = 0.0%)
   Risk: Medium — Usage is flat, but seat utilization is only 56.6%.

8. C-0BBC4E7A | CSM: Cole Ingram | ARR: $56,374 | Date used: 2026-10-10
   Seat utilization: 228 / 337 = 67.7%
   3-month usage trend: 142 -> 141 -> 139; 139 - 142 = -3 (-3 / 142 = -2.1%)
   Risk: Medium — Usage declined 2.1% and utilization is below 70% at 67.7%.

9. C-0FD551AB | CSM: Elena Sinclair | ARR: $48,815 | Date used: 2026-10-14
   Seat utilization: 210 / 376 = 55.9%
   3-month usage trend: 123 -> 122 -> 126; 126 - 123 = +3 (+3 / 123 = +2.4%)
   Risk: Medium — Usage increased 2.4%, but seat utilization is only 55.9%.

10. C-0F9F8F13 | CSM: Dana Mercer | ARR: $46,230 | Date used: 2026-10-18
    Seat utilization: 199 / 352 = 56.5%
    3-month usage trend: 185 -> 185 -> 182; 182 - 185 = -3 (-3 / 185 = -1.6%)
    Risk: Medium — Usage declined 1.6% and utilization is only 56.5%.

11. C-0BC34584 | CSM: Cole Ingram | ARR: $16,740 | Date used: 2026-10-22
    Seat utilization: 327 / 494 = 66.2%
    3-month usage trend: 104 -> 104 -> 106; 106 - 104 = +2 (+2 / 104 = +1.9%)
    Risk: Medium — Usage increased 1.9%, but utilization remains below 70% at 66.2%.

12. C-0B7A7546 | CSM: Elena Sinclair | ARR: $35,062 | Date used: 2026-10-25
    Seat utilization: 182 / 205 = 88.8%
    3-month usage trend: 64 -> 65 -> 63; 63 - 64 = -1 (-1 / 64 = -1.6%)
    Risk: Medium — Seat utilization is strong at 88.8%, but usage declined 1.6%.

13. C-0B369871 | CSM: Dana Mercer | ARR: $85,128 | Date used: 2026-10-29
    Seat utilization: 317 / 422 = 75.1%
    3-month usage trend: 326 -> 330 -> 333; 333 - 326 = +7 (+7 / 326 = +2.1%)
    Risk: Low — Utilization is 75.1% and usage increased 2.1%.

14. C-0B144C78 | CSM: Cole Ingram | ARR: $30,899 | Date used: 2026-11-02
    Seat utilization: 169 / 224 = 75.4%
    3-month usage trend: 101 -> 101 -> 106; 106 - 101 = +5 (+5 / 101 = +5.0%)
    Risk: Low — Utilization is 75.4% and usage increased 5.0%.

15. C-0FC4DBB8 | CSM: Elena Sinclair | ARR: $94,732 | Date used: 2026-11-05
    Seat utilization: 356 / 464 = 76.7%
    3-month usage trend: 189 -> 191 -> 193; 193 - 189 = +4 (+4 / 189 = +2.1%)
    Risk: Low — Utilization is 76.7% and usage increased 2.1%.

16. C-0D5BBE3A | CSM: Dana Mercer | ARR: $39,740 | Date used: 2026-11-09
    Seat utilization: 85 / 102 = 83.3%
    3-month usage trend: 88 -> 90 -> 91; 91 - 88 = +3 (+3 / 88 = +3.4%)
    Risk: Low — Utilization is 83.3% and usage increased 3.4%.

17. C-0FB9D5AF | CSM: Cole Ingram | ARR: $63,158 | Date used: 2026-11-13
    Seat utilization: 144 / 199 = 72.4%
    3-month usage trend: 173 -> 173 -> 176; 176 - 173 = +3 (+3 / 173 = +1.7%)
    Risk: Low — Utilization is 72.4% and usage increased 1.7%.

18. C-0B344485 | CSM: Elena Sinclair | ARR: $64,384 | Date used: 2026-11-16
    Seat utilization: 224 / 287 = 78.0%
    3-month usage trend: 238 -> 240 -> 244; 244 - 238 = +6 (+6 / 238 = +2.5%)
    Risk: Low — Utilization is 78.0% and usage increased 2.5%.

19. C-0CB2C1B4 | CSM: Dana Mercer | ARR: $40,628 | Date used: 2026-11-20
    Seat utilization: 386 / 473 = 81.6%
    3-month usage trend: 47 -> 48 -> 49; 49 - 47 = +2 (+2 / 47 = +4.3%)
    Risk: Low — Utilization is 81.6% and usage increased 4.3%.

20. C-22170CA1 | CSM: Cole Ingram | ARR: $45,646 | Date used: 2026-11-24
    Seat utilization: 251 / 294 = 85.4%
    3-month usage trend: 143 -> 148 -> 146; 146 - 143 = +3 (+3 / 143 = +2.1%)
    Risk: Low — Utilization is 85.4% and usage increased 2.1%.

Total ARR renewing:
$65,901 + $54,427 + $38,022 + $30,993 + $90,647 + $79,419 + $21,770 + $56,374 + $48,815 + $46,230 + $16,740 + $35,062 + $85,128 + $30,899 + $94,732 + $39,740 + $63,158 + $64,384 + $40,628 + $45,646 = $1,048,715

ARR at risk:
High-risk ARR only = $65,901 + $54,427 + $38,022 + $30,993 + $90,647 + $79,419 = $359,409
ticket-theme-synthesis0.80

? The Question

Synthesize the quarter's support tickets into 4 to 7 themes. Do not trust existing tags; read the text. Per theme: name, count, share, distinct accounts, ARR affected, two ticket ids, one-line recommendation. Separate broad patterns from single-account noise and rank by ARR exposure, not volume.
cost $0.2302 · 33s · in 11,178 / out 5,314 tok
✗ theme_count_range✓ counts_verifiable✓ single_account_flagged✓ arr_ranked✓ ticket_ids

✓ Correct Answer (ground truth)

quarter_window: 2026-06-01 through 2026-08-31
total_tickets: 80
themes: {"points_not_posting": {"count": 20, "share_pct": 25.0, "distinct_accounts": 9, "arr_affected": "31100.00", "example_ticket_ids": ["IC-460001", "IC-460002"], "single_account": false}, "slack_sync": {"
ranking_by_arr_exposure: hris_provisioning, redemption_failures, billing_dispute, points_not_posting, slack_sync
single_account_noise_theme: billing_dispute
note: existing tags are noisy/unreliable; classify by text
sample_ticket_ids: 80 items — e.g. IC-460004, IC-460016, IC-460001, IC-460041
graded checks: theme_count_range · counts_verifiable · single_account_flagged · arr_ranked · ticket_ids

Model's actual answer

show the model's full answer
Total tickets: 80

1. HRIS provisioning/sync failures
Count: 12
Share: 15.0% = 12 / 80
Distinct accounts: 3
ARR affected: $114,000 = C-0B2213A9 $36,000 + C-0DDFC9A7 $48,000 + C-0F6C0F34 $30,000
Ticket ids: IC-460059, IC-460055
Recommendation: Treat as the top broad-pattern risk; prioritize HRIS sync/provisioning reliability and error visibility for high-ARR accounts.

2. Redemption/gift card checkout failures
Count: 18
Share: 22.5% = 18 / 80
Distinct accounts: 7
ARR affected: $68,800 = C-0CEF69FD $8,900 + C-0B827671 $10,700 + C-0FCCD2DF $9,600 + C-0F876796 $8,700 + C-0D9CA315 $9,600 + C-14264ABD $11,000 + C-0B0F1BAB $10,300
Ticket ids: IC-460025, IC-460030
Recommendation: Fix checkout/redemption completion and points deduction failure paths; this is a broad pattern across 7 accounts.

3. Invoice/seat-count/tier-price errors
Count: 16
Share: 20.0% = 16 / 80
Distinct accounts: 1
ARR affected: $52,000 = C-0E9C27D1 $52,000
Ticket ids: IC-460071, IC-460069
Recommendation: Separate as single-account noise despite high ARR; assign C-0E9C27D1 an owner to resolve billing, seat-count, and renewal tier errors.

4. Recognition points not posting/missing balances
Count: 20
Share: 25.0% = 20 / 80
Distinct accounts: 9
ARR affected: $31,100 = C-0D3278C7 $3,500 + C-0BF20542 $4,500 + C-0D0B047C $4,500 + C-0BE96399 $2,700 + C-0D284E42 $3,400 + C-0D6CC8E3 $4,200 + C-21FEBCBB $2,900 + C-0DD0626C $2,500 + C-0B2895EF $2,900
Ticket ids: IC-460004, IC-460016
Recommendation: Investigate recognition-to-points posting pipeline; broadest account spread but lower ARR exposure than HRIS/redemption.

5. Slack recognition integration failures
Count: 14
Share: 17.5% = 14 / 80
Distinct accounts: 4
ARR affected: $18,900 = C-0B843542 $4,400 + C-10A56B0F $5,400 + C-0BA71F12 $3,900 + C-8C2E8F00 $5,200
Ticket ids: IC-460041, IC-460047
Recommendation: Stabilize Slack sync, re-auth persistence, and slash command errors; broad pattern, but lowest ARR exposure.
marketing
case-study-matchPASS 1.00

? The Question

Given the prospect, rank the three most similar existing customers with public case studies for use as social proof. Similarity on industry, size band, use case, region. Cite the fields that drove each match. Do not name any customer without a case study and never include billing data or contact names.
cost $0.0613 · 13s · in 2,588 / out 1,133 tok
✓ top3✓ decoys_excluded✓ cites_fields✓ no_billing

✓ Correct Answer (ground truth)

top3: company_alias=C-11C31562; similarity_score=3 | company_alias=C-64171065; similarity_score=3 | company_alias=C-A13C193D; similarity_score=3
decoys_excluded: company_alias=C-D6217CAA; similarity_score=4; reason=no_case_study | company_alias=C-C153868F; similarity_score=4; reason=no_case_study
constraints: no_billing_data, no_contact_names, only_customers_with_has_case_study_true_named
prospect_fields_used: industry, size_band, use_case, region
prospect_alias: C-82AF3719
graded checks: top3 · decoys_excluded · cites_fields · no_billing

Model's actual answer

show the model's full answer
Prospect: C-82AF3719
Fields: Technology, Mid-Market, employee_recognition, NA-West

Ranked social proof matches, only customers with has_case_study=true:

1. C-11C31562 — 3/4 match
   Arithmetic: industry 0 + size_band 1 + use_case 1 + region 1 = 3
   Matching fields:
   - size_band: Mid-Market
   - use_case: employee_recognition
   - region: NA-West
   Non-match:
   - industry: Manufacturing vs Technology

2. C-64171065 — 3/4 match
   Arithmetic: industry 1 + size_band 1 + use_case 1 + region 0 = 3
   Matching fields:
   - industry: Technology
   - size_band: Mid-Market
   - use_case: employee_recognition
   Non-match:
   - region: NA-East vs NA-West

3. C-A13C193D — 3/4 match
   Arithmetic: industry 1 + size_band 1 + use_case 0 + region 1 = 3
   Matching fields:
   - industry: Technology
   - size_band: Mid-Market
   - region: NA-West
   Non-match:
   - use_case: retention vs employee_recognition
channel-attribution-mathPASS 1.00

? The Question

Compute channel performance for the trailing 6 months. Paid: spend, SQMs, SQOs, cost per SQM, cost per SQO, SQM-to-SQO rate, pipeline amount, pipeline per dollar. Organic: volume, SQO rate, pipeline. Report a channel with spend and zero SQMs as undefined, not zero. Flag rows where SQO date precedes SQM date. Finish with a reallocation recommendation and your confidence given sample sizes.
cost $0.2113 · 36s · in 8,807 / out 5,079 tok
✓ undefined_channel✓ undefined_not_zero✓ flagged_rows✓ best_channel✓ reallocation

✓ Correct Answer (ground truth)

months: 2026-03, 2026-04, 2026-05, 2026-06, 2026-07, 2026-08
per_channel: {"linkedin_ads": {"spend_usd": 24000, "sqms": 25, "sqos": 8, "pipeline_amount": 96000, "cost_per_sqm": 960.0, "cost_per_sqo": 3000.0, "sqm_to_sqo_rate": 0.32, "pipeline_per_dollar": 4.0}, "organic_sea
undefined_channels: paid_social
flagged_rows_sqo_before_sqm: CT-000041, CT-000044
best_pipeline_per_dollar_channel: paid_search
reallocation_recommendation: from=paid_social; to=paid_search; rationale=paid_social has spend but zero SQMs (cost per SQM undefined, not zero); paid_search has the highest pipeline per dollar.
confidence_note: webinars and referral have small SQM samples (<20); treat rates as directional.
graded checks: undefined_channel · undefined_not_zero · flagged_rows · best_channel · reallocation

Model's actual answer

show the model's full answer
Trailing 6 months: 2026-03 through 2026-08.

Paid channel performance

| channel | spend | SQMs | SQOs | cost per SQM | cost per SQO | SQM-to-SQO rate | pipeline amount | pipeline per dollar |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| paid_search | $36,000 | 40 | 18 | $900.00 | $2,000.00 | 45.0% | $720,000 | $20.00 |
| linkedin_ads | $24,000 | 25 | 8 | $960.00 | $3,000.00 | 32.0% | $96,000 | $4.00 |
| paid_social | $18,000 | 0 | 0 | undefined | undefined | undefined | $0 | undefined |
| webinars | $9,000 | 12 | 5 | $750.00 | $1,800.00 | 41.7% | $60,000 | $6.67 |

Arithmetic:
- paid_search spend = $6,000 × 6 = $36,000
  - cost per SQM = $36,000 / 40 = $900.00
  - cost per SQO = $36,000 / 18 = $2,000.00
  - SQM-to-SQO rate = 18 / 40 = 45.0%
  - pipeline per dollar = $720,000 / $36,000 = $20.00

- linkedin_ads spend = $4,000 × 6 = $24,000
  - cost per SQM = $24,000 / 25 = $960.00
  - cost per SQO = $24,000 / 8 = $3,000.00
  - SQM-to-SQO rate = 8 / 25 = 32.0%
  - pipeline per dollar = $96,000 / $24,000 = $4.00

- paid_social spend = $3,000 × 6 = $18,000
  - SQMs = 0, so cost per SQM, cost per SQO, SQM-to-SQO rate, and pipeline per dollar are undefined

- webinars spend = $1,500 × 6 = $9,000
  - cost per SQM = $9,000 / 12 = $750.00
  - cost per SQO = $9,000 / 5 = $1,800.00
  - SQM-to-SQO rate = 5 / 12 = 41.7%
  - pipeline per dollar = $60,000 / $9,000 = $6.67

Organic channel performance

| channel | volume | SQOs | SQO rate | pipeline |
|---|---:|---:|---:|---:|
| organic_search | 30 | 10 | 33.3% | $90,000 |
| referral | 15 | 6 | 40.0% | $48,000 |

Arithmetic:
- organic_search SQO rate = 10 / 30 = 33.3%; pipeline = $90,000
- referral SQO rate = 6 / 15 = 40.0%; pipeline = $48,000

Rows where SQO date precedes SQM date:
- CT-000044, linkedin_ads: SQM date 2026-07-23, SQO date 2026-07-18, pipeline_amount $12,000
- CT-000041, linkedin_ads: SQM date 2026-06-14, SQO date 2026-06-09, pipeline_amount $12,000

Reallocation recommendation:
- Reallocate paid_social spend away from paid_social because it spent $18,000 and produced 0 SQMs.
- Prioritize reallocating that budget to paid_search first: paid_search has the strongest paid performance with $20.00 pipeline per dollar, 18 SQOs, and 40 SQMs.
- Consider a smaller test increase to webinars: webinars has the lowest cost per SQO at $1,800 and $6.67 pipeline per dollar, but the sample is smaller at 12 SQMs and 5 SQOs.
- Do not increase linkedin_ads until the two date-quality issues are resolved; linkedin_ads also trails paid_search and webinars on pipeline per dollar.

Confidence: moderate. paid_search has the strongest sample size among paid channels at 40 SQMs and 18 SQOs. webinars is promising but lower-confidence due to only 12 SQMs and 5 SQOs. paid_social has no SQM sample, so performance is undefined rather than confidently zero-converting.
competitor-battlecard-updatePASS 1.00

? The Question

Update the battlecard for the competitor. Sections: one-line positioning, pricing with source and date (newer source wins, note the conflict), where they win, where we win, objections and responses, recent changes, our 12-month win/loss record against them. Cite a snippet id for every factual claim. Rep opinion on a call is not a fact about the competitor. Anything from the old card you cannot re-source gets marked unverified.
cost $0.1296 · 27s · in 4,387 / out 3,129 tok
✓ competitor_named✓ pricing✓ win_loss✓ unverified_marked✓ citations

✓ Correct Answer (ground truth)

competitor: Rivally
pricing: value_per_user_per_month=7; source_snippet=S17; source_date=2026-08-12
win_loss_12mo: wins=13; losses=7
unverified_old_claims: Rivally lacks a Slack integration., Rivally was acquired by WorkHuman in 2025.
rep_opinion_snippet_ids_not_facts: S09, S21
citation_ids_available: 25 items — e.g. S01, S02, S03, S04
graded checks: competitor_named · pricing · win_loss · unverified_marked · citations

Model's actual answer

show the model's full answer
# Battlecard: Rivally

## One-line positioning
Rivally is a points-based recognition platform with an engaging recognition feed, plus Rivally Pulse as a lightweight engagement-survey add-on that is priced separately from Recognition. [S02, S06, S23]

## Pricing with source and date
- Newer source wins: Recognition Starter is $7 per user/month, annual billing required, per Rivally pricing page dated 2026-08-12. [S17]
- Conflict noted: older Rivally pricing pages showed $5 per user/month:
  - 2026-01-20: “$5 per user/month, annual billing required.” [S03]
  - 2026-04-01: pricing page “still shows $5 per user/month.” [S08]
- Deal-price mentions also show movement/conflict:
  - 2026-06-02: Rivally quoted $6.50/user/mo to a 500-seat prospect, annual term. [S13]
  - 2026-08-14: prospect said Rivally quoted $7/user/mo list and offered 15% discount for a 3-year term. [S18]
- Rivally Pulse is an add-on, not bundled; no numeric Pulse price was provided. [S23]

## Where they win
- Engaging points-based recognition feed. [S02, S16]
- Fast setup: one mid-market reviewer said setup took under a week. [S04]
- Slack integration: one reviewer said Slack integration worked out of the box. [S04]
- EU story: Rivally pitched EU data residency; EU data residency later became generally available. [S05, S15]
- EU enterprise fit: reviewer praised Rivally for distributed EU teams and multi-language support. [S12]
- Support responsiveness: reviewer praised support response time under 4 hours. [S22]

## Where we win
- Analytics depth: one 800-seat prospect picked Bonusly over Rivally citing analytics depth. [S25]
- Rivally analytics/reporting weaknesses:
  - Reviewer noted limited analytics. [S02]
  - Capterra review said reporting dashboards are basic compared to enterprise tools. [S07]
  - Reviewer said migration off Rivally was hard because analytics exports are CSV-only. [S20]
- Enterprise admin/provisioning gaps:
  - Enterprise reviewer said Rivally lacks SCIM provisioning and manual user management is painful. [S10]
  - Reviewer said admin tooling lags peers. [S16]
  - Reviewer said admin console still lacks bulk recognition editing. [S24]
- EMEA rewards catalog gap: TrustRadius review said Rivally’s rewards catalog in EMEA is thinner than its US catalog. [S14]

## Objections and responses
- Objection: “Rivally is cheaper at $5/user/month.”
  - Response: That price is from older pricing pages dated 2026-01-20 and 2026-04-01; the newer Rivally pricing page dated 2026-08-12 says Recognition Starter is now $7/user/month, annual billing required. Newer source wins. [S03, S08, S17]

- Objection: “Rivally is strong for EU teams.”
  - Response: Concede EU strength where relevant: EU data residency became generally available, and an EU enterprise reviewer praised distributed EU-team fit and multi-language support. Then probe gaps: EMEA rewards catalog was reviewed as thinner than the US catalog, and enterprise provisioning/admin issues were reported. [S12, S14, S15, S10, S24]

- Objection: “Rivally is easy to roll out.”
  - Response: Concede setup evidence: one mid-market reviewer said setup took under a week and Slack worked out of the box. Then shift to scale/admin: enterprise reviewer said Rivally lacks SCIM provisioning, manual user management is painful, admin tooling lags peers, and bulk recognition editing is still missing. [S04, S10, S16, S24]

- Objection: “Rivally’s recognition feed is engaging.”
  - Response: Concede: reviewers praised the points-based recognition feed and called it engaging. Then differentiate on analytics/reporting depth, where reviewers cited limited analytics, basic dashboards, and CSV-only analytics exports. [S02, S16, S07, S20]

## Recent changes
- 2025-11-04: Rivally announced a $40M Series C led by Northgate Ventures. [S01]
- 2026-03-05: Rivally launched Rivally Pulse, a lightweight engagement survey add-on. [S06]
- 2026-05-09: Rivally hired an ex-Workday VP EMEA to lead European expansion. [S11]
- 2026-07-01: Rivally opened a Dublin office and announced EU data residency generally available. [S15]
- 2026-08-12: Rivally pricing page updated Recognition Starter to $7 per user/month, annual billing required. [S17]
- 2026-08-19/20: Rivally announced Microsoft Teams app v2 in public preview. [S19]
- 2026-09-01: Rivally Pulse exited beta and is priced as an add-on, not bundled. [S23]

## Our 12-month win/loss record against Rivally
Data provided covers 2025-09 through 2026-08.

Arithmetic:
- Total deals = 20
- Wins = 13
- Losses = 7
- Win rate = 13 / 20 = 65%
- Loss rate = 7 / 20 = 35%

Wins:
Deal-A9FD43, Deal-7AA785, Deal-44C524, Deal-0D0CD6, Deal-D5B790, Deal-5C636E, Deal-67BE14, Deal-1B6969, Deal-F03E7B, Deal-072E31, Deal-F65C8F, Deal-E46EAB, Deal-1D2392

Losses:
Deal-7767F5, Deal-5645A5, Deal-C6FFAA, Deal-D263E0, Deal-935746, Deal-9066A6, Deal-72A02F

Monthly record:
- 2025-09: 1 win, 1 loss
- 2025-10: 2 wins, 0 losses
- 2025-11: 1 win, 1 loss
- 2025-12: 1 win, 1 loss
- 2026-01: 2 wins, 0 losses
- 2026-02: 2 wins, 0 losses
- 2026-03: 1 win, 1 loss
- 2026-04: 0 wins, 2 losses
- 2026-05: 0 wins, 1 loss
- 2026-06: 1 win, 0 losses
- 2026-07: 1 win, 0 losses
- 2026-08: 1 win, 0 losses

## Old-card claims that are unverified or corrected
- “Positioning: points-based recognition for mid-market.” Partially resourced: points-based recognition/feed is supported; “mid-market” appears only as a reviewer segment, not as company positioning. [S02, S04]
- “Pricing: starts at $5 per user/month, annual billing.” Correct historically, but superseded by newer $7/user/month pricing page. [S03, S08, S17]
- “Rivally lacks a Slack integration.” Corrected: a reviewer said Slack integration worked out of the box. [S04]
- “Rivally was acquired by WorkHuman in 2025.” Unverified: no provided snippet supports this.
- “Strong in EU enterprise with multi-language support.” Resourced: EU enterprise reviewer praised distributed EU teams and multi-language support. [S12]
nurture-sequence-diagnosisPASS 1.00

? The Question

Review the sequences. Per sequence: sent, open rate, reply rate, meeting rate, weakest step. Identify tracking errors (opened above sent), audience overlap between sequences, and the failure mode for anything under 2 percent reply. One change per weak sequence and which to fix first. Under 350 words.
cost $0.1667 · 25s · in 14,244 / out 2,551 tok
✓ tracking_error✓ overlap_pair✓ weakest_step✓ fix_priority

✓ Correct Answer (ground truth)

per_sequence: {"New Logo Nurture": {"sent": 1386, "opened": 490, "replied": 90, "meetings": 27, "open_rate": 0.3535, "reply_rate": 0.0649, "meeting_rate": 0.0195, "weakest_step": 3}, "Expansion Nurture": {"sent": 8
tracking_error: sequence=Expansion Nurture; step=2; sent=300; opened=340; issue=opened exceeds sent
audience_overlap_pair: Cold Outbound - HR Leaders, Cold Outbound - People Ops
all_pairwise_overlaps: ['Cold Outbound - HR Leaders', 'Cold Outbound - People Ops'], ['Expansion Nurture', 'New Logo Nurture']
under_2pct_reply: Cold Outbound - HR Leaders
failure_mode: Cold Outbound - HR Leaders=opens healthy (~37%) but replies near zero from step 1 -> body/CTA not landing, not a deliverability or subject-line problem
fix_priority_first: Cold Outbound - HR Leaders
graded checks: tracking_error · overlap_pair · weakest_step · fix_priority

Model's actual answer

show the model's full answer
New Logo Nurture: sent 1,386; open rate 490/1,386=35.4%; reply rate 90/1,386=6.5%; meeting rate 27/1,386=1.9%. Weakest step: step 3, replies 18/428=4.2%, meetings 6/428=1.4%. Change: rewrite step 3 with a stronger reason-to-reply/CTA.

Expansion Nurture: sent 875; open rate 565/875=64.6%, but unreliable due tracking error; reply rate 59/875=6.7%; meeting rate 12/875=1.4%. Weakest step: step 3, replies 12/275=4.4%, meetings 3/275=1.1%. Change: fix tracking first, then tighten step 3 around expansion trigger/value.

Cold Outbound - HR Leaders: sent 1,785; open rate 545/1,785=30.5%; reply rate 8/1,785=0.4%; meeting rate 0/1,785=0.0%. Weakest step: step 3, replies 1/590=0.2%, meetings 0/590=0.0%. Failure mode under 2% reply: opens exist but replies/meetings do not, so the issue is likely offer/message fit, not just deliverability. Change: replace the sequence with a new HR-specific pain/trigger and clearer CTA.

Cold Outbound - People Ops: sent 1,163; open rate 340/1,163=29.2%; reply rate 29/1,163=2.5%; meeting rate 6/1,163=0.5%. Weakest step: step 3, replies 6/377=1.6%, meetings 1/377=0.3%. Failure mode under 2% at step 3: late-step fatigue/weak follow-up. Change: cut or rewrite step 3 with a new angle.

Tracking errors: Expansion Nurture step 2 has opened 340 above sent 300; open rate 340/300=113.3%.

Audience overlap: New Logo Nurture overlaps Expansion Nurture on CT-000301, CT-000624. Cold Outbound - HR Leaders overlaps Cold Outbound - People Ops on 21 contacts including CT-000849, CT-001255, CT-001105, CT-000884, CT-001103, CT-001217, CT-001345, CT-001153, CT-001258, CT-001033, CT-001130, CT-000890, CT-001097, CT-000908, CT-001311, CT-001277, CT-001227, CT-001101, CT-001236, CT-001285, CT-001159.

Fix first: Cold Outbound - HR Leaders, because reply rate is 0.4% and meetings are 0.
weekly-marketing-goals-rollupPASS 1.00

? The Question

Produce the weekly marketing goals update for the active quarter: SQMs, SQOs, DS2s, closed-lost MIA rate, same-quarter close count, active pipeline coverage against target. For each: QTD actual, target, delta, pace (on/behind/ahead given days elapsed). One paragraph on what moved this week.
cost $0.1380 · 17s · in 16,360 / out 1,642 tok
✓ metrics_present✓ deltas✓ coverage_ratio✓ pace✓ paragraph

✓ Correct Answer (ground truth)

days_elapsed: 66
days_in_quarter: 92
elapsed_fraction: 0.7174
metrics: {"SQMs": {"qtd_actual": 230, "target": 300, "delta": -70, "expected_at_day_elapsed": 215.22, "pace": "ahead"}, "SQOs": {"qtd_actual": 84, "target": 120, "delta": -36, "expected_at_day_elapsed": 86.09,
coverage_ratio_active_pipeline: 0.75
pace_rule: expected = target * days_elapsed/days_in_quarter; within +/-5% of expected = on, above = ahead, below = behind; lower_better metrics invert.
this_week_paragraph_required: True
graded checks: metrics_present · deltas · coverage_ratio · pace · paragraph

Model's actual answer

show the model's full answer
Active quarter: Q3-2026. Days elapsed: 66 / 92 = 71.7% of quarter.

| Goal | QTD actual | Target | Delta | Pace |
|---|---:|---:|---:|---|
| SQMs | 230 | 300 | 230 - 300 = -70 | Ahead: paced target = 300 × 66 / 92 = 215.2; actual 230 is +14.8 ahead of pace |
| SQOs | 84 | 120 | 84 - 120 = -36 | Behind: paced target = 120 × 66 / 92 = 86.1; actual 84 is -2.1 behind pace |
| DS2s | 40 | 75 | 40 - 75 = -35 | Behind: paced target = 75 × 66 / 92 = 53.8; actual 40 is -13.8 behind pace |
| Closed-lost MIA rate | 5 / 25 = 20.0% | 10.0% | 20.0% - 10.0% = +10.0 pp | Behind: lower is better, and 20.0% is above the 10.0% target |
| Same-quarter closes | 10 | 20 | 10 - 20 = -10 | Behind: paced target = 20 × 66 / 92 = 14.3; actual 10 is -4.3 behind pace |
| Active pipeline coverage against target | $3,000,000 / $4,000,000 = 75.0% | $4,000,000 / $4,000,000 = 100.0% | $3,000,000 - $4,000,000 = -$1,000,000, or -25.0 pp coverage | Ahead: paced target = $4,000,000 × 66 / 92 = $2,869,565; actual $3,000,000 is +$130,435 ahead of pace |

What moved this week: Weekly movement cannot be determined from the provided data because only QTD actuals, targets, and quarter timing were provided; no prior-week or weekly-change data is included. Current QTD position shows SQMs and active pipeline ahead of elapsed-quarter pace, while SQOs, DS2s, same-quarter closes, and closed-lost MIA rate are behind target/pacing.
communication
ceo-slack-compressionPASS 1.00

? The Question

Compress the forecast analysis below into a single Slack message for the CEO. Rules: at most three sentences; under 80 words; lead with the decision or the so-what; keep every distinct number that matters and the one caveat that changes the conclusion; cut any sentence that does not carry a claim, a number, or a directive. No em dashes, no throat-clearing phrases.
cost $0.0354 · 11s · in 2,013 / out 631 tok
✓ max_3_sentences✓ under_80_words✓ weighted_present✓ caveat_present✓ no_em_dash

✓ Correct Answer (ground truth)

graded checks: max_3_sentences · under_80_words · weighted_present · caveat_present · no_em_dash

Model's actual answer

show the model's full answer
Use $115,976.75 as the Q3 forecast: $44,729 COMMIT + 35% x $203,565 BEST_CASE = $115,976.75, with PIPELINE at $0. Caveat: 32 deals worth $227,575 are excluded after 2026-09-30, including COMMIT Deal-D348E1 at $13,770 and 9 BEST_CASE totaling $28,240. Do not roll up by rep because owner is blank on 85 of 86 deals.
stalled-deal-followup-emailPASS 1.00

? The Question

Write a follow-up email for this stalled deal. Under 120 words. Reference one specific prior touch from the engagement history, make one clear ask, and match the tone of the prior thread. No invented details.
cost $0.0201 · 5s · in 1,835 / out 150 tok
✓ word_cap✓ references_touch✓ single_ask

✓ Correct Answer (ground truth)

deal_alias: Deal-0D2F7A
amount: 5100
stage: DS3
owner: Alex Franklin
days_since_last_engagement: 31
word_cap: 120
must_reference: engagement_id=ENG-00002; date=2026-07-28; type=meeting; why=most substantive prior touch; concrete prospect reactions to milestone awards and points catalog
single_cta: True
tone: warm, concise, professional (matches prior thread)
allowed_facts: 14 items — e.g. Deal-0D2F7A, $5,100, DS3, Alex Franklin
allowed_new: ENG-00001, ENG-00002, ENG-00003, ENG-00004, ENG-00005
graded checks: word_cap · references_touch · single_ask

Model's actual answer

show the model's full answer
Subject: Following up on Deal-0D2F7A

Hi,

Following up on my Aug. 5 recap of the July 28 demo with pricing for 150 seats.

Is the People team still evaluating this, or should we close the loop for now?

Best,
Alex
funnel-digest-paragraphPASS 1.00

? The Question

Write the weekly GTM digest: one paragraph per team (Marketing, Sales, CS, PLG) in Bonusly brand voice. Every number must trace to the extracts. No customer contact names, no employee performance commentary, no compensation. Four paragraphs, under 400 words total.
cost $0.0261 · 6s · in 1,699 / out 372 tok
✓ four_paragraphs✓ word_cap✓ teams

✓ Correct Answer (ground truth)

required_paragraphs: Marketing, Sales, CS, PLG
allowed_numbers: 16 items — e.g. 2, 3, 9, 08
word_cap: 400
banned: customer contact names, employee performance commentary, compensation
graded checks: four_paragraphs · word_cap · teams

Model's actual answer

show the model's full answer
Marketing: SQMs landed at 46 against a 52 target, which is 6 short (52 - 46 = 6) and 88.5% of goal (46 / 52 = 88.5%). Webinar contributed 18 SQMs, or 39.1% of the week’s SQMs (18 / 46 = 39.1%). Paid search cost per SQM held at $150.

Sales: Sales converted 14 SQOs and set 9 DS2 meetings. New pipeline created was $310,000, which averages $22,142.86 per SQO ($310,000 / 14 = $22,142.86). Same-quarter close count for the week was 3.

CS: CS saved 2 renewals this week. Team NPS moved to 61. There are 3 open red-flag accounts heading into next week, so the team is carrying more red-flag accounts than saved renewals by 1 (3 - 2 = 1).

PLG: PLG added 412 new signups with activation at 31%, which equals 127.72 activated signups (412 × 31% = 127.72). 38 companies reached the aha moment of 10 recognition gives, equal to 9.2% of new signups if compared directionally to signup volume (38 / 412 = 9.2%).
partner-digest-sectionPASS 1.00

? The Question

Write the partner digest for the period. Every partner in the roster appears once: active plays get status with numbers, quiet partners get one line. Partner-sourced pipeline count and amount must match the deal data. Nothing omitted, nothing added.
cost $0.0532 · 9s · in 3,311 / out 778 tok
✓ all_partners✓ pipeline_amount✓ quiet_one_line

✓ Correct Answer (ground truth)

roster: partner_name=Apex Rewards Co; status=active | partner_name=HRCloud Partners; status=active | partner_name=CultureBridge; status=active | partner_name=WorkWell Group; status=active | partner_name=Recogniq; status=quiet | partner_name=KudosWave; status=quiet … (+2 more)
per_partner_pipeline: {"Apex Rewards Co": {"count": 2, "amount": 275000}, "HRCloud Partners": {"count": 1, "amount": 140000}, "CultureBridge": {"count": 2, "amount": 135000}}
partner_sourced_pipeline: count=5; amount=550000
quiet_partners: Recogniq, KudosWave, PeopleFirst Advisors, TotalPerk
active_no_deals: WorkWell Group
rule: every roster partner named exactly once; quiet partners get one line; no partners outside roster.
graded checks: all_partners · pipeline_amount · quiet_one_line

Model's actual answer

show the model's full answer
Partner digest: 2026-08-24 to 2026-09-04

Partner-sourced pipeline total: 5 deals, $550,000
Arithmetic: $180,000 + $95,000 + $140,000 + $60,000 + $75,000 = $550,000

Apex Rewards Co — active. Co-webinar locked for 09-15. Sourced 2 DS1 opps: Deal-DDAAF2 ($180,000) and Deal-2CF33E ($95,000). Pipeline arithmetic: 2 deals; $180,000 + $95,000 = $275,000.

HRCloud Partners — active. Integration referral closed security review and moved to DS2. Sourced 1 opp: Deal-F1CDA5 ($140,000). Pipeline arithmetic: 1 deal; $140,000.

CultureBridge — active. Lunch-and-learn produced 2 early-stage sourced opps: Deal-096E1D ($60,000) and Deal-067213 ($75,000). Pipeline arithmetic: 2 deals; $60,000 + $75,000 = $135,000.

WorkWell Group — active. Wants to restart the joint playbook in Q4; no sourced deals this period; planning call booked for 09-09. Pipeline arithmetic: 0 deals; $0.

Recogniq — quiet. No Slack activity or partner-sourced deals provided for this period. Pipeline arithmetic: 0 deals; $0.

KudosWave — quiet. No Slack activity or partner-sourced deals provided for this period. Pipeline arithmetic: 0 deals; $0.

PeopleFirst Advisors — quiet. No Slack activity or partner-sourced deals provided for this period. Pipeline arithmetic: 0 deals; $0.

TotalPerk — quiet. No Slack activity or partner-sourced deals provided for this period. Pipeline arithmetic: 0 deals; $0.
rfp-answer-from-policyPASS 1.00

? The Question

Answer the 10 questionnaire items from the policy excerpts only. Cite the excerpt id for every answer. Where the excerpts do not answer the question, mark it unanswerable and say what document would; do not guess. Tone: vendor response, direct.
cost $0.0288 · 7s · in 1,884 / out 434 tok
✓ answered_count✓ unanswerable_flagged✓ citations✓ no_guess

✓ Correct Answer (ground truth)

answered_count: 7
answer_map: {"Q1": ["P1"], "Q2": ["P1"], "Q3": ["P2"], "Q4": ["P3"], "Q5": ["P4"], "Q6": ["P5"], "Q7": ["P6"]}
unanswerable: Q8, Q9, Q10
valid_citation_ids: P1, P2, P3, P4, P5, P6, P7, P8
rule: unanswerable items must be flagged and the document that would answer them named; do not guess.
graded checks: answered_count · unanswerable_flagged · citations · no_guess

Model's actual answer

show the model's full answer
Q1. Yes. Customer data is encrypted at rest using AES-256. [P1]

Q2. Yes. Data in transit is encrypted with TLS 1.2 or higher. [P1]

Q3. Yes. Bonusly maintains a SOC 2 Type II report, renewed annually. The report is available to prospects under NDA. [P2]

Q4. Customer data is retained for the duration of the contract plus 90 days after contract termination, then deleted from production systems. Arithmetic: contract duration + 90 days. [P3]

Q5. The current subprocessor list is published at trust.bonusly.com. Customers receive 30 days’ notice before a new subprocessor is added. [P4]

Q6. Yes. SAML 2.0 single sign-on is available on the Enterprise plan. [P5]

Q7. Production databases are backed up daily. Backups are encrypted and retained for 35 days. Arithmetic: 1 backup per day; backup retention = 35 days. [P6]

Q8. Unanswerable from the provided excerpts. The excerpts do not state whether Bonusly holds ISO 27001 certification. This would be answered by an ISO 27001 certificate, compliance certifications page, or security/compliance documentation.

Q9. Unanswerable from the provided excerpts. The excerpts do not state a contractual uptime SLA percentage. This would be answered by the service level agreement, master services agreement, or uptime SLA documentation.

Q10. Unanswerable from the provided excerpts. The excerpts do not state whether Bonusly will sign a HIPAA Business Associate Agreement. This would be answered by a HIPAA/BAA policy, legal terms, or healthcare compliance documentation.
ops-maintenance
skill-file-reconciliationPASS 1.00

? The Question

Reconcile the skill set below against its manifest. Find and report (with severity CRITICAL/WARNING/INFO and an action type TRIM_DESC, UPDATE_BODY, DELETE_SKILL, MERGE, or REVIEW): (1) any two skills whose ALWAYS-trigger phrases overlap/duplicate, (2) any circular delegation chain between skills (name it), (3) any delegation target that does not exist (dangling, name it), (4) any version conflict and which skill should survive, (5) how many manifest descriptions exceed 1,024 characters, (6) any hardcoded page ids, dates, or person names in skill bodies, (7) manifest drift in BOTH directions (files with no manifest row, and manifest rows with no file). Make one proposal per finding; rewrite nothing.
cost $0.5521 · 62s · in 66,520 / out 7,102 tok
✓ duplicate_trigger_found✓ circular_chain✓ dangling_target✓ oversized_desc_count✓ hardcoded_values✓ manifest_drift✓ severity_and_action

✓ Correct Answer (ground truth)

graded checks: duplicate_trigger_found · circular_chain · dangling_target · oversized_desc_count · hardcoded_values · manifest_drift · severity_and_action

Model's actual answer

show the model's full answer
Findings

| # | Severity | Action type | Finding | Evidence / arithmetic | Proposal |
|---|---|---|---|---|---|
| 1 | CRITICAL | MERGE | ALWAYS-trigger overlap: `comms-drafter` and `email-drafter` duplicate email-writing triggers. | `comms-drafter`: "write me an email", "draft a follow-up", "help me reply", "what should I say", "bump email", "contract nudge". `email-drafter`: same phrases plus "review this email", "rewrite this". | Merge or clearly route: make `comms-drafter` the broad external-communications umbrella and preserve `email-drafter` only as the email-specific execution/signature module. |
| 2 | WARNING | REVIEW | ALWAYS-trigger overlap: `pipeline-intelligence-report` and `weekly-pipeline-report` both claim generic pipeline-report/update asks. | `pipeline-intelligence-report`: "run the pipeline report", "pipeline review", "pipeline update", "what's the pipeline look like". `weekly-pipeline-report`: "run the pipeline update", "weekly pipeline report", "pipeline summary", "generate the pipeline report", "update the pipeline", "what does pipeline look like". | Clarify trigger boundary: `pipeline-intelligence-report` = full scored/tiered active-deal intelligence; `weekly-pipeline-report` = weekly performance metrics/SQM/SQO/DS2/bookings HTML. |
| 3 | WARNING | REVIEW | ALWAYS-trigger overlap: `sales-forecast` and `pipeline-intelligence-report` both claim forecast/pipeline-health asks. | `sales-forecast`: "pipeline forecast", "deal-level confidence", "quarter-close risk", "current quarter revenue outlook". `pipeline-intelligence-report`: "pipeline health or forecast context", "score the pipeline". | Route current-quarter revenue outlook to `sales-forecast`; route scored active-deal tiers to `pipeline-intelligence-report`. |
| 4 | WARNING | REVIEW | ALWAYS-trigger overlap: `sales-forecast` and `next-to-close` can both trigger on close-likelihood questions. | `sales-forecast`: "what do we think we're going to close", "deal-level confidence". `next-to-close`: "next to close", "what's about to close", "which deals are most likely to close", "closest to signature". | Route shortlists of specific near-close deals to `next-to-close`; route quarter forecast totals and forecast-category analysis to `sales-forecast`. |
| 5 | WARNING | REVIEW | Universal ALWAYS trigger in `model-selection` overlaps every skill. | `model-selection`: "ALWAYS run this skill at the start of every task, without exception". This conflicts by scope with all other ALWAYS-trigger skills, though it is positioned as a planning gate. | Keep only if intended as orchestration-level preflight; otherwise narrow to model-planning tasks. |
| 6 | CRITICAL | UPDATE_BODY | Circular delegation chain exists: `deal-strategy-coach` → `email-drafter` → `deal-strategy-coach`. | `deal-strategy-coach`: "When drafting manager-to-prospect emails, use the `email-drafter` skill". `email-drafter`: "For deal strategy, diagnosis, or coaching... use deal-strategy-coach instead." | Break loop: make `deal-strategy-coach` own diagnosis and call `email-drafter` as terminal drafting only; make `email-drafter` not route back when invoked by `deal-strategy-coach`. |
| 7 | CRITICAL | UPDATE_BODY | Larger circular delegation chain exists: `comms-drafter` → `deal-strategy-coach` → `email-drafter` → `deal-strategy-coach`. | `comms-drafter`: "For deep deal strategy, use deal-strategy-coach". `deal-strategy-coach`: "When drafting manager-to-prospect emails, use the `email-drafter` skill". `email-drafter`: "For deal strategy... use deal-strategy-coach instead." | Define a single owner by intent: strategy first = `deal-strategy-coach`; draft-only = `email-drafter` or `comms-drafter`; no recursive handoff. |
| 8 | CRITICAL | UPDATE_BODY | Dangling delegation target: `bonusly-brand` does not exist in the manifest/files provided. | Referenced by `comms-drafter`, `email-drafter`, and `sales-forecast`. Manifest rows/files contain 14 skills and no `bonusly-brand`. | Add `bonusly-brand` to manifest/files or remove/replace the dependency. |
| 9 | CRITICAL | UPDATE_BODY | Dangling delegation target: `prospect-research-multithreading` does not exist in the manifest/files provided. | Referenced by `comms-drafter`, `email-drafter`, and `deal-strategy-coach`. No manifest row/file exists. | Add `prospect-research-multithreading` or remove the handoff. |
| 10 | CRITICAL | UPDATE_BODY | Dangling specialist-skill targets in `analysis-validator`. | `analysis-validator` references `bonusly-data-questions`, `bonusly-product-questions`, `bonusly-business-reporting-questions`, `bonusly-rewards-questions`, `bonusly-ppp-questions`, `bonusly-feature-flag-questions`, `bonusly-deal-desk-questions`, `bonusly-datadog-questions`. None appear in manifest/files. | Add these specialist skills to manifest/files or replace the delegation table with existing targets. |
| 11 | WARNING | REVIEW | Possible dangling referenced skill/path: `signalforge-reports`. | `pipeline-intelligence-report` and `weekly-pipeline-report` instruct reading `/mnt/skills/organization/signalforge-reports/...`; no manifest row/file for `signalforge-reports` is provided. | Confirm whether `signalforge-reports` is intentionally external to this manifest. If it belongs to this set, add a manifest row/file. |
| 12 | WARNING | REVIEW | Possible dangling registry target: `skill-orchestrator`. | `signalforge-feedback` says skill should be registered in `skill-orchestrator`; no manifest row/file exists. | Confirm whether `skill-orchestrator` is outside scope. If not, add it or remove the registration dependency. |
| 13 | WARNING | UPDATE_BODY | Version conflict inside `analysis-validator`: current version is 3.6, but validation trail template still says v3.2. | Header: `Version: 3.6`. Footer: `Produced by SignalForge · analysis-validator v3.6`. Template: `Validator: analysis-validator v3.2`. | `analysis-validator` v3.6 should survive; update the stale v3.2 reference. |
| 14 | WARNING | UPDATE_BODY | Version conflict inside `sales-forecast`: changelog says quarter-agnostic, but body still contains Q2-specific instructions. | Changelog v1.1: "Quarter-agnostic (Q2 → current quarter throughout)." Body: "Open Q2 Deals", "Q2 total", "Q2 QTD vs Q1 full quarter", "Q2 Narrative", examples around Q2. | `sales-forecast` v1.1 quarter-agnostic version should survive; update Q2-specific body references. |
| 15 | INFO | REVIEW | Manifest descriptions exceeding 1,024 characters: none. | Arithmetic: 14 manifest rows checked. Values over 1,024 = 0. Max description_chars = 1,006 (`pipeline-intelligence-report`, `signalforge-claim-compressor`). 1,006 < 1,024. Count = 0 / 14. | No trim required. |
| 16 | WARNING | UPDATE_BODY | Hardcoded page IDs and object IDs exist in skill bodies. | Examples: `partner-digest` has Cloud ID `73fe98de-a4a3-4869-9f8a-bb1eeed4cf7f`, Space ID `1958248479`, folder ID `2286616609`, page IDs `2286321666`, `2265382925`, `2236940297`, `2237825028`, `2239365136`, `2238283777`. `signalforge-feedback` has Page ID `2295136266`, spaceId `2232811524`, Parent `2234417154`, Build Log page `2247295002`. `sales-forecast` has Parent page ID `2232582148`. `pipeline-intelligence-report` hardcodes HubSpot org ID `1973303`. | Move IDs to configurable references or manifest metadata; keep bodies free of environment-specific IDs where possible. |
| 17 | WARNING | UPDATE_BODY | Hardcoded spreadsheet IDs exist in skill bodies. | `weekly-pipeline-report`: `1CLZeOsElVDF_LF0ZG_t2nfwvhnZ6bpwqM_nX3WEYzcw`, `1ENuaEcCuLjdKhMvp8FK3Ys1ek5Aw9ZuOZhsHJJFoB_k`. | Move spreadsheet IDs to config/reference metadata. |
| 18 | WARNING | UPDATE_BODY | Hardcoded dates exist across skill bodies. | Examples: `analysis-validator`: `April 26, 2026`, `May 9, 2026`, `May 4, 2026`, `March 28, 2023`. `model-selection`: `2026-05-19`. `partner-digest`: `May 16, 2026`, `May 19, 2026`, `June 2, 2026`, `2026-05-17`. `sales-forecast`: `April 27, 2026`, Q2 references. `stale-pipeline-report`: `2026-06-10`, example dates `5/15`, `5/19`. `weekly-pipeline-report`: `April 1 – June 30, 2026`, `Q1 2026`. | Replace execution dates with dynamic date logic unless the date is truly historical changelog data. |
| 19 | WARNING | REVIEW | Hardcoded person names exist across skill bodies. | Examples: `analysis-validator`: Manish, Amani, Alaina Loori, Shealagh Coughlin, Bryce Harmon, Hugo Lindqvist, Dana Mercer, Alex Franklin, Cole Ingram, Gavin Porter, Colleen Perry, Ellie Barton, Ashley Reyer, Megan Franz, Elena Sinclair, Youssef Elkhateeb, Amanda Czenkus, Ben Castelli, John Thomas, Yasmin Wahid. `partner-digest`: Amani Phipps, Kelli, Jen Lee, Hani, Bryce, Sara. `weekly-pipeline-report`: Ben Lavin, Ben. `sales-forecast`: Alaina, Elena. `deal-strategy-coach`: Alaina Loori, Farid. | Review whether names are stable reference data or should be dynamically resolved from HubSpot/roster sources. |
| 20 | INFO | REVIEW | Manifest drift, files with no manifest row: none. | Arithmetic: files provided = 14. Manifest rows = 14. File set minus manifest set = 0. | No action needed. |
| 21 | INFO | REVIEW | Manifest drift, manifest rows with no file: none. | Arithmetic: manifest rows = 14. Files provided = 14. Manifest set minus file set = 0. | No action needed. |
runbook-from-slack-thread0.75

? The Question

Turn the resolved incident thread into a runbook: ordered steps with the exact command or action taken, who ran it, how success was verified, and rollback for each step that changed state. Every step must trace to a message in the thread. Add nothing the thread does not contain; where the thread is unclear, mark the step as needing confirmation.
cost $0.0438 · 11s · in 1,034 / out 1,057 tok
✗ ordered_steps✓ rollback✓ needs_confirmation✓ sourced

✓ Correct Answer (ground truth)

steps: order=1; msg=M01; actor=Bryce Harmon; action=acknowledge PagerDuty alert, take IC; state_change=False; verified_by=M01 | order=2; msg=M02; actor=Farid Osman; action=check queue depth; command=bundle exec rake sidekiq:queue_depth; state_change=False; verified_by=M02 (48,213 pending) | order=3; msg=M04; actor=Farid Osman; action=pause auto-enqueue via feature flag; command=bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'; state_change=True; rollback=bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'; verified_by=M07 (queue depth falling) | order=4; msg=M05; actor=Elena Sinclair; action=cleared the dead set; state_change=True; needs_confirmation=thread gives no command or method for clearing the dead set | order=5; msg=M06; actor=Bryce Harmon; action=scale reward-worker to 6 replicas; command=kubectl scale deployment/reward-worker --replicas=6; state_change=True; rollback=kubectl scale deployment/reward-worker --replicas=3; verified_by=M07/M08 | order=6; msg=M08; actor=Cole Ingram; action=verify queue drained and error rate baseline; command=bundle exec rake sidekiq:queue_depth; state_change=False; verified_by=M08 (returns 0) … (+2 more)
ambiguous_step: order=4; msg=M05; mark=needs-confirmation
rule: no steps beyond what the thread contains; every step traces to a message id.
graded checks: ordered_steps · rollback · needs_confirmation · sourced

Model's actual answer

show the model's full answer
Runbook — reward-worker queue depth incident

| Order | Trace | Who ran it | Exact command or action taken | State changed? | Success verification from thread | Rollback |
|---:|---|---|---|---|---|---|
| 1 | [M01] | Bryce Harmon | Acknowledged PagerDuty alert for `reward-worker` queue depth > 10k and took IC. | No | Alert condition stated: queue depth > 10k. No separate success verification stated. Needs confirmation if acknowledgement verification is required. | Not applicable. |
| 2 | [M02] | Farid Osman | Ran `bundle exec rake sidekiq:queue_depth` | No | `reward` queue was 48,213 pending jobs. Normal is under 500. Arithmetic over normal threshold: 48,213 - 500 = 47,713 above stated normal upper bound. | Not applicable. |
| 3 | [M03] | Farid Osman | Checked dead set. | No | Dead set had 112 jobs, all `Redis::TimeoutError` from around 13:58. | Not applicable. |
| 4 | [M04] | Farid Osman | Paused enqueue to stop the bleed: `bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'` | Yes | No direct verification stated immediately after this step. Later queue depth was down to 9,400 and falling ~1,200/min [M07]. Needs confirmation whether that was accepted as verification of this specific step. | `bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'` |
| 5 | [M05] | Elena Sinclair | Cleared out the dead set while in the console. | Yes | No success verification stated. Needs confirmation. Exact command/action is unclear because the thread only says “cleared out the dead set.” | No rollback stated. Needs confirmation. |
| 6 | [M06] | Bryce Harmon | Scaled workers up: `kubectl scale deployment/reward-worker --replicas=6` | Yes | Previous replica count was 3. Later queue depth was down to 9,400 and falling ~1,200/min [M07]. Arithmetic: 48,213 - 9,400 = 38,813 fewer pending jobs from [M02] to [M07]. Needs confirmation whether that was accepted as verification of the scale-up specifically. | `kubectl scale deployment/reward-worker --replicas=3` |
| 7 | [M07] | Farid Osman | Checked queue progress. | No | Queue depth was down to 9,400 and falling ~1,200/min. Arithmetic from initial queue depth: 48,213 - 9,400 = 38,813 jobs reduced. | Not applicable. |
| 8 | [M08] | Cole Ingram | Verified with `bundle exec rake sidekiq:queue_depth` | No | Command returned 0. Error rate in Datadog was back to baseline. Arithmetic from initial queue depth: 48,213 - 0 = 48,213 jobs reduced. | Not applicable. |
| 9 | [M09] | Bryce Harmon | Re-enabled enqueue: `bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'` | Yes | Verified 40 new jobs processed cleanly in the next 3 minutes. | Rollback would be the prior disable command from [M04]: `bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'`. Needs confirmation because [M09] does not explicitly state rollback. |
| 10 | [M10] | Bryce Harmon | Scaled workers back down: `kubectl scale deployment/reward-worker --replicas=3` | Yes | Queue stable at 0. Incident resolved. Arithmetic from scaled-up replica count: 6 - 3 = 3 replicas removed. | Rollback would be the prior scale-up command from [M06]: `kubectl scale deployment/reward-worker --replicas=6`. Needs confirmation because [M10] does not explicitly state rollback. |
datadog-incident-timelinePASS 1.00

? The Question

From the log slice, sequence the failure: first error with timestamp, the cascade in order, the service and job involved, and the Datadog query that would confirm the first error. State what the logs do not show.
cost $0.1139 · 24s · in 4,622 / out 2,549 tok
✓ first_error_ts✓ service_named✓ cascade_ordered✓ query✓ gaps_stated

✓ Correct Answer (ground truth)

first_error: timestamp=2026-09-03T14:01:12Z; service=reward-service; message=Redis::TimeoutError: Connection to Redis at redis-primary:6379 timed out after 5s
cascade_order: service=reward-service; first_error_at=2026-09-03T14:01:12Z | service=sidekiq; first_error_at=2026-09-03T14:01:40Z | service=api-gateway; first_error_at=2026-09-03T14:03:05Z | service=web-app; first_error_at=2026-09-03T14:03:30Z
job_class_involved: RewardGiveJob
confirming_query: service:reward-service status:error "@message:Redis::TimeoutError"
logs_do_not_show: root cause of the Redis timeout, redis-server's own logs, infrastructure metrics (CPU/memory/network), any deploy or config change preceding the incident, who applied mitigation or when (no actor fields in logs)
clean_services: postgres
graded checks: first_error_ts · service_named · cascade_ordered · query · gaps_stated

Model's actual answer

show the model's full answer
First error:
2026-09-03T14:01:12Z — reward-service — ERROR — “Redis::TimeoutError: Connection to Redis at redis-primary:6379 timed out after 5s”

Cascade in order:
1. 2026-09-03T14:01:12Z — reward-service hits Redis timeout to redis-primary:6379.
   Arithmetic: baseline T0 = 14:01:12.

2. 2026-09-03T14:01:20Z — reward-service logs “Redis::TimeoutError: retry exhausted for RewardGiveJob”.
   Arithmetic: 14:01:20 - 14:01:12 = 8 seconds after first error.

3. 2026-09-03T14:01:30Z — reward-service logs another “retry exhausted for RewardGiveJob”.
   Arithmetic: 14:01:30 - 14:01:12 = 18 seconds after first error.

4. 2026-09-03T14:01:40Z — reward-service logs another “retry exhausted for RewardGiveJob”.
   Arithmetic: 14:01:40 - 14:01:12 = 28 seconds after first error.

5. 2026-09-03T14:01:40Z — sidekiq logs “RewardGiveJob failed: Redis::TimeoutError; retrying in 60s”.
   Arithmetic: 14:01:40 - 14:01:12 = 28 seconds after first error.

6. 2026-09-03T14:01:46Z — sidekiq_jobs.csv first recorded job failure: J-00005, RewardGiveJob, Redis::TimeoutError.
   Arithmetic: 14:01:46 - 14:01:12 = 34 seconds after first error.

7. 2026-09-03T14:02:28Z — sidekiq logs more RewardGiveJob failures with Redis::TimeoutError.
   Arithmetic: 14:02:28 - 14:01:12 = 76 seconds after first error.

8. 2026-09-03T14:02:30Z — sidekiq warns “Queue reward depth above 10,000”.
   Arithmetic: 14:02:30 - 14:01:12 = 78 seconds after first error.

9. 2026-09-03T14:02:36Z — sidekiq_jobs.csv first RecognitionDigestJob failure: J-00013, Redis::TimeoutError.
   Arithmetic: 14:02:36 - 14:01:12 = 84 seconds after first error.

10. 2026-09-03T14:03:05Z — api-gateway logs “502 upstream timeout calling reward-service /gives”.
    Arithmetic: 14:03:05 - 14:01:12 = 113 seconds after first error.

11. 2026-09-03T14:03:30Z — web-app logs “Give form submission failed: upstream 502 from api-gateway”.
    Arithmetic: 14:03:30 - 14:01:12 = 138 seconds after first error.

12. 2026-09-03T14:22:10Z — reward-service logs “Redis connection restored; resuming job processing”.
    Arithmetic: 14:22:10 - 14:01:12 = 20 minutes 58 seconds after first error.

13. 2026-09-03T14:24:45Z — sidekiq logs “Queue reward depth below 500”.
    Arithmetic: 14:24:45 - 14:01:12 = 23 minutes 33 seconds after first error.

Service and job involved:
service: reward-service
primary job shown in datadog_logs.csv: RewardGiveJob
worker/service emitting job failures: sidekiq
additional job shown only in sidekiq_jobs.csv: RecognitionDigestJob
upstream cascade services: api-gateway, web-app

Datadog query to confirm the first error:
service:reward-service level:error "Redis::TimeoutError" "redis-primary:6379"

What the logs do not show:
- They do not show the underlying cause of the Redis timeout at redis-primary:6379.
- They do not show Redis server logs, Redis metrics, host health, network status, or failover events.
- They do not show whether a deploy, config change, traffic spike, or infrastructure event caused the timeout.
- They do not show exact queue peak depth; only “above 10,000” and later “below 500”.
- They do not show total user impact or number of failed Give submissions.
- They do not show Postgres errors; postgres entries are only “checkpoint complete”.
- They do not prove there were no earlier errors outside this log slice.
feature-flag-state-summaryPASS 1.00

? The Question

Summarize the state of the listed feature flags: what each controls per the code excerpt, which companies or segments have it on, targeting rules, and any flag with no code reference or no targets. Company counts must match the export. Do not describe a flag that is not in the export.
cost $0.0392 · 10s · in 1,922 / out 773 tok
✓ orphans✓ no_targets✓ counts_match✓ no_invented

✓ Correct Answer (ground truth)

flags: {"recognition_streaks_v2": {"state": "on", "targeting_rules": "segment:beta_companies", "company_count": 42}, "points_budget_guardrails": {"state": "on", "targeting_rules": "all_companies", "company_c
code_referenced: recognition_streaks_v2, points_budget_guardrails, slack_dm_nudges, redeem_flow_redesign, analytics_dashboard_v3, ms_teams_app_v2
orphan_flags_no_code_reference: legacy_give_modal, survey_boosters_q3, paused_offboard_cleanup
flags_with_no_targets: paused_offboard_cleanup
rule: company counts must equal the export; do not describe flags absent from the export.
graded checks: orphans · no_targets · counts_match · no_invented

Model's actual answer

show the model's full answer
Feature flag state summary from export only:

| Flag | State | What it controls per code excerpt | Targeting / companies or segments | Company count |
|---|---:|---|---|---:|
| recognition_streaks_v2 | on | Enables `StreakTracker.record(give)` in `app/models/recognition.rb` | `segment:beta_companies` | 42 |
| points_budget_guardrails | on | Enables `BudgetService.new(company).enforce!(giver, points)` in `app/services/budget_service.rb` | `all_companies` | 220 |
| slack_dm_nudges | on | Allows `SlackDm.send_nudge(user)` in `app/jobs/nudge_job.rb` | `segment:region_na` | 87 |
| redeem_flow_redesign | off | Controls redeem UI: when enabled renders `RedeemV2Component`; otherwise renders `RedeemV1Component` | `targeted_list` | 12 |
| analytics_dashboard_v3 | on | Enables `@dashboard = AnalyticsV3.new(company)` in `app/controllers/analytics_controller.rb` | `segment:tier_three` | 65 |
| ms_teams_app_v2 | off | Enables `TeamsAppV2.install(company)` in `app/services/teams_installer.rb` | `targeted_list` | 9 |
| legacy_give_modal | off | No code reference provided | `segment:legacy_plan` | 14 |
| survey_boosters_q3 | on | No code reference provided | `segment:legacy_plan` | 7 |
| paused_offboard_cleanup | off | No code reference provided | No targeting rule provided | 0 |

Flags that are on:
- `recognition_streaks_v2`: `segment:beta_companies`, 42 companies
- `points_budget_guardrails`: `all_companies`, 220 companies
- `slack_dm_nudges`: `segment:region_na`, 87 companies
- `analytics_dashboard_v3`: `segment:tier_three`, 65 companies
- `survey_boosters_q3`: `segment:legacy_plan`, 7 companies

Arithmetic:
- On-flag company counts: 42 + 220 + 87 + 65 + 7 = 421
- Off-flag company counts: 12 + 9 + 14 + 0 = 35
- Total company-count entries across export: 421 + 35 = 456
- Total flags in export: 9

Flags with no code reference:
- `legacy_give_modal`: off, `segment:legacy_plan`, 14 companies
- `survey_boosters_q3`: on, `segment:legacy_plan`, 7 companies
- `paused_offboard_cleanup`: off, no targeting rule, 0 companies

Flags with no targets:
- `paused_offboard_cleanup`: targeting rule is blank, company count is 0
nda-triagePASS 1.00

? The Question

Triage each NDA as GREEN (standard approval), YELLOW (counsel review), or RED (full legal review). Cite the clause that drives each classification by quoting it. Catch embedded non-solicits, non-competes, missing carve-outs, and unusual term or governing law. One line of reasoning per NDA.
cost $0.0293 · 6s · in 1,962 / out 437 tok
✓ green✓ yellow✓ red✓ non_solicit_caught✓ clause_quoted✓ correct_labels

✓ Correct Answer (ground truth)

classifications: {"NDA-1": {"classification": "GREEN", "driving_reason": "standard mutual NDA: 2-year term, standard carve-outs, Delaware law", "driving_clause_quote": "Carve-outs: information that (a) is or becomes p
non_solicit_in: NDA-2
checks: classification_per_nda, clause_quotes_exist_in_document, non_solicit_caught, carveouts_checked
graded checks: green · yellow · red · non_solicit_caught · clause_quoted · correct_labels

Model's actual answer

show the model's full answer
NDA-1 — GREEN (standard approval) — Standard mutual NDA with normal term/survival and carve-outs: “Term: two (2) years from the Effective Date; obligations survive for three (3) years after termination” and “Carve-outs: information that (a) is or becomes publicly available through no breach, (b) was known prior to disclosure, (c) is received from a third party without duty of confidence, (d) is independently developed, or (e) must be disclosed by law or court order”; arithmetic: 2-year term + 3-year post-termination survival, with Delaware law and no non-solicit/non-compete/exclusivity.

NDA-2 — YELLOW (counsel review) — Embedded non-solicit requires review despite otherwise standard terms: “During the term of this Agreement and for eighteen (18) months thereafter, neither party shall solicit for employment or hire any employee of the other party with whom it came into contact under this Agreement”; arithmetic: 3-year term + 18-month post-term restriction = up to 4.5 years of restricted hiring/solicitation exposure.

NDA-3 — RED (full legal review) — Contains a broad non-compete, unusual governing law/forum, one-way structure, long term, and missing carve-outs: “For a period of three (3) years following the Effective Date, Recipient shall not, directly or indirectly, engage in or provide services to any business that competes with Discloser's business,” “This Agreement shall be governed by and construed in accordance with the laws of the Republic of Ireland, and the parties submit to the exclusive jurisdiction of its courts,” and “No confidentiality carve-outs are stated in this Agreement”; arithmetic: 5-year confidentiality term + 3-year non-compete restriction, with zero stated carve-outs.