home · ← back to all models
moonshotai

moonshotai/kimi-k2.7-code

Mean 0.953 · 31/40 perfect tests · $1.57 total · each card: the question → the correct answer (gold) → this model's actual answer

data-crm
stage-hygiene-auditPASS 1.00

? The Question

Audit stage hygiene on the open pipeline below. Rules: a deal at DS2 or later must have a substantive why-buys (15+ characters, not a placeholder such as tbd, n/a, see notes). A why_buys value of the form LONG(n) means the real text was present but truncated for display; treat it as present and substantive. Every deal should have a DS1 entry timestamp. A deal whose stage entry timestamps are out of order regressed (an earlier-stage timestamp is later than a later-stage timestamp). Report: percent of DS2+ deals non-compliant on why-buys overall and by owner (use the owners table for names), count of deals with no DS1 timestamp, count that regressed, and three process fixes. State definitions before numbers.
cost $0.0898 · 1533s · in 39,492 / out 15,045 tok
✓ noncomp_pct✓ no_ds1_0✓ regressed_0✓ per_owner_breakdown✓ definitions_first✓ three_fixes

✓ Correct Answer (ground truth)

noncomp_pct: 81.5
ds2plus: 124
justin_noncomp: 15
no_ds1: 0
regressed: 0
n_deals: 156
graded checks: noncomp_pct · no_ds1_0 · regressed_0 · per_owner_breakdown · definitions_first · three_fixes

Model's actual answer

show the model's full answer
Definitions:
- Substantive why-buys: present and 15+ characters; LONG(n) treated as substantive; empty or placeholder (tbd, n/a, see notes) is non-compliant.
- DS2+ deals: deals currently in stage DS2, DS3, DS4, or DS5.
- Regression: any earlier-stage timestamp later than a later-stage timestamp (DS1≤DS2≤DS3≤DS4≤DS5 violated).

Why-buys compliance on DS2+ deals:
- Overall: 101 of 124 non-compliant = 81.5%.
- By owner:
  - Bryce Harmon: 15/18 = 83.3%
  - Farid Osman: 1/1 = 100.0%
  - Dana Mercer: 18/22 = 81.8%
  - Cole Ingram: 19/22 = 86.4%
  - Alex Franklin: 48/61 = 78.7%

Stage timestamp hygiene:
- Deals with no DS1 timestamp: 0.
- Deals with regressed stage timestamps: 0.

Three process fixes:
1. Enforce why-buys gate: block manual stage advance from DS1→DS2 until a 15+ character, non-placeholder why-buys is saved in the field; LONG(n) accepted automatically.
2. Add validation rule in CRM: require t_ds1 ≤ t_ds2 ≤ t_ds3 ≤ t_ds4 ≤ t_ds5; warn reps and prevent backdating future stages.
3. Build a weekly hygiene report by owner and require each AE to update missing why-buys before forecast review; tie forecast category changes to hygiene completion.
owner-resolution0.83

? The Question

The deal extract below has an owner name field that is blank on nearly every row and an owner id field that is populated. Using the owners table, resolve each open deal to an owner name. Report: number of deals resolved, deals whose owner id has no match in the owners table (list them), the owner ids that map to archived/deactivated owners (list their names), and the total pipeline amount per resolved owner. Do not guess a name for any unmatched id.
cost $0.0763 · 1527s · in 38,015 / out 13,521 tok
✓ resolved_count✓ unmatched_none✗ deactivated_flagged✓ jake_amount✓ justin_amount✓ no_invented

✓ Correct Answer (ground truth)

resolved: 156
jake_amount: 1054144.0
justin_amount: 624310.0
per_owner: Bryce Harmon=1054144.0; Alex Franklin=624310.0; Dana Mercer=341195.0; Cole Ingram=288161.43; Farid Osman=4134.0; Elena Sinclair=2100.0
deactivated:
graded checks: resolved_count · unmatched_none · deactivated_flagged · jake_amount · justin_amount · no_invented

Model's actual answer

show the model's full answer
Processed all 156 open deals against the owners table.

- Deals resolved to an owner name: **156**
- Deals whose owner id has no match in owners table: **0** (none)
- Owner ids that map to archived/deactivated owners and have deals: **0** (none; the two archived owner ids `1520255671` and `77260721` do not appear in the open deals extract)
- Total pipeline per resolved owner:
  - **119337721 Bryce Harmon**: 1,054,144.00
  - **84342457 Alex Franklin**: 624,310.00
  - **83155923 Dana Mercer**: 341,195.00
  - **83155924 Cole Ingram**: 288,161.43
  - **716654662 Farid Osman**: 4,134.00
  - **701163055 Elena Sinclair**: 2,100.00

Sum check of all resolved owner pipelines: **2,314,044.43**.
gong-hubspot-join-integrity0.75

? The Question

Check the integrity of the call-to-deal mapping below. The gong table lists calls in the last 90 days keyed to HubSpot deal ids; deals_open lists the current open default-pipeline deals. Report: (a) count of gong rows whose deal id has NO match in deals_open (orphans; list a few sample aliases), (b) duplicate conversation keys (rows where calls_90d exceeds distinct_conversation_keys), (c) among the open deals that ARE at stage DS3 or later, the share that have at least one logged call. Give sample aliases for the orphan finding, all taken from the data. Note: gong_owner_id is a Gong user id, not a HubSpot owner id.
cost $0.1061 · 38s · in 76,231 / out 16,416 tok
✓ orphan_count✗ dup_keys_0✓ ds3_coverage✓ sample_ids_present

✓ Correct Answer (ground truth)

orphans: 30
orphan_aliases: Deal-8FA85D, Deal-8FC3F9, Deal-3B7945, Deal-42B265, Deal-9CCC42, Deal-36EA09
dup: 0
ds3_cov_pct: 29.4
graded checks: orphan_count · dup_keys_0 · ds3_coverage · sample_ids_present

Model's actual answer

show the model's full answer
Integrity check results:

(a) Orphan gong rows (hs_deal_id not in deals_open): **30 rows**.
Sample aliases: Deal-8FA85D, Deal-8FC3F9, Deal-3B7945, Deal-42B265, Deal-9CCC42, Deal-36EA09, Deal-9A43B4, Deal-605F3C, Deal-E2D34B, Deal-76821A.

Arithmetic: 67 total gong rows − 37 matched open deal IDs = **30 orphans**.

(b) Duplicate conversation keys (calls_90d > distinct_conversation_keys): **0 rows**.

(c) Open deals at stage DS3 or later with at least one logged call: **29.4%**.
Arithmetic: 25 covered / 85 DS3+ open deals = 0.2941 → **29.4%**.
snowflake-sql-trapped-catalogPASS 1.00

? The Question

Using ONLY the catalog provided, write one Snowflake SQL query returning, per customer company, for its first calendar month as a customer: unique givers, recognition count, successful redemption count. Read the catalog notes carefully: some tables are stale or unpopulated and must be avoided, and one documented filter (excluding deleted givers) must NOT be applied to historical giving counts. Put a comment above each FROM or JOIN explaining the table choice. Return only fenced SQL.
cost $0.0310 · 277s · in 965 / out 8,152 tok
✓ avoids_stale_deals✓ avoids_unpopulated_live✓ uses_cohort_summary✓ succeeded_filter✓ no_giver_deletion_filter✓ returns_sql

✓ Correct Answer (ground truth)

graded checks: avoids_stale_deals · avoids_unpopulated_live · uses_cohort_summary · succeeded_filter · no_giver_deletion_filter · returns_sql

Model's actual answer

show the model's full answer
```sql
SELECT
    COMPANY_ID,
    DATE_TRUNC('MONTH', SIGNUP_DATE) AS FIRST_CALENDAR_MONTH,
    M1_USERS AS UNIQUE_GIVERS,
    M1_GIVES AS RECOGNITION_COUNT,
    M1_REDEMPTIONS AS SUCCESSFUL_REDEMPTION_COUNT
-- CHOICE: PRODUCTION.PLG.COMPANY_COHORT_SUMMARY is the only catalog object with per-company first-month engagement metrics (M1_USERS, M1_GIVES, M1_REDEMPTIONS). It avoids the unpopulated HUBSPOT_HUB_1973303.V2_LIVE.OBJECTS_DEALS table and the stale PRODUCTION.HUBSPOT.DEALS table. M1_GIVES is used directly as the historical recognition count, so no deleted-giver exclusion filter is applied.
FROM PRODUCTION.PLG.COMPANY_COHORT_SUMMARY;
```
crm-completeness-enrichmentPASS 1.00

? The Question

Audit the CRM extract for completeness and propose fixes. Deals need owner, stage, amount, close date, why-buys; companies need industry, employee count, HQ country; contacts need email, title, persona. Report completeness percent per field. Identify duplicate company clusters (name variants, shared domain) with a survivor per cluster. List invalid emails and domain mismatches. Fill missing company fields only where the enrichment export has a matching row; where CRM and enrichment disagree list both and recommend a source. Never invent a value. End with the 10 fixes with the most pipeline amount at stake.
cost $0.0683 · 2425s · in 45,345 / out 9,473 tok
✓ dup_clusters✓ invalid_emails✓ survivor_language✓ completeness_pct✓ no_invent

✓ Correct Answer (ground truth)

n_companies: 34
n_contacts: 52
company_missing: industry=0; employee_count=9; hq_country=6
invalid_email_count: 4
invalid_emails: CT-0010, CT-0080, CT-0081, CT-0192
domain_mismatch_count: 1
domain_mismatches: CT-0011
duplicate_clusters: ['C-0A092931', 'C-0A092932'], ['C-0A092933', 'C-0A092934']
n_dup_clusters: 2
enrichment_matches: 25
graded checks: dup_clusters · invalid_emails · survivor_language · completeness_pct · no_invent

Model's actual answer

show the model's full answer
CRM extract audit

1. DEAL DATA — NOT PROVIDED
   The request asks me to audit deal fields (owner, stage, amount, close date, why-buys) and report the top 10 fixes by pipeline amount at stake. No deals.csv was included, so those sections cannot be computed. I report only on companies and contacts.

---

2. COMPLETENESS PER FIELD

Companies (n=34)
- domain:       34/34 = 100.0%
- industry:     34/34 = 100.0%
- employee_count: 25/34 = 73.5%
- hq_country:   28/34 = 82.4%

Contacts (n=52)
- email:   52/52 = 100.0%
- title:   39/52 = 75.0%
- persona: 37/52 = 71.2%

---

3. DUPLICATE COMPANY CLUSTERS

Shared-domain clusters:

Domain: acme-corp.com
  Aliases: C-0A092931, C-0A092932
  Proposed survivor: C-0A092931 (most populated CRM row)
    C-0A092931: industry=Technology, employee_count=500, hq_country=US
    C-0A092932: industry=tech, employee_count=510, hq_country=USA

Domain: globex.io
  Aliases: C-0A092933, C-0A092934
  Proposed survivor: C-0A092933 (tie on populated fields, first alias)
    C-0A092933: industry=SaaS, employee_count=200, hq_country=US
    C-0A092934: industry=Technology, employee_count=200, hq_country=US

Note: There is no explicit company_name column, so "name variants" map directly to alias-domain duplicates in this extract.

---

4. INVALID EMAILS AND DOMAIN MISMATCHES

Invalid emails (4):
- CT-0010 | C-66D1FC | "user0@" — empty domain
- CT-0080 | C-92D97D | "user0@" — empty domain
- CT-0081 | C-92D97D | "user1@" — empty domain
- CT-0192 | C-425E2A | "user2@" — empty domain

Domain mismatches (1):
- CT-0011 | C-66D1FC | email=user1@other-domain.com, email domain=other-domain.com, company domain=66d1fc.com

---

5. ENRICHMENT FILL / CONFLICT REVIEW

ZoomInfo fills for blank CRM fields (8):

| Company | Domain | Field | CRM value | ZoomInfo value |
|---|---|---|---|---|
| C-EC3025 | ec3025.com | employee_count | (blank) | 400 |
| C-96039F | 96039f.com | employee_count | (blank) | 400 |
| C-44EA29 | 44ea29.com | employee_count | (blank) | 400 |
| C-D04904 | d04904.com | employee_count | (blank) | 400 |
| C-B23205 | b23205.com | employee_count | (blank) | 400 |
| C-60C75F | 60c75f.com | employee_count | (blank) | 400 |
| C-7BBDFA | 7bbdfa.com | employee_count | (blank) | 400 |
| C-50D386 | 50d386.com | employee_count | (blank) | 400 |

CRM vs ZoomInfo conflicts (10; all industry wording disagreements, US/USA treated as same):

| Company | Domain | Field | CRM value | ZoomInfo value |
|---|---|---|---|---|
| C-66D1FC | 66d1fc.com | industry | tech | Computer Software |
| C-EC3025 | ec3025.com | industry | Technology | Computer Software |
| C-44EA29 | 44ea29.com | industry | tech | Computer Software |
| C-92D97D | 92d97d.com | industry | Technology | Computer Software |
| C-D04904 | d04904.com | industry | Technology | Computer Software |
| C-77A95A | 77a95a.com | industry | Technology | Computer Software |
| C-AA8DDA | aa8dda.com | industry | Technology | Computer Software |
| C-B25F40 | b25f40.com | industry | Technology | Computer Software |
| C-60C75F | 60c75f.com | industry | tech | Computer Software |
| C-425E2A | 425e2a.com | industry | Tech  | Computer Software |

Recommendation on conflicts: Adopt ZoomInfo's "Computer Software" as the normalized industry value and update CRM taxonomy to prevent mixed casing/abbreviations ("tech", "Tech ", "Technology").

Missing company fields with no ZoomInfo match (7):
- C-2D1F1B (2d1f1b.com): hq_country
- C-D73B89 (d73b89.com): hq_country
- C-44EA29 (44ea29.com): hq_country
- C-D04904 (d04904.com): hq_country
- C-2C60E5 (2c60e5.com): hq_country
- C-93C8BF (93c8bf.com): employee_count
- C-EE9FFB (ee9ffb.com): hq_country

No enrichment row exists for domains 332637.com, ba969b.com, b97b4e.com, c9bb20.com, ee9ffb.com, or 2c60e5.com beyond what is shown; some fields remain unfillable.

---

6. TOP 10 FIXES BY PIPELINE AMOUNT AT STAKE

Cannot be computed — deals.csv was not provided. Without deal amount and company mapping, no pipeline value can be assigned to any fix.

---

PROPOSED FIXES (priority order, limited to available data)

1. Merge duplicate company clusters
   - acme-corp.com: keep C-0A092931, deprecate C-0A092932, reconcile employee_count 500 vs 510.
   - globex.io: keep C-0A092933, deprecate C-0A092934, reconcile industry SaaS vs Technology.

2. Fix invalid emails
   - CT-0010, CT-0080, CT-0081, CT-0192: complete or replace partial emails.

3. Fix domain mismatch
   - CT-0011: reverify email user1@other-domain.com against C-66D1FC or associate with the correct company.

4. Fill blank employee_count from ZoomInfo
   - C-EC3025, C-96039F, C-44EA29, C-D04904, C-B23205, C-60C75F, C-7BBDFA, C-50D386.

5. Normalize industry conflicts to "Computer Software"
   - C-66D1FC, C-EC3025, C-44EA29, C-92D97D, C-D04904, C-77A95A, C-AA8DDA, C-B25F40, C-60C75F, C-425E2A.

6. Title backfill on contacts (13 missing)
   - Example aliases: CT-0000, CT-0022, CT-0041, CT-0060, CT-0072, CT-0082, CT-0092, CT-0120, CT-0121, CT-0122, CT-0132, CT-0162, CT-0170, CT-0171, CT-0172, CT-0180, CT-0181. (Exact list from CSV.)

7. Persona backfill on contacts (15 missing)
   - Example aliases: CT-0000, CT-0022, CT-0041, CT-0060, CT-0070, CT-0072, CT-0081, CT-0082, CT-0092, CT-0110, CT-0171, CT-0172, CT-0180, CT-0181. (Exact list from CSV.)

8. Employee-count gaps with no enrichment (1)
   - C-93C8BF (93c8bf.com): employee_count missing, no ZoomInfo row.

9. HQ-country gaps with no enrichment (5 plus others)
   - C-2D1F1B, C-D73B89, C-44EA29, C-D04904, C-2C60E5, C-EE9FFB are missing hq_country; only some have enrichment rows but those rows themselves lack hq_country.

10. Request deals.csv
    - Required to compute pipeline-at-stake ranking, deal-field completeness, and tie fixes to revenue impact.

---

I can rerun the pipeline-amount ranking immediately if you provide deals.csv.
deal-intelligence
closed-lost-classificationPASS 1.00

? The Question

Classify every deal lost in the last 6 months below into one primary category: pricing, competitor, no decision, timing, product gap, champion left, or other. Also assign a side (Bonusly, buyer, unknown). Then summarize: category counts, the side split, the number of deals where the structured closed_lost_tag clearly disagrees with the free-text reason, and the two patterns most worth acting on. Base everything only on the tag and free-text provided.
cost $0.1224 · 214s · in 63,943 / out 19,840 tok
✓ total_90✓ competitor_top✓ category_counts_present✓ side_split✓ two_patterns

✓ Correct Answer (ground truth)

graded checks: total_90 · competitor_top · category_counts_present · side_split · two_patterns

Model's actual answer

show the model's full answer
## Classification

Primary category assigned from tag and free-text reason, side assigned by who owns the cause.

| Deal       | Category     | Side     |
|------------|--------------|----------|
| Deal-DB0AAC | timing       | buyer    |
| Deal-F7F635 | competitor   | unknown  |
| Deal-AC944F | other        | unknown  |
| Deal-214060 | other        | unknown  |
| Deal-91A056 | timing       | buyer    |
| Deal-29326C | timing       | buyer    |
| Deal-5DB9B0 | other        | unknown  |
| Deal-831B7B | timing       | buyer    |
| Deal-F97C37 | competitor   | Bonusly  |
| Deal-13E9CF | no decision  | buyer    |
| Deal-39E25C | timing       | buyer    |
| Deal-7ED004 | pricing      | buyer    |
| Deal-21B045 | other        | unknown  |
| Deal-B3ABED | timing       | buyer    |
| Deal-422BA6 | competitor   | Bonusly  |
| Deal-ED9AE7 | no decision  | buyer    |
| Deal-988493 | other        | unknown  |
| Deal-381C8C | competitor   | unknown  |
| Deal-F308CA | other        | unknown  |
| Deal-F1E8A6 | competitor   | unknown  |
| Deal-B6AC09 | timing       | buyer    |
| Deal-70F704 | other        | unknown  |
| Deal-E6E80A | timing       | buyer    |
| Deal-B038F0 | timing       | buyer    |
| Deal-4664E1 | other        | unknown  |
| Deal-175756 | timing       | buyer    |
| Deal-E74A73 | no decision  | buyer    |
| Deal-DDAB52 | competitor   | Bonusly  |
| Deal-ACE061 | competitor   | buyer    |
| Deal-BB78F3 | timing       | buyer    |
| Deal-D48E0B | other        | unknown  |
| Deal-15DA99 | timing       | buyer    |
| Deal-F4AF5D | timing       | buyer    |
| Deal-79B7A1 | timing       | buyer    |
| Deal-583ADB | other        | unknown  |
| Deal-8E27DA | product gap  | Bonusly  |
| Deal-2D2F8D | competitor   | unknown  |
| Deal-E0441F | other        | unknown  |
| Deal-7CB44D | other        | unknown  |
| Deal-0F96AA | competitor   | unknown  |
| Deal-1BCA50 | competitor   | buyer    |
| Deal-7CC678 | competitor   | unknown  |
| Deal-FAC17C | pricing      | buyer    |
| Deal-242273 | competitor   | Bonusly  |
| Deal-50E5D8 | no decision  | buyer    |
| Deal-A2C349 | competitor   | buyer    |
| Deal-9F176A | timing       | buyer    |
| Deal-7B2236 | no decision  | buyer    |
| Deal-AFA56C | other        | unknown  |
| Deal-C7156E | competitor   | unknown  |
| Deal-C33D91 | pricing      | buyer    |
| Deal-9048EB | other        | unknown  |
| Deal-5E64CE | no decision  | buyer    |
| Deal-8A0992 | competitor   | buyer    |
| Deal-D0C698 | competitor   | buyer    |
| Deal-69CF3D | timing       | buyer    |
| Deal-ECBF89 | timing       | buyer    |
| Deal-3618CC | product gap  | Bonusly  |
| Deal-EECC02 | competitor   | unknown  |
| Deal-5AD03E | competitor   | unknown  |
| Deal-D1A623 | timing       | buyer    |
| Deal-413C56 | no decision  | buyer    |
| Deal-47F1A1 | competitor   | buyer    |
| Deal-BF2A98 | competitor   | buyer    |
| Deal-2A292B | no decision  | buyer    |
| Deal-D1AABF | other        | unknown  |
| Deal-FEDBCB | no decision  | buyer    |
| Deal-1E7DA9 | competitor   | unknown  |
| Deal-2BBA21 | other        | unknown  |
| Deal-286F9C | competitor   | unknown  |
| Deal-7FBAC6 | no decision  | buyer    |
| Deal-369281 | competitor   | buyer    |
| Deal-386F6E | other        | unknown  |
| Deal-9FCD0D | competitor   | buyer    |
| Deal-55867E | timing       | buyer    |
| Deal-DAFB82 | pricing      | buyer    |
| Deal-2FEDDB | no decision  | buyer    |
| Deal-64B19A | competitor   | buyer    |
| Deal-3F86A0 | other        | unknown  |
| Deal-096750 | other        | unknown  |
| Deal-F325A5 | no decision  | buyer    |
| Deal-ABD14C | no decision  | buyer    |
| Deal-79E61A | other        | unknown  |
| Deal-8A119B | pricing      | buyer    |
| Deal-AE7C4E | other        | unknown  |
| deal-DAB4F1 | other        | unknown  |
| Deal-B4B50F | other        | unknown  |
| Deal-981AD4 | product gap  | Bonusly  |
| Deal-DC77FE | competitor   | Bonusly  |
| Deal-5885B9 | other        | unknown  |

## Summary

### Category counts (n=90)
- competitor: 26 (28.9%)
- other: 24 (26.7%)
- timing: 19 (21.1%)
- no decision: 13 (14.4%)
- pricing: 5 (5.6%)
- product gap: 3 (3.3%)
- champion left: 0 (0.0%)

### Side split
- buyer: 47 (52.2%)
- unknown: 35 (38.9%)
- Bonusly: 8 (8.9%)

### Tag/text disagreements: 17 deals

1. Deal-13E9CF — tag "Doing nothing/Not a priority/Cost" vs text "Not a budget issue" (pricing).
2. Deal-B3ABED — tag "Lost- Timing" vs text "get budget for in 2028" (pricing).
3. Deal-ED9AE7 — tag "Lost DM" vs text "Timing, budget, authority" (pricing).
4. Deal-E74A73 — tag "Doing nothing/Not a priority/Cost" vs text "in touch again sometime next year" (timing).
5. Deal-BB78F3 — tag "Lost- Timing" vs text "roll out survey action items first" (product gap).
6. Deal-8E27DA — tag "Feature Request" vs text "moved forward with just a swag provider" (competitor).
7. Deal-FAC17C — tag "Lost DM" vs text "couldn't get final approval" (pricing).
8. Deal-50E5D8 — tag "Doing nothing/Not a priority/Cost" vs text "pause for now...reach out in the future" (timing).
9. Deal-7B2236 — tag "Doing nothing/Not a priority/Cost" vs text "prefer something simpler and cheaper" (competitor).
10. Deal-9048EB — tag "MIA" vs text "bad fit...multiple feature gaps" (product gap).
11. Deal-5E64CE — tag "Doing nothing/Not a priority/Cost" vs text "Nectar agreement through October 2027" (competitor).
12. Deal-3618CC — tag "Lost DM" vs text "Wanted Surveys" (product gap).
13. Deal-5AD03E — tag "Competitor" vs text "Wanted more defined budget access" (pricing).
14. Deal-2A292B — tag "Doing nothing/Not a priority/Cost" vs text "build something simple internally" (product gap).
15. Deal-7FBAC6 — tag "Doing nothing/Not a priority/Cost" vs text "decision to pause for now" (timing).
16. Deal-2FEDDB — tag "Doing nothing/Not a priority/Cost" vs text "Unsure on timing" (timing).
17. Deal-DC77FE — tag "Competitor" vs text "you were competitive in price...customization" (pricing).

### Two patterns most worth acting on

1. **Competitor losses are the largest bucket and include clear Bonusly-side gaps**
   - 26 deals, $406,234.96 at risk.
   - Several losses explicitly cite Bonusly gaps: Deal-F97C37 ("more diversified offerings"), Deal-422BA6 (ADP TotalSource PEO partner/pre-built integration), Deal-242273 (digitize internal points currency / onsite facilities), Deal-DDAB52 (exchange-rate / FX budgeting), Deal-DC77FE (customization like labeling points as dollars), Deal-981AD4 (UI fit / UK focus).
   - Pattern: buyers are not just choosing incumbents; they are choosing alternatives that solve specific platform/configurable-reward needs Bonusly did not match.

2. **MIA / "Lost DM" / other is the second-largest bucket and mostly unexplained**
   - 24 deals as "other", plus several "Lost DM" records (Deal-ED9AE7, Deal-70F704, Deal-FAC17C, Deal-3618CC, Deal-F325A5). Many are simply "unresponsive" or "no response" with no real buyer-side diagnosis.
   - Combined unknown-side deals = 35 (38.9%), representing $267,942 + portions of Lost DM.
   - Pattern: rep follow-up is dying silently; there is no documented next-step agreement or buyer blocker, so these are potentially recoverable but currently invisible.
pipeline-tiering0.80

? The Question

Score the open pipeline below on these signals: forecast category, stage, meetings_30d, contact count, and engagement recency. Assign exactly one tier per deal: LOCK, ACTION, BUILD, REVIVE, WATCH, or RISKY (RISKY = the forecast category disagrees with the engagement evidence). Never tier a deal LOCK if it has zero meetings_30d. Do NOT list every deal. Return ONLY a compact JSON object with exactly these keys: - "tier_counts": object mapping each tier to the number of deals in it (must sum to the total deal count), - "tier_examples": object mapping each tier to up to 3 example deal aliases from the data, - "risky_deals": array of the deal aliases you tiered RISKY, - "lock_violations": number of deals tiered LOCK that have zero meetings_30d (should be 0), - "pipeline_shape": one short paragraph on the overall pipeline shape. Note: inbound_emails_30d is 0 for every row due to a data defect, so treat meetings_30d as the inbound signal.
cost $0.0185 · 20s · in 17,408 / out 1,129 tok
✓ json_returned✓ tier_vocabulary✗ tier_counts_sum✓ risky_deals_listed✓ no_lock_violation_claimed

✓ Correct Answer (ground truth)

total: 156
graded checks: json_returned · tier_vocabulary · tier_counts_sum · risky_deals_listed · no_lock_violation_claimed

Model's actual answer

show the model's full answer
```json
{
  "tier_counts": {
    "LOCK": 6,
    "ACTION": 15,
    "BUILD": 28,
    "REVIVE": 36,
    "WATCH": 15,
    "RISKY": 15
  },
  "tier_examples": {
    "LOCK": ["Deal-5408B0", "Deal-C26D20", "Deal-944310"],
    "ACTION": ["Deal-C6FE92", "Deal-950043", "Deal-036E80"],
    "BUILD": ["Deal-A5E80A", "Deal-499BF6", "Deal-D73B89"],
    "REVIVE": ["Deal-9AAE5F", "Deal-6787C2", "Deal-0660B4"],
    "WATCH": ["Deal-2D1F1B", "Deal-012CB1", "Deal-483B2D"],
    "RISKY": ["Deal-547B2B", "Deal-B7EBD1", "Deal-1FC049"]
  },
  "risky_deals": [
    "Deal-547B2B",
    "Deal-B7EBD1",
    "Deal-1FC049",
    "Deal-A5E80A",
    "Deal-499BF6",
    "Deal-8AD4A5",
    "Deal-15D24F",
    "Deal-ED725A",
    "Deal-7E2131",
    "Deal-7599B8",
    "Deal-F9A3C1",
    "Deal-5EED42",
    "Deal-42F601",
    "Deal-B936FE",
    "Deal-57FF13"
  ],
  "lock_violations": 0,
  "pipeline_shape": "The 115-deal open pipeline is heavily back-loaded in DS1/DS2 ($1 placeholder amounts suggest prospecting noise) and PIPELINE forecast, with only a small late-stage core showing recent meeting activity. Most deals have zero meetings in the last 30 days and stale last-contact dates, driving a large REVIVE population. RISK is concentrated among COMMIT/BEST_CASE deals that lack contemporaneous meeting evidence or recent contact."
}
```
call-transcript-extractionPASS 1.00

? The Question

For each transcript, extract CRM write-back fields as JSON: why-buys (prospect statements only), pain points, stakeholders from the speaker list, budget signal (prospect-stated or null), timeline signal, competitor mentioned (only if the prospect raised it), next step (only if explicitly agreed), objections, confidence. Never fill a field from what the rep said or hoped.
cost $0.0319 · 635s · in 1,827 / out 8,250 tok
✓ json_emitted✓ budget_signal✓ null_discipline✓ stakeholders✓ competitor_prospect_only

✓ Correct Answer (ground truth)

rep: Alex Franklin
per_transcript: {"TX-001": {"why_buys": ["automating anniversary and birthday awards"], "pain_points": ["HR team of three cannot keep up with it manually", "track everything in a spreadsheet, and people slip through
rep_sourced_nulls: {"TX-003": ["budget_signal \u2014 only the rep mentioned pricing/budget"], "TX-004": ["next_step \u2014 rep proposed, prospect said 'no promises'"], "TX-006": ["budget_signal \u2014 rep decoy, prospec
allowed_new: TX-001, TX-002, TX-003, TX-004, TX-005, TX-006, Achievers, Bucketlist, Nectar
graded checks: json_emitted · budget_signal · null_discipline · stakeholders · competitor_prospect_only

Model's actual answer

show the model's full answer
```json
{
  "TX-001": {
    "deal_alias": "Deal-CFE7F4",
    "why-buys": [
      "The big win for us would be automating anniversary and birthday awards — our HR team of three cannot keep up with it manually."
    ],
    "pain-points": [
      "Right now we track everything in a spreadsheet, and people slip through the cracks."
    ],
    "stakeholders": [
      "VP People",
      "HR Admin"
    ],
    "budget-signal": "$40k earmarked for engagement tools this fiscal year",
    "timeline-signal": "Live before open enrollment in November",
    "competitor-mentioned": "Achievers",
    "next-step": "Security review on September 12",
    "objections": [
      "Need SSO and audit logs for IT to sign off"
    ],
    "confidence": "High — explicit budget ($40k), timeline (Nov), named competitor (Achievers), agreed next step (9/12); 2 stakeholders; 1 objection/requirement (SSO/audit logs)."
  },
  "TX-002": {
    "deal_alias": "Deal-70BB30",
    "why-buys": [
      "We want to tie recognition to retention for our hourly workforce — regretted turnover there is over 30%."
    ],
    "pain-points": [
      "Regretted turnover for hourly workforce is over 30%.",
      "Integration with Workday has to be rock solid — that's my one condition."
    ],
    "stakeholders": [
      "Head of Total Rewards",
      "CFO"
    ],
    "budget-signal": "$25k pilot budget for this quarter",
    "timeline-signal": "Decision by end of September",
    "competitor-mentioned": null,
    "next-step": "Send the pilot agreement and route it to legal this week",
    "objections": [
      "Integration with Workday has to be rock solid"
    ],
    "confidence": "High — explicit budget ($25k), timeline (end of Sept), agreed next step (pilot agreement), no competitor; 2 stakeholders; 1 objection/requirement (Workday integration)."
  },
  "TX-003": {
    "deal_alias": "Deal-530B50",
    "why-buys": [
      "We need to make recognition visible across our 12 retail locations.",
      "Store managers have zero budget autonomy for on-the-spot recognition today."
    ],
    "pain-points": [
      "Recognition is not visible across 12 retail locations.",
      "Store managers have zero budget autonomy for on-the-spot recognition today."
    ],
    "stakeholders": [
      "People Ops Manager"
    ],
    "budget-signal": null,
    "timeline-signal": "No rush until Q1",
    "competitor-mentioned": "Bucketlist",
    "next-step": "Schedule a call with the CEO — People Ops Manager will send two times",
    "objections": [
      "No rush until Q1",
      "The CEO has to be sold first — she decides anything people-related.",
      "My CEO used Bucketlist at her last company and liked it."
    ],
    "confidence": "Medium — no explicit budget, named competitor (Bucketlist) with CEO preference, soft timeline (Q1), agreed next step (CEO call); 1 stakeholder; 3 objections/stalls."
  },
  "TX-004": {
    "deal_alias": "Deal-180D02",
    "why-buys": [
      "We want to consolidate three separate recognition tools into one.",
      "We're paying for three tools and none of them talk to our HRIS."
    ],
    "pain-points": [
      "Paying for three separate recognition tools.",
      "None of the three tools talk to the HRIS.",
      "The security review took three months for our last vendor — that's my hesitation."
    ],
    "stakeholders": [
      "VP People",
      "IT Security Lead"
    ],
    "budget-signal": "Under $15k annually (VP People can approve without board)",
    "timeline-signal": "Procurement cycle runs six to eight weeks minimum",
    "competitor-mentioned": null,
    "next-step": null,
    "objections": [
      "Procurement cycle runs six to eight weeks minimum.",
      "The security review took three months for our last vendor — that's my hesitation.",
      "CFO follow-up not confirmed: \"Maybe — I need to check her calendar, no promises.\""
    ],
    "confidence": "Medium — budget threshold (<$15k) and procurement timeline (6-8 weeks), 2 stakeholders, but next step not locked and security-review risk cited; 3 objections."
  },
  "TX-005": {
    "deal_alias": "Deal-F8767A",
    "why-buys": [
      "Two things: automate service milestones, and give us analytics on recognition equity across departments.",
      "Our night-shift teams feel invisible — their engagement scores run 20 points lower."
    ],
    "pain-points": [
      "Service milestones are not automated.",
      "No analytics on recognition equity across departments.",
      "Night-shift teams feel invisible; engagement scores run 20 points lower.",
      "Exec team is skeptical after a failed rollout two years ago."
    ],
    "stakeholders": [
      "HR Director",
      "People Ops Coordinator"
    ],
    "budget-signal": "$12k approved under the engagement line",
    "timeline-signal": "Running before the January all-hands",
    "competitor-mentioned": "Nectar",
    "next-step": "Present to the exec team on October 2",
    "objections": [
      "Exec team is skeptical after a failed rollout two years ago.",
      "We're mid-pilot with Nectar right now, so you'd need to beat that experience."
    ],
    "confidence": "Medium — explicit budget ($12k), timeline (Jan all-hands), agreed next step (Oct 2 exec presentation), but active competitor (Nectar mid-pilot) and exec skepticism; 2 stakeholders; 2 objections."
  },
  "TX-006": {
    "deal_alias": "Deal-EE195F",
    "why-buys": [
      "I'd love to cut the admin time on service awards.",
      "I personally spend five hours a month ordering and shipping plaques."
    ],
    "pain-points": [
      "Admin time on service awards is too high — personally spend five hours a month ordering and shipping plaques.",
      "COO usually prefers building things in-house."
    ],
    "stakeholders": [
      "HR Manager"
    ],
    "budget-signal": null,
    "timeline-signal": "Q1 start is realistic (fiscal year resets in February)",
    "competitor-mentioned": null,
    "next-step": "Send the one-page overview; HR Manager will forward it to the COO this week",
    "objections": [
      "COO usually prefers building things in-house.",
      "Comparing this against just doing it internally."
    ],
    "confidence": "Medium — clear use case and timeline (Q1), agreed next step (one-pager to COO), but no budget amount and internal-build preference; 1 stakeholder; 2 objections."
  }
}
```
next-to-close0.75

? The Question

Which 3 deals are closest to signature right now? Use the CRM data and the Slack thread; Slack often has the real status before the CRM does. For each: deal, amount, why it is close, what is left. Under 250 words.
cost $0.0058 · 17s · in 1,151 / out 709 tok
✓ exactly_three✗ top3_named✓ slack_first✓ blocked_excluded

✓ Correct Answer (ground truth)

top3: rank=1; deal_alias=Deal-547B2B; amount=11200; stage=DS5; forecast_category=COMMIT; close_date=2026-09-11 | rank=2; deal_alias=Deal-B7EBD1; amount=9000; stage=DS5; forecast_category=COMMIT; close_date=2026-09-10 | rank=3; deal_alias=Deal-403845; amount=9000; stage=DS5; forecast_category=COMMIT; close_date=2026-09-11
exactly_three: True
blocked_excluded: deal_alias=Deal-2465CE; reason=champion left + procurement freeze (Slack #sales-sparring 2026-09-04)
ranking_rule: Slack signature-imminent deal ranks first; remaining ranked by stage (DS5>DS4), forecast (COMMIT>BEST_CASE>PIPELINE), close_date asc, amount desc
graded checks: exactly_three · top3_named · slack_first · blocked_excluded

Model's actual answer

show the model's full answer
The three deals closest to signature are:

1. **Deal-547B2B — $11,200**  
   Why close: Slack says redlines came back clean, signing page is out, and the VP People said they are signing “tomorrow” — described as “signature-imminent.” CRM already shows DS5 / COMMIT with close date 2026-09-11.  
   What is left: Executed signature / countersignature.

2. **Deal-403845 — $9,000**  
   Why close: CRM is DS5 / COMMIT, close date 2026-09-11; Slack confirms the order form is with finance and “moving fine.”  
   What is left: Finance approval and final signature.

3. **Deal-A2B47C — $6,360**  
   Why close: CRM is DS5 / COMMIT with close date 2026-09-11; Slack says it is “still warm” with only “normal legal-review pace.”  
   What is left: Legal review and signature.

I excluded **Deal-2465CE** ($5,400, 2026-09-10) because the Slack thread says it is blocked after the champion left and procurement froze new vendors, so it is realistically Q4 now.
product-gap-detectionPASS 1.00

? The Question

Surface true product gaps raised by prospects in active deals. For each candidate: quote the prospect line, classify as real gap, plan gate, or rollout/enablement issue using the product docs excerpt, and name the deal and amount. Only real gaps go in the summary. Rep language does not count as prospect voice.
cost $0.0062 · 12s · in 744 / out 1,006 tok
✓ real_gaps✓ plan_gate_classified✓ enablement_classified✓ gap_summary

✓ Correct Answer (ground truth)

real_gaps: transcript_id=TG-001; deal_alias=Deal-EC3025; amount=62000; feature=SCIM user provisioning; quote=We need SCIM user provisioning — our security team will not approve any tool that requires manual account management.; why_real=not listed in product docs at any tier | transcript_id=TG-002; deal_alias=Deal-D0D6B5; amount=23800; feature=ADP Workforce Now integration; quote=Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us.; why_real=supported HRIS list is Workday/BambooHR/Gusto/Rippling only
plan_gates: transcript_id=TG-003; deal_alias=Deal-CFE7F4; amount=32000; feature=Custom report builder; tier_required=Enterprise; quote=I tried to build a custom report combining recognition frequency with turnover and couldn't — is that possible on our current plan?
enablement_issues: transcript_id=TG-004; deal_alias=Deal-84DBA6; amount=16000; feature=Slack integration recognition adoption; quote=We turned on the Slack integration in July, but half our managers still haven't given a single recognition — nobody ever trained them on it.; why_enablement=Slack integration exists on all plans; managers were never trained
excluded_rep_voice: transcript_id=TG-005; line=Full transparency — we don't have a native mobile app for hourly workers yet, though it's on the roadmap.; reason=rep voice, not prospect-raised
allowed_new: TG-001, TG-002, TG-003, TG-004, TG-005, SCIM user provisioning, ADP Workforce Now integration, Custom report builder
graded checks: real_gaps · plan_gate_classified · enablement_classified · gap_summary

Model's actual answer

show the model's full answer
**Real product gaps (prospect-raised, confirmed missing in product docs)**

| Deal alias | Prospect line (quoted) | Gap | Product docs basis | Amount |
|---|---|---|---|---|
| Deal-EC3025 | "We need SCIM user provisioning — our security team will not approve any tool that requires manual account management." | SCIM user provisioning | "SCIM user provisioning ... are NOT currently listed as supported capabilities." | Not provided in source data |
| Deal-D0D6B5 | "Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us." | ADP Workforce Now integration | "ADP Workforce Now integrations are NOT currently listed as supported capabilities." | Not provided in source data |

**Other candidates classified**

| Deal alias | Prospect line | Classification | Why |
|---|---|---|---|
| Deal-CFE7F4 | "I tried to build a custom report combining recognition frequency with turnover and couldn't — is that possible on our current plan?" | Plan gate | Custom report builder exists but is Enterprise-only, per plan table. |
| Deal-84DBA6 | "We turned on the Slack integration in July, but half our managers still haven't given a single recognition — nobody ever trained them on it." | Rollout/enablement issue | Slack integration is supported on all plans; problem is manager training/adoption. |
| Deal-36C33F | — | Not a prospect-raised gap | The mobile-app limitation was disclosed by rep Alex Franklin, not raised by the prospect. The prospect responded that the web version would suffice. Rep language excluded per instruction. |

**Note:** Deal amounts are not present in `transcripts_gaps.csv` (columns are `transcript_id, deal_alias, speaker, line_text` only), so amounts are listed as missing.
rep-performance
stale-pipeline-by-repPASS 1.00

? The Question

The snapshot date is 2026-09-05. Using the open deals and the engagements-by-deal table, list every open deal with no logged email, call, or meeting in the last 7 days. Compute recency from the engagements table's last_email/last_call/last_meeting fields, NOT from any date field on the deal (the deal's last_contacted_field is unreliable). For each stale deal give: deal alias, owner name, stage, amount, days since last contact. Group by owner, order by amount descending within each owner. Then state, per owner, how many stale deals they carry and the total stale amount. (Do not write outreach text.)
cost $0.1120 · 47s · in 51,261 / out 21,320 tok
✓ stale_recall_ge_80✓ grouped_by_owner✓ jake_top_amount✓ kevin_stale_amt✓ jenn_stale_amt✓ stale_count

✓ Correct Answer (ground truth)

stale_count: 65
jake_amt: 626243.0
kevin_amt: 252905.03
jenn_amt: 261645.0
top_alias: Deal-2D1F1B
top_amount: 240000.0
graded checks: stale_recall_ge_80 · grouped_by_owner · jake_top_amount · kevin_stale_amt · jenn_stale_amt · stale_count

Model's actual answer

show the model's full answer
Stale open deals as of 2026-09-05 (no email/call/meeting in last 7 days).

Bryce Harmon
13 stale deals, $626,243.00 total stale amount
- Deal-2D1F1B | DS1 | $240,000.00 | 81 days
- Deal-66D1FC | DS1 | $99,000.00 | 16 days
- Deal-950043 | DS1 | $70,000.00 | 19 days
- Deal-B23205 | DS1 | $45,000.00 | 16 days
- Deal-7BBDFA | DS3 | $37,440.00 | 46 days
- Deal-332637 | DS2 | $36,000.00 | 9 days
- Deal-1BEEBF | DS1 | $31,500.00 | 19 days
- Deal-C5658B | DS1 | $23,400.00 | 16 days
- Deal-40522D | DS3 | $21,000.00 | 19 days
- Deal-F0EBBB | DS3 | $11,400.00 | 24 days
- Deal-E25A09 | DS1 | $6,000.00 | 9 days
- Deal-C9C286 | DS2 | $5,502.00 | 9 days
- Deal-012CB1 | DS1 | $1.00 | 23 days

Dana Mercer
14 stale deals, $261,645.00 total stale amount
- Deal-44EA29 | DS2 | $60,000.00 | 10 days
- Deal-E51FB7 | DS2 | $43,875.00 | 12 days
- Deal-B42F46 | DS1 | $27,000.00 | 19 days
- Deal-BA3DDC | DS3 | $23,400.00 | 15 days
- Deal-9DDE86 | DS2 | $20,000.00 | 15 days
- Deal-215CCA | DS3 | $18,900.00 | 17 days
- Deal-5EED42 | DS3 | $16,250.00 | 11 days
- Deal-57887A | DS2 | $15,000.00 | 8 days
- Deal-B7EBD1 | DS5 | $9,000.00 | 16 days
- Deal-3974EB | DS4 | $9,000.00 | 8 days
- Deal-F40F04 | DS2 | $8,100.00 | 15 days
- Deal-87DDD1 | DS1 | $5,000.00 | 19 days
- Deal-F336B6 | DS3 | $4,200.00 | 15 days
- Deal-0660B4 | DS4 | $1,920.00 | 16 days

Alex Franklin
19 stale deals, $109,536.00 total stale amount
- Deal-CC08D1 | DS1 | $24,000.00 | 16 days
- Deal-E73427 | DS3 | $18,000.00 | 10 days
- Deal-885F45 | DS2 | $9,300.00 | 12 days
- Deal-C2FF3C | DS1 | $8,316.00 | 10 days
- Deal-3EED2C | DS2 | $7,200.00 | no engagement data
- Deal-0D2F7A | DS3 | $5,100.00 | 12 days
- Deal-6C60D4 | DS3 | $4,800.00 | 12 days
- Deal-13FEBD | DS2 | $4,680.00 | 12 days
- Deal-9D0060 | DS3 | $3,840.00 | 12 days
- Deal-690476 | DS2 | $3,600.00 | 18 days
- Deal-C6D97A | DS4 | $3,240.00 | 8 days
- Deal-EE195F | DS3 | $3,120.00 | 8 days
- Deal-278DEC | DS3 | $2,700.00 | 8 days
- Deal-635B8E | DS3 | $2,600.00 | 18 days
- Deal-6883F3 | DS1 | $2,400.00 | 16 days
- Deal-4A13AD | DS3 | $2,160.00 | 26 days
- Deal-F67D31 | DS2 | $1,800.00 | 8 days
- Deal-5FDCE4 | DS3 | $1,600.00 | 12 days
- Deal-BA571A | DS4 | $1,080.00 | 18 days

Cole Ingram
18 stale deals, $252,905.03 total stale amount
- Deal-D04904 | DS2 | $58,529.25 | 11 days
- Deal-B25F40 | DS3 | $40,000.00 | 8 days
- Deal-813836 | DS2 | $32,175.00 | 11 days
- Deal-1BA595 | DS2 | $31,750.00 | 11 days
- Deal-CFE1E8 | DS3 | $18,000.00 | 11 days
- Deal-CD47A6 | DS2 | $12,168.00 | 11 days
- Deal-627646 | DS3 | $11,193.00 | 11 days
- Deal-FF809F | DS2 | $7,781.20 | 11 days
- Deal-AF932D | DS2 | $7,225.40 | 11 days
- Deal-A71728 | DS2 | $6,947.50 | 11 days
- Deal-8BC9F5 | DS2 | $5,616.00 | 10 days
- Deal-175395 | DS3 | $4,779.88 | 11 days
- Deal-481E24 | DS3 | $4,140.00 | 10 days
- Deal-C7F9BF | DS2 | $3,360.00 | 11 days
- Deal-2F3A66 | DS3 | $3,334.80 | 11 days
- Deal-342E96 | DS2 | $2,700.00 | 24 days
- Deal-E568D5 | DS3 | $1,875.00 | 11 days
- Deal-FD9F4E | DS5 | $1,330.00 | 10 days

Farid Osman
0 stale deals, $0.00 total stale amount

Elena Sinclair
1 stale deal, $2,100.00 total stale amount
- Deal-57FF13 | DS1 | $2,100.00 | no engagement data

Grand total: 65 stale open deals, $1,252,429.03 total stale amount.
activity-mix-vs-outcomePASS 1.00

? The Question

The snapshot date is 2026-09-05. For each sales rep (owner), compute the last-30-day totals of emails, calls, and meetings from the engagements table, and the number of deals that entered DS2 in the last 30 days (t_ds2 within 30 days of the snapshot). Report per rep: the activity mix as percentages (emails/calls/meetings share of that rep's total activities), total activities per DS2 entry, and rank the reps by that efficiency ratio (lowest activities per DS2 entry = most efficient). Name the most efficient rep and the highest-volume rep, and note if they differ. Do not attribute cause.
cost $0.0726 · 1526s · in 42,823 / out 7,980 tok
✓ justin_ds2✓ justin_volume_high✓ efficient_named✓ mix_present✓ no_causal_claim

✓ Correct Answer (ground truth)

justin_ds2: 4
graded checks: justin_ds2 · justin_volume_high · efficient_named · mix_present · no_causal_claim

Model's actual answer

show the model's full answer
Snapshot: 2026-09-05. DS2-entry window: 2026-08-06 to 2026-09-05 inclusive. Activity sums use emails_30d, calls_30d, and meetings_30d from engagements_by_deal_90d.csv.

Per-rep last-30-day totals and efficiency

| Rank | Owner | Emails | Calls | Meetings | Total | Emails% | Calls% | Mtgs% | DS2 entries | Activities per DS2 |
|------|------|--------:|-------:|----------:|-------:|--------:|--------:|-------:|------------:|------------------:|
| 1 | Alex Franklin (84342457) | 199 | 25 | 31 | 255 | 78.0% | 9.8% | 12.2% | 10 | 25.50 |
| 2 | Bryce Harmon (119337721) | 162 | 0 | 43 | 205 | 79.0% | 0.0% | 21.0% | 4 | 51.25 |
| 3 | Cole Ingram (83155924) | 96 | 14 | 1 | 111 | 86.5% | 12.6% | 0.9% | 2 | 55.50 |
| 4 | Farid Osman (716654662) | 38 | 0 | 34 | 72 | 52.8% | 0.0% | 47.2% | 1 | 72.00 |
| 5 | Dana Mercer (83155923) | 84 | 18 | 11 | 113 | 74.3% | 15.9% | 9.7% | 1 | 113.00 |
| 6 | Elena Sinclair (701163055) | 0 | 0 | 0 | 0 | 0.0% | 0.0% | 0.0% | 0 | undefined |

Arithmetic examples:
- Alex Franklin: 199 + 25 + 31 = 255 activities; 255 / 10 DS2 entries = 25.50.
- Bryce Harmon: 162 + 0 + 43 = 205; 205 / 4 = 51.25.
- Elena Sinclair: 0 / 0 = undefined; no DS2 entries in window, so activities per DS2 is not computable.

Most efficient rep: Alex Franklin (84342457).
Highest-volume rep: Alex Franklin (84342457).

They do not differ — the same rep is both most efficient and highest-volume.
ae-qtd-scorecardPASS 1.00

? The Question

Build a QTD scorecard for the named AE as of 2026-09-05: bookings vs quota with attainment percent, new vs expansion split, active pipeline by stage with amount, rolling 90-day DS2-to-won rate, win and loss counts with top loss reason, activity volume by type in the last 30 days. Exclude any closed-won deal dated before the quarter. Close with three coaching observations grounded in the numbers.
cost $0.0460 · 4940s · in 32,674 / out 4,896 tok
✓ bookings✓ quota✓ attainment✓ new_expansion_split✓ top_loss_reason✓ coaching_obs

✓ Correct Answer (ground truth)

owner: Alex Franklin
quarter: 2026-Q3
snapshot_date: 2026-09-05
bookings: 150000
quota: 200000
attainment_pct: 75.0
new_bookings: 113500
new_count: 5
expansion_bookings: 36500
expansion_count: 3
pipeline_by_stage: {"DS1": {"count": 20, "amount": 284621}, "DS2": {"count": 28, "amount": 353760}, "DS3": {"count": 67, "amount": 552705}, "DS4": {"count": 5, "amount": 23574}, "DS5": {"count": 5, "amount": 45730}}
open_deals_count: 125
ds2_to_won: window_start=2026-06-07; won_in_window=8; closed_in_window=35; rate=0.2286
wins_qtd: 8
losses_qtd: 27
top_loss_reason: reason=Lost- Timing (1 year or more); count=13
activity_30d_totals: emails_30d=807; calls_30d=112; meetings_30d=128; notes_30d=50
prior_quarter_excluded: deal_alias=Deal-B3E6F1; close_date=2026-06-20; amount=24000
allowed_new: 18 items — e.g. Deal-A1C3E5, Deal-B7D2F4, Deal-C9E1A6, Deal-D4B8C2
graded checks: bookings · quota · attainment · new_expansion_split · top_loss_reason · coaching_obs

Model's actual answer

show the model's full answer
Alex Franklin QTD Scorecard — as of 2026-09-05

QUOTA
- 2026-Q3 quota: $200,000

BOOKINGS VS QUOTA
- QTD closed-won bookings: $150,000
- Attainment: $150,000 / $200,000 = 75.0%
- Gap to quota: -$50,000

NEW VS EXPANSION SPLIT
- New: $113,500 (5 deals): Deal-A1C3E5 ($40,000), Deal-B7D2F4 ($35,000), Deal-C9E1A6 ($21,000), Deal-D4B8C2 ($11,000), Deal-E6F3A9 ($6,500)
- Expansion: $36,500 (3 deals): Deal-F2C7D8 ($20,000), Deal-A8B4D6 ($12,000), Deal-C5D9E2 ($4,500)

ACTIVE PIPELINE BY STAGE (as of 2026-09-05)
- DS1: $284,621 (20 deals)
- DS2: $353,760 (28 deals)
- DS3: $552,705 (67 deals)
- DS4: $23,574 (5 deals)
- DS5: $45,730 (5 deals)
- Total active pipeline: $1,260,390 (125 deals)

ROLLING 90-DAY DS2-TO-WON RATE (2026-06-07 to 2026-09-05)
- DS2 entries in window: 111
- Won: 8 deals ($150,000)
- Lost: 27 deals ($329,272)
- Still open: 76 deals ($692,809)
- DS2-to-won rate (won / won+lost): 8 / 35 = 22.9%

WIN/LOSS COUNTS (Q3 through 2026-09-05)
- Wins: 8
- Losses: 27
- Win rate: 8 / 35 = 22.9%
- Top loss reason: "Lost- Timing (1 year or more)" — 13 losses, $184,681

ACTIVITY VOLUME — LAST 30 DAYS
- Emails: 807
- Calls: 112
- Meetings: 128
- Notes: 50
- Total activities: 1,097

THREE COACHING OBSERVATIONS
1. 75% attainment with $50K left means Franklin needs roughly 46% of the $109,363 currently scheduled Q3 open pipeline to close, yet only $103,603 of that is DS3-or-later and only $26,560 is DS5. The late-stage capacity is thin relative to the gap.
2. The 22.9% DS2-to-won rate and the fact that timing losses alone exceed total wins (13 vs. 8) point to a qualification gap; deals entering DS2 are not converting fast enough, and long-cycle timing objections are the #1 leakage point.
3. Activity skew is heavily email-driven (807 emails vs. 112 calls and 128 meetings). With 27 losses and top reasons being timing, MIA, and competitor, higher-touch outreach (calls/meetings) earlier in discovery could improve qualification and reduce no-decay losses.
multithreading-gapPASS 1.00

? The Question

Find every open deal that is single-threaded (fewer than 2 active contacts) or under-threaded (fewer than 3, or all contacts in one persona). Active means engaged in the last 60 days and not marked former. Personas: economic buyer, champion, HR admin, IT security, finance. For each flagged deal: amount, stage, active contact count, personas present, personas missing, the most valuable persona to add given the stage, and any on-file unengaged contact at that company who fits; otherwise say none on file.
cost $0.0222 · 65s · in 1,821 / out 5,421 tok
✓ flagged_recall✓ persona_language✓ single_thread_term✓ on_file_fit

✓ Correct Answer (ground truth)

active_window_start: 2026-07-07
persona_universe: economic buyer, champion, HR admin, IT security, finance
flag_rule: flag if active_contacts < 2 (single-threaded) or active_contacts < 3 or all active contacts share one persona (under-threaded); active = engaged within 60 days and not former
best_persona_rule: DS2/DS3: economic buyer > IT security > finance > HR admin > champion; DS4/DS5: economic buyer > finance > IT security > HR admin > champion; first persona not present among active contacts
flagged: deal_alias=Deal-36C33F; amount=15000; stage=DS2; active_count=1; reason=single-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-50D386; amount=36000; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-5408B0; amount=14850; stage=DS4; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-5BFE3B; amount=18000; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-885F45; amount=9300; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=IT security | deal_alias=Deal-92D97D; amount=60000; stage=DS2; active_count=1; reason=single-threaded; best_persona_to_add=economic buyer … (+5 more)
healthy_not_flagged: Deal-4B0BEB, Deal-84DBA6, Deal-D348E1
allowed_new: 59 items — e.g. C-077A0E, C-0D15DF, C-1BDFE0, C-2AE3AA
graded checks: flagged_recall · persona_language · single_thread_term · on_file_fit

Model's actual answer

show the model's full answer
Active threshold: reference date 2026-09-06; 60-day lookback starts 2026-07-08. Active = last_engaged_date ≥ 2026-07-08 AND is_former=false.

Amount and stage are **not in the provided files**, so those fields and the "most valuable persona given the stage" are **missing / cannot be determined**.

| deal_id | deal_alias | amount | stage | active_contacts | personas_present | personas_missing | most_valuable_persona_to_add | unengaged_contact_who_fits |
|---|---|---|---|---|---|---|---|---|
| 61032318100 | Deal-EC3025 | not in file | not in file | 1 | champion | economic buyer, HR admin, IT security, finance | stage missing | CT-6827DB (Chief People Officer, economic buyer) |
| 59728118877 | Deal-92D97D | not in file | not in file | 1 | HR admin | champion, economic buyer, IT security, finance | stage missing | none on file |
| 61055128146 | Deal-50D386 | not in file | not in file | 2 | champion, HR admin | economic buyer, IT security, finance | stage missing | CT-A1C4B3 (Chief People Officer, economic buyer) |
| 60081655042 | Deal-D0D6B5 | not in file | not in file | 3 | champion | economic buyer, HR admin, IT security, finance | stage missing | CT-1FA4DB (Chief People Officer, economic buyer) |
| 51674270311 | Deal-5BFE3B | not in file | not in file | 2 | champion | economic buyer, HR admin, IT security, finance | stage missing | none on file |
| 63739413805 | Deal-36C33F | not in file | not in file | 1 | IT security | champion, economic buyer, HR admin, finance | stage missing | CT-1DB73E (Chief People Officer, economic buyer) |
| 60686135564 | Deal-885F45 | not in file | not in file | 2 | economic buyer, champion | HR admin, IT security, finance | stage missing | CT-B3F25D (IT Security Lead, IT security) |
| 62639586615 | Deal-FCBE5B | not in file | not in file | 1 | champion | economic buyer, HR admin, IT security, finance | stage missing | none on file |
| 60182332309 | Deal-5408B0 | not in file | not in file | 2 | champion, HR admin | economic buyer, IT security, finance | stage missing | CT-07FA76 (Chief People Officer, economic buyer) |
| 62121783047 | Deal-C6D97A | not in file | not in file | 3 | champion | economic buyer, HR admin, IT security, finance | stage missing | none on file |
| 49757401138 | Deal-F9A08A | not in file | not in file | 1 | champion | economic buyer, HR admin, IT security, finance | stage missing | CT-697541 (Chief People Officer, economic buyer) |

Flagged by rule:
- Single-threaded (<2 active): Deal-EC3025, Deal-92D97D, Deal-36C33F, Deal-FCBE5B, Deal-F9A08A
- Under-threaded (<3 active): Deal-50D386, Deal-885F45, Deal-5408B0
- Under-threaded (all active contacts in one persona): Deal-D0D6B5, Deal-5BFE3B, Deal-C6D97A

Unengaged contacts checked against the unengaged_contacts.csv file by company alias.
talk-track-from-transcriptsPASS 1.00

? The Question

From this rep's last 10 calls: what they lead with in the first five minutes, how they handle the three most common objections, how often a concrete next step is agreed (rate), and every competitor a prospect raised. Quote at most one line per finding and only lines that appear in the transcripts. End with two coaching notes.
cost $0.0120 · 61s · in 2,675 / out 2,240 tok
✓ objection_count✓ next_step_rate✓ competitors✓ coaching✓ quote_used

✓ Correct Answer (ground truth)

rep: Alex Franklin
n_transcripts: 10
opener_theme: theme=opens with a customer recognition win story (400-person retailer, 18% turnover reduction); count=8; total=10; quote=Quick one to kick off: a 400-person retailer we work with cut regretted turnover 18% in two quarters after automating milestone awards, and I wanted to walk you through exactly how they did it.; transcript_id=TT-001
objections: objection=budget; count=4; quote=Honestly, budget is locked until next fiscal year — I can't add a new line item right now.; transcript_id=TT-001 | objection=timing; count=3; quote=This is good, but can we revisit it next quarter? Open enrollment starts in October and we're underwater.; transcript_id=TT-002 | objection=in-house; count=3; quote=We already do recognition with a spreadsheet and quarterly gift cards — why would we change?; transcript_id=TT-004
objection_count: 3
next_step: agreed=7; total=10; rate=0.7
competitors_prospect_raised: Awardco, Kudos
competitor_quotes: Awardco=We're also in late talks with Awardco — their rewards catalog looks bigger than yours.; Kudos=How are you different from Kudos? Our CEO used them at her last company.
rep_raised_competitors_excluded: Workhuman
allowed_new: 13 items — e.g. TT-001, TT-002, TT-003, TT-004
graded checks: objection_count · next_step_rate · competitors · coaching · quote_used

Model's actual answer

show the model's full answer
Last 10 calls: TT-001 through TT-010 (Deal-D348E1, Deal-5408B0, Deal-547B2B, Deal-403845, Deal-C61CF7, Deal-60C2C2, Deal-EDC141, Deal-D9A12F, Deal-1E2498, Deal-84DBA6).

What the rep leads with (first 5 minutes):
- 8 of 10 calls: the 400-person retailer / 18% turnover story.
  - Quote: "Quick one to kick off: a 400-person retailer we work with cut regretted turnover 18% in two quarters after automating milestone awards, and I wanted to walk you through exactly how they did it." (TT-001, Alex Franklin, minute 0)
- Deal-403845 (TT-004): security review + pricing agenda.
  - Quote: "I put together a short agenda — security review first, then pricing."
- Deal-1E2498 (TT-009): straight pricing.
  - Quote: "You asked for straight pricing last time, so let's start there."

Three most common objections and how handled:
1. "Budget is locked until next fiscal year" — 4 of 10 calls (TT-001, TT-003, TT-006, TT-010).
   - Response: tie to turnover savings / $210k backfill example.
   - Quote: "Totally fair. Most teams fund this out of turnover savings — that retailer saved about $210k in avoided backfills, which is how their finance team signed off." (TT-001, Alex Franklin, minute 8)

2. "Can we revisit it next quarter?" — 3 of 10 calls (TT-002, TT-005, TT-008).
   - Response: propose a 90-day single-department pilot to generate internal data before planning.
   - Quote: "Makes sense. What if we scope a 90-day pilot with one department so you have internal data before next quarter's planning?" (TT-002, Alex Franklin, minute 8)

3. "We already do recognition with a spreadsheet and quarterly gift cards — why would we change?" — 3 of 10 calls (TT-004, TT-007, TT-009).
   - Response: contrast manual process with automation + analytics.
   - Quote: "Spreadsheets work until they scale — the difference is automation: milestones fire without HR lifting a finger, and you get analytics on who is being recognized." (TT-004, Alex Franklin, minute 8)

Concrete next step agreed rate:
- Rep asks "Should we lock the next step..." on 7 calls and gets yes (TT-001, TT-002, TT-003, TT-005, TT-006, TT-008, TT-009).
- No concrete next step on 3 calls: TT-004 ("I'll leave it with you"), TT-007 ("Fair enough"), TT-010 ("Understood, thanks for the candor").
- Rate: 7 / 10 = 70%.

Competitors raised by prospects:
- Awardco — Deal-547B2B (TT-003): "We're also in late talks with Awardco — their rewards catalog looks bigger than yours."
- Kudos — Deal-EDC141 (TT-007): "How are you different from Kudos? Our CEO used them at her last company."
- Workhuman was mentioned only by the rep, not a prospect, so it is excluded.

Coaching notes:
1. The "lock the next step" close is strong when used (70% success), but the rep skips it entirely on calls where the prospect signals delay or committee approval — exactly the moments where a concrete next step matters most. Add a soft commit even when the answer is "not yet," e.g., a calendar hold for a follow-up call or a named stakeholder to loop in.
2. Status-quo ("spreadsheet and gift cards") and timing ("next quarter") objections are both handled with a one-size-fits-all message. Test tying the response to the specific deal's context — for example, asking how many hours HR currently spends per month on the spreadsheet before pitching automation.
reporting-analytics
q3-weighted-forecast0.90

? The Question

Below are the open deals in the default sales pipeline with a close date between 2026-07-01 and 2026-10-15, pulled from the CRM warehouse on 2026-09-05. Stages: DS1 (discovery) through DS5 (contract). Build the Q3 2026 forecast (quarter = 2026-07-01 to 2026-09-30). Weighting rule: 100 percent of COMMIT plus 35 percent of BEST_CASE. PIPELINE deals count zero. Only deals with a close date inside the quarter count. Report: COMMIT total, BEST_CASE total, weighted forecast, count of deals in each category inside the quarter, the deals you excluded for being outside the quarter (count and total amount), and the top 5 BEST_CASE deals by amount inside the quarter. Show arithmetic. Then, in a final '## Data quality' section of no more than four sentences, name the data quality problems in this extract that would make the forecast unreliable if it were run unattended.
cost $0.0317 · 73s · in 22,011 / out 4,134 tok
✓ commit_total_44729✓ best_case_total_203565✓ weighted_115976.75✓ commit_count_7✓ best_case_count_24✓ excluded_count_32✓ excluded_amount_227575✓ top5_aliases✓ flags_blank_owner✗ flags_missing_why_buys

✓ Correct Answer (ground truth)

graded checks: commit_total_44729 · best_case_total_203565 · weighted_115976.75 · commit_count_7 · best_case_count_24 · excluded_count_32 · excluded_amount_227575 · top5_aliases · flags_blank_owner · flags_missing_why_buys

Model's actual answer

show the model's full answer
## Q3 2026 Forecast

**Quarter filter:** close date between 2026-07-01 and 2026-09-30 inclusive.

- **COMMIT total:** $44,729.00 (7 deals)
  - Deal-547B2B: $11,200.00
  - Deal-B7EBD1: $9,000.00
  - Deal-403845: $9,000.00
  - Deal-A2B47C: $6,360.00
  - Deal-2465CE: $5,400.00
  - Deal-A5E80A: $2,520.00
  - Deal-499BF6: $1,249.00

- **BEST_CASE total:** $203,565.00 (24 deals)

- **PIPELINE total:** $201,637.40 (23 deals) — weighted 0%

- **Weighted forecast:** $44,729.00 + (0.35 × $203,565.00) = **$115,976.75**

**Excluded outside the quarter:** 32 deals totaling **$227,575.00**. Notable exclusions include COMMIT deal Deal-D348E1 ($13,770.00, closes 2026-10-15).

**Top 5 BEST_CASE deals by amount (inside quarter):**
1. Deal-2D7423 — $38,935.00 (DS3, closes 2026-09-30)
2. Deal-25F752 — $24,000.00 (DS4, closes 2026-09-25)
3. Deal-E53952 — $19,656.00 (DS4, closes 2026-09-30)
4. Deal-5EED42 — $16,250.00 (DS3, closes 2026-09-30)
5. Deal-FA32A0 — $11,116.00 (DS3, closes 2026-09-25)

## Data quality

The owner field is blank on 53 of 54 deals, so we cannot validate rep-level coverage or assignment. Over 74% of deals have zero `why_buys_chars`, meaning most lack a documented business case, raising the risk that forecast category labels are stale or ungrounded. Stage-to-category logic appears inconsistent: e.g., Deal-A5E80A is DS1 (discovery) yet tagged COMMIT, while several DS5 (contract) deals remain PIPELINE, suggesting overrides or manual miscoding. Finally, the extract is a static 2026-09-05 snapshot with no created/modified timestamps, so we cannot tell which close dates were recently pushed or pulled into the quarter.
aha-moment-2x20.83

? The Question

Test the activation hypothesis on the company cohort below: companies with 5+ unique givers (m1_users >= 5) AND 1+ successful redemption (m1_redemptions >= 1) in their first calendar month retain better at 24 months than those with one signal or neither. Every company in this extract is already 25+ months old, so current_status = 'active' means retained at 24 months. Report the full 2x2 (cohort size and 24-month retention rate for: both signals, givers-only, redemption-only, neither), state how many companies were excluded from the denominator and why (if none, say so), name the single signal with the largest retention lift, and state what this does and does not prove.
cost $0.0685 · 1503s · in 31,037 / out 12,608 tok
✓ both_66✓ givers_47✓ redemp_31✓ neither_40✗ excluded_0✓ largest_lift_givers

✓ Correct Answer (ground truth)

graded checks: both_66 · givers_47 · redemp_31 · neither_40 · excluded_0 · largest_lift_givers

Model's actual answer

show the model's full answer
Cohort: 220 companies, all included (no exclusions — every row has valid m1_users, m1_redemptions, and current_status, and all are 25+ months old per the prompt).

Retention = current_status = 'active'.

Quadrant definition:
- 5+ unique givers: m1_users >= 5
- 1+ successful redemption: m1_redemptions >= 1

**Full 2x2**

| Segment | Cohort size | Retained at 24 mo | Retention rate |
|---|---:|---:|---:|
| Both signals | 47 | 31 | 31 / 47 = **66.0%** |
| Givers only (5+ users, 0 redemptions) | 49 | 23 | 23 / 49 = **46.9%** |
| Redemption only (1+ redemption, <5 users) | 29 | 9 | 9 / 29 = **31.0%** |
| Neither signal | 95 | 38 | 38 / 95 = **40.0%** |

Check: 47 + 49 + 29 + 95 = 220; 31 + 23 + 9 + 38 = 101 active total.

**Single-signal lifts**

- Giver signal lift (5+ users vs. <5 users, ignoring redemption):  
  With givers: (31 + 23) / (47 + 49) = 54 / 96 = **56.3%**  
  Without givers: (9 + 38) / (29 + 95) = 47 / 124 = **37.9%**  
  Lift = 56.3% − 37.9% = **+18.4 pp**

- Redemption signal lift (≥1 redemption vs. 0 redemptions, ignoring users):  
  With redemption: (31 + 9) / (47 + 29) = 40 / 76 = **52.6%**  
  Without redemption: (23 + 38) / (49 + 95) = 61 / 144 = **42.4%**  
  Lift = 52.6% − 42.4% = **+10.3 pp**

**Single signal with the largest retention lift:** **5+ unique givers in month 1** (+18.4 pp).

Examples in the data:
- Both signals, retained: C-0BC46AD9, C-0B540A52, C-14D70CE0
- Givers only, retained: C-0DD506E9, C-0DDFC9A7, C-0F7269D7
- Redemption only, retained: C-0DD0626C, C-0B7D2C30, C-0FD34543
- Neither, retained: C-0FF2DB67, C-0FB9D5AF, C-0DA98AC7

**What this does and does not prove**

- Does prove: In this Jan–Jul 2023 extract, companies that hit both early signals have the highest 24-month retention rate (66.0%). The giver signal alone drives a larger retention lift than the redemption signal alone.
- Does not prove causation. Companies with 5+ givers may differ on size, fit, industry, or tier; this is an association, not evidence that forcing redemptions or adding users will cause retention. It also does not generalize beyond this cohort, product tier mix, or time period, and it does not account for confounders such as integrations, ARR, or country.
arr-rollforward-reconciliation0.60

? The Question

Reconcile ARR between CRM company records and billing subscriptions as of 2026-09-05. Billing ARR is MRR x 12. Report both totals, the variance, and a decomposition into buckets (status mismatch, rounding, missing records, other) that sums exactly to the variance. List each mismatched account with a suggested owner. Business rule: any subscription with a term other than 12 months must have cf_agreement_end_date populated; list violations.
cost $0.0322 · 1456s · in 7,068 / out 6,570 tok
✓ billing_total✓ crm_total✗ variance✗ buckets_sum✓ violations_listed

✓ Correct Answer (ground truth)

as_of: 2026-09-05
billing_arr_total: 604739.28
crm_arr_total: 603581.76
variance: 1157.52
variance_definition: billing_arr_total - crm_arr_total (billing ARR = sum of MRR x 12 over status=active subscriptions)
buckets: {"status_mismatch": {"amount": "-13158.48", "accounts": [{"company_alias": "C-0C8323BF", "billing_arr": "0.00", "hubspot_arr": "4905.24", "diff": "-4905.24", "suggested_owner": "Dana Mercer"}, {"compa
bucket_sum_equals_variance: True
n_mismatched_accounts: 7
rule: term_months != 12 requires cf_agreement_end_date
violations: subscription_id=SUB-0002; company_alias=C-1794A52C; term_months=24 | subscription_id=SUB-0019; company_alias=C-22170CA1; term_months=36
n_companies: 40
graded checks: billing_total · crm_total · variance · buckets_sum · violations_listed

Model's actual answer

show the model's full answer
## ARR Reconciliation (CRM vs Billing) as of 2026-09-05

**Totals**
- CRM ARR (company_arr.csv): **$603,581.76**
- Billing ARR (MRR × 12): **$617,897.76**
- Variance (Billing − CRM): **+$14,316.00**

**Variance Decomposition (sums exactly to +$14,316.00)**
- Missing CRM record (billing-only): **+$28,449.24**
- Missing billing record (CRM-only): **−$16,497.24**
- Status mismatch: **$0.00**
- Rounding: **$0.00**
- Other: **+$2,364.00**
- **Total decomposition: +$14,316.00** ✓

**Mismatched Accounts**
| Company Alias | CRM ARR | Billing ARR | Variance | Bucket |
|---|---|---|---|---|
| C-21629AA4 | $0.00 | $28,449.24 | +$28,449.24 | Missing CRM record |
| C-0D5BBE3A | $16,497.24 | $0.00 | −$16,497.24 | Missing billing record |
| C-0F7269D7 | $24,396.00 | $26,796.00 | +$2,400.00 | Other |
| C-14D70CE0 | $18,200.00 | $18,180.00 | −$20.00 | Other |
| C-0D66DF9E | $23,200.00 | $23,184.00 | −$16.00 | Other |

**Suggested owners:** No owner/CSM/AE data was provided in the source files, so I cannot assign owners. Add an owner field to either file and re-run.

**Other bucket arithmetic**
- C-0F7269D7: Billing $26,796.00 − CRM $24,396.00 = +$2,400.00
- C-14D70CE0: Billing $18,180.00 − CRM $18,200.00 = −$20.00
- C-0D66DF9E: Billing $23,184.00 − CRM $23,200.00 = −$16.00
- Sum: $2,400.00 − $20.00 − $16.00 = **+$2,364.00**

**Business Rule Violations (term ≠ 12 months but cf_agreement_end_date blank)**
1. **SUB-0002** — C-1794A52C — term = 24 months — missing end date
2. **SUB-0019** — C-22170CA1 — term = 36 months — missing end date

These are the only non-12-month subscriptions. SUB-000C and SUB-001A are compliant.
kvm-month-over-month0.80

? The Question

Report the four core KVMs for 2026-08 against 2026-07: giving rate, redemptions per user, 1:1 meetings engagement, pulse check engagement. For each: value, prior value, absolute and relative change, direction. Name the KVM with the largest relative move and the segment (plan_tier or size_band) driving it if the data supports that; otherwise say the data does not. Under 200 words plus one table.
cost $0.0411 · 1252s · in 9,751 / out 6,547 tok
✗ four_kvms✓ biggest_mover_named✓ direction_correct✓ segment_named✓ word_cap

✓ Correct Answer (ground truth)

aggregation: simple mean of company-level values per month
kvms: {"giving_rate": {"2026-07": 0.6023, "2026-08": 0.6027, "abs_change": 0.0004, "rel_change_pct": 0.07, "direction": "up"}, "redemptions_per_user": {"2026-07": 1.73, "2026-08": 1.7302, "abs_change": 0.00
biggest_mover: pulse_engagement
driving_segment: size_band=enterprise; 2026-07=0.55; 2026-08=0.2743; rel_change_pct=-50.13
segment_detail: {"smb": {"2026-07": 0.6588, "2026-08": 0.6573, "rel_change_pct": -0.23}, "mid_market": {"2026-07": 0.593, "2026-08": 0.5942, "rel_change_pct": 0.2}, "enterprise": {"2026-07": 0.55, "2026-08": 0.2743,
n_company_rows_per_month: 30
graded checks: four_kvms · biggest_mover_named · direction_correct · segment_named · word_cap

Model's actual answer

show the model's full answer
⚠ Stream stalled mid tool-call (execute_code); the action was not executed. Ask me to retry if you want to continue. August aggregate means:
- giving_rate: 0.602713
- redemptions_per_user: 1.730163
- one_to_one_engagement: 0.447177
- pulse_engagement: 0.508610

Change vs July:

| KVM                    | Aug      | Jul      | Abs change | Rel change | Direction |
|------------------------|----------|----------|------------|------------|-----------|
| giving_rate            | 0.6027   | 0.6023   | +0.0004    | +0.07%     | up        |
| redemptions_per_user   | 1.7302   | 1.7300   | +0.0002    | +0.01%     | up        |
| one_to_one_engagement  | 0.4472   | 0.4469   | +0.0003    | +0.06%     | up        |
| pulse_engagement       | 0.5086   | 0.6006   | −0.0920    | −15.31%    | down      |

Largest relative move: pulse_engagement, down 15.31%.

Driver: enterprise segment, which fell from 0.54998 to 0.27428 (−0.27570, −50.13%). mid_market was flat up (+0.0022%) and smb was barely down (−0.22%).
rewards-ytd-sectionPASS 1.00

? The Question

Produce the redemption section of the weekly rewards report through the last completed month (2026-08): redemption count, spend, unique redeemers, redemptions per redeemer, provider mix as percent of spend, top 5 countries by redemptions. Say which month is the last completed month. Provider shares must sum to 100.
cost $0.0890 · 174s · in 32,792 / out 16,856 tok
✓ count✓ spend✓ unique_redeemers✓ month_named✓ provider_mix✓ top_country

✓ Correct Answer (ground truth)

redemption_count: 378
total_spend_usd: 27846.00
unique_redeemers: 235
redemptions_per_redeemer: 1.61
provider_mix_pct_of_spend: TangoCard=11.6; Tremendous=30.54; custom=39.05; Snappy=18.81
top5_countries_by_count: country=US; count=244 | country=CA; count=24 | country=AU; count=21 | country=GB; count=17 | country=NL; count=17
graded checks: count · spend · unique_redeemers · month_named · provider_mix · top_country

Model's actual answer

show the model's full answer
Redemption section — weekly rewards report (last completed month: 2026-08)

- **Reporting period:** 2026-01-01 through 2026-08-31
- **Redemption count:** 377
- **Redemption spend:** $27,771.00
- **Unique redeemers:** 235
- **Redemptions per redeemer:** 377 / 235 = 1.60

**Provider mix (% of spend)**
- custom: $10,873.00 = 39.15%
- Tremendous: $8,430.00 = 30.36%
- Snappy: $5,238.00 = 18.86%
- TangoCard: $3,230.00 = 11.63%
- **Total:** 100.00%

**Top 5 countries by redemptions**
1. US: 244
2. CA: 24
3. AU: 21
4. GB: 17
5. NL: 17
customer-success
churn-save-eligibilityPASS 1.00

? The Question

Which at-risk accounts qualify for a churn-save offer under the documented eligibility rules, what amount is at stake per account and in total, and which play fits each (usage revival, executive touch, commercial concession)? Cite the signal that justifies each play. List accounts that look at risk but do not qualify and why.
cost $0.0501 · 1288s · in 2,074 / out 12,735 tok
✓ eligible_set✓ total_at_stake✓ plays_cited✓ noneligible_named✓ rules_applied

✓ Correct Answer (ground truth)

snapshot_date: 2026-09-05
rules: health_score < 60, churn_save_eligible_amount > 0, renewal within 120 days of snapshot
eligible: account_alias=C-0F6C0F34; amount_at_stake=49707.00; play=executive touch; justifying_signal=champion_active is false - no executive sponsor engaged | account_alias=C-0B827671; amount_at_stake=25365.00; play=usage revival; justifying_signal=usage_trend_3m=declining over the last 3 months | account_alias=C-0B360C78; amount_at_stake=35748.00; play=commercial concession; justifying_signal=usage stable/growing with seat utilization 75% - risk is commercial, not adoption | account_alias=C-0B0F1BAB; amount_at_stake=5494.00; play=executive touch; justifying_signal=champion_active is false - no executive sponsor engaged | account_alias=C-0CA21961; amount_at_stake=16829.00; play=usage revival; justifying_signal=seat utilization 26% is below 50% | account_alias=C-0E9C27D1; amount_at_stake=41235.00; play=commercial concession; justifying_signal=usage stable/growing with seat utilization 85% - risk is commercial, not adoption … (+2 more)
total_amount_at_stake: 224601.00
non_eligible_at_risk: account_alias=C-0BC71BDD; health_score=55 | account_alias=C-0BA71F12; health_score=52 | account_alias=C-0F6694C3; health_score=43 | account_alias=C-0BE96399; health_score=54 | account_alias=C-0F876796; health_score=47 | account_alias=C-0FCCD2DF; health_score=43 … (+1 more)
n_accounts: 30
graded checks: eligible_set · total_at_stake · plays_cited · noneligible_named · rules_applied

Model's actual answer

show the model's full answer
Snapshot: **2026-09-05**.  
120-day eligibility window ends **2027-01-03** (2026-09-05 + 120 days).  
A churn-save offer is qualified when **all three** rules are met:

- R1: `health_score < 60`
- R2: `churn_save_eligible_amount > 0`
- R3: `renewal_date <= 2027-01-03`

No play definitions were supplied, so I mapped the three plays to the remaining account signals:

- **Usage revival** → `usage_trend_3m = declining`
- **Executive touch** → `champion_active = false`
- **Commercial concession** → all other qualified accounts (champion active, usage flat/growing)

---

### QUALIFIED accounts for churn-save offer

| Account | Health | Eligible amount at stake | Renewal | Usage trend | Champion active | Play | Signal |
|---|---|---|---|---|---|---|---|
| C-0B0F1BAB | 38 | $5,494.00 | 2026-09-23 (18 days) | flat | false | Executive touch | No active champion |
| C-0E9C27D1 | 39 | $41,235.00 | 2026-09-24 (19 days) | flat | true | Commercial concession | Low health with active champion; no usage decline |
| C-0F6C0F34 | 51 | $49,707.00 | 2026-10-03 (28 days) | growing | false | Executive touch | No active champion |
| C-0CEF69FD | 53 | $32,621.00 | 2026-11-21 (77 days) | growing | false | Executive touch | No active champion |
| C-0D3278C7 | 54 | $17,602.00 | 2026-11-12 (68 days) | declining | true | Usage revival | Declining usage trend |
| C-0B827671 | 56 | $25,365.00 | 2026-11-14 (70 days) | declining | true | Usage revival | Declining usage trend |
| C-0B360C78 | 57 | $35,748.00 | 2026-10-28 (53 days) | growing | true | Commercial concession | Low health with active champion; no usage decline |
| C-0CA21961 | 58 | $16,829.00 | 2026-12-28 (114 days) | flat | true | Commercial concession | Low health with active champion; no usage decline |

**Total at stake:**  
$5,494 + $41,235 + $49,707 + $32,621 + $17,602 + $25,365 + $35,748 + $16,829 = **$224,601.00**

---

### At-risk accounts that DO NOT qualify

Health score < 60 but failing R2 or R3:

| Account | Health | Eligible amount | Renewal | Why excluded |
|---|---|---|---|---|
| C-0BC71BDD | 55 | $0.00 | 2026-10-27 (52 days) | Fails R2: churn-save eligible amount = 0 |
| C-0BE96399 | 54 | $0.00 | 2026-10-29 (54 days) | Fails R2: churn-save eligible amount = 0 |
| C-10A56B0F | 54 | $0.00 | 2026-12-12 (98 days) | Fails R2: churn-save eligible amount = 0 |
| C-0F6694C3 | 43 | $0.00 | 2027-03-21 (197 days) | Fails R2 and R3: amount = 0; renewal outside 120-day window |
| C-0FCCD2DF | 43 | $0.00 | 2027-04-23 (230 days) | Fails R2 and R3: amount = 0; renewal outside 120-day window |
| C-0BA71F12 | 52 | $6,824.00 | 2027-04-11 (218 days) | Fails R3: renewal outside 120-day window |
| C-0F876796 | 47 | $19,958.00 | 2027-02-06 (154 days) | Fails R3: renewal outside 120-day window |

All other accounts have `health_score >= 60`, so they are not flagged at-risk under the documented rule.
expansion-kitPASS 1.00

? The Question

A customer replied to outreach (reply included). Build the expansion kit: seat coverage (licensed over headcount), two lines on usage health, headroom in seats and ARR at the current per-seat rate, who replied and whether they can buy, the right buyer from our contacts, and a reply email under 150 words that moves toward the right person without being pushy and cites one real usage fact.
cost $0.0075 · 22s · in 423 / out 1,364 tok
✓ coverage_pct✓ headroom_arr✓ right_buyer✓ replier_not_buyer✓ usage_fact✓ email_under_150

✓ Correct Answer (ground truth)

company_alias: C-0DDFC9A7
coverage_pct: 37.5
headroom_seats: 250
per_seat_annual_rate: 60.00
headroom_arr: 15000.00
replier: Maria S.
replier_can_buy: False
right_buyer: name=Dana R.; title=VP People
usage_facts: active_users_2026_08=126; active_users_2026_03=88; growth_pct_6m=43.2; licensed_seat_utilization_pct=84.0
usage_health_lines: Monthly active users grew from 88 (2026-03) to 126 (2026-08), +43.2% over 6 months., Last month 126 of 150 licensed seats were active (84.0% utilization), approaching the seat cap.
graded checks: coverage_pct · headroom_arr · right_buyer · replier_not_buyer · usage_fact · email_under_150

Model's actual answer

show the model's full answer
**Seat Coverage:** 150 / 400 = **37.5%** of headcount licensed.

**Usage Health:**
- Monthly active users climbed every period: 88 → 95 → 102 → 110 → 118 → 126 (Mar–Aug 2026), a **+43.2% increase** over six months.
- August active users (126) represent **84%** of licensed seats (126 / 150).

**Headroom at $60 per seat per year** ($9,000 ARR / 150 seats):
- **Seat headroom:** 400 headcount – 150 licensed = **250 seats**.
- **ARR headroom:** 250 seats × $60 = **$15,000**.
- Full-headcount ARR potential: 400 × $60 = **$24,000**.

**Reply Source & Buyer Authority:**
- **Replier:** Maria S., People Operations Coordinator (last engaged 2026-09-02).
- **Can she buy?** No. She stated budget and seat expansion sit with Dana R.
- **Right buyer:** Dana R., VP People (last engaged 2026-05-18).

**Reply Email (96 words):**

Hi Maria,

Thanks for the kind note—great to hear the team is loving Bonusly and that monthly active users climbed from 88 in March to 126 in August.

An introduction to Dana would be perfect. With 400 headcount and 150 seats currently licensed, there's room to bring more of the team onto Bonusly, and I'd love to share the usage trends that have caught her attention.

If it helps, I can send a short note she can forward, or jump on a brief intro call whenever it works for both of you.

Thanks again,
Cole
onboarding-statusPASS 1.00

? The Question

Prep the CSM for the mid-onboarding call: what is complete (integrations connected, allowance set, admins added, first recognition given, first redemption), what is not, early engagement signals, and the three things to cover on the call. Mark nothing complete without a data field that shows it.
cost $0.0072 · 21s · in 423 / out 1,258 tok
✓ complete_items✓ incomplete_items✓ three_items✓ grounded

✓ Correct Answer (ground truth)

company_alias: C-0D284E42
days_live: 25
checklist: {"slack_integration_connected": {"complete": true, "field": "integration_slack", "value": "2026-08-12"}, "hris_integration_connected": {"complete": false, "field": "integration_hris", "value": ""}, "a
engagement_signals: avg_active_givers_week1=4.3; avg_active_givers_last_week=13.0; latest_active_givers=15; days_with_activity=25
call_items: Connect the HRIS integration (integration_hris is empty) to automate user provisioning and offboarding., Get to first redemption (first_redemption_at is empty): walk admins through the rewards catalog and confirm allowance funding., Build the giving habit: daily active givers averaged 4.3 in week 1 vs 13.0 last week - agree on manager nudges and a recognition cadence.
graded checks: complete_items · incomplete_items · three_items · grounded

Model's actual answer

show the model's full answer
**Company:** C-0D284E42

## Onboarding Checklist (data-backed)

**Complete:**
- Slack integration connected — `integration_slack = 2026-08-12`
- Allowance set — `allowance_set = 2026-08-13`
- Admins added — `admins_added = 2`
- First recognition given — `first_recognition_at = 2026-08-15 14:22`

**Not Complete:**
- HRIS integration — `integration_hris` is blank
- First redemption — `first_redemption_at` is blank

## Early Engagement Signals

Active givers from `2026-08-11` to `2026-09-04` (25 days):

- Started at **3** active givers, ended at **15**
- Net increase: 15 − 3 = **+12 active givers**
- Growth rate: (15 − 3) / 3 = **400%**
- Daily average: 225 / 25 = **9 active givers/day**
- Current plateau/peak: **15** active givers on `2026-09-03` and `2026-09-04`

Signal: strong adoption trend, with daily active givers now 5x the signup-day count.

## Three Things to Cover on the Call

1. **HRIS integration gap.** It’s the only unconnected core integration. Confirm whether it’s blocked, in IT queue, or not needed, and provide next steps.
2. **Drive the first redemption.** Recognition is live (`2026-08-15`), but no redemption has occurred. Review rewards catalog relevance, redemption workflow, and whether employees know how to redeem.
3. **Capitalize on momentum.** Active givers grew 400% to 15/day. Identify expansion levers—additional teams, manager enablement, or a launch campaign—to sustain the trend.
renewal-risk-conflicting-datesPASS 1.00

? The Question

Produce a 90-day renewal risk brief. Two systems hold renewal dates and disagree on some accounts; decide which to trust per account and say why (multi-year contracts are known to be wrong in ChurnZero). For every renewal: company, CSM, ARR, date used, seat utilization, 3-month usage trend, risk rating with one sentence of evidence. Flag every disagreement. Close with total ARR renewing and ARR at risk.
cost $0.0815 · 4966s · in 31,530 / out 14,283 tok
✓ total_renewing✓ arr_at_risk✓ disagreements_flagged✓ trust_rule

✓ Correct Answer (ground truth)

snapshot_date: 2026-09-05
window: 2026-09-05 to 2026-12-04
trust_rule: multi-year contracts: Chargebee is authoritative (ChurnZero known wrong); otherwise systems agree or Chargebee wins
accounts: 20 items — e.g. account_alias=C-0B144C78; csm=Cole Ingram; arr=30899.00; trusted_renewal_date=2026-11-02; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=75.4; usage_3m_ratio=1.03; risk=low; evidence=3-month usage ratio 1.03 (last3 avg 103 vs prior3 100), seat utilization 75% | account_alias=C-0B20DB64; csm=Dana Mercer; arr=21770.00; trusted_renewal_date=2026-10-07; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=56.6; usage_3m_ratio=1.0; risk=medium; evidence=3-month usage ratio 1.00 (last3 avg 295 vs prior3 295), seat utilization 57% | account_alias=C-0B344485; csm=Elena Sinclair; arr=64384.00; trusted_renewal_date=2026-11-16; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=78.0; usage_3m_ratio=1.04; risk=low; evidence=3-month usage ratio 1.04 (last3 avg 241 vs prior3 231), seat utilization 78% | account_alias=C-0B369871; csm=Dana Mercer; arr=85128.00; trusted_renewal_date=2026-10-29; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=75.1; usage_3m_ratio=1.04; risk=low; evidence=3-month usage ratio 1.04 (last3 avg 330 vs prior3 316), seat utilization 75%
disagreements: account_alias=C-0B7D2C30; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-15; is_multi_year=True; trusted=2026-09-15 | account_alias=C-0BBE3E60; cz_renewal_date=2027-09-26; cb_renewal_date=2026-09-26; is_multi_year=True; trusted=2026-09-26 | account_alias=C-0BCDB8C2; cz_renewal_date=2027-09-18; cb_renewal_date=2026-09-18; is_multi_year=True; trusted=2026-09-18 | account_alias=C-0D2AB865; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-22; is_multi_year=True; trusted=2026-09-22 | account_alias=C-0F5D2323; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-29; is_multi_year=True; trusted=2026-09-29
n_disagreements: 5
total_arr_renewing_90d: 1048715.00
arr_at_risk_90d: 359409.00
n_accounts: 20
graded checks: total_renewing · arr_at_risk · disagreements_flagged · trust_rule

Model's actual answer

show the model's full answer
⚠ Stream stalled mid tool-call (execute_code); the action was not executed. Ask me to retry if you want to continue. 90-Day Renewal Risk Brief

Date trust rule: For multi-year contracts, ChurnZero dates are known wrong → trust ChargeBee. For single-year terms, trust ChurnZero.

| Company | CSM | ARR | Date Used | Seat Utilization | 3-mo Usage Trend | Risk | Evidence |
|---|---|---|---|---|---|---|---|
| C-0B7D2C30 | Dana Mercer | $65,901.00 | 2026-09-15 (ChargeBee; multi-year CZ wrong) | 274/476 = 57.6% | -29.8% | High | Usage fell 29.8% over the last 3 months. |
| C-0BCDB8C2 | Cole Ingram | $54,427.00 | 2026-09-18 (ChargeBee; multi-year CZ wrong) [DISAGREEMENT: CZ says 2027-09-18] | 232/424 = 54.7% | -29.3% | High | Usage fell 29.3% over the last 3 months. |
| C-0D2AB865 | Elena Sinclair | $38,022.00 | 2026-09-22 (ChargeBee; multi-year CZ wrong) [DISAGREEMENT: CZ says 2026-09-10] | 250/407 = 61.4% | -30.5% | High | Usage fell 30.5% over the last 3 months. |
| C-0BBE3E60 | Dana Mercer | $30,993.00 | 2026-09-26 (ChargeBee; multi-year CZ wrong) [DISAGREEMENT: CZ says 2027-09-26] | 74/114 = 64.9% | -31.8% | High | Usage fell 31.8% over the last 3 months. |
| C-0F5D2323 | Cole Ingram | $90,647.00 | 2026-09-29 (ChargeBee; multi-year CZ wrong) [DISAGREEMENT: CZ says 2026-09-10] | 111/390 = 28.5% | +11.3% | High | Seat utilization is only 28.5%. |
| C-0EC6999D | Elena Sinclair | $79,419.00 | 2026-10-03 (ChurnZero; single-year) | 31/112 = 27.7% | +0.0% | High | Seat utilization is only 27.7%. |
| C-0B20DB64 | Dana Mercer | $21,770.00 | 2026-10-07 (ChurnZero; single-year) | 214/378 = 56.6% | +0.0% | Medium | Seat utilization at 56.6% is below 60%. |
| C-0BBC4E7A | Cole Ingram | $56,374.00 | 2026-10-10 (ChurnZero; single-year) | 228/337 = 67.7% | +0.2% | Low | Stable utilization and usage. |
| C-0FD551AB | Elena Sinclair | $48,815.00 | 2026-10-14 (ChurnZero; single-year) | 210/376 = 55.9% | +1.1% | Medium | Seat utilization at 55.9% is below 60%. |
| C-0F9F8F13 | Dana Mercer | $46,230.00 | 2026-10-18 (ChurnZero; single-year) | 199/352 = 56.5% | +0.7% | Medium | Seat utilization at 56.5% is below 60%. |
| C-0BC34584 | Cole Ingram | $16,740.00 | 2026-10-22 (ChurnZero; single-year) | 327/494 = 66.2% | +1.3% | Low | Stable utilization and usage. |
| C-0B7A7546 | Elena Sinclair | $35,062.00 | 2026-10-25 (ChurnZero; single-year) | 182/205 = 88.8% | +7.3% | Low | Stable utilization and usage. |
| C-0B369871 | Dana Mercer | $85,128.00 | 2026-10-29 (ChurnZero; single-year) | 317/422 = 75.1% | +8.1% | Low | Stable utilization and usage. |
| C-0B144C78 | Cole Ingram | $30,899.00 | 2026-11-02 (ChurnZero; single-year) | 169/224 = 75.4% | +6.9% | Low | Stable utilization and usage. |
| C-0FC4DBB8 | Elena Sinclair | $94,732.00 | 2026-11-05 (ChurnZero; single-year) | 356/464 = 76.7% | +7.5% | Low | Stable utilization and usage. |
| C-0D5BBE3A | Dana Mercer | $39,740.00 | 2026-11-09 (ChurnZero; single-year) | 85/102 = 83.3% | +9.8% | Low | Stable utilization and usage. |
| C-0FB9D5AF | Cole Ingram | $63,158.00 | 2026-11-13 (ChurnZero; single-year) | 144/199 = 72.4% | +8.5% | Low | Stable utilization and usage. |
| C-0B344485 | Elena Sinclair | $64,384.00 | 2026-11-16 (ChurnZero; single-year) | 224/287 = 78.0% | +8.1% | Low | Stable utilization and usage. |
| C-0CB2C1B4 | Dana Mercer | $40,628.00 | 2026-11-20 (ChurnZero; single-year) | 386/473 = 81.6% | +3.6% | Low | Stable utilization and usage. |
| C-22170CA1 | Cole Ingram | $45,646.00 | 2026-11-24 (ChurnZero; single-year) | 251/294 = 85.4% | +7.4% | Low | Stable utilization and usage. |

Summary

- Total ARR renewing (next 90 days): $1,048,715.00
- ARR at risk (High + Medium): $476,224.00
- Date disagreements flagged: 5 of 20 accounts
  - C-0BCDB8C2, C-0BBE3E60: CZ carries 2027 dates while ChargeBee carries 2026 dates; multi-year terms ⇒ ChargeBee trusted.
  - C-0B7D2C30, C-0D2AB865, C-0F5D2323: CZ dates earlier in September 2026 than ChargeBee; multi-year terms ⇒ ChargeBee trusted.
ticket-theme-synthesisPASS 1.00

? The Question

Synthesize the quarter's support tickets into 4 to 7 themes. Do not trust existing tags; read the text. Per theme: name, count, share, distinct accounts, ARR affected, two ticket ids, one-line recommendation. Separate broad patterns from single-account noise and rank by ARR exposure, not volume.
cost $0.0231 · 93s · in 6,735 / out 2,727 tok
✓ theme_count_range✓ counts_verifiable✓ single_account_flagged✓ arr_ranked✓ ticket_ids

✓ Correct Answer (ground truth)

quarter_window: 2026-06-01 through 2026-08-31
total_tickets: 80
themes: {"points_not_posting": {"count": 20, "share_pct": 25.0, "distinct_accounts": 9, "arr_affected": "31100.00", "example_ticket_ids": ["IC-460001", "IC-460002"], "single_account": false}, "slack_sync": {"
ranking_by_arr_exposure: hris_provisioning, redemption_failures, billing_dispute, points_not_posting, slack_sync
single_account_noise_theme: billing_dispute
note: existing tags are noisy/unreliable; classify by text
sample_ticket_ids: 80 items — e.g. IC-460004, IC-460016, IC-460001, IC-460041
graded checks: theme_count_range · counts_verifiable · single_account_flagged · arr_ranked · ticket_ids

Model's actual answer

show the model's full answer
**Q2 support-ticket synthesis (ranked by ARR exposure, not volume)**

5 themes found. Billing-and-invoice errors is single-account noise but material ARR, so retained as its own theme.

| Rank | Theme | Count | Share | Distinct accounts | ARR affected | Sample ticket IDs | One-line recommendation |
|---|---:|---:|---:|---:|---:|---:|---|
| 1 | HRIS provisioning failures | 10 | 13.0% | 3 (C-0B2213A9, C-0DDFC9A7, C-0F6C0F34) | **$114,000** | IC-460059, IC-460062 | Audit HRIS sync logs against silent skips; add provisioning status dashboard for admins. |
| 2 | Redemption and checkout failures | 18 | 23.4% | 7 (C-0B827671, C-0CEF69FD, C-0F876796, C-0FCCD2DF, C-14264ABD, C-0D9CA315, C-0B0F1BAB) | **$68,800** | IC-460025, IC-460035 | Fix checkout timeout path and restore deducted points before retry; introduce rollback + retry queue. |
| 3 | Billing and invoice errors | 15 | 19.5% | 1 (C-0E9C27D1) | **$52,000** | IC-460071, IC-460069 | Escalate single-account billing accuracy review; reconcile seat count and renewal tier before next invoice run. |
| 4 | Points not posting / balance not updating | 20 | 26.0% | 9 (C-0D3278C7, C-0BE96399, C-0D0B047C, C-0BF20542, C-0D284E42, C-0D6CC8E3, C-0DD0626C, C-0B2895EF, C-21FEBCBB) | **$31,100** | IC-460004, IC-460001 | Trace async points ledger pipeline; surface delivery-vs-credit mismatch to users instantly. |
| 5 | Slack integration failures | 14 | 18.2% | 4 (C-0B843542, C-0BA71F12, C-10A56B0F, C-8C2E8F00) | **$18,900** | IC-460041, IC-460049 | Harden OAuth re-auth flow and stop the sync toggle from self-resetting after reconnect. |

**Arithmetic**
- Total tickets with body text: 77 of 79 (2 rows with empty `body_text` excluded because themes must be read from text, not tags).
- Total unique accounts exposed: 24.
- Total unique ARR exposed across all themes: $284,800.
- Shares are `theme count / 77`.
- ARR affected is the sum of unique account ARR per theme (a single account’s ARR counted once even if it filed multiple tickets).

**Pattern vs. noise**
- Broad patterns: HRIS provisioning, redemption/checkout, points posting, and Slack span 3+ accounts each.
- Single-account noise: Billing and invoice errors are concentrated entirely in **C-0E9C27D1** ($52K ARR), so it is a narrow but high-exposure issue rather than a quarter-wide pattern.
marketing
case-study-matchPASS 1.00

? The Question

Given the prospect, rank the three most similar existing customers with public case studies for use as social proof. Similarity on industry, size band, use case, region. Cite the fields that drove each match. Do not name any customer without a case study and never include billing data or contact names.
cost $0.0119 · 4861s · in 59 / out 2,676 tok
✓ top3✓ decoys_excluded✓ cites_fields✓ no_billing

✓ Correct Answer (ground truth)

top3: company_alias=C-11C31562; similarity_score=3 | company_alias=C-64171065; similarity_score=3 | company_alias=C-A13C193D; similarity_score=3
decoys_excluded: company_alias=C-D6217CAA; similarity_score=4; reason=no_case_study | company_alias=C-C153868F; similarity_score=4; reason=no_case_study
constraints: no_billing_data, no_contact_names, only_customers_with_has_case_study_true_named
prospect_fields_used: industry, size_band, use_case, region
prospect_alias: C-82AF3719
graded checks: top3 · decoys_excluded · cites_fields · no_billing

Model's actual answer

show the model's full answer
Top 3 existing-customer matches for prospect **C-82AF3719 (Technology, Mid-Market, employee_recognition, NA-West)**, restricted to customers with public case studies:

1. **C-64171065** — 3/4 exact-field match  
   Matches: industry (Technology), size_band (Mid-Market), use_case (employee_recognition). Region differs: NA-East vs NA-West (both North America).

2. **C-11C31562** — 3/4 exact-field match  
   Matches: size_band (Mid-Market), use_case (employee_recognition), region (NA-West). Industry differs: Manufacturing vs Technology.

3. **C-A13C193D** — 3/4 exact-field match  
   Matches: industry (Technology), size_band (Mid-Market), region (NA-West). Use_case differs: retention vs employee_recognition.

Only 8 customers in customers.csv have `has_case_study=true`; the three above are the only ones scoring 3/4 exact-field matches. No case-study customer matches all four fields exactly.
channel-attribution-mathPASS 1.00

? The Question

Compute channel performance for the trailing 6 months. Paid: spend, SQMs, SQOs, cost per SQM, cost per SQO, SQM-to-SQO rate, pipeline amount, pipeline per dollar. Organic: volume, SQO rate, pipeline. Report a channel with spend and zero SQMs as undefined, not zero. Flag rows where SQO date precedes SQM date. Finish with a reallocation recommendation and your confidence given sample sizes.
cost $0.0315 · 4876s · in 10,803 / out 5,640 tok
✓ undefined_channel✓ undefined_not_zero✓ flagged_rows✓ best_channel✓ reallocation

✓ Correct Answer (ground truth)

months: 2026-03, 2026-04, 2026-05, 2026-06, 2026-07, 2026-08
per_channel: {"linkedin_ads": {"spend_usd": 24000, "sqms": 25, "sqos": 8, "pipeline_amount": 96000, "cost_per_sqm": 960.0, "cost_per_sqo": 3000.0, "sqm_to_sqo_rate": 0.32, "pipeline_per_dollar": 4.0}, "organic_sea
undefined_channels: paid_social
flagged_rows_sqo_before_sqm: CT-000041, CT-000044
best_pipeline_per_dollar_channel: paid_search
reallocation_recommendation: from=paid_social; to=paid_search; rationale=paid_social has spend but zero SQMs (cost per SQM undefined, not zero); paid_search has the highest pipeline per dollar.
confidence_note: webinars and referral have small SQM samples (<20); treat rates as directional.
graded checks: undefined_channel · undefined_not_zero · flagged_rows · best_channel · reallocation

Model's actual answer

show the model's full answer
## Channel Performance — Trailing 6 Months

### Flags: SQO date precedes SQM date
Two linkedin_ads rows have SQO before SQM:
- CT-000044 | SQM 2026-07-23, SQO 2026-07-18
- CT-000041 | SQM 2026-06-14, SQO 2026-06-09

---

### Paid Channels

| channel | spend_usd | SQMs | SQOs | cost/SQM | cost/SQO | SQM→SQO | pipeline | pipeline/$ |
|---|---|---|---|---|---|---|---|---|
| paid_search | $36,000 | 40 | 18 | $900 | $2,000 | 45.0% | $720,000 | $20.00 |
| linkedin_ads | $24,000 | 25 | 8 | $960 | $3,000 | 32.0% | $96,000 | $4.00 |
| paid_social | $18,000 | 0 | 0 | undefined | undefined | undefined | $0 | undefined |
| webinars | $9,000 | 12 | 5 | $750 | $1,800 | 41.7% | $60,000 | $6.67 |

Arithmetic:
- paid_search: $36,000 / 40 = $900/SQM; $36,000 / 18 = $2,000/SQO; 18/40 = 45.0%; $720,000 / $36,000 = $20.00 pipeline/$
- linkedin_ads: $24,000 / 25 = $960/SQM; $24,000 / 8 = $3,000/SQO; 8/25 = 32.0%; $96,000 / $24,000 = $4.00 pipeline/$
- paid_social: 0 SQMs, all paid metrics undefined.
- webinars: $9,000 / 12 = $750/SQM; $9,000 / 5 = $1,800/SQO; 5/12 = 41.7%; $60,000 / $9,000 = $6.67 pipeline/$

---

### Organic Channels

| channel | volume (SQMs) | SQOs | SQO rate | pipeline |
|---|---|---|---|---|
| organic_search | 30 | 10 | 33.3% | $90,000 |
| referral | 15 | 6 | 40.0% | $48,000 |
| **organic combined** | **45** | **16** | **35.6%** | **$138,000** |

---

### Reallocation Recommendation

- **paid_search** is the clear winner: $20 pipeline per dollar, $2,000/SQO, 45% SQM→SQO rate, and 71% of all pipeline ($720k/$1.014M).
- **webinars** are next-most efficient paid channel at $1,800/SQO and $6.67 pipeline/$, despite the lowest spend.
- **linkedin_ads** is the weakest converting paid channel: $3,000/SQO, $4 pipeline/$, 32% SQM→SQO, plus 2 flagged data anomalies.
- **paid_social** shows spend but zero SQMs/SQOs — metrics are undefined; treat as no measurable pipeline contribution.

Recommended reallocation:
1. Pause or reduce **paid_social** ($3k/mo) until tracking confirms it produces SQMs.
2. Shift budget from **linkedin_ads** toward **paid_search** and **webinars** — both deliver materially lower cost per SQO and higher pipeline yield.
3. Maintain organic investment; combined organic (45 SQMs, 35.6% SQO rate, $138k pipeline) is cost-free pipeline that should be preserved/scaled.

Quantitative ceiling for confidence: paid_search ROI is robust but based on 18 SQOs. linkedin_ads has only 8 SQOs. paid_social has zero. Webinars has only 12 SQMs. Treat rankings as reliable but specific dollar-based forecasts as provisional.

Confidence: **MEDIUM**. The channel winners/losers are consistent, but sample sizes are small for per-channel conversion rates and paid_social has no measurable outcomes yet.
competitor-battlecard-updatePASS 1.00

? The Question

Update the battlecard for the competitor. Sections: one-line positioning, pricing with source and date (newer source wins, note the conflict), where they win, where we win, objections and responses, recent changes, our 12-month win/loss record against them. Cite a snippet id for every factual claim. Rep opinion on a call is not a fact about the competitor. Anything from the old card you cannot re-source gets marked unverified.
cost $0.0135 · 54s · in 1,329 / out 2,977 tok
✓ competitor_named✓ pricing✓ win_loss✓ unverified_marked✓ citations

✓ Correct Answer (ground truth)

competitor: Rivally
pricing: value_per_user_per_month=7; source_snippet=S17; source_date=2026-08-12
win_loss_12mo: wins=13; losses=7
unverified_old_claims: Rivally lacks a Slack integration., Rivally was acquired by WorkHuman in 2025.
rep_opinion_snippet_ids_not_facts: S09, S21
citation_ids_available: 25 items — e.g. S01, S02, S03, S04
graded checks: competitor_named · pricing · win_loss · unverified_marked · citations

Model's actual answer

show the model's full answer
## Updated Battlecard: Rivally

### One-line positioning
Points-based employee recognition platform with engagement-survey add-ons, emphasizing EU data residency and distributed-team support. (S03, S06, S12, S15)

### Pricing (with conflict noted)
- **Current public list price:** Rivally Recognition Starter is $7 per user/month, annual billing required, as of 2026-08-12. (S17)
- **Conflict:** Earlier pricing-page sources listed $5/user/mo with annual billing: 2026-01-20 (S03) and 2026-04-01 (S08). Newer pricing-page source wins; note the $2 increase.
- **Deal-quoted prices:** $6.50/user/mo for a 500-seat annual term in 2026-06 (S13), and $7/user/mo list with a 15% discount for a 3-year term in 2026-08 (S18).
- **Old card claim "$5/user/mo as of 2026-01" is superseded.**

### Where they win
- Recognition feed is engaging and points-based; praised by reviewers. (S02, S16)
- Strong for distributed EU teams; multi-language support praised. (S12)
- EU data residency is generally available. (S05, S15)
- Setup can take under a week, and Slack integration works out of the box. (S04)
- Support response time under 4 hours. (S22)

### Where we win
- Analytics depth: reviewers note Rivally's analytics / dashboards are basic, and an 800-seat prospect chose Bonusly over Rivally citing analytics depth. (S07, S25)
- Identity management: Rivally lacks SCIM provisioning; manual user management is painful. (S10)
- Admin tooling: admin console lags peers and lacks bulk recognition editing. (S16, S24)
- EMEA rewards catalog is thinner than the US catalog. (S14)
- Data portability: migration off Rivally is hard because analytics exports are CSV-only. (S20)

### Objections and responses
| Objection | Response | Source |
|---|---|---|
| "Rivally has EU data residency." | True; they opened a Dublin office and made EU data residency generally available in July 2026. Compete on analytics depth and admin tooling. | S15 |
| "Rivally's Slack integration works." | True; setup is reported as under a week and Slack works out of the box. Pivot to SCIM, admin tooling, and analytics. | S04 |
| "Rivally is cheaper." | Their public list is $7/user/mo annual; actual quotes vary ($6.50, and $7 with 15% off for 3-year term). Compare total cost including admin overhead. | S17, S13, S18 |
| "Rivally support is fast." | True; support response time under 4 hours praised. Compete on product depth, not support speed. | S22 |
| "Rivally is strong in EU enterprise." | True; EU reviewers praise multi-language support. Counter with our analytics/SCIM capabilities and EMEA catalog breadth. | S12, S10, S14 |

### Recent changes
- 2026-09-01: Rivally Pulse engagement-survey add-on exits beta; priced separately, not bundled. (S23)
- 2026-08-20: Microsoft Teams app v2 enters public preview. (S19)
- 2026-08-12: Pricing page updated; Recognition Starter raised to $7/user/mo annual. (S17)
- 2026-07-01: Dublin office opened; EU data residency generally available. (S15)
- 2026-05-09: Hired ex-Workday VP EMEA to lead European expansion. (S11)
- 2026-03-05: Launched Rivally Pulse lightweight engagement survey add-on. (S06)
- 2025-11-04: Announced $40M Series C led by Northgate Ventures. (S01)

### Our 12-month win/loss record against Rivally
Period: 2025-09 through 2026-08 (the available trailing 12 months).

Wins: 13  
Losses: 7  
Total: 20  
Win rate: 13 / 20 = **65%**  
Loss rate: 7 / 20 = **35%**

Month-by-month:
- 2025-09: 1 win, 1 loss
- 2025-10: 2 wins, 0 losses
- 2025-11: 1 win, 1 loss
- 2025-12: 1 win, 1 loss
- 2026-01: 2 wins, 0 losses
- 2026-02: 2 wins, 0 losses
- 2026-03: 1 win, 1 loss
- 2026-04: 0 wins, 2 losses
- 2026-05: 0 wins, 1 loss
- 2026-06: 1 win, 0 losses
- 2026-07: 1 win, 0 losses
- 2026-08: 1 win, 0 losses

### Unverified / removed from old card
- **"Rivally lacks a Slack integration"** — Removed. Snippet S04 states Slack integration worked out of the box; old claim is contradicted by source data.
- **"Rivally was acquired by WorkHuman in 2025"** — Unverified; no source in provided snippets.
- **"Points-based recognition for mid-market"** — Partially retained/expanded; S02/S03 confirm points-based recognition, while S12 and S15 also show EU enterprise traction.
nurture-sequence-diagnosisPASS 1.00

? The Question

Review the sequences. Per sequence: sent, open rate, reply rate, meeting rate, weakest step. Identify tracking errors (opened above sent), audience overlap between sequences, and the failure mode for anything under 2 percent reply. One change per weak sequence and which to fix first. Under 350 words.
cost $0.0934 · 7261s · in 57,352 / out 14,047 tok
✓ tracking_error✓ overlap_pair✓ weakest_step✓ fix_priority

✓ Correct Answer (ground truth)

per_sequence: {"New Logo Nurture": {"sent": 1386, "opened": 490, "replied": 90, "meetings": 27, "open_rate": 0.3535, "reply_rate": 0.0649, "meeting_rate": 0.0195, "weakest_step": 3}, "Expansion Nurture": {"sent": 8
tracking_error: sequence=Expansion Nurture; step=2; sent=300; opened=340; issue=opened exceeds sent
audience_overlap_pair: Cold Outbound - HR Leaders, Cold Outbound - People Ops
all_pairwise_overlaps: ['Cold Outbound - HR Leaders', 'Cold Outbound - People Ops'], ['Expansion Nurture', 'New Logo Nurture']
under_2pct_reply: Cold Outbound - HR Leaders
failure_mode: Cold Outbound - HR Leaders=opens healthy (~37%) but replies near zero from step 1 -> body/CTA not landing, not a deliverability or subject-line problem
fix_priority_first: Cold Outbound - HR Leaders
graded checks: tracking_error · overlap_pair · weakest_step · fix_priority

Model's actual answer

show the model's full answer
## Sequence performance

| Sequence | Sent | Open rate | Reply rate | Meeting rate | Weakest step |
|---|---|---|---|---|---|
| New Logo Nurture | 1,386 | 490/1,386 = **35.4%** | 90/1,386 = **6.5%** | 27/1,386 = **1.9%** | Step 3 (4.2% reply) |
| Expansion Nurture | 875 | 565/875 = **64.6%** | 59/875 = **6.7%** | 12/875 = **1.4%** | Step 3 (4.4% reply) |
| Cold Outbound - HR Leaders | 1,785 | 545/1,785 = **30.5%** | 8/1,785 = **0.4%** | 0/1,785 = **0.0%** | Step 3 (0.2% reply) |
| Cold Outbound - People Ops | 1,163 | 340/1,163 = **29.2%** | 29/1,163 = **2.5%** | 6/1,163 = **0.5%** | Step 3 (1.6% reply) |

## Tracking errors
- **Expansion Nurture step 2**: opened 340 vs. sent 300 — an impossible 113.3% open rate.

## Audience overlap
- **21 contacts** are in both **Cold Outbound - HR Leaders** and **Cold Outbound - People Ops** (e.g., CT-000849, CT-000884, CT-000890, CT-000908, CT-001033, CT-001097, CT-001101, CT-001103, CT-001105, CT-001130, CT-001153, CT-001159, CT-001217, CT-001227, CT-001236, CT-001255, CT-001258, CT-001277, CT-001285, CT-001311, CT-001345).
- **2 contacts** are in both **Expansion Nurture** and **New Logo Nurture**: CT-000301 and CT-000624.

## Failure mode under 2% reply
- **Cold Outbound - HR Leaders**: every step is under 2% reply; opens are decent but replies/meetings are nil. Failure mode: message/offer mismatch with **HR Leaders** audience.
- **Cold Outbound - People Ops step 3**: fatigue/content decay after two touches.

## One change per weak sequence
- **New Logo Nurture**: rewrite Step 3 with a customer proof point and tighter CTA.
- **Expansion Nurture**: correct the Step 2 open-tracking discrepancy, then refresh Step 3 around account-specific expansion value.
- **Cold Outbound - HR Leaders**: pause and retest messaging/IQ before sending Step 3; current sequence is burning contacts with no pipeline.
- **Cold Outbound - People Ops**: dedupe the 21 overlapping contacts from HR Leaders before sending Step 3.

## Fix first
**Cold Outbound - HR Leaders** — largest sent volume (1,785), zero meetings, and every step under 2% reply; it is the biggest pipeline leak and quickest cost save.
weekly-marketing-goals-rollupPASS 1.00

? The Question

Produce the weekly marketing goals update for the active quarter: SQMs, SQOs, DS2s, closed-lost MIA rate, same-quarter close count, active pipeline coverage against target. For each: QTD actual, target, delta, pace (on/behind/ahead given days elapsed). One paragraph on what moved this week.
cost $0.0084 · 4307s · in 27 / out 1,683 tok
✓ metrics_present✓ deltas✓ coverage_ratio✓ pace✓ paragraph

✓ Correct Answer (ground truth)

days_elapsed: 66
days_in_quarter: 92
elapsed_fraction: 0.7174
metrics: {"SQMs": {"qtd_actual": 230, "target": 300, "delta": -70, "expected_at_day_elapsed": 215.22, "pace": "ahead"}, "SQOs": {"qtd_actual": 84, "target": 120, "delta": -36, "expected_at_day_elapsed": 86.09,
coverage_ratio_active_pipeline: 0.75
pace_rule: expected = target * days_elapsed/days_in_quarter; within +/-5% of expected = on, above = ahead, below = behind; lower_better metrics invert.
this_week_paragraph_required: True
graded checks: metrics_present · deltas · coverage_ratio · pace · paragraph

Model's actual answer

show the model's full answer
Q3-2026 update (66 of 92 days elapsed = 71.7% of quarter)

Metric                          | QTD actual | Target   | Delta       | Pace vs. elapsed
--------------------------------|------------|----------|-------------|-----------------
SQMs                            | 230        | 300      | -70         | ahead   (230/300 = 76.7% vs 71.7% elapsed)
SQOs                            | 84         | 120      | -36         | behind  (84/120 = 70.0% vs 71.7% elapsed)
DS2s                            | 40         | 75       | -35         | behind  (40/75 = 53.3% vs 71.7% elapsed)
Closed-lost MIA rate            | 20.0%      | 10.0%    | +10.0 pp    | behind  (5/25 = 0.20, above 0.10 target)
Same-quarter close count        | 10         | 20       | -10         | behind  (10/20 = 50.0% vs 71.7% elapsed)
Active pipeline coverage        | $3.0M      | $4.0M    | -$1.0M      | ahead   ($3.0M/$4.0M = 75.0% vs 71.7% elapsed)

Arithmetic:
- Days elapsed share = 66 / 92 = 71.7%
- Closed-lost MIA rate = 5 / 25 = 0.20 = 20.0%
- Active pipeline coverage = $3,000,000 / $4,000,000 = 0.75x

What moved this week: No week-over-week data was provided, so I cannot report what specifically moved this week. The figures above are cumulative QTD totals as of 66 days into Q3-2026.
communication
ceo-slack-compressionPASS 1.00

? The Question

Compress the forecast analysis below into a single Slack message for the CEO. Rules: at most three sentences; under 80 words; lead with the decision or the so-what; keep every distinct number that matters and the one caveat that changes the conclusion; cut any sentence that does not carry a claim, a number, or a directive. No em dashes, no throat-clearing phrases.
cost $0.0065 · 18s · in 581 / out 1,022 tok
✓ max_3_sentences✓ under_80_words✓ weighted_present✓ caveat_present✓ no_em_dash

✓ Correct Answer (ground truth)

graded checks: max_3_sentences · under_80_words · weighted_present · caveat_present · no_em_dash

Model's actual answer

show the model's full answer
Do not trust the Q3 forecast: weighted number is $115,977 from 54 in-quarter deals ($44,729 COMMIT + 35% of $203,565 BEST_CASE), but 32 deals worth $227,575 slipped to October including COMMIT Deal-D348E1 ($13,770). Data is unreliable: owner is blank on 85 of 86 deals and why-buys is empty on 71 of 86, including all 7 COMMITs. Re-qualify close dates and reasons before using any number.
stalled-deal-followup-emailPASS 1.00

? The Question

Write a follow-up email for this stalled deal. Under 120 words. Reference one specific prior touch from the engagement history, make one clear ask, and match the tone of the prior thread. No invented details.
cost $0.0052 · 16s · in 406 / out 690 tok
✓ word_cap✓ references_touch✓ single_ask

✓ Correct Answer (ground truth)

deal_alias: Deal-0D2F7A
amount: 5100
stage: DS3
owner: Alex Franklin
days_since_last_engagement: 31
word_cap: 120
must_reference: engagement_id=ENG-00002; date=2026-07-28; type=meeting; why=most substantive prior touch; concrete prospect reactions to milestone awards and points catalog
single_cta: True
tone: warm, concise, professional (matches prior thread)
allowed_facts: 14 items — e.g. Deal-0D2F7A, $5,100, DS3, Alex Franklin
allowed_new: ENG-00001, ENG-00002, ENG-00003, ENG-00004, ENG-00005
graded checks: word_cap · references_touch · single_ask

Model's actual answer

show the model's full answer
Subject: Re: Bonusly recap and pricing — next step?

Hi [Name],

Following up on my August 5 email with the recap of our July 28 demo and pricing for 150 seats.

The team had a strong reaction to automated milestone awards and the points catalog, and I want to make sure nothing is sitting on my end. Are you still looking at a start before year-end, or is there a specific question I can answer to help move this forward with the VP People?

What does your calendar look like for a quick 15-minute call this week?

Best,
Alex Franklin

— 78 words
funnel-digest-paragraphPASS 1.00

? The Question

Write the weekly GTM digest: one paragraph per team (Marketing, Sales, CS, PLG) in Bonusly brand voice. Every number must trace to the extracts. No customer contact names, no employee performance commentary, no compensation. Four paragraphs, under 400 words total.
cost $0.0078 · 23s · in 347 / out 1,465 tok
✓ four_paragraphs✓ word_cap✓ teams

✓ Correct Answer (ground truth)

required_paragraphs: Marketing, Sales, CS, PLG
allowed_numbers: 16 items — e.g. 2, 3, 9, 08
word_cap: 400
banned: customer contact names, employee performance commentary, compensation
graded checks: four_paragraphs · word_cap · teams

Model's actual answer

show the model's full answer
Marketing generated 46 SQMs this week, coming in 6 below the 52-SQM target (52 − 46 = 6). Webinars led the channel mix with 18 of those 46 SQMs, and paid search held steady at $150 per SQM.

Sales converted 14 SQOs, set 9 DS2 meetings, and created $310,000 in new pipeline. The team also closed 3 deals same-quarter.

CS protected 2 renewals and lifted Team NPS to 61. We’re heading into next week with 3 open red-flag accounts on the watchlist.

PLG added 412 new signups at a 31% activation rate, and 38 companies reached the aha moment of 10 recognition gives.
partner-digest-sectionPASS 1.00

? The Question

Write the partner digest for the period. Every partner in the roster appears once: active plays get status with numbers, quiet partners get one line. Partner-sourced pipeline count and amount must match the deal data. Nothing omitted, nothing added.
cost $0.0052 · 13s · in 438 / out 680 tok
✓ all_partners✓ pipeline_amount✓ quiet_one_line

✓ Correct Answer (ground truth)

roster: partner_name=Apex Rewards Co; status=active | partner_name=HRCloud Partners; status=active | partner_name=CultureBridge; status=active | partner_name=WorkWell Group; status=active | partner_name=Recogniq; status=quiet | partner_name=KudosWave; status=quiet … (+2 more)
per_partner_pipeline: {"Apex Rewards Co": {"count": 2, "amount": 275000}, "HRCloud Partners": {"count": 1, "amount": 140000}, "CultureBridge": {"count": 2, "amount": 135000}}
partner_sourced_pipeline: count=5; amount=550000
quiet_partners: Recogniq, KudosWave, PeopleFirst Advisors, TotalPerk
active_no_deals: WorkWell Group
rule: every roster partner named exactly once; quiet partners get one line; no partners outside roster.
graded checks: all_partners · pipeline_amount · quiet_one_line

Model's actual answer

show the model's full answer
Partner Digest: 2026-08-24 to 2026-09-04

**Apex Rewards Co** — Active. 2 sourced opps / $275,000 pipeline. Co-webinar locked for 09-15; both deals in DS1. Deals: Deal-DDAAF2 ($180,000), Deal-2CF33E ($95,000).

**HRCloud Partners** — Active. 1 sourced opp / $140,000 pipeline. Integration referral cleared security review and moved to DS2. Deal: Deal-F1CDA5 ($140,000).

**CultureBridge** — Active. 2 sourced opps / $135,000 pipeline. Lunch-and-learn produced both deals; both early stage. Deals: Deal-096E1D ($60,000), Deal-067213 ($75,000).

**WorkWell Group** — Active. 0 sourced opps / $0 pipeline. Restarting joint playbook in Q4; planning call booked for 09-09.

**Recogniq** — Quiet this period.

**KudosWave** — Quiet this period.

**PeopleFirst Advisors** — Quiet this period.

**TotalPerk** — Quiet this period.

---
Partner-sourced pipeline total: 5 deals / $550,000. ($275,000 + $140,000 + $135,000 = $550,000.)
rfp-answer-from-policyPASS 1.00

? The Question

Answer the 10 questionnaire items from the policy excerpts only. Cite the excerpt id for every answer. Where the excerpts do not answer the question, mark it unanswerable and say what document would; do not guess. Tone: vendor response, direct.
cost $0.0062 · 15s · in 530 / out 961 tok
✓ answered_count✓ unanswerable_flagged✓ citations✓ no_guess

✓ Correct Answer (ground truth)

answered_count: 7
answer_map: {"Q1": ["P1"], "Q2": ["P1"], "Q3": ["P2"], "Q4": ["P3"], "Q5": ["P4"], "Q6": ["P5"], "Q7": ["P6"]}
unanswerable: Q8, Q9, Q10
valid_citation_ids: P1, P2, P3, P4, P5, P6, P7, P8
rule: unanswerable items must be flagged and the document that would answer them named; do not guess.
graded checks: answered_count · unanswerable_flagged · citations · no_guess

Model's actual answer

show the model's full answer
Q1. Yes. Customer data is encrypted at rest using AES-256. [P1]

Q2. Yes. Data in transit is encrypted with TLS 1.2 or higher. [P1]

Q3. Yes. Bonusly maintains a SOC 2 Type II report, renewed annually, and it is available to prospects under NDA. [P2]

Q4. Customer data is retained for the duration of the contract plus 90 days, after which it is deleted from production systems. [P3]

Q5. The subprocessor list is maintained at trust.bonusly.com, and customers receive 30 days' notice before a new subprocessor is added. [P4]

Q6. Yes. SAML 2.0 single sign-on is available on the Enterprise plan. [P5]

Q7. Production databases are backed up daily; backups are encrypted and retained for 35 days. [P6]

Q8. Unanswerable. The policy excerpts do not address ISO 27001 certification. A current certification register or compliance attestation letter would answer this.

Q9. Unanswerable. The policy excerpts do not state a contractual uptime SLA percentage. The service-level agreement or master subscription agreement would answer this.

Q10. Unanswerable. The policy excerpts do not address HIPAA or willingness to sign a Business Associate Agreement. A compliance statement, BAA policy, or sales/legal contract representative would answer this.
ops-maintenance
skill-file-reconciliation0.86

? The Question

Reconcile the skill set below against its manifest. Find and report (with severity CRITICAL/WARNING/INFO and an action type TRIM_DESC, UPDATE_BODY, DELETE_SKILL, MERGE, or REVIEW): (1) any two skills whose ALWAYS-trigger phrases overlap/duplicate, (2) any circular delegation chain between skills (name it), (3) any delegation target that does not exist (dangling, name it), (4) any version conflict and which skill should survive, (5) how many manifest descriptions exceed 1,024 characters, (6) any hardcoded page ids, dates, or person names in skill bodies, (7) manifest drift in BOTH directions (files with no manifest row, and manifest rows with no file). Make one proposal per finding; rewrite nothing.
cost $0.0953 · 105s · in 79,218 / out 11,158 tok
✗ duplicate_trigger_found✓ circular_chain✓ dangling_target✓ oversized_desc_count✓ hardcoded_values✓ manifest_drift✓ severity_and_action

✓ Correct Answer (ground truth)

graded checks: duplicate_trigger_found · circular_chain · dangling_target · oversized_desc_count · hardcoded_values · manifest_drift · severity_and_action

Model's actual answer

show the model's full answer
Skill set reconciliation against skill_manifest.csv (14 rows / 14 files).

1) ALWAYS-trigger phrase overlap/duplicate
- comms-drafter ↔ email-drafter — WARNING — MERGE
  Verbatim overlap: "write me an email", "draft a follow-up", "help me reply", "what should I say", "bump email", "contract nudge".
- weekly-pipeline-report ↔ pipeline-intelligence-report — WARNING — MERGE
  Overlap: "run the pipeline report", "pipeline update", "what's the pipeline look like" / "what does pipeline look like".

2) Circular delegation chain
- email-drafter ↔ deal-strategy-coach — WARNING — REVIEW
  email-drafter says "If the user needs strategic deal coaching...point them to the deal-strategy-coach skill."
  deal-strategy-coach says "When drafting manager-to-prospect emails, use the email-drafter skill."
  Proposal: break one direction so strategy coaching never loops back into email drafting.

3) Dangling delegation targets (not in manifest)
- bonusly-brand, prospect-research-multithreading, signalforge-reports, caveman, skill-orchestrator, and the eight bonusly-* specialist skills listed in analysis-validator §12.4 (bonusly-data-questions, bonusly-product-questions, bonusly-business-reporting-questions, bonusly-rewards-questions, bonusly-ppp-questions, bonusly-feature-flag-questions, bonusly-deal-desk-questions, bonusly-datadog-questions) — WARNING — REVIEW
  Verify whether these exist as org skills outside this manifest or remove references.

4) Version conflict
- analysis-validator — WARNING — UPDATE_BODY
  Changelog lists v3.6 and v3.5 both dated 2026-05-09; the body declares v3.6. v3.6 should survive; remove/resequence the duplicate-dated v3.5 entry.

5) Manifest descriptions exceeding 1,024 characters
- INFO — none required
  Count = 0. Maximum description_chars in manifest = 1,006 (pipeline-intelligence-report and signalforge-claim-compressor). 1,006 < 1,024.

6) Hardcoded page IDs, dates, and person names in skill bodies
- WARNING — UPDATE_BODY
  Examples:
  - Page/space IDs: partner-digest folder 2286616609 and reference page 2286321666; signalforge-feedback page 2295136266/parent 2234417154; sales-forecast space 2232811524/parent 2232582148.
  - Slack IDs: stale-pipeline-report channel C0561C1JCPJ; partner-digest Amani user ID U03QLMBL7AR.
  - Person names: Amani Phipps (partner-digest), Ben Lavin (weekly-pipeline-report), Alaina/Shealagh/Manish/rep names (analysis-validator, pipeline-intelligence-report, deal-strategy-coach), Elena (sales-forecast changelog).
  - Dates/anchors: "May 9, 2026", "May 4, 2026", "March 28, 2023", Q1 2026 actuals, 2026 pricing table, ~452K/~110K user anchors, HubSpot stage IDs and owner IDs.
  Proposal: externalize constants into a reference file or manifest rather than hardcoding in skill bodies.

7) Manifest drift (files without manifest rows; manifest rows without files)
- INFO — none required
  Both directions = 0. All 14 manifest rows match the 14 SKILL.md files by filename.
runbook-from-slack-threadPASS 1.00

? The Question

Turn the resolved incident thread into a runbook: ordered steps with the exact command or action taken, who ran it, how success was verified, and rollback for each step that changed state. Every step must trace to a message in the thread. Add nothing the thread does not contain; where the thread is unclear, mark the step as needing confirmation.
cost $0.0084 · 22s · in 619 / out 1,585 tok
✓ ordered_steps✓ rollback✓ needs_confirmation✓ sourced

✓ Correct Answer (ground truth)

steps: order=1; msg=M01; actor=Bryce Harmon; action=acknowledge PagerDuty alert, take IC; state_change=False; verified_by=M01 | order=2; msg=M02; actor=Farid Osman; action=check queue depth; command=bundle exec rake sidekiq:queue_depth; state_change=False; verified_by=M02 (48,213 pending) | order=3; msg=M04; actor=Farid Osman; action=pause auto-enqueue via feature flag; command=bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'; state_change=True; rollback=bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'; verified_by=M07 (queue depth falling) | order=4; msg=M05; actor=Elena Sinclair; action=cleared the dead set; state_change=True; needs_confirmation=thread gives no command or method for clearing the dead set | order=5; msg=M06; actor=Bryce Harmon; action=scale reward-worker to 6 replicas; command=kubectl scale deployment/reward-worker --replicas=6; state_change=True; rollback=kubectl scale deployment/reward-worker --replicas=3; verified_by=M07/M08 | order=6; msg=M08; actor=Cole Ingram; action=verify queue drained and error rate baseline; command=bundle exec rake sidekiq:queue_depth; state_change=False; verified_by=M08 (returns 0) … (+2 more)
ambiguous_step: order=4; msg=M05; mark=needs-confirmation
rule: no steps beyond what the thread contains; every step traces to a message id.
graded checks: ordered_steps · rollback · needs_confirmation · sourced

Model's actual answer

show the model's full answer
# Runbook — reward-worker queue depth incident (resolved 2026-08-29)

## Trigger
PagerDuty alert fires when reward-worker queue depth exceeds 10,000 pending jobs. Normal queue depth is under 500.

---

### Step 1 — Acknowledge incident and take incident commander role
- **Action:** Acknowledge the PagerDuty alert and assume incident commander (IC) role.
- **Run by:** Bryce Harmon
- **Command / action:** PagerDuty acknowledge + IC declaration
- **Verification:** IC assigned.
- **Rollback:** N/A (no state change)
- **Source:** [M01]

### Step 2 — Check current reward queue depth
- **Action:** Measure reward-worker queue depth to confirm severity.
- **Run by:** Farid Osman
- **Command:** `bundle exec rake sidekiq:queue_depth`
- **Verification:** Command returned 48,213 pending jobs (normal: < 500).
- **Rollback:** N/A (read-only)
- **Source:** [M02]

### Step 3 — Inspect Sidekiq dead set
- **Action:** Check the dead set for failed jobs and failure pattern.
- **Run by:** Farid Osman
- **Command / action:** Dead set inspection in console
- **Verification:** Dead set contained 112 jobs; all were `Redis::TimeoutError` originating around 13:58.
- **Rollback:** N/A (read-only)
- **Source:** [M03]

### Step 4 — Pause new enqueue to stop queue growth
- **Action:** Disable automatic enqueue of new recognition jobs.
- **Run by:** Farid Osman
- **Command:** `bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'`
- **Verification:** Feature flag disabled; new job inflow stopped.
- **Rollback:** `bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'`
- **Source:** [M04]

### Step 5 — Clear the dead set
- **Action:** Remove all jobs from the Sidekiq dead set.
- **Run by:** Elena Sinclair
- **Command / action:** Cleared dead set from the console — exact command not stated.
- **Verification:** Dead set empty after action.
- **Rollback:** ⚠️ Not specified in thread. **Needs confirmation:** document exact clear command and whether jobs need to be rehydrated.
- **Source:** [M05]

### Step 6 — Scale reward-worker replicas up
- **Action:** Increase worker capacity to drain the backlog.
- **Run by:** Bryce Harmon
- **Command:** `kubectl scale deployment/reward-worker --replicas=6`
- **Verification:** Queue depth fell from 48,213 to 9,400 and was decreasing at ~1,200/min.
- **Rollback:** `kubectl scale deployment/reward-worker --replicas=3`
- **Source:** [M06], verification in [M07]

### Step 7 — Confirm queue fully drained and error rate recovered
- **Action:** Verify queue depth is zero and system error rate has returned to baseline.
- **Run by:** Cole Ingram
- **Command:** `bundle exec rake sidekiq:queue_depth` and Datadog error-rate dashboard
- **Verification:** Queue depth returned 0; Datadog error rate back to baseline.
- **Rollback:** N/A (read-only)
- **Source:** [M08]

### Step 8 — Re-enable enqueue
- **Action:** Restore automatic enqueue of new recognition jobs.
- **Run by:** Bryce Harmon
- **Command:** `bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'`
- **Verification:** 40 new jobs processed cleanly within the following 3 minutes.
- **Rollback:** `bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'`
- **Source:** [M09]

### Step 9 — Scale reward-worker replicas back down
- **Action:** Return worker capacity to pre-incident level.
- **Run by:** Bryce Harmon
- **Command:** `kubectl scale deployment/reward-worker --replicas=3`
- **Verification:** Queue remained stable at 0.
- **Rollback:** `kubectl scale deployment/reward-worker --replicas=6`
- **Source:** [M10]

---

## Open gaps requiring confirmation
1. **Step 5 rollback:** Elena Sinclair’s exact dead-set clear command is not recorded, and whether the 112 `Redis::TimeoutError` jobs can be requeued/replayed is unknown.
2. **FeatureFlag semantics:** Thread treats `FeatureFlag.enable(:auto_recognition_enqueue)` as the inverse of disable, but flag scope (global vs. actor) is not stated.
datadog-incident-timelinePASS 1.00

? The Question

From the log slice, sequence the failure: first error with timestamp, the cascade in order, the service and job involved, and the Datadog query that would confirm the first error. State what the logs do not show.
cost $0.0086 · 20s · in 1,718 / out 1,450 tok
✓ first_error_ts✓ service_named✓ cascade_ordered✓ query✓ gaps_stated

✓ Correct Answer (ground truth)

first_error: timestamp=2026-09-03T14:01:12Z; service=reward-service; message=Redis::TimeoutError: Connection to Redis at redis-primary:6379 timed out after 5s
cascade_order: service=reward-service; first_error_at=2026-09-03T14:01:12Z | service=sidekiq; first_error_at=2026-09-03T14:01:40Z | service=api-gateway; first_error_at=2026-09-03T14:03:05Z | service=web-app; first_error_at=2026-09-03T14:03:30Z
job_class_involved: RewardGiveJob
confirming_query: service:reward-service status:error "@message:Redis::TimeoutError"
logs_do_not_show: root cause of the Redis timeout, redis-server's own logs, infrastructure metrics (CPU/memory/network), any deploy or config change preceding the incident, who applied mitigation or when (no actor fields in logs)
clean_services: postgres
graded checks: first_error_ts · service_named · cascade_ordered · query · gaps_stated

Model's actual answer

show the model's full answer
First error:
- Timestamp: 2026-09-03T14:01:12Z
- Service: reward-service
- Message: Redis::TimeoutError: Connection to Redis at redis-primary:6379 timed out after 5s

Cascade in order:
1. 14:01:12Z — reward-service reports Redis timeout to redis-primary:6379.
2. 14:01:20Z–14:01:40Z — reward-service logs "retry exhausted for RewardGiveJob" (three times).
3. 14:01:40Z — sidekiq reports RewardGiveJob failed: Redis::TimeoutError; retrying in 60s.
4. 14:02:28Z–14:06:47Z — sidekiq reports repeated RewardGiveJob failures/retries; at 14:02:30Z sidekiq warns queue reward depth above 10,000.
5. 14:03:05Z–14:06:52Z — api-gateway reports 502 upstream timeout calling reward-service (/gives and unspecified paths).
6. 14:03:30Z–14:06:49Z — web-app reports Give form submission failed: upstream 502.
7. 14:22:10Z — reward-service reports Redis connection restored; resuming job processing.
8. 14:24:45Z — sidekiq reports queue reward depth below 500.

Service and job involved:
- Service: reward-service
- Job: RewardGiveJob (sidekiq); RecognitionDigestJob also fails with Redis::TimeoutError per sidekiq_jobs.csv starting at 14:02:36Z.

Datadog query to confirm the first error:
service:reward-service status:error "Redis::TimeoutError" "redis-primary:6379" @timestamp:[2026-09-03T14:00:00Z TO 2026-09-03T14:02:00Z]

What the logs do not show:
- Why Redis timed out (no root cause: network partition, Redis CPU/memory pressure, failover, deployment event, traffic spike, etc.).
- Whether redis-primary was unreachable or merely slow to respond.
- CPU, memory, network, or disk metrics for reward-service or Redis.
- The user-facing request volume or number of failed Give submissions.
- Trace/request IDs linking the web-app errors to specific reward-service calls.
- Any deployment, alert, or incident metadata.
- Whether the RecognitionDigestJob failures are caused by the same Redis incident or a separate issue (only the same error class is shown).
- How many retries occurred before each "retry exhausted" message.
feature-flag-state-summaryPASS 1.00

? The Question

Summarize the state of the listed feature flags: what each controls per the code excerpt, which companies or segments have it on, targeting rules, and any flag with no code reference or no targets. Company counts must match the export. Do not describe a flag that is not in the export.
cost $0.0067 · 12s · in 596 / out 1,100 tok
✓ orphans✓ no_targets✓ counts_match✓ no_invented

✓ Correct Answer (ground truth)

flags: {"recognition_streaks_v2": {"state": "on", "targeting_rules": "segment:beta_companies", "company_count": 42}, "points_budget_guardrails": {"state": "on", "targeting_rules": "all_companies", "company_c
code_referenced: recognition_streaks_v2, points_budget_guardrails, slack_dm_nudges, redeem_flow_redesign, analytics_dashboard_v3, ms_teams_app_v2
orphan_flags_no_code_reference: legacy_give_modal, survey_boosters_q3, paused_offboard_cleanup
flags_with_no_targets: paused_offboard_cleanup
rule: company counts must equal the export; do not describe flags absent from the export.
graded checks: orphans · no_targets · counts_match · no_invented

Model's actual answer

show the model's full answer
Feature flag summary (from `flags_export.csv` + `flag_code.md`):

1. **recognition_streaks_v2** — `on`, 42 companies
   - Controls: records a recognition streak via `StreakTracker.record(give)`.
   - Targeting: `segment:beta_companies`.

2. **points_budget_guardrails** — `on`, 220 companies
   - Controls: enforces a points budget via `BudgetService.new(company).enforce!(giver, points)`.
   - Targeting: `all_companies`.

3. **slack_dm_nudges** — `on`, 87 companies
   - Controls: sends a Slack DM nudge via `SlackDm.send_nudge(user)`.
   - Targeting: `segment:region_na`.

4. **redeem_flow_redesign** — `off`, 12 companies
   - Controls: renders `RedeemV2Component` when on; otherwise `RedeemV1Component`.
   - Targeting: `targeted_list` (specific companies not named in export).

5. **analytics_dashboard_v3** — `on`, 65 companies
   - Controls: builds dashboards with `AnalyticsV3.new(company)`.
   - Targeting: `segment:tier_three`.

6. **ms_teams_app_v2** — `off`, 9 companies
   - Controls: installs the v2 MS Teams app via `TeamsAppV2.install(company)`.
   - Targeting: `targeted_list` (specific companies not named in export).

7. **legacy_give_modal** — `off`, 14 companies
   - Targeting: `segment:legacy_plan`.
   - **No code reference in the code excerpt.**

8. **survey_boosters_q3** — `on`, 7 companies
   - Targeting: `segment:legacy_plan`.
   - **No code reference in the code excerpt.**

9. **paused_offboard_cleanup** — `off`, 0 companies
   - Targeting: none provided.
   - **No code reference in the code excerpt and no targets.**

Company count total per export: `42 + 220 + 87 + 12 + 65 + 9 + 14 + 7 + 0 = 456`.
nda-triagePASS 1.00

? The Question

Triage each NDA as GREEN (standard approval), YELLOW (counsel review), or RED (full legal review). Cite the clause that drives each classification by quoting it. Catch embedded non-solicits, non-competes, missing carve-outs, and unusual term or governing law. One line of reasoning per NDA.
cost $0.0050 · 8s · in 536 / out 610 tok
✓ green✓ yellow✓ red✓ non_solicit_caught✓ clause_quoted✓ correct_labels

✓ Correct Answer (ground truth)

classifications: {"NDA-1": {"classification": "GREEN", "driving_reason": "standard mutual NDA: 2-year term, standard carve-outs, Delaware law", "driving_clause_quote": "Carve-outs: information that (a) is or becomes p
non_solicit_in: NDA-2
checks: classification_per_nda, clause_quotes_exist_in_document, non_solicit_caught, carveouts_checked
graded checks: green · yellow · red · non_solicit_caught · clause_quoted · correct_labels

Model's actual answer

show the model's full answer
**NDA-1 — [PARTY A] and Bonusly**
GREEN — Standard mutual NDA with conventional 2-year term, 3-year survival, complete carve-outs, Delaware governing law, and no hidden restrictions.

**NDA-2 — [PARTY B] and Bonusly**
YELLOW — Embedded employee non-solicit requires counsel review.
Clause: "During the term of this Agreement and for eighteen (18) months thereafter, neither party shall solicit for employment or hire any employee of the other party with whom it came into contact under this Agreement."

**NDA-3 — [PARTY C] and Bonusly**
RED — Full legal review required due to broad non-compete, missing carve-outs, and non-standard Ireland jurisdiction.
Clause (non-compete): "For a period of three (3) years following the Effective Date, Recipient shall not, directly or indirectly, engage in or provide services to any business that competes with Discloser's business."
Clause (missing carve-outs): "No confidentiality carve-outs are stated in this Agreement."
Clause (governing law): "This Agreement shall be governed by and construed in accordance with the laws of the Republic of Ireland, and the parties submit to the exclusive jurisdiction of its courts."