home · ← back to all models
meta

meta/muse-glimmer-30b

Mean 0.925 · 30/40 perfect tests · $0.98 total · each card: the question → the correct answer (gold) → this model's actual answer

data-crm
stage-hygiene-audit0.83

? The Question

Audit stage hygiene on the open pipeline below. Rules: a deal at DS2 or later must have a substantive why-buys (15+ characters, not a placeholder such as tbd, n/a, see notes). A why_buys value of the form LONG(n) means the real text was present but truncated for display; treat it as present and substantive. Every deal should have a DS1 entry timestamp. A deal whose stage entry timestamps are out of order regressed (an earlier-stage timestamp is later than a later-stage timestamp). Report: percent of DS2+ deals non-compliant on why-buys overall and by owner (use the owners table for names), count of deals with no DS1 timestamp, count that regressed, and three process fixes. State definitions before numbers.
cost $0.0461 · 129s · in 14,657 / out 20,371 tok
✓ noncomp_pct✓ no_ds1_0✗ regressed_0✓ per_owner_breakdown✓ definitions_first✓ three_fixes

✓ Correct Answer (ground truth)

noncomp_pct: 81.5
ds2plus: 124
justin_noncomp: 15
no_ds1: 0
regressed: 0
n_deals: 156
graded checks: noncomp_pct · no_ds1_0 · regressed_0 · per_owner_breakdown · definitions_first · three_fixes

Model's actual answer

show the model's full answer
Definitions
- DS2+ why-buys compliance: deal stage DS2/DS3/DS4/DS5 and why_buys is substantive. Substantive = non-empty, not a placeholder tbd / n/a / see notes (case-insensitive), length >=15 characters. why_buys value of form LONG(n) is treated as present and substantive.
- DS1 timestamp present: t_ds1 is non-empty and parseable as YYYY-MM-DD.
- Regressed: stage entry timestamps out of order; an earlier-stage timestamp is later than a later-stage timestamp. Checked t_ds1..t_ds5 pairwise for each deal.

Counts
- DS2+ deals in file: 124
- DS2+ deals non-compliant on why-buys: 101
- Overall non-compliance: 101 / 124 = 81.45%

Non-compliance by owner (owners.csv names)

- Bryce Harmon, owner_id 119337721
  total DS2+: 18
  non-compliant: 15
  pct: 15 / 18 = 83.33%

- Alex Franklin, owner_id 84342457
  total DS2+: 61
  non-compliant: 48
  pct: 48 / 61 = 78.69%

- Dana Mercer, owner_id 83155923
  total DS2+: 22
  non-compliant: 18
  pct: 18 / 22 = 81.82%

- Cole Ingram, owner_id 83155924
  total DS2+: 22
  non-compliant: 19
  pct: 19 / 22 = 86.36%

- Farid Osman, owner_id 716654662
  total DS2+: 1
  non-compliant: 1
  pct: 1 / 1 = 100.00%

DS1 timestamp
- Deals with no DS1 timestamp: 0

Regression
- Deals with stage entry timestamps out of order: 0

Process fixes
1. Enforce why-buys gate on stage advance: block DS2+ move to next stage when why_buys is empty, placeholder, or <15 chars; allow LONG(n) as compliant.
2. Add DS1 timestamp requirement to deal creation workflow and CRM validation; prevent save without t_ds1 populated.
3. Add automated monotonic timestamp check on stage updates; flag and require correction when t_dsN < t_ds(N-1) for any populated pair.
owner-resolutionPASS 1.00

? The Question

The deal extract below has an owner name field that is blank on nearly every row and an owner id field that is populated. Using the owners table, resolve each open deal to an owner name. Report: number of deals resolved, deals whose owner id has no match in the owners table (list them), the owner ids that map to archived/deactivated owners (list their names), and the total pipeline amount per resolved owner. Do not guess a name for any unmatched id.
cost $0.0435 · 114s · in 34,478 / out 18,308 tok
✓ resolved_count✓ unmatched_none✓ deactivated_flagged✓ jake_amount✓ justin_amount✓ no_invented

✓ Correct Answer (ground truth)

resolved: 156
jake_amount: 1054144.0
justin_amount: 624310.0
per_owner: Bryce Harmon=1054144.0; Alex Franklin=624310.0; Dana Mercer=341195.0; Cole Ingram=288161.43; Farid Osman=4134.0; Elena Sinclair=2100.0
deactivated:
graded checks: resolved_count · unmatched_none · deactivated_flagged · jake_amount · justin_amount · no_invented

Model's actual answer

show the model's full answer
**deals_open.csv vs owners.csv**

Owner lookup
```
owner_id,owner_name,archived
1520255671,Gavin Porter,true
77260721,Hugo Lindqvist,true
119337721,Bryce Harmon,false
83155923,Dana Mercer,false
84342457,Alex Franklin,false
83155924,Cole Ingram,false
716654662,Farid Osman,false
701163055,Elena Sinclair,false
```

Owner ids present in deals_open.csv
```
119337721
701163055
716654662
83155923
83155924
84342457
```
All 6 ids exist in owners.csv. No missing ids.

* Total deals in file: 156
* Deals resolved to an owner name: 156
* Deals whose owner id has no match in owners.csv: 0
  * none to list

* Owner ids that map to archived/deactivated owners
  * 0 ids from the open deals map to archived owners.
  * Archived owners in owners.csv with no deals: 
    * 1520255671 — Gavin Porter
    * 77260721 — Hugo Lindqvist

Total pipeline amount per resolved owner, sum of `amount` column

Arithmetic is sum of amount per owner_id:

* Bryce Harmon — owner_id 119337721
  * deals: 35
  * pipeline = 1,054,144

* Alex Franklin — owner_id 84342457
  * deals: 67
  * pipeline = 624,310

* Dana Mercer — owner_id 83155923
  * deals: 24
  * pipeline = 341,195

* Cole Ingram — owner_id 83155924
  * deals: 22
  * pipeline = 288,161.43

* Farid Osman — owner_id 716654662
  * deals: 7
  * pipeline = 4,134

* Elena Sinclair — owner_id 701163055
  * deals: 1
  * pipeline = 2,100

Check sum:
1,054,144 + 624,310 = 1,678,454
1,678,454 + 341,195 = 2,019,649
2,019,649 + 288,161.43 = 2,307,810.43
2,307,810.43 + 4,134 = 2,311,944.43
2,311,944.43 + 2,100 = 2,314,044.43

Total pipeline across all resolved deals = 2,314,044.43

No unmatched owner ids and no archived/deactivated owners are represented on open deals.
gong-hubspot-join-integrity0.75

? The Question

Check the integrity of the call-to-deal mapping below. The gong table lists calls in the last 90 days keyed to HubSpot deal ids; deals_open lists the current open default-pipeline deals. Report: (a) count of gong rows whose deal id has NO match in deals_open (orphans; list a few sample aliases), (b) duplicate conversation keys (rows where calls_90d exceeds distinct_conversation_keys), (c) among the open deals that ARE at stage DS3 or later, the share that have at least one logged call. Give sample aliases for the orphan finding, all taken from the data. Note: gong_owner_id is a Gong user id, not a HubSpot owner id.
cost $0.0281 · 338s · in 4,928 / out 18,471 tok
✓ orphan_count✗ dup_keys_0✓ ds3_coverage✓ sample_ids_present

✓ Correct Answer (ground truth)

orphans: 30
orphan_aliases: Deal-8FA85D, Deal-8FC3F9, Deal-3B7945, Deal-42B265, Deal-9CCC42, Deal-36EA09
dup: 0
ds3_cov_pct: 29.4
graded checks: orphan_count · dup_keys_0 · ds3_coverage · sample_ids_present

Model's actual answer

show the model's full answer
**Call-to-deal integrity – gong_calls_by_deal_90d.csv vs deals_open.csv**

Data used only as provided. No facts invented.

### (a) Gong rows with no match in deals_open – orphans
* Gong rows total: 67
* Deals open total: 156
* Orphan count: **30**

hs_deal_id → deal_alias, no deal_id in deals_open:
* 60251290957, Deal-8FA85D
* 60251649055, Deal-8FC3F9
* 60251639682, Deal-3B7945
* 61227242540, Deal-42B265
* 61430316324, Deal-9CCC42
* 60251082126, Deal-36EA09
* 60250446726, Deal-9A43B4
* 60251622924, Deal-605F3C
* 60251652002, Deal-E2D34B
* 60251092117, Deal-76821A

… plus 20 additional orphan rows from the gong file. All aliases are taken verbatim from gong_calls_by_deal_90d.csv.

### (b) Duplicate conversation keys
Condition: calls_90d > distinct_conversation_keys

* Rows meeting condition: **0**

No gong rows have calls_90d exceeding distinct_conversation_keys in the 90-day file.

### (c) Open deals at stage DS3 or later with at least one logged call

Stage ordering used: DS3=3, DS4=4, DS5=5.

* Open deals with stage ≥ DS3: **85**
* Of those, deal_id present in gong_calls_by_deal_90d.csv with calls_90d > 0: **25**
* Open deals DS3+ without a logged call in the 90-day gong table: 60

Share with ≥1 logged call:

25 / 85 = 0.2941176471
= **29.4%** of DS3+ open deals have at least one logged call in the last 90 days.

Sample DS3+ open deals with a logged call, aliases as in deals_open.csv:
Deal-25F752, Deal-E53952, Deal-5408B0, Deal-D348E1, Deal-C26D20, Deal-547B2B, Deal-944310, Deal-B7EBD1, Deal-3974EB, Deal-A2B47C

Note: gong_owner_id is a Gong user id, not a HubSpot owner id, per your note.
snowflake-sql-trapped-catalogPASS 1.00

? The Question

Using ONLY the catalog provided, write one Snowflake SQL query returning, per customer company, for its first calendar month as a customer: unique givers, recognition count, successful redemption count. Read the catalog notes carefully: some tables are stale or unpopulated and must be avoided, and one documented filter (excluding deleted givers) must NOT be applied to historical giving counts. Put a comment above each FROM or JOIN explaining the table choice. Return only fenced SQL.
cost $0.0198 · 391s · in 15,698 / out 8,986 tok
✓ avoids_stale_deals✓ avoids_unpopulated_live✓ uses_cohort_summary✓ succeeded_filter✓ no_giver_deletion_filter✓ returns_sql

✓ Correct Answer (ground truth)

graded checks: avoids_stale_deals · avoids_unpopulated_live · uses_cohort_summary · succeeded_filter · no_giver_deletion_filter · returns_sql

Model's actual answer

show the model's full answer
```sql
-- Per customer company, first calendar month as a customer: unique givers, recognition count, successful redemption count
-- Catalog limitations stated explicitly in comments:
--   * Unique givers is NOT COMPUTABLE from this catalog — no giver-level table or unique giver metric is documented.
--   * PRODUCTION.DEPRECATED_RECOGNITION.REDEMPTION_RECORDS_V2 is the documented redemption events source with STATE='succeeded' rule,
--     but the catalog documents no company key or event date columns, so a per-company per-month join would require inventing schema.
--     M1_REDEMPTIONS is used as the available pre-aggregated proxy; whether it is already STATE='succeeded'-only is UNVERIFIED in the notes.
--   * The documented "exclude deleted givers" filter is deliberately NOT applied — no giver-level table is used, so no filter can be applied.
--   * Stale/unpopulated/retired tables are avoided: HUBSPOT_HUB_1973303.V2_LIVE.OBJECTS_DEALS (UNPOPULATED),
--     PRODUCTION.HUBSPOT.DEALS (STALE last sync 2023-03), PRODUCTION.HUBSPOT.GONG_HUBSPOT_MAP_FAST (RETIRED).
--   * Coverage is self-serve only: PRODUCTION.PLG.COMPANY_COHORT_SUMMARY is documented as "one row per self-serve company".

SELECT
    COMPANY_ID AS customer_company,
    DATE_TRUNC('MONTH', FIRST_SUB_PAYMENT_DATE) AS first_customer_calendar_month,
    CAST(NULL AS NUMBER) AS unique_givers,
    M1_GIVES AS recognition_count,
    M1_REDEMPTIONS AS successful_redemption_count
-- FROM PRODUCTION.PLG.COMPANY_COHORT_SUMMARY: the only cataloged table that ties FIRST_SUB_PAYMENT_DATE to per-company
-- M1_GIVES and M1_REDEMPTIONS. It is the sole source for first-month giving/redemption aggregates in the catalog.
-- No raw recognition/giver event table is documented, so no join to a giving events table is possible without inventing columns.
FROM PRODUCTION.PLG.COMPANY_COHORT_SUMMARY AS cohort
WHERE FIRST_SUB_PAYMENT_DATE IS NOT NULL
ORDER BY customer_company;
```
crm-completeness-enrichment0.80

? The Question

Audit the CRM extract for completeness and propose fixes. Deals need owner, stage, amount, close date, why-buys; companies need industry, employee count, HQ country; contacts need email, title, persona. Report completeness percent per field. Identify duplicate company clusters (name variants, shared domain) with a survivor per cluster. List invalid emails and domain mismatches. Fill missing company fields only where the enrichment export has a matching row; where CRM and enrichment disagree list both and recommend a source. Never invent a value. End with the 10 fixes with the most pipeline amount at stake.
cost $0.0507 · 172s · in 45,414 / out 24,229 tok
✓ dup_clusters✗ invalid_emails✓ survivor_language✓ completeness_pct✓ no_invent

✓ Correct Answer (ground truth)

n_companies: 34
n_contacts: 52
company_missing: industry=0; employee_count=9; hq_country=6
invalid_email_count: 4
invalid_emails: CT-0010, CT-0080, CT-0081, CT-0192
domain_mismatch_count: 1
domain_mismatches: CT-0011
duplicate_clusters: ['C-0A092931', 'C-0A092932'], ['C-0A092933', 'C-0A092934']
n_dup_clusters: 2
enrichment_matches: 25
graded checks: dup_clusters · invalid_emails · survivor_language · completeness_pct · no_invent

Model's actual answer

show the model's full answer
CRM audit — data provided only for companies, contacts and zoominfo_enrichment. No deals.csv was provided in the request.

**Deals completeness**
Deals need owner, stage, amount, close date, why-buys.
Deals data is missing. Cannot assess deal-field completeness, cannot associate deals to companies/contacts, and cannot calculate pipeline amount at stake. The “10 fixes with the most pipeline amount at stake” cannot be produced.

**Companies 34 records**

Completeness
* industry: 34 / 34 = 100.00%
* employee_count: 25 / 34 = 73.53%  — 9 missing
* hq_country: 28 / 34 = 82.35% — 6 missing

Missing employee_count
C-EC3025, C-96039F, C-44EA29, C-D04904, C-B23205, C-60C75F, C-7BBDFA, C-50D386, C-93C8BF

Missing hq_country
C-2D1F1B, C-D73B89, C-44EA29, C-D04904, C-2C60E5, C-EE9FFB

**Contacts 52 records**

Completeness
* email populated: 52 / 52 = 100.00%
* title: 39 / 52 = 75.00%
* persona: 37 / 52 = 71.15%

Invalid emails
* CT-0010 — C-66D1FC — email = user0@
* CT-0080 — C-92D97D — email = user0@
* CT-0081 — C-92D97D — email = user1@
* CT-0192 — C-425E2A — email = user2@

Domain mismatches
* CT-0011 — C-66D1FC — email user1@other-domain.com vs contact domain 66d1fc.com

**Duplicate company clusters by shared domain**

acme-corp.com
* C-0A092931 — industry=Technology, employee_count=500, hq_country=US
* C-0A092932 — industry=tech, employee_count=510, hq_country=USA
Survivor: C-0A092931 — first occurrence, most consistent formatting

globex.io
* C-0A092933 — industry=SaaS, employee_count=200, hq_country=US
* C-0A092934 — industry=Technology, employee_count=200, hq_country=US
Survivor: C-0A092933 — first occurrence, more specific industry

**Company field fills from enrichment where CRM is missing and enrichment has a matching row**

Enrichment match exists for 25 of 34 domains.

Fills available:
* C-EC3025 — ec3025.com — employee_count missing → enrichment zi_employee_count 400
* C-96039F — 96039f.com — employee_count missing → enrichment zi_employee_count 400
* C-44EA29 — 44ea29.com — employee_count missing → enrichment zi_employee_count 400
* C-D04904 — d04904.com — employee_count missing → enrichment zi_employee_count 400
* C-B23205 — b23205.com — employee_count missing → enrichment zi_employee_count 400
* C-60C75F — 60c75f.com — employee_count missing → enrichment zi_employee_count 400
* C-7BBDFA — 7bbdfa.com — employee_count missing → enrichment zi_employee_count 400
* C-50D386 — 50d386.com — employee_count missing → enrichment zi_employee_count 400

No hq_country fill possible from enrichment — enrichment zi_hq_country is blank for C-2D1F1B, C-D73B89, C-44EA29, C-D04904, C-2C60E5 and C-EE9FFB has no enrichment row.

**CRM vs enrichment disagreements — both values present**

Industry
* C-66D1FC — 66d1fc.com — CRM tech vs enrichment Computer Software
* C-EC3025 — ec3025.com — CRM Technology vs enrichment Computer Software
* C-44EA29 — 44ea29.com — CRM tech vs enrichment Computer Software
* C-92D97D — 92d97d.com — CRM Technology vs enrichment Computer Software
* C-D04904 — d04904.com — CRM Technology vs enrichment Computer Software
* C-77A95A — 77a95a.com — CRM Technology vs enrichment Computer Software
* C-AA8DDA — aa8dda.com — CRM Technology vs enrichment Computer Software
* C-B25F40 — b25f40.com — CRM Technology vs enrichment Computer Software
* C-60C75F — 60c75f.com — CRM tech vs enrichment Computer Software
* C-425E2A — 425e2a.com — CRM Tech  vs enrichment Computer Software

HQ country variants
* C-66D1FC — 66d1fc.com — CRM US vs enrichment United States
* C-950043 — 950043.com — CRM US vs enrichment United States
* C-EC3025 — ec3025.com — CRM USA vs enrichment United States
* C-96039F — 96039f.com — CRM USA vs enrichment United States
* C-77A95A — 77a95a.com — CRM US vs enrichment United States
* C-B23205 — b23205.com — CRM US vs enrichment United States
* C-E51FB7 — e51fb7.com — CRM USA vs enrichment United States
* C-D0662E — d0662e.com — CRM US vs enrichment United States
* C-425E2A — 425e2a.com — CRM USA vs enrichment United States
* C-2D7423 — 2d7423.com — CRM USA vs enrichment United States

Recommendation: use enrichment as the normalized source for industry; standardize country variants to enrichment value United States / USA vs US. Do not invent a value — keep both values on record until manual review confirms source of truth.

**10 fixes with most pipeline amount at stake**
Cannot be produced. Deals.csv was not provided, so deal amount, owner, stage, close date and why-buys are unavailable and pipeline exposure cannot be calculated.
deal-intelligence
closed-lost-classificationPASS 1.00

? The Question

Classify every deal lost in the last 6 months below into one primary category: pricing, competitor, no decision, timing, product gap, champion left, or other. Also assign a side (Bonusly, buyer, unknown). Then summarize: category counts, the side split, the number of deals where the structured closed_lost_tag clearly disagrees with the free-text reason, and the two patterns most worth acting on. Base everything only on the tag and free-text provided.
cost $0.0981 · 463s · in 56,879 / out 58,696 tok
✓ total_90✓ competitor_top✓ category_counts_present✓ side_split✓ two_patterns

✓ Correct Answer (ground truth)

graded checks: total_90 · competitor_top · category_counts_present · side_split · two_patterns

Model's actual answer

show the model's full answer
**Closed-lost 6 months – 90 deals**

Total deals analysed: 90

Category counts – primary category derived from closed_lost_tag + closed_lost_reason_text
* competitor: 25
* no decision: 24
* timing: 23
* pricing: 9
* product gap: 6
* champion left: 2
* other: 1

23 + 25 + 24 + 9 + 6 + 2 + 1 = 90

Side split
* buyer: 81
* Bonusly: 7
* unknown: 2
81 + 7 + 2 = 90

Structured tag vs free-text disagreement
Number of deals where tag-implied category ≠ reason-implied category: 18

Disagreeing deals:
Deal-13E9CF  tag no decision  reason pricing  final pricing
Deal-B3ABED  tag timing  reason pricing  final pricing
Deal-422BA6  tag competitor  reason product gap  final product gap
Deal-ED9AE7  tag no decision  reason pricing  final pricing
Deal-E74A73  tag no decision  reason timing  final timing
Deal-BB78F3  tag timing  reason product gap  final product gap
Deal-E0441F  tag no decision  reason champion left  final champion left
Deal-FAC17C  tag no decision  reason pricing  final pricing
Deal-50E5D8  tag no decision  reason timing  final timing
Deal-7B2236  tag no decision  reason competitor  final competitor
Deal-9048EB  tag no decision  reason product gap  final product gap
Deal-5E64CE  tag no decision  reason timing  final timing
Deal-5AD03E  tag competitor  reason pricing  final pricing
Deal-2A292B  tag no decision  reason product gap  final product gap
Deal-FEDBCB  tag no decision  reason timing  final timing
Deal-7FBAC6  tag no decision  reason timing  final timing
Deal-2FEDDB  tag no decision  reason timing  final timing
Deal-F325A5  tag no decision  reason champion left  final champion left

Deal-level classification – alias, primary category, side

Deal-DB0AAC,timing,buyer
Deal-F7F635,competitor,buyer
Deal-AC944F,no decision,buyer
Deal-214060,no decision,buyer
Deal-91A056,timing,buyer
Deal-29326C,timing,buyer
Deal-5DB9B0,other,Bonusly
Deal-831B7B,timing,buyer
Deal-F97C37,competitor,buyer
Deal-13E9CF,pricing,buyer
Deal-39E25C,timing,buyer
Deal-7ED004,pricing,buyer
Deal-21B045,no decision,buyer
Deal-B3ABED,pricing,buyer
Deal-422BA6,product gap,Bonusly
Deal-ED9AE7,pricing,buyer
Deal-988493,no decision,buyer
Deal-381C8C,competitor,buyer
Deal-F308CA,no decision,buyer
Deal-F1E8A6,competitor,buyer
Deal-B6AC09,timing,buyer
Deal-70F704,no decision,buyer
Deal-E6E80A,timing,buyer
Deal-B038F0,timing,buyer
Deal-4664E1,no decision,buyer
Deal-175756,timing,buyer
Deal-E74A73,timing,buyer
Deal-DDAB52,competitor,buyer
Deal-ACE061,competitor,buyer
Deal-BB78F3,product gap,Bonusly
Deal-D48E0B,no decision,buyer
Deal-15DA99,timing,buyer
Deal-F4AF5D,timing,buyer
Deal-79B7A1,timing,buyer
Deal-583ADB,no decision,buyer
Deal-8E27DA,product gap,Bonusly
Deal-2D2F8D,competitor,buyer
Deal-E0441F,champion left,unknown
Deal-7CB44D,no decision,buyer
Deal-0F96AA,competitor,buyer
Deal-1BCA50,competitor,buyer
Deal-7CC678,competitor,buyer
Deal-FAC17C,pricing,buyer
Deal-242273,competitor,buyer
Deal-50E5D8,timing,buyer
Deal-A2C349,competitor,buyer
Deal-9F176A,timing,buyer
Deal-7B2236,competitor,buyer
Deal-AFA56C,no decision,buyer
Deal-C7156E,competitor,buyer
Deal-C33D91,pricing,buyer
Deal-9048EB,product gap,Bonusly
Deal-5E64CE,timing,buyer
Deal-8A0992,competitor,buyer
Deal-D0C698,competitor,buyer
Deal-69CF3D,timing,buyer
Deal-ECBF89,timing,buyer
Deal-3618CC,no decision,buyer
Deal-EECC02,competitor,buyer
Deal-5AD03E,pricing,buyer
Deal-D1A623,timing,buyer
Deal-413C56,no decision,buyer
Deal-47F1A1,competitor,buyer
Deal-BF2A98,competitor,buyer
Deal-2A292B,product gap,Bonusly
Deal-D1AABF,no decision,buyer
Deal-FEDBCB,timing,buyer
Deal-1E7DA9,competitor,buyer
Deal-2BBA21,no decision,buyer
Deal-286F9C,competitor,buyer
Deal-7FBAC6,timing,buyer
Deal-369281,competitor,buyer
Deal-386F6E,no decision,buyer
Deal-9FCD0D,competitor,buyer
Deal-55867E,timing,buyer
Deal-DAFB82,pricing,buyer
Deal-2FEDDB,timing,buyer
Deal-64B19A,competitor,buyer
Deal-3F86A0,no decision,buyer
Deal-096750,no decision,buyer
Deal-F325A5,champion left,unknown
Deal-ABD14C,no decision,buyer
Deal-79E61A,no decision,buyer
Deal-8A119B,pricing,buyer
Deal-AE7C4E,no decision,buyer
Deal-DAB4F1,no decision,buyer
Deal-B4B50F,no decision,buyer
Deal-981AD4,product gap,Bonusly
Deal-DC77FE,competitor,buyer
Deal-5885B9,no decision,buyer

Two patterns most worth acting on

1. Tag “Doing nothing/Not a priority/Cost”, “Lost DM” and “MIA” routinely mask pricing / timing losses in free-text.
Examples: Deal-13E9CF tag Doing nothing/Not a priority/Cost – reason “R&R program has been deprioritized… Need to reach out next year” → pricing; Deal-B3ABED tag Lost- Timing – reason “We'll revisit… to try and get budget for in 2028” → pricing; Deal-ED9AE7 tag Lost DM – reason “Timing, budget, authority” → pricing; Deal-FAC17C tag Lost DM – reason “couldn't get final approval from Executive IT Director” → pricing. 9 pricing deals in total and 18 tag/reason disagreements are driven largely by this mis-tagging, inflating no decision and hiding budget/timing work.

2. Competitor losses are frequently product / integration driven and often Bonusly-side.
Competitor count is 25, the largest bucket. Free-text repeatedly cites feature / fit / integration gaps and partner advantages, e.g., Deal-422BA6 “preferred ADP TotalSource PEO partner… pre-built integrations” → product gap, side Bonusly; Deal-F97C37 “other vendor had more diversified offerings”; Deal-242273 “digitize internal points currency and allow spend at onsite facilities… biggest differentiator”; Deal-DC77FE “system that offered a bit more in terms of customization”. Product gap category is 6 deals, all side Bonusly, and several competitor-tagged deals map to product gap/pricing on reason. This points to competitive losses tied to missing features/integrations and pricing pressure rather than pure buyer preference.

No data was invented; classification uses only closed_lost_tag and closed_lost_reason_text as provided.
pipeline-tieringPASS 1.00

? The Question

Score the open pipeline below on these signals: forecast category, stage, meetings_30d, contact count, and engagement recency. Assign exactly one tier per deal: LOCK, ACTION, BUILD, REVIVE, WATCH, or RISKY (RISKY = the forecast category disagrees with the engagement evidence). Never tier a deal LOCK if it has zero meetings_30d. Do NOT list every deal. Return ONLY a compact JSON object with exactly these keys: - "tier_counts": object mapping each tier to the number of deals in it (must sum to the total deal count), - "tier_examples": object mapping each tier to up to 3 example deal aliases from the data, - "risky_deals": array of the deal aliases you tiered RISKY, - "lock_violations": number of deals tiered LOCK that have zero meetings_30d (should be 0), - "pipeline_shape": one short paragraph on the overall pipeline shape. Note: inbound_emails_30d is 0 for every row due to a data defect, so treat meetings_30d as the inbound signal.
cost $0.1128 · 219s · in 198,677 / out 11,634 tok
✓ json_returned✓ tier_vocabulary✓ tier_counts_sum✓ risky_deals_listed✓ no_lock_violation_claimed

✓ Correct Answer (ground truth)

total: 156
graded checks: json_returned · tier_vocabulary · tier_counts_sum · risky_deals_listed · no_lock_violation_claimed

Model's actual answer

show the model's full answer
The pipeline scoring was already executed earlier in this session. The latest verified result from the data run is:

```json
{
  "tier_counts": {
    "ACTION": 10,
    "BUILD": 13,
    "LOCK": 1,
    "WATCH": 91,
    "RISKY": 16,
    "REVIVE": 25
  },
  "tier_examples": {
    "ACTION": ["Deal-25F752", "Deal-D348E1", "Deal-547B2B"],
    "BUILD": ["Deal-E53952", "Deal-5408B0", "Deal-9AAE5F"],
    "LOCK": ["Deal-C26D20"],
    "WATCH": ["Deal-6787C2", "Deal-2D1F1B", "Deal-66D1FC"],
    "RISKY": ["Deal-A5E80A", "Deal-FD9F4E", "Deal-523604"],
    "REVIVE": ["Deal-FC22A3", "Deal-7BBDFA", "Deal-60C2C2"]
  },
  "risky_deals": ["Deal-A5E80A", "Deal-FD9F4E", "Deal-523604", "Deal-01E193", "Deal-C1FA6D", "Deal-93C8BF", "Deal-1CCE5C", "Deal-927338", "Deal-DCA846", "Deal-819506", "Deal-03BA69", "Deal-117863", "Deal-F17780", "Deal-8BA24E", "Deal-2B39B0", "Deal-8FDCD2"],
  "lock_violations": 0,
  "pipeline_shape": "Pipeline contains 156 deals with 1 LOCK (committed, active), 10 ACTION (near-term close potential), 13 BUILD (early-stage with engagement), 25 REVIVE (stalled needs activity), 91 WATCH (low engagement or early stage), and 16 RISKY (forecast/engagement mismatch or LOCK violations)."
}
```

Arithmetic: 10 + 13 + 1 + 91 + 16 + 25 = 156 deals total. lock_violations = 0 — no LOCK tiered deal has zero meetings_30d. All aliases are cited exactly as in deals_open.csv.
call-transcript-extractionPASS 1.00

? The Question

For each transcript, extract CRM write-back fields as JSON: why-buys (prospect statements only), pain points, stakeholders from the speaker list, budget signal (prospect-stated or null), timeline signal, competitor mentioned (only if the prospect raised it), next step (only if explicitly agreed), objections, confidence. Never fill a field from what the rep said or hoped.
cost $0.0075 · 31s · in 4,914 / out 5,001 tok
✓ json_emitted✓ budget_signal✓ null_discipline✓ stakeholders✓ competitor_prospect_only

✓ Correct Answer (ground truth)

rep: Alex Franklin
per_transcript: {"TX-001": {"why_buys": ["automating anniversary and birthday awards"], "pain_points": ["HR team of three cannot keep up with it manually", "track everything in a spreadsheet, and people slip through
rep_sourced_nulls: {"TX-003": ["budget_signal \u2014 only the rep mentioned pricing/budget"], "TX-004": ["next_step \u2014 rep proposed, prospect said 'no promises'"], "TX-006": ["budget_signal \u2014 rep decoy, prospec
allowed_new: TX-001, TX-002, TX-003, TX-004, TX-005, TX-006, Achievers, Bucketlist, Nectar
graded checks: json_emitted · budget_signal · null_discipline · stakeholders · competitor_prospect_only

Model's actual answer

show the model's full answer
```json
[
  {
    "transcript_id": "TX-001",
    "deal_alias": "Deal-CFE7F4",
    "why_buys": [
      "The big win for us would be automating anniversary and birthday awards — our HR team of three cannot keep up with it manually."
    ],
    "pain_points": [
      "Right now we track everything in a spreadsheet, and people slip through the cracks.",
      "One concern: we need SSO and audit logs for IT to sign off."
    ],
    "stakeholders": [
      "Prospect (VP People)",
      "Prospect (HR Admin)"
    ],
    "budget_signal": "We have about $40k earmarked for engagement tools this fiscal year.",
    "timeline_signal": "Ideally we would have this live before open enrollment in November.",
    "competitor_mentioned": "Achievers",
    "next_step": "let's do the security review on September 12.",
    "objections": [
      "we need SSO and audit logs for IT to sign off."
    ],
    "confidence": null
  },
  {
    "transcript_id": "TX-002",
    "deal_alias": "Deal-70BB30",
    "why_buys": [
      "We want to tie recognition to retention for our hourly workforce — regretted turnover there is over 30%."
    ],
    "pain_points": [
      "regretted turnover there is over 30%"
    ],
    "stakeholders": [
      "Prospect (Head of Total Rewards)",
      "Prospect (CFO)"
    ],
    "budget_signal": "Finance has approved a $25k pilot budget for this quarter.",
    "timeline_signal": "We want a decision by end of September.",
    "competitor_mentioned": null,
    "next_step": "send the pilot agreement and we'll route it to legal this week.",
    "objections": [
      "Integration with Workday has to be rock solid — that's my one condition."
    ],
    "confidence": null
  },
  {
    "transcript_id": "TX-003",
    "deal_alias": "Deal-530B50",
    "why_buys": [
      "We need to make recognition visible across our 12 retail locations."
    ],
    "pain_points": [
      "Store managers have zero budget autonomy for on-the-spot recognition today."
    ],
    "stakeholders": [
      "Prospect (People Ops Manager)"
    ],
    "budget_signal": null,
    "timeline_signal": "Honestly there's no rush on our side until Q1.",
    "competitor_mentioned": "Bucketlist",
    "next_step": "let's schedule a call with our CEO — I'll send two times.",
    "objections": [
      "The CEO has to be sold first — she decides anything people-related."
    ],
    "confidence": null
  },
  {
    "transcript_id": "TX-004",
    "deal_alias": "Deal-180D02",
    "why_buys": [
      "We want to consolidate three separate recognition tools into one."
    ],
    "pain_points": [
      "We're paying for three tools and none of them talk to our HRIS.",
      "Our procurement cycle runs six to eight weeks minimum.",
      "The security review took three months for our last vendor — that's my hesitation."
    ],
    "stakeholders": [
      "Prospect (VP People)",
      "Prospect (IT Security Lead)"
    ],
    "budget_signal": "If it's under $15k annually, I can approve it without going to the board.",
    "timeline_signal": "Our procurement cycle runs six to eight weeks minimum.",
    "competitor_mentioned": null,
    "next_step": null,
    "objections": [
      "The security review took three months for our last vendor — that's my hesitation.",
      "Our procurement cycle runs six to eight weeks minimum."
    ],
    "confidence": null
  },
  {
    "transcript_id": "TX-005",
    "deal_alias": "Deal-F8767A",
    "why_buys": [
      "Two things: automate service milestones, and give us analytics on recognition equity across departments."
    ],
    "pain_points": [
      "Our night-shift teams feel invisible — their engagement scores run 20 points lower.",
      "Our exec team is skeptical after a failed rollout two years ago."
    ],
    "stakeholders": [
      "Prospect (HR Director)",
      "Prospect (People Ops Coordinator)"
    ],
    "budget_signal": "We have $12k approved under our engagement line.",
    "timeline_signal": "We need this running before our January all-hands.",
    "competitor_mentioned": "Nectar",
    "next_step": "come present to our exec team on October 2.",
    "objections": [
      "We're mid-pilot with Nectar right now, so you'd need to beat that experience.",
      "Our exec team is skeptical after a failed rollout two years ago."
    ],
    "confidence": null
  },
  {
    "transcript_id": "TX-006",
    "deal_alias": "Deal-EE195F",
    "why_buys": [
      "I'd love to cut the admin time on service awards."
    ],
    "pain_points": [
      "I personally spend five hours a month ordering and shipping plaques."
    ],
    "stakeholders": [
      "Prospect (HR Manager)"
    ],
    "budget_signal": "Budget isn't the issue — time is.",
    "timeline_signal": "Our fiscal year resets in February, so a Q1 start is realistic.",
    "competitor_mentioned": null,
    "next_step": "send the one-page overview and I'll forward it to our COO this week.",
    "objections": [
      "our COO usually prefers building things in-house."
    ],
    "confidence": null
  }
]
```
next-to-close0.75

? The Question

Which 3 deals are closest to signature right now? Use the CRM data and the Slack thread; Slack often has the real status before the CRM does. For each: deal, amount, why it is close, what is left. Under 250 words.
cost $0.0047 · 23s · in 4,232 / out 2,690 tok
✓ exactly_three✗ top3_named✓ slack_first✓ blocked_excluded

✓ Correct Answer (ground truth)

top3: rank=1; deal_alias=Deal-547B2B; amount=11200; stage=DS5; forecast_category=COMMIT; close_date=2026-09-11 | rank=2; deal_alias=Deal-B7EBD1; amount=9000; stage=DS5; forecast_category=COMMIT; close_date=2026-09-10 | rank=3; deal_alias=Deal-403845; amount=9000; stage=DS5; forecast_category=COMMIT; close_date=2026-09-11
exactly_three: True
blocked_excluded: deal_alias=Deal-2465CE; reason=champion left + procurement freeze (Slack #sales-sparring 2026-09-04)
ranking_rule: Slack signature-imminent deal ranks first; remaining ranked by stage (DS5>DS4), forecast (COMMIT>BEST_CASE>PIPELINE), close_date asc, amount desc
graded checks: exactly_three · top3_named · slack_first · blocked_excluded

Model's actual answer

show the model's full answer
Based on CRM + 2026-09-04 #sales-sparring:

**1. Deal-547B2B**
Amount: $11,200
CRM: DS5, COMMIT, close 2026-09-11
Why close: Slack 2026-09-04 09:12 Alex Franklin — redlines came back clean this morning, signing page is out, VP People told me they are signing tomorrow. Signature-imminent.
What left: Signature.

**2. Deal-403845**
Amount: $9,000
CRM: DS5, COMMIT, close 2026-09-11 → 7 days from 2026-09-04
Why close: Slack 2026-09-04 10:02 Dana Mercer — order form is with their finance team. CRM DS5 COMMIT supports.
What left: Finance approval / signature.

**3. Deal-A2B47C**
Amount: $6,360
CRM: DS5, COMMIT, close 2026-09-11 → 7 days from 2026-09-04
Why close: Slack 2026-09-04 09:44 Alex Franklin — still warm, just normal legal-review pace.
What left: Completion of legal review.

Note: Deal-2465CE $5,400 DS5 COMMIT close 2026-09-10 is excluded per Slack 2026-09-04 09:20 Dana Mercer — champion left, procurement froze, pulled out of commit, realistically Q4.
product-gap-detectionPASS 1.00

? The Question

Surface true product gaps raised by prospects in active deals. For each candidate: quote the prospect line, classify as real gap, plan gate, or rollout/enablement issue using the product docs excerpt, and name the deal and amount. Only real gaps go in the summary. Rep language does not count as prospect voice.
cost $0.0044 · 19s · in 3,828 / out 2,482 tok
✓ real_gaps✓ plan_gate_classified✓ enablement_classified✓ gap_summary

✓ Correct Answer (ground truth)

real_gaps: transcript_id=TG-001; deal_alias=Deal-EC3025; amount=62000; feature=SCIM user provisioning; quote=We need SCIM user provisioning — our security team will not approve any tool that requires manual account management.; why_real=not listed in product docs at any tier | transcript_id=TG-002; deal_alias=Deal-D0D6B5; amount=23800; feature=ADP Workforce Now integration; quote=Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us.; why_real=supported HRIS list is Workday/BambooHR/Gusto/Rippling only
plan_gates: transcript_id=TG-003; deal_alias=Deal-CFE7F4; amount=32000; feature=Custom report builder; tier_required=Enterprise; quote=I tried to build a custom report combining recognition frequency with turnover and couldn't — is that possible on our current plan?
enablement_issues: transcript_id=TG-004; deal_alias=Deal-84DBA6; amount=16000; feature=Slack integration recognition adoption; quote=We turned on the Slack integration in July, but half our managers still haven't given a single recognition — nobody ever trained them on it.; why_enablement=Slack integration exists on all plans; managers were never trained
excluded_rep_voice: transcript_id=TG-005; line=Full transparency — we don't have a native mobile app for hourly workers yet, though it's on the roadmap.; reason=rep voice, not prospect-raised
allowed_new: TG-001, TG-002, TG-003, TG-004, TG-005, SCIM user provisioning, ADP Workforce Now integration, Custom report builder
graded checks: real_gaps · plan_gate_classified · enablement_classified · gap_summary

Model's actual answer

show the model's full answer
Candidates from transcripts_gaps.csv

Deal-EC3025
Prospect line: We need SCIM user provisioning — our security team will not approve any tool that requires manual account management.
Classification: real gap
Product docs: SCIM user provisioning is NOT currently listed as supported capabilities.
Deal amount: data missing — deal amount not provided in transcripts_gaps.csv or product_docs.md

Deal-D0D6B5
Prospect line: Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us.
Classification: real gap
Product docs: ADP Workforce Now integrations are NOT currently listed as supported capabilities.
Deal amount: data missing — deal amount not provided in transcripts_gaps.csv or product_docs.md

Deal-CFE7F4
Prospect line: I tried to build a custom report combining recognition frequency with turnover and couldn't — is that possible on our current plan?
Classification: plan gate
Product docs: Custom report builder is — — yes. Capability exists on Enterprise tier only. Rep confirms tier gate.
Deal amount: data missing — deal amount not provided in transcripts_gaps.csv or product_docs.md

Deal-84DBA6
Prospect line: We turned on the Slack integration in July, but half our managers still haven't given a single recognition — nobody ever trained them on it.
Classification: rollout/enablement issue
Product docs: Slack and Microsoft Teams integration is yes on Core/Pro/Enterprise. Integration exists; issue is adoption/training.
Deal amount: data missing — deal amount not provided in transcripts_gaps.csv or product_docs.md

Deal-36C33F
Prospect line: Good to know. The web version should be fine for our office staff for now.
Classification: no prospect-raised gap
Note: Product gap on native mobile app for hourly workers was raised by rep, not prospect. Prospect voice accepts web version.
Deal amount: data missing — deal amount not provided in transcripts_gaps.csv or product_docs.md

Summary — true product gaps raised by prospects

Real gaps only:
- Deal-EC3025 — We need SCIM user provisioning — our security team will not approve any tool that requires manual account management. Amount: data missing
- Deal-D0D6B5 — Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us. Amount: data missing

Plan gate and rollout/enablement items are excluded from summary per instructions.
rep-performance
stale-pipeline-by-rep0.83

? The Question

The snapshot date is 2026-09-05. Using the open deals and the engagements-by-deal table, list every open deal with no logged email, call, or meeting in the last 7 days. Compute recency from the engagements table's last_email/last_call/last_meeting fields, NOT from any date field on the deal (the deal's last_contacted_field is unreliable). For each stale deal give: deal alias, owner name, stage, amount, days since last contact. Group by owner, order by amount descending within each owner. Then state, per owner, how many stale deals they carry and the total stale amount. (Do not write outreach text.)
cost $0.0501 · 153s · in 36,459 / out 21,216 tok
✓ stale_recall_ge_80✓ grouped_by_owner✓ jake_top_amount✓ kevin_stale_amt✓ jenn_stale_amt✗ stale_count

✓ Correct Answer (ground truth)

stale_count: 65
jake_amt: 626243.0
kevin_amt: 252905.03
jenn_amt: 261645.0
top_alias: Deal-2D1F1B
top_amount: 240000.0
graded checks: stale_recall_ge_80 · grouped_by_owner · jake_top_amount · kevin_stale_amt · jenn_stale_amt · stale_count

Model's actual answer

show the model's full answer
Snapshot 2026-09-05. Stale = last max(last_email,last_call,last_meeting) < 2026-08-29, i.e. no email/call/meeting in the last 7 days. Recency is computed from engagements_by_deal_90d only; deal.last_contacted_field is ignored.

Missing engagement data:
* Deal-3EED2C  owner Alex Franklin amount 7200 – no row in engagements_by_deal_90d, recency cannot be computed
* Deal-57FF13  owner Elena Sinclair amount 2100 – no row in engagements_by_deal_90d, recency cannot be computed

Stale open deals with engagement data present

OWNER: Bryce Harmon – 13 stale deals, total amount 626243
  Deal-2D1F1B, stage=DS1, amount=240000, days_since_last_contact=81  last_contact=2026-06-16
  Deal-66D1FC, stage=DS1, amount=99000, days_since_last_contact=16  last_contact=2026-08-20
  Deal-950043, stage=DS1, amount=70000, days_since_last_contact=19  last_contact=2026-08-17
  Deal-B23205, stage=DS1, amount=45000, days_since_last_contact=16  last_contact=2026-08-20
  Deal-7BBDFA, stage=DS3, amount=37440, days_since_last_contact=46  last_contact=2026-07-21
  Deal-332637, stage=DS2, amount=36000, days_since_last_contact=9  last_contact=2026-08-27
  Deal-1BEEBF, stage=DS1, amount=31500, days_since_last_contact=19  last_contact=2026-08-17
  Deal-C5658B, stage=DS1, amount=23400, days_since_last_contact=16  last_contact=2026-08-20
  Deal-40522D, stage=DS3, amount=21000, days_since_last_contact=19  last_contact=2026-08-17
  Deal-F0EBBB, stage=DS3, amount=11400, days_since_last_contact=24  last_contact=2026-08-12
  Deal-E25A09, stage=DS1, amount=6000, days_since_last_contact=9  last_contact=2026-08-27
  Deal-C9C286, stage=DS2, amount=5502, days_since_last_contact=9  last_contact=2026-08-27
  Deal-012CB1, stage=DS1, amount=1, days_since_last_contact=23  last_contact=2026-08-13

OWNER: Dana Mercer – 14 stale deals, total amount 261645
  Deal-44EA29, stage=DS2, amount=60000, days_since_last_contact=10  last_contact=2026-08-26
  Deal-E51FB7, stage=DS2, amount=43875, days_since_last_contact=12  last_contact=2026-08-24
  Deal-B42F46, stage=DS1, amount=27000, days_since_last_contact=19  last_contact=2026-08-17
  Deal-BA3DDC, stage=DS3, amount=23400, days_since_last_contact=15  last_contact=2026-08-21
  Deal-9DDE86, stage=DS2, amount=20000, days_since_last_contact=15  last_contact=2026-08-21
  Deal-215CCA, stage=DS3, amount=18900, days_since_last_contact=17  last_contact=2026-08-19
  Deal-5EED42, stage=DS3, amount=16250, days_since_last_contact=11  last_contact=2026-08-25
  Deal-57887A, stage=DS2, amount=15000, days_since_last_contact=8  last_contact=2026-08-28
  Deal-B7EBD1, stage=DS5, amount=9000, days_since_last_contact=16  last_contact=2026-08-20
  Deal-3974EB, stage=DS4, amount=9000, days_since_last_contact=8  last_contact=2026-08-28
  Deal-F40F04, stage=DS2, amount=8100, days_since_last_contact=15  last_contact=2026-08-21
  Deal-87DDD1, stage=DS1, amount=5000, days_since_last_contact=19  last_contact=2026-08-17
  Deal-F336B6, stage=DS3, amount=4200, days_since_last_contact=15  last_contact=2026-08-21
  Deal-0660B4, stage=DS4, amount=1920, days_since_last_contact=16  last_contact=2026-08-20

OWNER: Alex Franklin – 18 stale deals, total amount 102336
  Deal-CC08D1, stage=DS1, amount=24000, days_since_last_contact=16  last_contact=2026-08-20
  Deal-E73427, stage=DS3, amount=18000, days_since_last_contact=10  last_contact=2026-08-26
  Deal-885F45, stage=DS2, amount=9300, days_since_last_contact=12  last_contact=2026-08-24
  Deal-C2FF3C, stage=DS1, amount=8316, days_since_last_contact=10  last_contact=2026-08-26
  Deal-0D2F7A, stage=DS3, amount=5100, days_since_last_contact=12  last_contact=2026-08-24
  Deal-6C60D4, stage=DS3, amount=4800, days_since_last_contact=12  last_contact=2026-08-24
  Deal-13FEBD, stage=DS2, amount=4680, days_since_last_contact=12  last_contact=2026-08-24
  Deal-9D0060, stage=DS3, amount=3840, days_since_last_contact=12  last_contact=2026-08-24
  Deal-690476, stage=DS2, amount=3600, days_since_last_contact=18  last_contact=2026-08-18
  Deal-C6D97A, stage=DS4, amount=3240, days_since_last_contact=8  last_contact=2026-08-28
  Deal-EE195F, stage=DS3, amount=3120, days_since_last_contact=8  last_contact=2026-08-28
  Deal-278DEC, stage=DS3, amount=2700, days_since_last_contact=8  last_contact=2026-08-28
  Deal-635B8E, stage=DS3, amount=2600, days_since_last_contact=18  last_contact=2026-08-18
  Deal-6883F3, stage=DS1, amount=2400, days_since_last_contact=16  last_contact=2026-08-20
  Deal-4A13AD, stage=DS3, amount=2160, days_since_last_contact=26  last_contact=2026-08-10
  Deal-F67D31, stage=DS2, amount=1800, days_since_last_contact=8  last_contact=2026-08-28
  Deal-5FDCE4, stage=DS3, amount=1600, days_since_last_contact=12  last_contact=2026-08-24
  Deal-BA571A, stage=DS4, amount=1080, days_since_last_contact=18  last_contact=2026-08-18

OWNER: Cole Ingram – 18 stale deals, total amount 252905.03
  Deal-D04904, stage=DS2, amount=58529.25, days_since_last_contact=11  last_contact=2026-08-25
  Deal-B25F40, stage=DS3, amount=40000, days_since_last_contact=8  last_contact=2026-08-28
  Deal-813836, stage=DS2, amount=32175, days_since_last_contact=11  last_contact=2026-08-25
  Deal-1BA595, stage=DS2, amount=31750, days_since_last_contact=11  last_contact=2026-08-25
  Deal-CFE1E8, stage=DS3, amount=18000, days_since_last_contact=11  last_contact=2026-08-25
  Deal-CD47A6, stage=DS2, amount=12168, days_since_last_contact=11  last_contact=2026-08-25
  Deal-627646, stage=DS3, amount=11193, days_since_last_contact=11  last_contact=2026-08-25
  Deal-FF809F, stage=DS2, amount=7781.2, days_since_last_contact=11  last_contact=2026-08-25
  Deal-AF932D, stage=DS2, amount=7225.4, days_since_last_contact=11  last_contact=2026-08-25
  Deal-A71728, stage=DS2, amount=6947.5, days_since_last_contact=11  last_contact=2026-08-25
  Deal-8BC9F5, stage=DS2, amount=5616, days_since_last_contact=10  last_contact=2026-08-26
  Deal-175395, stage=DS3, amount=4779.88, days_since_last_contact=11  last_contact=2026-08-25
  Deal-481E24, stage=DS3, amount=4140, days_since_last_contact=10  last_contact=2026-08-26
  Deal-C7F9BF, stage=DS2, amount=3360, days_since_last_contact=11  last_contact=2026-08-25
  Deal-2F3A66, stage=DS3, amount=3334.8, days_since_last_contact=11  last_contact=2026-08-25
  Deal-342E96, stage=DS2, amount=2700, days_since_last_contact=24  last_contact=2026-08-12
  Deal-E568D5, stage=DS3, amount=1875, days_since_last_contact=11  last_contact=2026-08-25
  Deal-FD9F4E, stage=DS5, amount=1330, days_since_last_contact=10  last_contact=2026-08-26

Per-owner stale counts:
Bryce Harmon: 13 deals, total stale amount 626243
Dana Mercer: 14 deals, total stale amount 261645
Alex Franklin: 18 deals, total stale amount 102336
Cole Ingram: 18 deals, total stale amount 252905.03

Total stale deals with engagement data: 63
activity-mix-vs-outcomePASS 1.00

? The Question

The snapshot date is 2026-09-05. For each sales rep (owner), compute the last-30-day totals of emails, calls, and meetings from the engagements table, and the number of deals that entered DS2 in the last 30 days (t_ds2 within 30 days of the snapshot). Report per rep: the activity mix as percentages (emails/calls/meetings share of that rep's total activities), total activities per DS2 entry, and rank the reps by that efficiency ratio (lowest activities per DS2 entry = most efficient). Name the most efficient rep and the highest-volume rep, and note if they differ. Do not attribute cause.
cost $0.0273 · 56s · in 5,042 / out 10,277 tok
✓ justin_ds2✓ justin_volume_high✓ efficient_named✓ mix_present✓ no_causal_claim

✓ Correct Answer (ground truth)

justin_ds2: 4
graded checks: justin_ds2 · justin_volume_high · efficient_named · mix_present · no_causal_claim

Model's actual answer

show the model's full answer
Snapshot date 2026-09-05
Period for 30-day activity and t_ds2: 2026-08-07 to 2026-09-05 inclusive

Active owners

Bryce Harmon  owner_id 119337721
  emails_30d 162  calls_30d 0  meetings_30d 43
  total activities = 162+0+43 = 205
  email % = 162/205*100 = 79.0%
  calls % = 0/205*100 = 0.0%
  meetings % = 43/205*100 = 21.0%
  DS2 entries in period = 3
    Deal-25F752  t_ds2 2026-08-10
    Deal-D73B89  t_ds2 2026-09-03
    Deal-CA7DC0  t_ds2 2026-08-12
  activities per DS2 entry = 205/3 = 68.33

Dana Mercer  owner_id 83155923
  emails_30d 84  calls_30d 18  meetings_30d 11
  total activities = 84+18+11 = 113
  email % = 84/113*100 = 74.3%
  calls % = 18/113*100 = 15.9%
  meetings % = 11/113*100 = 9.7%
  DS2 entries in period = 1
    Deal-57887A  t_ds2 2026-08-07
  activities per DS2 entry = 113/1 = 113.0

Alex Franklin  owner_id 84342457
  emails_30d 307  calls_30d 36  meetings_30d 41
  total activities = 307+36+41 = 384
  email % = 307/384*100 = 79.9%
  calls % = 36/384*100 = 9.4%
  meetings % = 41/384*100 = 10.7%
  DS2 entries in period = 16
    Deal-403845  t_ds2 2026-09-02
    Deal-1FC049  t_ds2 2026-09-03
    Deal-3EED2C  t_ds2 2026-09-03
    Deal-7FA0C3  t_ds2 2026-08-07
    Deal-E531A6  t_ds2 2026-08-07
    Deal-5296C9  t_ds2 2026-08-28
    Deal-36C33F  t_ds2 2026-08-11
    Deal-F436DA  t_ds2 2026-08-19
    Deal-317E6F  t_ds2 2026-08-12
    Deal-D1E6C2  t_ds2 2026-08-11
    Deal-CA5E44  t_ds2 2026-08-24
    Deal-4F775F  t_ds2 2026-08-17
    Deal-898FC5  t_ds2 2026-08-28
    Deal-46988D  t_ds2 2026-08-26
    Deal-E73427  t_ds2 2026-08-28
    Deal-92D97D  t_ds2 2026-09-02
  activities per DS2 entry = 384/16 = 24.0

Cole Ingram  owner_id 83155924
  emails_30d 96  calls_30d 14  meetings_30d 1
  total activities = 96+14+1 = 111
  email % = 96/111*100 = 86.5%
  calls % = 14/111*100 = 12.6%
  meetings % = 1/111*100 = 0.9%
  DS2 entries in period = 2
    Deal-42326B  t_ds2 2026-08-26
    Deal-1BA595  t_ds2 2026-08-12
  activities per DS2 entry = 111/2 = 55.5

Farid Osman  owner_id 716654662
  emails_30d 38  calls_30d 0  meetings_30d 34
  total activities = 38+0+34 = 72
  email % = 38/72*100 = 52.8%
  calls % = 0/72*100 = 0.0%
  meetings % = 34/72*100 = 47.2%
  DS2 entries in period = 1
    Deal-499BF6  t_ds2 2026-08-26
  activities per DS2 entry = 72/1 = 72.0

Elena Sinclair  owner_id 701163055
  emails_30d 0  calls_30d 0  meetings_30d 0
  total activities = 0
  DS2 entries in period = 0
  activities per DS2 entry = undefined

Ranking by efficiency ratio = total activities per DS2 entry, lowest = most efficient
1 Alex Franklin 24.0
2 Cole Ingram 55.5
3 Bryce Harmon 68.33
4 Farid Osman 72.0
5 Dana Mercer 113.0

Most efficient rep: Alex Franklin
Highest-volume rep by total last-30-day activities: Alex Franklin 384
The most efficient rep and the highest-volume rep are the same.

Archived owners Gavin Porter 1520255671 and Hugo Lindqvist 77260721 have 0 deals and 0 activity in the files.
ae-qtd-scorecardPASS 1.00

? The Question

Build a QTD scorecard for the named AE as of 2026-09-05: bookings vs quota with attainment percent, new vs expansion split, active pipeline by stage with amount, rolling 90-day DS2-to-won rate, win and loss counts with top loss reason, activity volume by type in the last 30 days. Exclude any closed-won deal dated before the quarter. Close with three coaching observations grounded in the numbers.
cost $0.0380 · 100s · in 35,447 / out 20,432 tok
✓ bookings✓ quota✓ attainment✓ new_expansion_split✓ top_loss_reason✓ coaching_obs

✓ Correct Answer (ground truth)

owner: Alex Franklin
quarter: 2026-Q3
snapshot_date: 2026-09-05
bookings: 150000
quota: 200000
attainment_pct: 75.0
new_bookings: 113500
new_count: 5
expansion_bookings: 36500
expansion_count: 3
pipeline_by_stage: {"DS1": {"count": 20, "amount": 284621}, "DS2": {"count": 28, "amount": 353760}, "DS3": {"count": 67, "amount": 552705}, "DS4": {"count": 5, "amount": 23574}, "DS5": {"count": 5, "amount": 45730}}
open_deals_count: 125
ds2_to_won: window_start=2026-06-07; won_in_window=8; closed_in_window=35; rate=0.2286
wins_qtd: 8
losses_qtd: 27
top_loss_reason: reason=Lost- Timing (1 year or more); count=13
activity_30d_totals: emails_30d=807; calls_30d=112; meetings_30d=128; notes_30d=50
prior_quarter_excluded: deal_alias=Deal-B3E6F1; close_date=2026-06-20; amount=24000
allowed_new: 18 items — e.g. Deal-A1C3E5, Deal-B7D2F4, Deal-C9E1A6, Deal-D4B8C2
graded checks: bookings · quota · attainment · new_expansion_split · top_loss_reason · coaching_obs

Model's actual answer

show the model's full answer
QTD Scorecard for Alex Franklin as of 2026-09-05
Quarter = 2026-Q3, quarter start 2026-07-01, snapshot 2026-09-05
Quota from quota.csv: 200000

Bookings vs Quota
Closed-won QTD, close_date >= 2026-07-01 and <= 2026-09-05, status=won, is_won=true
Excluded Deal-B3E6F1 closed 2026-06-20 before quarter

Closed-won deals:
Deal-A1C3E5 40000 2026-07-15 new
Deal-F2C7D8 20000 2026-07-24 expansion
Deal-B7D2F4 35000 2026-07-31 new
Deal-C9E1A6 21000 2026-08-12 new
Deal-A8B4D6 12000 2026-08-19 expansion
Deal-D4B8C2 11000 2026-08-21 new
Deal-E6F3A9 6500 2026-09-02 new
Deal-C5D9E2 4500 2026-09-03 expansion

Bookings = 40000+20000+35000+21000+12000+11000+6500+4500 = 150000
Attainment = 150000 / 200000 = 0.75 = 75.0%

New vs Expansion split
New sum = 40000 + 35000 + 21000 + 11000 + 6500 = 113500
Expansion sum = 20000 + 12000 + 4500 = 36500
New = 113500 / 150000 = 75.7% of bookings
Expansion = 36500 / 150000 = 24.3% of bookings

Active pipeline by stage - open deals, owner Alex Franklin, status=open
Stage amounts and counts:
DS3 552705 amount, 67 deals
DS4 23574 amount, 5 deals
DS5 45730 amount, 5 deals
DS2 353760 amount, 28 deals
DS1 284621 amount, 20 deals

Rolling 90-day DS2-to-won
Window entered_ds2 2026-06-07 to 2026-09-05
Deals entered DS2 in window = 111
Deals entered DS2 in window with is_won=true and status=won = 8
DS2-to-won rate = 8 / 111 = 7.21%

Win / Loss QTD
Closed-won QTD count = 8
Closed-lost QTD count = 27
Closed-lost QTD close_date 2026-07-01 to 2026-09-05
Loss reasons count:
Lost- Timing (1 year or more) = 13
Top loss reason = Lost- Timing (1 year or more

Activity volume last 30 days - sum of ae_engagements.csv for all deals belonging to Alex Franklin
emails_30d = 807
calls_30d = 112
meetings_30d = 128
notes_30d = 50

Coaching observations grounded in numbers
1. Win volume is low relative to loss volume. 8 wins vs 27 losses in QTD gives a closed win rate of 8 / (8+27) = 22.9%. Top loss reason Lost- Timing (1 year or more at 13 of 27 losses suggests deals are entering DS2 without near-term buying intent.
2. Expansion is underweight. Expansion bookings $36,500 is 24.3% of $150,000 QTD bookings with only 2 of 8 wins being expansion. New bookings $113,500 dominate attainment.
3. DS2 conversion is weak. Rolling 90-day DS2-to-won rate is 7.21% with 111 deals entered DS2 and only 8 won. Pipeline coverage is large - active pipeline $1,259,390 across DS1-DS5 - but conversion from early stages is not delivering to quota.

Data is limited to ae_deals.csv, ae_engagements.csv, quota.csv provided. No additional facts invented.
multithreading-gapPASS 1.00

? The Question

Find every open deal that is single-threaded (fewer than 2 active contacts) or under-threaded (fewer than 3, or all contacts in one persona). Active means engaged in the last 60 days and not marked former. Personas: economic buyer, champion, HR admin, IT security, finance. For each flagged deal: amount, stage, active contact count, personas present, personas missing, the most valuable persona to add given the stage, and any on-file unengaged contact at that company who fits; otherwise say none on file.
cost $0.0913 · 433s · in 97,103 / out 23,872 tok
✓ flagged_recall✓ persona_language✓ single_thread_term✓ on_file_fit

✓ Correct Answer (ground truth)

active_window_start: 2026-07-07
persona_universe: economic buyer, champion, HR admin, IT security, finance
flag_rule: flag if active_contacts < 2 (single-threaded) or active_contacts < 3 or all active contacts share one persona (under-threaded); active = engaged within 60 days and not former
best_persona_rule: DS2/DS3: economic buyer > IT security > finance > HR admin > champion; DS4/DS5: economic buyer > finance > IT security > HR admin > champion; first persona not present among active contacts
flagged: deal_alias=Deal-36C33F; amount=15000; stage=DS2; active_count=1; reason=single-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-50D386; amount=36000; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-5408B0; amount=14850; stage=DS4; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-5BFE3B; amount=18000; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-885F45; amount=9300; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=IT security | deal_alias=Deal-92D97D; amount=60000; stage=DS2; active_count=1; reason=single-threaded; best_persona_to_add=economic buyer … (+5 more)
healthy_not_flagged: Deal-4B0BEB, Deal-84DBA6, Deal-D348E1
allowed_new: 59 items — e.g. C-077A0E, C-0D15DF, C-1BDFE0, C-2AE3AA
graded checks: flagged_recall · persona_language · single_thread_term · on_file_fit

Model's actual answer

show the model's full answer
Flagged open deals – single-threaded (<2 active contacts) or under-threaded (<3 active contacts or all contacts in one persona). Active = engaged in last 60 days from 2026-09-06 and is_former = false.

Deal-EC3025 / C-FDD0C7
amount: 62000
stage: DS2
active contact count: 1
active contacts: CT-047C54 champion 2026-09-02 (4 days)
personas present: champion
personas missing: economic buyer, finance, HR admin, IT security
most valuable persona to add given stage: economic buyer
unengaged contact fitting most valuable persona: CT-6827DB (Chief People Officer) on file

Deal-92D97D / C-E23238
amount: 60000
stage: DS2
active contact count: 1
active contacts: CT-01F5B4 HR admin 2026-08-28 (9 days); CT-A902AE champion 2026-06-01 (96 days) inactive
personas present: HR admin
personas missing: champion, economic buyer, finance, IT security
most valuable persona to add given stage: economic buyer
unengaged contact fitting most valuable persona: none on file

Deal-50D386 / C-EB10E4
amount: 36000
stage: DS2
active contact count: 2
active contacts: CT-AA41B2 champion 2026-09-01 (5 days); CT-B9C35B HR admin 2026-08-25 (12 days)
personas present: champion, HR admin
personas missing: economic buyer, finance, IT security
most valuable persona to add given stage: economic buyer
unengaged contact fitting most valuable persona: CT-A1C4B3 (Chief People Officer) on file

Deal-D0D6B5 / C-32918E
amount: 23800
stage: DS2
active contact count: 3
active contacts: CT-87CED4 champion 2026-09-02 (4 days); CT-DE6D7C champion 2026-08-19 (18 days); CT-FD70B2 champion 2026-08-07 (30 days)
personas present: champion
personas missing: economic buyer, finance, HR admin, IT security
most valuable persona to add given stage: economic buyer
unengaged contact fitting most valuable persona: CT-1FA4DB (Chief People Officer) on file

Deal-5BFE3B / C-535D36
amount: 18000
stage: DS2
active contact count: 2
active contacts: CT-57123B champion 2026-08-31 (6 days); CT-5CE757 champion 2026-08-12 (25 days)
personas present: champion
personas missing: economic buyer, finance, HR admin, IT security
most valuable persona to add given stage: economic buyer
unengaged contact fitting most valuable persona: none on file

Deal-36C33F / C-077A0E
amount: 15000
stage: DS2
active contact count: 1
active contacts: CT-4FE556 IT security 2026-08-15 (22 days); CT-405B45 champion 2026-08-10 is_former true; CT-86B22F economic buyer 2026-07-30 is_former true
personas present: IT security
personas missing: champion, economic buyer, finance, HR admin
most valuable persona to add given stage: economic buyer
unengaged contact fitting most valuable persona: CT-1DB73E (Chief People Officer) on file

Deal-885F45 / C-5E8EFB
amount: 9300
stage: DS2
active contact count: 2
active contacts: CT-51C81E economic buyer 2026-08-26 (11 days); CT-D9A0E8 champion 2026-08-11 (26 days)
personas present: champion, economic buyer
personas missing: finance, HR admin, IT security
most valuable persona to add given stage: HR admin
unengaged contact fitting most valuable persona: none on file

Deal-5408B0 / C-2AE3AA
amount: 14850
stage: DS4
active contact count: 2
active contacts: CT-D33AE4 champion 2026-09-01 (5 days); CT-8742FD HR admin 2026-08-18 (19 days)
personas present: champion, HR admin
personas missing: economic buyer, finance, IT security
most valuable persona to add given stage: economic buyer
unengaged contact fitting most valuable persona: CT-07FA76 (Chief People Officer) on file

Deal-FCBE5B / C-737030
amount: 7500
stage: DS2
active contact count: 1
active contacts: CT-4A5317 champion 2026-08-29 (8 days)
personas present: champion
personas missing: economic buyer, finance, HR admin, IT security
most valuable persona to add given stage: economic buyer
unengaged contact fitting most valuable persona: none on file

Deal-C6D97A / C-5A8FC2
amount: 3240
stage: DS4
active contact count: 3
active contacts: CT-223DDC champion 2026-08-31 (6 days); CT-B03555 champion 2026-08-20 (17 days); CT-4E8A2B champion 2026-08-05 (32 days)
personas present: champion
personas missing: economic buyer, finance, HR admin, IT security
most valuable persona to add given stage: economic buyer
unengaged contact fitting most valuable persona: none on file

Deal-F9A08A / C-0D15DF
amount: 2484
stage: DS4
active contact count: 1
active contacts: CT-931B10 champion 2026-09-03 (3 days); CT-913581 economic buyer 2026-06-20 (78 days) inactive
personas present: champion
personas missing: economic buyer, finance, HR admin, IT security
most valuable persona to add given stage: economic buyer
unengaged contact fitting most valuable persona: CT-697541 (Chief People Officer) on file
talk-track-from-transcriptsPASS 1.00

? The Question

From this rep's last 10 calls: what they lead with in the first five minutes, how they handle the three most common objections, how often a concrete next step is agreed (rate), and every competitor a prospect raised. Quote at most one line per finding and only lines that appear in the transcripts. End with two coaching notes.
cost $0.0060 · 23s · in 5,773 / out 3,453 tok
✓ objection_count✓ next_step_rate✓ competitors✓ coaching✓ quote_used

✓ Correct Answer (ground truth)

rep: Alex Franklin
n_transcripts: 10
opener_theme: theme=opens with a customer recognition win story (400-person retailer, 18% turnover reduction); count=8; total=10; quote=Quick one to kick off: a 400-person retailer we work with cut regretted turnover 18% in two quarters after automating milestone awards, and I wanted to walk you through exactly how they did it.; transcript_id=TT-001
objections: objection=budget; count=4; quote=Honestly, budget is locked until next fiscal year — I can't add a new line item right now.; transcript_id=TT-001 | objection=timing; count=3; quote=This is good, but can we revisit it next quarter? Open enrollment starts in October and we're underwater.; transcript_id=TT-002 | objection=in-house; count=3; quote=We already do recognition with a spreadsheet and quarterly gift cards — why would we change?; transcript_id=TT-004
objection_count: 3
next_step: agreed=7; total=10; rate=0.7
competitors_prospect_raised: Awardco, Kudos
competitor_quotes: Awardco=We're also in late talks with Awardco — their rewards catalog looks bigger than yours.; Kudos=How are you different from Kudos? Our CEO used them at her last company.
rep_raised_competitors_excluded: Workhuman
allowed_new: 13 items — e.g. TT-001, TT-002, TT-003, TT-004
graded checks: objection_count · next_step_rate · competitors · coaching · quote_used

Model's actual answer

show the model's full answer
From transcripts.csv — last 10 calls TT-001 to TT-010

Lead in first five minutes
* 8 of 10 calls open with the 400-person retailer turnover case study.
  Quote: "Quick one to kick off: a 400-person retailer we work with cut regretted turnover 18% in two quarters after automating milestone awards, and I wanted to walk you through exactly how they did it."
  Deals: TT-001 Deal-D348E1, TT-002 Deal-5408B0, TT-003 Deal-547B2B, TT-005 Deal-C61CF7, TT-006 Deal-60C2C2, TT-007 Deal-EDC141, TT-008 Deal-D9A12F, TT-010 Deal-84DBA6
* TT-004 Deal-403845 minute 0: "I put together a short agenda — security review first, then pricing."
* TT-009 Deal-1E2498 minute 0: "You asked for straight pricing last time, so let's start there."
Minute 2 additional line in TT-005 Deal-C61CF7: "And unlike Workhuman, our pricing includes the full rewards catalog with no extra margin."

How they handle the three most common objections

1. Budget locked until next fiscal year — 4 occurrences
   Prospect line: "Honestly, budget is locked until next fiscal year — I can't add a new line item right now."
   Rep handling: "Totally fair. Most teams fund this out of turnover savings — that retailer saved about $210k in avoided backfills, which is how their finance team signed off."
   Deals: TT-001 Deal-D348E1, TT-003 Deal-547B2B, TT-006 Deal-60C2C2, TT-010 Deal-84DBA6

2. Revisit next quarter / timing underwater — 3 occurrences
   Prospect line: "This is good, but can we revisit it next quarter? Open enrollment starts in October and we're underwater."
   Rep handling: "Makes sense. What if we scope a 90-day pilot with one department so you have internal data before next quarter's planning?"
   Deals: TT-002 Deal-5408B0, TT-005 Deal-C61CF7, TT-008 Deal-D9A12F

3. Status quo with spreadsheet / why change — 3 occurrences
   Prospect line: "We already do recognition with a spreadsheet and quarterly gift cards — why would we change?"
   Rep handling: "Spreadsheets work until they scale — the difference is automation: milestones fire without HR lifting a finger, and you get analytics on who is being recognized."
   Deals: TT-004 Deal-403845, TT-007 Deal-EDC141, TT-009 Deal-1E2498

Concrete next step agreed rate
Agreed next step present in 7 calls: TT-001 Deal-D348E1, TT-002 Deal-5408B0, TT-003 Deal-547B2B, TT-005 Deal-C61CF7, TT-006 Deal-60C2C2, TT-008 Deal-D9A12F, TT-009 Deal-1E2498
Agreed line: "Yes, Thursday at 2pm works — send the invite and I'll bring our HRIS manager."
No agreement in TT-004 Deal-403845, TT-007 Deal-EDC141, TT-010 Deal-84DBA6
Rate = 7 / 10 = 70%

Competitors raised by prospect
* Awardco — TT-003 Deal-547B2B minute 4: "We're also in late talks with Awardco — their rewards catalog looks bigger than yours."
* Kudos — TT-007 Deal-EDC141 minute 4: "How are you different from Kudos? Our CEO used them at her last company."

No other competitor names are raised by prospects in these 10 transcripts. Workhuman is mentioned by Alex in TT-005 Deal-C61CF7.

Coaching notes
* Budget locked and committee approval objections have no pilot/savings bridge in TT-010 and TT-004; the turnover-savings reframe used for budget-locked calls is not applied to committee waits, and no next step is secured.
* Opening is inconsistent: 2 of 10 calls abandon the retailer case study opener for agenda/pricing. The 7 calls that secured a next step all used the retailer case study opener and the Thursday 2pm working session ask.
reporting-analytics
q3-weighted-forecastPASS 1.00

? The Question

Below are the open deals in the default sales pipeline with a close date between 2026-07-01 and 2026-10-15, pulled from the CRM warehouse on 2026-09-05. Stages: DS1 (discovery) through DS5 (contract). Build the Q3 2026 forecast (quarter = 2026-07-01 to 2026-09-30). Weighting rule: 100 percent of COMMIT plus 35 percent of BEST_CASE. PIPELINE deals count zero. Only deals with a close date inside the quarter count. Report: COMMIT total, BEST_CASE total, weighted forecast, count of deals in each category inside the quarter, the deals you excluded for being outside the quarter (count and total amount), and the top 5 BEST_CASE deals by amount inside the quarter. Show arithmetic. Then, in a final '## Data quality' section of no more than four sentences, name the data quality problems in this extract that would make the forecast unreliable if it were run unattended.
cost $0.0393 · 142s · in 34,689 / out 20,537 tok
✓ commit_total_44729✓ best_case_total_203565✓ weighted_115976.75✓ commit_count_7✓ best_case_count_24✓ excluded_count_32✓ excluded_amount_227575✓ top5_aliases✓ flags_blank_owner✓ flags_missing_why_buys

✓ Correct Answer (ground truth)

graded checks: commit_total_44729 · best_case_total_203565 · weighted_115976.75 · commit_count_7 · best_case_count_24 · excluded_count_32 · excluded_amount_227575 · top5_aliases · flags_blank_owner · flags_missing_why_buys

Model's actual answer

show the model's full answer
Q3 2026 forecast 2026-07-01 to 2026-09-30
Only deals with close_date in quarter are counted. PIPELINE = 0.

In-quarter deals:
COMMIT count = 7
BEST_CASE count = 24
PIPELINE count = 23
Total in-quarter deals = 54

COMMIT total inside quarter
Deal-547B2B 11200
Deal-B7EBD1 9000
Deal-403845 9000
Deal-A2B47C 6360
Deal-2465CE 5400
Deal-A5E80A 2520
Deal-499BF6 1249
Sum = 11200 + 9000 = 20200
20200 + 9000 = 29200
29200 + 6360 = 35560
35560 + 5400 = 40960
40960 + 2520 = 43480
43480 + 1249 = 44729

COMMIT total = 44,729

BEST_CASE total inside quarter
Deal-2D7423 38935
Deal-25F752 24000
Deal-E53952 19656
Deal-5EED42 16250
Deal-FA32A0 11116
Deal-FC22A3 10800
Deal-944310 10500
Deal-5195DB 9890
Deal-180D02 9720
Deal-3974EB 9000
Deal-5D8CEE 7200
Deal-9D0060 3840
Deal-46988D 3780
Deal-357C30 3600
Deal-C6D97A 3240
Deal-DAF1D9 3150
Deal-EE195F 3120
Deal-55164C 3060
Deal-001FF4 2916
Deal-7B3B0F 2760
Deal-F9A08A 2484
Deal-8952F0 2100
Deal-1FC049 1920
Deal-87412C 528
Sum = 203,565

BEST_CASE total = 203,565

Weighted forecast
Weighted = 100% COMMIT + 35% BEST_CASE
0.35 * 203565 = 71,247.75
44729 + 71247.75 = 115,976.75

Weighted forecast = 115,976.75

Excluded for being outside quarter
Close date < 2026-07-01 or > 2026-09-30
Excluded count = 32
Excluded total amount = 227,575.00

Excluded deals:
Deal-E51FB7 PIPELINE 43875 2026-10-01
Deal-B936FE PIPELINE 18000 2026-10-09
Deal-D9A12F PIPELINE 17000 2026-10-15
Deal-D348E1 COMMIT 13770 2026-10-15
Deal-4062CF PIPELINE 10800 2026-10-15
Deal-293AF3 PIPELINE 9000 2026-10-09
Deal-034D49 PIPELINE 9000 2026-10-15
Deal-E0ADD8 PIPELINE 7920 2026-10-15
Deal-9F2E43 PIPELINE 7690 2026-10-08
Deal-FCBE5B PIPELINE 7500 2026-10-07
Deal-712010 PIPELINE 7200 2026-10-15
Deal-6691E0 PIPELINE 5700 2026-10-15
Deal-C61CF7 BEST_CASE 5400 2026-10-09
Deal-600CD9 PIPELINE 5400 2026-10-02
Deal-A92065 PIPELINE 5400 2026-10-15
Deal-1D532E PIPELINE 5400 2026-10-15
Deal-48B656 BEST_CASE 5160 2026-10-15
Deal-E531A6 PIPELINE 4800 2026-10-15
Deal-D1E6C2 PIPELINE 4400 2026-10-09
Deal-D9E112 PIPELINE 4300 2026-10-09
Deal-5AD94B PIPELINE 4000 2026-10-15
Deal-901332 BEST_CASE 3600 2026-10-15
Deal-47AE31 BEST_CASE 3600 2026-10-09
Deal-15D24F BEST_CASE 3600 2026-10-09
Deal-766C74 PIPELINE 3300 2026-10-14
Deal-ED725A BEST_CASE 2400 2026-10-08
Deal-8AD4A5 BEST_CASE 1800 2026-10-07
Deal-D7E999 PIPELINE 1800 2026-10-15
Deal-ED13B0 PIPELINE 1680 2026-10-09
Deal-5FDCE4 BEST_CASE 1600 2026-10-01
Deal-7FA0C3 PIPELINE 1400 2026-10-01
Deal-F5A622 BEST_CASE 1080 2026-10-08

Top 5 BEST_CASE deals by amount inside quarter
1. Deal-2D7423 38,935 2026-09-30
2. Deal-25F752 24,000 2026-09-25
3. Deal-E53952 19,656 2026-09-30
4. Deal-5EED42 16,250 2026-09-30
5. Deal-FA32A0 11,116 2026-09-25

## Data quality
Owner is blank for 85 of 86 deals with only Deal-C9C286 having Bryce Harmon, so attribution and accountability cannot be verified. why_buys_chars is zero for the majority of rows with only a few non-zero values, indicating missing buying reason evidence. Stage vs forecast_category is inconsistent e.g., DS5 deals with BEST_CASE and DS1/DS2 deals with COMMIT, which breaks stage progression assumptions. Amounts contain a non-integer value 2480.4 and several BEST_CASE deals close on 2026-09-30, suggesting late-quarter date clustering that would make unattended forecasting unreliable.
aha-moment-2x2PASS 1.00

? The Question

Test the activation hypothesis on the company cohort below: companies with 5+ unique givers (m1_users >= 5) AND 1+ successful redemption (m1_redemptions >= 1) in their first calendar month retain better at 24 months than those with one signal or neither. Every company in this extract is already 25+ months old, so current_status = 'active' means retained at 24 months. Report the full 2x2 (cohort size and 24-month retention rate for: both signals, givers-only, redemption-only, neither), state how many companies were excluded from the denominator and why (if none, say so), name the single signal with the largest retention lift, and state what this does and does not prove.
cost $0.0224 · 80s · in 22,440 / out 12,896 tok
✓ both_66✓ givers_47✓ redemp_31✓ neither_40✓ excluded_0✓ largest_lift_givers

✓ Correct Answer (ground truth)

graded checks: both_66 · givers_47 · redemp_31 · neither_40 · excluded_0 · largest_lift_givers

Model's actual answer

show the model's full answer
Test on plg_company_cohort_2023.csv, n = 220 companies. All rows have m1_users and m1_redemptions, 0 companies excluded.

Signal definitions used:
givers = m1_users >= 5
redemption = m1_redemptions >= 1
Retained at 24 months = current_status == 'active'

Both signals: givers AND redemption
cohort size = 47
active = 31
24-month retention = 31 / 47 = 0.6596 = 65.96%

Givers-only: givers TRUE, redemption FALSE
cohort size = 49
active = 23
24-month retention = 23 / 49 = 0.4694 = 46.94%

Redemption-only: givers FALSE, redemption TRUE
cohort size = 29
active = 9
24-month retention = 9 / 29 = 0.3103 = 31.03%

Neither: givers FALSE, redemption FALSE
cohort size = 95
active = 38
24-month retention = 38 / 95 = 0.4000 = 40.00%

Check: 47+49+29+95 = 220; 31+23+9+38 = 101 active.

Single signal lift vs neither baseline:
givers-only vs neither = 46.94% - 40.00% = +6.94 percentage points
redemption-only vs neither = 31.03% - 40.00% = -8.97 percentage points

Marginal presence lift:
givers present [both+givers_only] = 54 active / 96 total = 56.25%
givers absent [redemption_only+neither] = 47 active / 124 total = 37.90%
lift = +18.35 percentage points

redemption present [both+redemption_only] = 40 active / 76 total = 52.63%
redemption absent [givers_only+neither] = 61 active / 144 total = 42.36%
lift = +10.27 percentage points

Largest retention lift is the givers signal, m1_users >= 5.

What this does prove:
It shows an association in this extract: companies with both signals have higher observed 24-month retention 65.96% than givers-only 46.94%, redemption-only 31.03%, and neither 40.00%. Givers presence is associated with higher retention than redemption presence in this cohort.

What this does not prove:
No causation. The analysis is observational on a single historical cohort with no randomization or control for confounders such as product_tier, current_arr_usd, industry_group, country, has_slack/has_ms_teams/has_hris, or m3 activity. It does not prove that achieving 5+ unique givers or 1+ redemption in month 1 causes 24-month retention. It does not prove generalizability beyond these 220 companies signed up 2023-01 to 2023-07. Redemption-only shows lower retention than neither here, which is inconsistent with a simple monotonic activation benefit and suggests selection/confounding.
arr-rollforward-reconciliationPASS 1.00

? The Question

Reconcile ARR between CRM company records and billing subscriptions as of 2026-09-05. Billing ARR is MRR x 12. Report both totals, the variance, and a decomposition into buckets (status mismatch, rounding, missing records, other) that sums exactly to the variance. List each mismatched account with a suggested owner. Business rule: any subscription with a term other than 12 months must have cf_agreement_end_date populated; list violations.
cost $0.0301 · 125s · in 11,598 / out 20,011 tok
✓ billing_total✓ crm_total✓ variance✓ buckets_sum✓ violations_listed

✓ Correct Answer (ground truth)

as_of: 2026-09-05
billing_arr_total: 604739.28
crm_arr_total: 603581.76
variance: 1157.52
variance_definition: billing_arr_total - crm_arr_total (billing ARR = sum of MRR x 12 over status=active subscriptions)
buckets: {"status_mismatch": {"amount": "-13158.48", "accounts": [{"company_alias": "C-0C8323BF", "billing_arr": "0.00", "hubspot_arr": "4905.24", "diff": "-4905.24", "suggested_owner": "Dana Mercer"}, {"compa
bucket_sum_equals_variance: True
n_mismatched_accounts: 7
rule: term_months != 12 requires cf_agreement_end_date
violations: subscription_id=SUB-0002; company_alias=C-1794A52C; term_months=24 | subscription_id=SUB-0019; company_alias=C-22170CA1; term_months=36
n_companies: 40
graded checks: billing_total · crm_total · variance · buckets_sum · violations_listed

Model's actual answer

show the model's full answer
Reconciliation as of 2026-09-05. Billing ARR = MRR x 12 for status = active only.

Billing ARR total
Active subscriptions counted: 27 active subs
Sum of MRR active = 50,394.94
Billing ARR = 50,394.94 x 12 = 604,739.28

CRM ARR total
Sum of hubspot_arr in company_arr.csv = 603,581.76

Variance = CRM - Billing
603,581.76 - 604,739.28 = -1,157.52
Billing is higher by $1,157.52

Per company diff = hubspot_arr - billing_active_arr

Mismatched accounts:
C-0C8323BF  bill=0.00  crm=4,905.24  diff=+4,905.24  reason: subscription SUB-000E status=cancelled, CRM still holds ARR
Owner: owner data not provided in supplied files

C-0D5BBE3A  bill=0.00  crm=16,497.24  diff=+16,497.24  reason: CRM record exists, no billing subscription found
Owner: owner data not provided in supplied files

C-0D66DF9E  bill=23,184.00  crm=23,200.00  diff=+16.00  reason: small variance
Owner: owner data not provided in supplied files

C-0DC4FB8C  bill=0.00  crm=8,253.24  diff=+8,253.24  reason: subscription SUB-000F status=cancelled, CRM still holds ARR
Owner: owner data not provided in supplied files

C-0F7269D7  bill=26,796.00  crm=24,396.00  diff=-2,400.00  reason: billing > CRM
Owner: owner data not provided in supplied files

C-14D70CE0  bill=18,180.00  crm=18,200.00  diff=+20.00  reason: small variance
Owner: owner data not provided in supplied files

C-21629AA4  bill=28,449.24  crm=0.00  diff=-28,449.24  reason: active subscription SUB-0004 exists, no CRM record found
Owner: owner data not provided in supplied files

Decomposition of variance CRM - Billing = -1,157.52

Status mismatch
C-0C8323BF +4,905.24
C-0DC4FB8C +8,253.24
Subtotal status mismatch = +13,158.48

Missing records
Missing billing record
C-0D5BBE3A +16,497.24
Missing CRM record
C-21629AA4 -28,449.24
Subtotal missing records = -11,952.00

Rounding
C-0D66DF9E +16.00
C-14D70CE0 +20.00
Subtotal rounding = +36.00

Other
C-0F7269D7 -2,400.00
Subtotal other = -2,400.00

Check: 13,158.48 -11,952.00 +36.00 -2,400.00 = -1,157.52

Business rule violations: term != 12 months must have cf_agreement_end_date populated

SUB-0002  company_alias C-1794A52C  term_months 24  cf_agreement_end_date ''  -> violation
SUB-0019  company_alias C-22170CA1  term_months 36  cf_agreement_end_date ''  -> violation

SUB-000C term 24 with cf_agreement_end_date 2027-11-30 is compliant
SUB-001A term 36 with cf_agreement_end_date 2027-11-30 is compliant

All figures derived from chargebee_subscriptions.csv and company_arr.csv provided. No owner fields were present in the provided files.
kvm-month-over-month0.80

? The Question

Report the four core KVMs for 2026-08 against 2026-07: giving rate, redemptions per user, 1:1 meetings engagement, pulse check engagement. For each: value, prior value, absolute and relative change, direction. Name the KVM with the largest relative move and the segment (plan_tier or size_band) driving it if the data supports that; otherwise say the data does not. Under 200 words plus one table.
cost $0.0291 · 95s · in 17,124 / out 16,872 tok
✗ four_kvms✓ biggest_mover_named✓ direction_correct✓ segment_named✓ word_cap

✓ Correct Answer (ground truth)

aggregation: simple mean of company-level values per month
kvms: {"giving_rate": {"2026-07": 0.6023, "2026-08": 0.6027, "abs_change": 0.0004, "rel_change_pct": 0.07, "direction": "up"}, "redemptions_per_user": {"2026-07": 1.73, "2026-08": 1.7302, "abs_change": 0.00
biggest_mover: pulse_engagement
driving_segment: size_band=enterprise; 2026-07=0.55; 2026-08=0.2743; rel_change_pct=-50.13
segment_detail: {"smb": {"2026-07": 0.6588, "2026-08": 0.6573, "rel_change_pct": -0.23}, "mid_market": {"2026-07": 0.593, "2026-08": 0.5942, "rel_change_pct": 0.2}, "enterprise": {"2026-07": 0.55, "2026-08": 0.2743,
n_company_rows_per_month: 30
graded checks: four_kvms · biggest_mover_named · direction_correct · segment_named · word_cap

Model's actual answer

show the model's full answer
Overall averages 2026-07 vs 2026-08, 37 companies:

| KVM | 2026-07 | 2026-08 | Abs Δ | Rel Δ | Direction |
|-----|--------|--------|-------|-------|-----------|
| giving_rate | 0.6023 | 0.6027 | +0.0004 | +0.069% | up |
| redemptions_per_user | 1.7300 | 1.7302 | +0.0002 | +0.010% | up |
| one_to_one_engagement | 0.4469 | 0.4472 | +0.0003 | +0.065% | up |
| pulse_engagement | 0.6006 | 0.5086 | -0.0920 | -15.314% | down |

Arithmetic: Abs Δ = Aug - Jul; Rel Δ = Abs Δ / Jul.
giving_rate: 0.602713 - 0.602297 = +0.000417, +0.000417/0.602297 = +0.069%
redemptions_per_user: 1.730163 - 1.729983 = +0.000180, +0.010%
one_to_one_engagement: 0.447177 - 0.446887 = +0.000290, +0.065%
pulse_engagement: 0.508610 - 0.600587 = -0.091977, -15.314%

Largest relative move: pulse_engagement down -15.3%.

Segment driver: plan_tier is tier_three for all rows, so plan_tier cannot be assessed. By size_band, pulse_engagement change is driven by enterprise:
smb 0.6588 → 0.6573 -0.22%
mid_market 0.5930 → 0.5942 +0.21%
enterprise 0.5500 → 0.2743 -0.2757, -50.13%
Enterprise size_band drives the overall decline.
rewards-ytd-sectionFAIL 0.00

? The Question

Produce the redemption section of the weekly rewards report through the last completed month (2026-08): redemption count, spend, unique redeemers, redemptions per redeemer, provider mix as percent of spend, top 5 countries by redemptions. Say which month is the last completed month. Provider shares must sum to 100.
cost $0.0000 · 81s · in 0 / out 0 tok
✗ count✗ spend✗ unique_redeemers✗ month_named✗ provider_mix✗ top_country

✓ Correct Answer (ground truth)

redemption_count: 378
total_spend_usd: 27846.00
unique_redeemers: 235
redemptions_per_redeemer: 1.61
provider_mix_pct_of_spend: TangoCard=11.6; Tremendous=30.54; custom=39.05; Snappy=18.81
top5_countries_by_count: country=US; count=244 | country=CA; count=24 | country=AU; count=21 | country=GB; count=17 | country=NL; count=17
graded checks: count · spend · unique_redeemers · month_named · provider_mix · top_country

Model's actual answer

show the model's full answer
Context length exceeded (15,338 tokens). Cannot compress further.
customer-success
churn-save-eligibilityPASS 1.00

? The Question

Which at-risk accounts qualify for a churn-save offer under the documented eligibility rules, what amount is at stake per account and in total, and which play fits each (usage revival, executive touch, commercial concession)? Cite the signal that justifies each play. List accounts that look at risk but do not qualify and why.
cost $0.0144 · 42s · in 15,328 / out 7,126 tok
✓ eligible_set✓ total_at_stake✓ plays_cited✓ noneligible_named✓ rules_applied

✓ Correct Answer (ground truth)

snapshot_date: 2026-09-05
rules: health_score < 60, churn_save_eligible_amount > 0, renewal within 120 days of snapshot
eligible: account_alias=C-0F6C0F34; amount_at_stake=49707.00; play=executive touch; justifying_signal=champion_active is false - no executive sponsor engaged | account_alias=C-0B827671; amount_at_stake=25365.00; play=usage revival; justifying_signal=usage_trend_3m=declining over the last 3 months | account_alias=C-0B360C78; amount_at_stake=35748.00; play=commercial concession; justifying_signal=usage stable/growing with seat utilization 75% - risk is commercial, not adoption | account_alias=C-0B0F1BAB; amount_at_stake=5494.00; play=executive touch; justifying_signal=champion_active is false - no executive sponsor engaged | account_alias=C-0CA21961; amount_at_stake=16829.00; play=usage revival; justifying_signal=seat utilization 26% is below 50% | account_alias=C-0E9C27D1; amount_at_stake=41235.00; play=commercial concession; justifying_signal=usage stable/growing with seat utilization 85% - risk is commercial, not adoption … (+2 more)
total_amount_at_stake: 224601.00
non_eligible_at_risk: account_alias=C-0BC71BDD; health_score=55 | account_alias=C-0BA71F12; health_score=52 | account_alias=C-0F6694C3; health_score=43 | account_alias=C-0BE96399; health_score=54 | account_alias=C-0F876796; health_score=47 | account_alias=C-0FCCD2DF; health_score=43 … (+1 more)
n_accounts: 30
graded checks: eligible_set · total_at_stake · plays_cited · noneligible_named · rules_applied

Model's actual answer

show the model's full answer
Eligibility rules from eligibility_rules.csv
R1 health_score lt 60
R2 churn_save_eligible_amount gt 0
R3 renewal_date within_days 120 of snapshot 2026-09-05

Snapshot 2026-09-05 + 120 days = 2027-01-03
Renewal must be <= 2027-01-03

Qualifying accounts = R1 AND R2 AND R3

C-0F6C0F34
health_score 51 <60, churn_save_eligible_amount 49707.00 >0, renewal_date 2026-10-03 <=2027-01-03
Amount at stake 49,707.00

C-0B827671
health_score 56 <60, churn_save_eligible_amount 25,365.00 >0, renewal_date 2026-11-14 <=2027-01-03
Amount at stake 25,365.00

C-0B360C78
health_score 57 <60, churn_save_eligible_amount 35,748.00 >0, renewal_date 2026-10-28 <=2027-01-03
Amount at stake 35,748.00

C-0B0F1BAB
health_score 38 <60, churn_save_eligible_amount 5,494.00 >0, renewal_date 2026-09-23 <=2027-01-03
Amount at stake 5,494.00

C-0CA21961
health_score 58 <60, churn_save_eligible_amount 16,829.00 >0, renewal_date 2026-12-28 <=2027-01-03
Amount at stake 16,829.00

C-0E9C27D1
health_score 39 <60, churn_save_eligible_amount 41,235.00 >0, renewal_date 2026-09-24 <=2027-01-03
Amount at stake 41,235.00

C-0CEF69FD
health_score 53 <60, churn_save_eligible_amount 32,621.00 >0, renewal_date 2026-11-21 <=2027-01-03
Amount at stake 32,621.00

C-0D3278C7
health_score 54 <60, churn_save_eligible_amount 17,602.00 >0, renewal_date 2026-11-12 <=2027-01-03
Amount at stake 17,602.00

Total at stake
49,707.00 + 25,365.00 = 75,072.00
75,072.00 + 35,748.00 = 110,820.00
110,820.00 + 5,494.00 = 116,314.00
116,314.00 + 16,829.00 = 133,143.00
133,143.00 + 41,235.00 = 174,378.00
174,378.00 + 32,621.00 = 206,999.00
206,999.00 + 17,602.00 = 224,601.00

Total = $224,601.00

Play assignment from available signals only. No documented play-to-signal rules are provided in the files. Assignment below uses direct data fields:

C-0F6C0F34 – executive touch
Signal: champion_active=false, usage_trend_3m=growing

C-0B827671 – usage revival
Signal: usage_trend_3m=declining, champion_active=true

C-0B360C78 – commercial concession
Signal: champion_active=true, usage_trend_3m=growing, health_score=57 <60

C-0B0F1BAB – executive touch
Signal: champion_active=false, usage_trend_3m=flat

C-0CA21961 – commercial concession
Signal: champion_active=true, usage_trend_3m=flat

C-0E9C27D1 – commercial concession
Signal: champion_active=true, usage_trend_3m=flat

C-0CEF69FD – executive touch
Signal: champion_active=false, usage_trend_3m=growing

C-0D3278C7 – usage revival
Signal: usage_trend_3m=declining, champion_active=true

At-risk but does NOT qualify for churn-save offer – health_score <60 but fails R2 and/or R3

C-0BC71BDD
health_score 55 <60, churn_save_eligible_amount 0.00 not >0
Fails R2. Renewal 2026-10-27 is within window.

C-0BA71F12
health_score 52 <60, churn_save_eligible_amount 6,824.00 >0
Fails R3. Renewal 2027-04-11 > 2027-01-03 cutoff.

C-0F6694C3
health_score 43 <60, churn_save_eligible_amount 0.00 not >0
Fails R2. Renewal 2027-03-21 > 2027-01-03 cutoff. Fails R3.

C-0BE96399
health_score 54 <60, churn_save_eligible_amount 0.00 not >0
Fails R2. Renewal 2026-10-29 is within window.

C-0F876796
health_score 47 <60, churn_save_eligible_amount 19,958.00 >0
Fails R3. Renewal 2027-02-06 > 2027-01-03 cutoff.

C-0FCCD2DF
health_score 43 <60, churn_save_eligible_amount 0.00 not >0
Fails R2. Renewal 2027-04-23 > 2027-01-03 cutoff. Fails R3.

C-10A56B0F
health_score 54 <60, churn_save_eligible_amount 0.00 not >0
Fails R2. Renewal 2026-12-12 is within window.

Data provided does not contain explicit documented play selection rules; plays above are inferred solely from usage_trend_3m and champion_active fields in churnzero_accounts.csv.
expansion-kitPASS 1.00

? The Question

A customer replied to outreach (reply included). Build the expansion kit: seat coverage (licensed over headcount), two lines on usage health, headroom in seats and ARR at the current per-seat rate, who replied and whether they can buy, the right buyer from our contacts, and a reply email under 150 words that moves toward the right person without being pushy and cites one real usage fact.
cost $0.0035 · 15s · in 3,506 / out 1,755 tok
✓ coverage_pct✓ headroom_arr✓ right_buyer✓ replier_not_buyer✓ usage_fact✓ email_under_150

✓ Correct Answer (ground truth)

company_alias: C-0DDFC9A7
coverage_pct: 37.5
headroom_seats: 250
per_seat_annual_rate: 60.00
headroom_arr: 15000.00
replier: Maria S.
replier_can_buy: False
right_buyer: name=Dana R.; title=VP People
usage_facts: active_users_2026_08=126; active_users_2026_03=88; growth_pct_6m=43.2; licensed_seat_utilization_pct=84.0
usage_health_lines: Monthly active users grew from 88 (2026-03) to 126 (2026-08), +43.2% over 6 months., Last month 126 of 150 licensed seats were active (84.0% utilization), approaching the seat cap.
graded checks: coverage_pct · headroom_arr · right_buyer · replier_not_buyer · usage_fact · email_under_150

Model's actual answer

show the model's full answer
**Expansion kit for C-0DDFC9A7**

Seat coverage
Licensed seats / headcount = 150 / 400 = 0.375 = 37.5%

Usage health
MAU is climbing month-over-month: 88 in 2026-03 → 95 in 2026-04 → 102 in 2026-05 → 110 in 2026-06 → 118 in 2026-07 → 126 in 2026-08.
August 2026 MAU 126 is 84% of licensed seats 150 and +38 users vs March 2026.

Headroom
Per-seat ARR = current_arr / licensed_seats = 9,000.00 / 150 = $60 per seat per year
Seats headroom vs headcount = 400 - 150 = 250 seats
ARR headroom at current per-seat rate = 250 × $60 = $15,000

Who replied / buying authority
Maria S., People Operations Coordinator, replied 2026-09-02. Cannot buy. Reply states: “I'm not the right person for purchasing decisions here. Budget and seat expansion sit with Dana R.”

Right buyer from contacts
Dana R., VP People, last_engaged 2026-05-18

Reply email to Maria S. <150 words
Subject: Re: Growing your team's recognition program

Hi Maria,

Thank you for the update and for the kind words about Bonusly. It’s great to hear the recognition feed is busy.

I noticed your monthly active users rose from 88 in March 2026 to 126 in August 2026 for C-0DDFC9A7. If helpful, I can share a brief usage summary for Dana R. as she’s been asking about usage numbers.

Would you be comfortable making an introduction, or would you prefer I reach out to her directly?

Best,
Cole
onboarding-statusPASS 1.00

? The Question

Prep the CSM for the mid-onboarding call: what is complete (integrations connected, allowance set, admins added, first recognition given, first redemption), what is not, early engagement signals, and the three things to cover on the call. Mark nothing complete without a data field that shows it.
cost $0.0056 · 22s · in 4,081 / out 2,943 tok
✓ complete_items✓ incomplete_items✓ three_items✓ grounded

✓ Correct Answer (ground truth)

company_alias: C-0D284E42
days_live: 25
checklist: {"slack_integration_connected": {"complete": true, "field": "integration_slack", "value": "2026-08-12"}, "hris_integration_connected": {"complete": false, "field": "integration_hris", "value": ""}, "a
engagement_signals: avg_active_givers_week1=4.3; avg_active_givers_last_week=13.0; latest_active_givers=15; days_with_activity=25
call_items: Connect the HRIS integration (integration_hris is empty) to automate user provisioning and offboarding., Get to first redemption (first_redemption_at is empty): walk admins through the rewards catalog and confirm allowance funding., Build the giving habit: daily active givers averaged 4.3 in week 1 vs 13.0 last week - agree on manager nudges and a recognition cadence.
graded checks: complete_items · incomplete_items · three_items · grounded

Model's actual answer

show the model's full answer
Company: C-0D284E42

What is complete - data field present
- Integration Slack: connected 2026-08-12
- Allowance set: 2026-08-13
- Admins added: 2
- First recognition given: 2026-08-15 14:22

What is not complete - data field missing / blank
- Integration HRIS: no date in onboarding_account.csv
- First redemption: no date in onboarding_account.csv, field is blank

Early engagement signals from onboarding_usage.csv
Active givers by date 2026-08-11 to 2026-09-04 = 25 days
Sum = 214
Average active givers = 214 / 25 = 8.56

First day 2026-08-11 = 3
Last day 2026-09-04 = 15
Absolute growth = 15 - 3 = 12
Percent growth = 12 / 3 * 100 = 400%

First 7 days avg = (3+3+4+4+5+4+7) / 7 = 30 / 7 = 4.29
Last 7 days avg = (11+13+11+13+13+15+15) / 7 = 91 / 7 = 13.00
Change = 13.00 - 4.29 = 8.71
Percent change = 8.71 / 4.29 * 100 = 203.3% increase

Peak active givers = 15 on 2026-09-03 and 2026-09-04

Three things to cover on the call
1. HRIS integration - no connection date recorded. Confirm status and unblock.
2. First redemption - first recognition given 2026-08-15 14:22, first redemption is blank. Discuss redemption flow and enable first redemption.
3. Engagement momentum - active givers rose from 3 to 15 over the period with last 7-day average 13.00 vs first 7-day 4.29. Discuss sustaining giver growth and admin enablement with 2 admins added.
renewal-risk-conflicting-datesPASS 1.00

? The Question

Produce a 90-day renewal risk brief. Two systems hold renewal dates and disagree on some accounts; decide which to trust per account and say why (multi-year contracts are known to be wrong in ChurnZero). For every renewal: company, CSM, ARR, date used, seat utilization, 3-month usage trend, risk rating with one sentence of evidence. Flag every disagreement. Close with total ARR renewing and ARR at risk.
cost $0.0132 · 49s · in 7,966 / out 9,402 tok
✓ total_renewing✓ arr_at_risk✓ disagreements_flagged✓ trust_rule

✓ Correct Answer (ground truth)

snapshot_date: 2026-09-05
window: 2026-09-05 to 2026-12-04
trust_rule: multi-year contracts: Chargebee is authoritative (ChurnZero known wrong); otherwise systems agree or Chargebee wins
accounts: 20 items — e.g. account_alias=C-0B144C78; csm=Cole Ingram; arr=30899.00; trusted_renewal_date=2026-11-02; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=75.4; usage_3m_ratio=1.03; risk=low; evidence=3-month usage ratio 1.03 (last3 avg 103 vs prior3 100), seat utilization 75% | account_alias=C-0B20DB64; csm=Dana Mercer; arr=21770.00; trusted_renewal_date=2026-10-07; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=56.6; usage_3m_ratio=1.0; risk=medium; evidence=3-month usage ratio 1.00 (last3 avg 295 vs prior3 295), seat utilization 57% | account_alias=C-0B344485; csm=Elena Sinclair; arr=64384.00; trusted_renewal_date=2026-11-16; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=78.0; usage_3m_ratio=1.04; risk=low; evidence=3-month usage ratio 1.04 (last3 avg 241 vs prior3 231), seat utilization 78% | account_alias=C-0B369871; csm=Dana Mercer; arr=85128.00; trusted_renewal_date=2026-10-29; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=75.1; usage_3m_ratio=1.04; risk=low; evidence=3-month usage ratio 1.04 (last3 avg 330 vs prior3 316), seat utilization 75%
disagreements: account_alias=C-0B7D2C30; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-15; is_multi_year=True; trusted=2026-09-15 | account_alias=C-0BBE3E60; cz_renewal_date=2027-09-26; cb_renewal_date=2026-09-26; is_multi_year=True; trusted=2026-09-26 | account_alias=C-0BCDB8C2; cz_renewal_date=2027-09-18; cb_renewal_date=2026-09-18; is_multi_year=True; trusted=2026-09-18 | account_alias=C-0D2AB865; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-22; is_multi_year=True; trusted=2026-09-22 | account_alias=C-0F5D2323; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-29; is_multi_year=True; trusted=2026-09-29
n_disagreements: 5
total_arr_renewing_90d: 1048715.00
arr_at_risk_90d: 359409.00
n_accounts: 20
graded checks: total_renewing · arr_at_risk · disagreements_flagged · trust_rule

Model's actual answer

show the model's full answer
90-Day Renewal Risk Brief
As of 2026-09-06, 90-day window = 2026-09-06 to 2026-12-05.
Date trust rule: is_multi_year = true → ChurnZero renewal is known wrong, use Chargebee date. is_multi_year = false → dates match, use ChurnZero/Chargebee date.

Disagreements flagged:

* C-0B7D2C30 CZ 2026-09-10 vs CB 2026-09-15 → DISAGREE, multi-year true, use CB
* C-0BCDB8C2 CZ 2027-09-18 vs CB 2026-09-18 → DISAGREE, multi-year true, use CB
* C-0D2AB865 CZ 2026-09-10 vs CB 2026-09-22 → DISAGREE, multi-year true, use CB
* C-0BBE3E60 CZ 2027-09-26 vs CB 2026-09-26 → DISAGREE, multi-year true, use CB
* C-0F5D2323 CZ 2026-09-10 vs CB 2026-09-29 → DISAGREE, multi-year true, use CB
Accounts C-0EC6999D to C-22170CA1: CZ = CB, no disagreement.

Renewals in window:

C-0B7D2C30 – CSM Dana Mercer – ARR $65,901 – Date used 2026-09-15 Chargebee – Seat utilization 274/476 = 57.5% – 3-mo usage 2026-06 97 → 2026-08 84 = -13 users, -13.4% – Risk: Medium – utilization moderate with 3-month active user decline.
C-0BCDB8C2 – CSM Cole Ingram – ARR $54,427 – Date used 2026-09-18 Chargebee – Seat utilization 232/424 = 54.7% – 3-mo usage 127 → 110 = -17 users, -13.4% – Risk: Medium – declining active users over last 3 months.
C-0D2AB865 – CSM Elena Sinclair – ARR $38,022 – Date used 2026-09-22 Chargebee – Seat utilization 250/407 = 61.4% – 3-mo usage 125 → 109 = -16 users, -12.8% – Risk: Medium – active users down 12.8% Jun to Aug.
C-0BBE3E60 – CSM Dana Mercer – ARR $30,993 – Date used 2026-09-26 Chargebee – Seat utilization 74/114 = 64.9% – 3-mo usage 39 → 33 = -6 users, -15.4% – Risk: Medium – steep 3-month active user decline despite 64.9% seat utilization.
C-0F5D2323 – CSM Cole Ingram – ARR $90,647 – Date used 2026-09-29 Chargebee – Seat utilization 111/390 = 28.5% – 3-mo usage 20 → 18 = -2 users, -10.0% – Risk: High – very low seat utilization 28.5% with declining usage.
C-0EC6999D – CSM Elena Sinclair – ARR $79,419 – Date used 2026-10-03 – Seat utilization 31/112 = 27.7% – 3-mo usage 17 → 15 = -2 users, -11.8% – Risk: High – utilization 27.7% and active users falling.
C-0B20DB64 – CSM Dana Mercer – ARR $21,770 – Date used 2026-10-07 – Seat utilization 214/378 = 56.5% – 3-mo usage 294 → 294 = 0 users, 0.0% – Risk: Low – flat usage with stable seat utilization.
C-0BBC4E7A – CSM Cole Ingram – ARR $56,374 – Date used 2026-10-10 – Seat utilization 228/337 = 67.7% – 3-mo usage 142 → 139 = -3 users, -2.1% – Risk: Low – high utilization with essentially flat usage.
C-0FD551AB – CSM Elena Sinclair – ARR $48,815 – Date used 2026-10-14 – Seat utilization 210/376 = 55.9% – 3-mo usage 123 → 126 = +3 users, +2.4% – Risk: Low – stable utilization with slight usage growth.
C-0F9F8F13 – CSM Dana Mercer – ARR $46,230 – Date used 2026-10-18 – Seat utilization 199/352 = 56.6% – 3-mo usage 185 → 182 = -3 users, -1.6% – Risk: Low – utilization 56.6% and usage essentially flat.
C-0BC34584 – CSM Cole Ingram – ARR $16,740 – Date used 2026-10-22 – Seat utilization 327/494 = 66.2% – 3-mo usage 104 → 106 = +2 users, +1.9% – Risk: Low – solid utilization with growing active users.
C-0B7A7546 – CSM Elena Sinclair – ARR $35,062 – Date used 2026-10-25 – Seat utilization 182/205 = 88.8% – 3-mo usage 64 → 63 = -1 users, -1.6% – Risk: Low – very high utilization with stable usage.
C-0B369871 – CSM Dana Mercer – ARR $85,128 – Date used 2026-10-29 – Seat utilization 317/422 = 75.1% – 3-mo usage 326 → 333 = +7 users, +2.1% – Risk: Low – high utilization and 3-month growth.
C-0B144C78 – CSM Cole Ingram – ARR $30,899 – Date used 2026-11-02 – Seat utilization 169/224 = 75.5% – 3-mo usage 101 → 106 = +5 users, +5.0% – Risk: Low – high utilization with growing usage.
C-0FC4DBB8 – CSM Elena Sinclair – ARR $94,732 – Date used 2026-11-05 – Seat utilization 356/464 = 76.6% – 3-mo usage 189 → 193 = +4 users, +2.1% – Risk: Low – strong utilization and upward usage trend.
C-0D5BBE3A – CSM Dana Mercer – ARR $39,740 – Date used 2026-11-09 – Seat utilization 85/102 = 83.3% – 3-mo usage 88 → 91 = +3 users, +3.4% – Risk: Low – high utilization with growing usage.
C-0FB9D5AF – CSM Cole Ingram – ARR $63,158 – Date used 2026-11-13 – Seat utilization 144/199 = 72.4% – 3-mo usage 173 → 176 = +3 users, +1.7% – Risk: Low – healthy utilization and stable growth.
C-0B344485 – CSM Elena Sinclair – ARR $64,384 – Date used 2026-11-16 – Seat utilization 224/287 = 78.1% – 3-mo usage 238 → 244 = +6 users, +2.5% – Risk: Low – high utilization with growing usage.
C-0CB2C1B4 – CSM Dana Mercer – ARR $40,628 – Date used 2026-11-20 – Seat utilization 386/473 = 81.6% – 3-mo usage 47 → 49 = +2 users, +4.3% – Risk: Low – high utilization with usage growth.
C-22170CA1 – CSM Cole Ingram – ARR $45,646 – Date used 2026-11-24 – Seat utilization 251/294 = 85.4% – 3-mo usage 143 → 146 = +3 users, +2.1% – Risk: Low – very high utilization with stable growth.

Totals
Total ARR renewing in 90 days: $1,048,715
Sum = 65,901+54,427+38,022+30,993+90,647+79,419+21,770+56,374+48,815+46,230+16,740+35,062+85,128+30,899+94,732+39,740+63,158+64,384+40,628+45,646 = $1,048,715

ARR at risk
High risk ARR = C-0F5D2323 $90,647 + C-0EC6999D $79,419 = $170,066
Medium risk ARR = C-0B7D2C30 $65,901 + C-0BCDB8C2 $54,427 + C-0D2AB865 $38,022 + C-0BBE3E60 $30,993 = $189,343
Medium+High ARR at risk = $359,409

No data missing for requested fields.
ticket-theme-synthesis0.80

? The Question

Synthesize the quarter's support tickets into 4 to 7 themes. Do not trust existing tags; read the text. Per theme: name, count, share, distinct accounts, ARR affected, two ticket ids, one-line recommendation. Separate broad patterns from single-account noise and rank by ARR exposure, not volume.
cost $0.0476 · 140s · in 27,087 / out 29,831 tok
✗ theme_count_range✓ counts_verifiable✓ single_account_flagged✓ arr_ranked✓ ticket_ids

✓ Correct Answer (ground truth)

quarter_window: 2026-06-01 through 2026-08-31
total_tickets: 80
themes: {"points_not_posting": {"count": 20, "share_pct": 25.0, "distinct_accounts": 9, "arr_affected": "31100.00", "example_ticket_ids": ["IC-460001", "IC-460002"], "single_account": false}, "slack_sync": {"
ranking_by_arr_exposure: hris_provisioning, redemption_failures, billing_dispute, points_not_posting, slack_sync
single_account_noise_theme: billing_dispute
note: existing tags are noisy/unreliable; classify by text
sample_ticket_ids: 80 items — e.g. IC-460004, IC-460016, IC-460001, IC-460041
graded checks: theme_count_range · counts_verifiable · single_account_flagged · arr_ranked · ticket_ids

Model's actual answer

show the model's full answer
Total tickets in file: 80

**Broad patterns – ranked by ARR exposure**

1. HRIS provisioning – new hire creation failure
* Count: 9  Share: 9/80 = 11.25%
* Distinct accounts: 3 – C-0B2213A9, C-0DDFC9A7, C-0F6C0F34
* ARR affected (distinct): 36,000 + 48,000 + 30,000 = 114,000
* Example tickets: IC-460059, IC-460055
* Recommendation: Audit HRIS field mapping and add provisioning success/failure alerts; 9 tickets across 3 large accounts indicate a systemic sync break.

2. HRIS provisioning – sync skip silent
* Count: 3  Share: 3/80 = 3.75%
* Distinct accounts: 2 – C-0F6C0F34, C-0DDFC9A7
* ARR affected (distinct): 30,000 + 48,000 = 78,000
* Example tickets: IC-460062, IC-460060
* Recommendation: Enable provisioning log error surfacing and a reconciliation job; “no errors” with 12 hires skipped is a silent failure.

3. Redemption failures
* Count: 18  Share: 18/80 = 22.5%
* Distinct accounts: 7 – C-0CEF69FD, C-0B827671, C-0F876796, C-14264ABD, C-0FCCD2DF, C-0D9CA315, C-0B0F1BAB
* ARR affected (distinct): 8,900 + 10,700 + 8,700 + 11,000 + 9,600 + 9,600 + 10,300 = 68,800
* Example tickets: IC-460025, IC-460035
* Recommendation: Investigate checkout timeout / payment gateway idempotency and gift-card email delivery; failures include hangs, missing codes and points deducted without fulfillment.

4. Points crediting issues
* Count: 20  Share: 20/80 = 25.0%
* Distinct accounts: 9 – C-0D3278C7, C-0BE96399, C-0DD0626C, C-0B2895EF, C-0D284E42, C-0BF20542, C-0D0B047C, C-0D6CC8E3, C-21FEBCBB
* ARR affected (distinct): 3,500 + 2,700 + 2,500 + 2,900 + 3,400 + 4,500 + 4,500 + 4,200 + 2,900 = 31,100
* Example tickets: IC-460004, IC-460016
* Recommendation: Review points ledger reconciliation and weekend batch job; patterns cover delivered-but-not-awarded, team batch delay and stale balances.

5. Slack integration issues
* Count: 14  Share: 14/80 = 17.5%
* Distinct accounts: 4 – C-0B843542, C-0BA71F12, C-10A56B0F, C-8C2E8F00
* ARR affected (distinct): 4,400 + 3,900 + 5,400 + 5,200 = 18,900
* Example tickets: IC-460041, IC-460047
* Recommendation: Stabilize Slack OAuth refresh and sync toggle persistence; recurring sync stop, toggle reset, auth disconnect and slash-command errors affect multiple accounts.

**Single-account noise – high volume / ARR but not broad**

* Billing errors – single account C-0E9C27D1
  Count: 16  Share: 16/80 = 20.0%
  Distinct accounts: 1 – C-0E9C27D1
  ARR affected: 52,000
  Example tickets: IC-460071, IC-460078
  Note: 5x “Third invoice in a row with the same seat-count error”, 5x “Invoice discrepancy - charged for 200 seats but we license 150”, 6x “Billing charged the annual renewal at the wrong tier price” / “Our invoice shows a seat count we never approved”. This is single-account noise; prioritize white-glove billing config review for C-0E9C27D1 rather than a product fix.

All themes derived from body_text only; tags were ignored. ARR affected is sum of distinct account ARR per theme to avoid double-counting multiple tickets from the same account.
marketing
case-study-matchPASS 1.00

? The Question

Given the prospect, rank the three most similar existing customers with public case studies for use as social proof. Similarity on industry, size band, use case, region. Cite the fields that drove each match. Do not name any customer without a case study and never include billing data or contact names.
cost $0.0045 · 18s · in 3,774 / out 2,589 tok
✓ top3✓ decoys_excluded✓ cites_fields✓ no_billing

✓ Correct Answer (ground truth)

top3: company_alias=C-11C31562; similarity_score=3 | company_alias=C-64171065; similarity_score=3 | company_alias=C-A13C193D; similarity_score=3
decoys_excluded: company_alias=C-D6217CAA; similarity_score=4; reason=no_case_study | company_alias=C-C153868F; similarity_score=4; reason=no_case_study
constraints: no_billing_data, no_contact_names, only_customers_with_has_case_study_true_named
prospect_fields_used: industry, size_band, use_case, region
prospect_alias: C-82AF3719
graded checks: top3 · decoys_excluded · cites_fields · no_billing

Model's actual answer

show the model's full answer
Prospect C-82AF3719
industry = Technology
size_band = Mid-Market
use_case = employee_recognition
region = NA-West

Case study customers evaluated on 4 fields. Match count = number of fields equal to prospect.

C-64171065
has_case_study = true
industry: Technology = Technology → match
size_band: Mid-Market = Mid-Market → match
use_case: employee_recognition = employee_recognition → match
region: NA-East ≠ NA-West → no match
Matches = 3/4

C-A13C193D
has_case_study = true
industry: Technology = Technology → match
size_band: Mid-Market = Mid-Market → match
use_case: retention ≠ employee_recognition → no match
region: NA-West = NA-West → match
Matches = 3/4

C-11C31562
has_case_study = true
industry: Manufacturing ≠ Technology → no match
size_band: Mid-Market = Mid-Market → match
use_case: employee_recognition = employee_recognition → match
region: NA-West = NA-West → match
Matches = 3/4

Ranked most similar case study customers:

1. C-64171065 — matches industry, size_band, use_case
2. C-A13C193D — matches industry, size_band, region
3. C-11C31562 — matches size_band, use_case, region

No case study customer matches all 4 fields for this prospect. Next best case study matches are 2/4:
C-9D69D147 — matches size_band, region
C-CD4829A7 — matches industry, size_band
channel-attribution-mathPASS 1.00

? The Question

Compute channel performance for the trailing 6 months. Paid: spend, SQMs, SQOs, cost per SQM, cost per SQO, SQM-to-SQO rate, pipeline amount, pipeline per dollar. Organic: volume, SQO rate, pipeline. Report a channel with spend and zero SQMs as undefined, not zero. Flag rows where SQO date precedes SQM date. Finish with a reallocation recommendation and your confidence given sample sizes.
cost $0.0189 · 53s · in 13,147 / out 11,612 tok
✓ undefined_channel✓ undefined_not_zero✓ flagged_rows✓ best_channel✓ reallocation

✓ Correct Answer (ground truth)

months: 2026-03, 2026-04, 2026-05, 2026-06, 2026-07, 2026-08
per_channel: {"linkedin_ads": {"spend_usd": 24000, "sqms": 25, "sqos": 8, "pipeline_amount": 96000, "cost_per_sqm": 960.0, "cost_per_sqo": 3000.0, "sqm_to_sqo_rate": 0.32, "pipeline_per_dollar": 4.0}, "organic_sea
undefined_channels: paid_social
flagged_rows_sqo_before_sqm: CT-000041, CT-000044
best_pipeline_per_dollar_channel: paid_search
reallocation_recommendation: from=paid_social; to=paid_search; rationale=paid_social has spend but zero SQMs (cost per SQM undefined, not zero); paid_search has the highest pipeline per dollar.
confidence_note: webinars and referral have small SQM samples (<20); treat rates as directional.
graded checks: undefined_channel · undefined_not_zero · flagged_rows · best_channel · reallocation

Model's actual answer

show the model's full answer
**Trailing 6 months: 2026-03-01 to 2026-08-31. SQM date in range.**

Spend from channel_spend.csv
* paid_search: 6,000 x 6 = 36,000
* linkedin_ads: 4,000 x 6 = 24,000
* paid_social: 3,000 x 6 = 18,000
* webinars: 1,500 x 6 = 9,000

Contacts first touch filtered to sqm_date in period = 122 rows
Counter by channel: paid_search 40, organic_search 30, linkedin_ads 25, referral 15, webinars 12
paid_social contacts = 0

### Paid channels

paid_search
* spend_usd = 36,000
* SQMs = 40
* SQOs = 18
* cost per SQM = 36,000 / 40 = 900
* cost per SQO = 36,000 / 18 = 2,000
* SQM-to-SQO rate = 18 / 40 = 0.45 = 45%
* pipeline amount = 18 x 40,000 = 720,000
* pipeline per dollar = 720,000 / 36,000 = 20.0

linkedin_ads
* spend_usd = 24,000
* SQMs = 25
* SQOs = 8
* cost per SQM = 24,000 / 25 = 960
* cost per SQO = 24,000 / 8 = 3,000
* SQM-to-SQO rate = 8 / 25 = 0.32 = 32%
* pipeline amount = 8 x 12,000 = 96,000
* pipeline per dollar = 96,000 / 24,000 = 4.0

webinars
* spend_usd = 9,000
* SQMs = 12
* SQOs = 5
* cost per SQM = 9,000 / 12 = 750
* cost per SQO = 9,000 / 5 = 1,800
* SQM-to-SQO rate = 5 / 12 = 0.4167 = 41.7%
* pipeline amount = 5 x 12,000 = 60,000
* pipeline per dollar = 60,000 / 9,000 = 6.67

paid_social
* spend_usd = 18,000
* SQMs = 0
* SQOs = 0
* cost per SQM = undefined
* cost per SQO = undefined
* SQM-to-SQO rate = undefined
* pipeline amount = 0
* pipeline per dollar = 0 / 18,000 = 0

Channel with spend and zero SQMs reported as undefined, not zero.

### Organic channels

organic_search
* volume = SQMs = 30
* SQOs = 10
* SQO rate = 10 / 30 = 0.3333 = 33.3%
* pipeline = 10 x 9,000 = 90,000

referral
* volume = SQMs = 15
* SQOs = 6
* SQO rate = 6 / 15 = 0.40 = 40%
* pipeline = 6 x 8,000 = 48,000

### Data quality flags

SQO date precedes SQM date:
* CT-000044 linkedin_ads sqm_date 2026-07-23 sqo_date 2026-07-18
* CT-000041 linkedin_ads sqm_date 2026-06-14 sqo_date 2026-06-09

These two rows are included in the counts above.

### Reallocation recommendation
* paid_search is best paid ROI: pipeline per dollar 20, cost per SQO 2,000, SQM-to-SQO 45% with 40 SQMs / 18 SQOs.
* webinars delivers second best paid efficiency: pipeline per dollar 6.67, cost per SQO 1,800, SQM-to-SQO 41.7% with 12 SQMs / 5 SQOs.
* linkedin_ads is least efficient paid: pipeline per dollar 4, cost per SQO 3,000, SQM-to-SQO 32% with 25 SQMs / 8 SQOs and two date anomalies.
* paid_social has $18,000 spend with 0 SQMs in period → pause/reallocate until tracking is fixed.
* Organic_search and referral generate pipeline with no spend: organic_search 90,000 pipeline from 30 volume, referral 48,000 pipeline from 15 volume.

Suggested move: increase paid_search budget, maintain webinars, hold/redirect paid_social spend to paid_search/webinars pending tracking fix, investigate LinkedIn Ads date quality and conversion path.

Confidence: MEDIUM. Sample sizes are small for paid channels: paid_search 40 SQMs/18 SQOs, linkedin_ads 25 SQMs/8 SQOs, webinars 12 SQMs/5 SQOs. paid_social has zero observed SQMs. Two SQO-before-SQM flags reduce confidence in LinkedIn Ads timing data.
competitor-battlecard-updatePASS 1.00

? The Question

Update the battlecard for the competitor. Sections: one-line positioning, pricing with source and date (newer source wins, note the conflict), where they win, where we win, objections and responses, recent changes, our 12-month win/loss record against them. Cite a snippet id for every factual claim. Rep opinion on a call is not a fact about the competitor. Anything from the old card you cannot re-source gets marked unverified.
cost $0.0089 · 35s · in 4,423 / out 6,419 tok
✓ competitor_named✓ pricing✓ win_loss✓ unverified_marked✓ citations

✓ Correct Answer (ground truth)

competitor: Rivally
pricing: value_per_user_per_month=7; source_snippet=S17; source_date=2026-08-12
win_loss_12mo: wins=13; losses=7
unverified_old_claims: Rivally lacks a Slack integration., Rivally was acquired by WorkHuman in 2025.
rep_opinion_snippet_ids_not_facts: S09, S21
citation_ids_available: 25 items — e.g. S01, S02, S03, S04
graded checks: competitor_named · pricing · win_loss · unverified_marked · citations

Model's actual answer

show the model's full answer
# Battlecard: Rivally — updated

**One-line positioning**
Points-based recognition feed for mid-market. Reviewer praises Rivally's points-based recognition feed [S02]; mid-market reviewer notes setup under a week [S04].

**Pricing**
* Newer official source wins
  * 2026-08-12 pricing_page: Recognition Starter now $7 per user/month, annual billing required [S17]
* Prior official pricing
  * 2026-01-20 pricing_page: Rivally Recognition listed at $5 per user/month, annual billing required [S03]
  * 2026-04-01 pricing_page: pricing page still shows $5 per user/month for Recognition Starter tier [S08]
* Conflict / deal quotes
  * 2026-06-02 call_notes: Rivally quoted $6.50/user/mo to a 500-seat prospect, annual term [S13]
  * 2026-08-14 call_notes: prospect says Rivally quoted $7/user/mo list, offered 15% discount for 3-year term [S18]
Official list price moved $5 → $7 between 2026-04-01 and 2026-08-12.

**Where they win**
* Points-based recognition feed praised; recognition feed is engaging [S02][S16]
* Setup speed and Slack integration: mid-market reviewer: setup took under a week, Slack integration worked out of the box [S04]
* EU focus
  * 2026-05-21 g2_review: EU enterprise reviewer: Rivally is strong for distributed EU teams, multi-language support praised [S12]
  * 2026-07-01 press: Rivally opens Dublin office; announces EU data residency generally available [S15]
  * 2026-02-18 call_notes: Rivally pitched EU data residency [S05]
* Support: G2 review: Rivally support response time praised (under 4 hours) [S22]
* Product motion: Microsoft Teams app v2 in public preview [S19]

**Where we win**
* Analytics depth
  * 2026-09-03 call_notes: 800-seat prospect picked Bonusly over Rivally citing analytics depth [S25]
  * Reviewer notes limited analytics [S02]
  * Capterra review: reporting dashboards are basic compared to enterprise tools [S07]
* Admin / IT capabilities
  * Enterprise reviewer: Rivally lacks SCIM provisioning; manual user management is painful [S10]
  * Reviewer: Rivally's recognition feed is engaging; admin tooling lags peers [S16]
  * Reviewer: migration off Rivally was hard because analytics exports are CSV-only [S20]
  * Reviewer: Rivally admin console still lacks bulk recognition editing [S24]
* EMEA rewards coverage
  * TrustRadius review: rewards catalog in EMEA is thinner than US catalog [S14]

**Objections and responses**
* Objection: limited / basic analytics / reporting
  * Evidence: limited analytics [S02]; reporting dashboards are basic compared to enterprise tools [S07]
  * Sourced response: 800-seat prospect picked Bonusly over Rivally citing analytics depth [S25]
* Objection: weak admin / identity management
  * Evidence: lacks SCIM provisioning; manual user management is painful [S10]; admin tooling lags peers [S16]; admin console still lacks bulk recognition editing [S24]; analytics exports CSV-only [S20]
  * Sourced response: No sourced response in data.
* Objection: thinner EMEA rewards
  * Evidence: rewards catalog in EMEA is thinner than US catalog [S14]
  * Sourced response: No sourced response in data.

**Recent changes**
* 2025-11-04 press: Rivally announced a Series C round of $40M led by Northgate Ventures [S01]
* 2026-03-05 press: Rivally launches 'Rivally Pulse', a lightweight engagement survey add-on [S06]
* 2026-05-09 press: Rivally hires ex-Workday VP EMEA to lead European expansion [S11]
* 2026-07-01 press: Rivally opens Dublin office; announces EU data residency generally available [S15]
* 2026-08-12 pricing_page: Recognition Starter now $7 per user/month, annual billing required [S17]
* 2026-08-20 press: Rivally announces Microsoft Teams app v2 in public preview [S19]
* 2026-09-01 press: Rivally Pulse add-on exits beta; priced as add-on, not bundled [S23]

**12-month win/loss record vs Rivally**
Window used: 2025-09 to 2026-08, all deals in deals_with_competitor.csv.

Wins = 13
Deal-A9FD43 2025-10 win
Deal-7AA785 2025-11 win
Deal-44C524 2025-12 win
Deal-0D0CD6 2026-01 win
Deal-D5B790 2026-02 win
Deal-5C636E 2026-03 win
Deal-67BE14 2026-06 win
Deal-1B6969 2026-07 win
Deal-F03E7B 2026-08 win
Deal-072E31 2025-09 win
Deal-F65C8F 2025-10 win
Deal-E46EAB 2026-01 win
Deal-1D2392 2026-02 win

Losses = 7
Deal-7767F5 2025-09 loss
Deal-5645A5 2026-04 loss
Deal-C6FFAA 2026-05 loss
Deal-D263E0 2025-11 loss
Deal-935746 2025-12 loss
Deal-9066A6 2026-03 loss
Deal-72A02F 2026-04 loss

Arithmetic: 13 wins + 7 losses = 20 deals
Win rate = 13 / 20 = 0.65 = 65%

**Legacy battlecard items — verification**
* Positioning: points-based recognition for mid-market — verified via S02, S04
* Pricing: starts at $5 per user/month, annual billing as of 2026-01 — superseded; newer official price is $7 as of 2026-08-12 [S17]
* Rivally lacks a Slack integration — UNVERIFIED / contradicted. S04 states Slack integration worked out of the box.
* Rivally was acquired by WorkHuman in 2025 — UNVERIFIED. No source in snippets.
* Strong in EU enterprise with multi-language support — verified [S12]

Rep opinions excluded as non-factual about competitor: AE opinion Bryce Harmon UI clunky [S09]; AE opinion Elena Sinclair discounting aggressively [S21].
nurture-sequence-diagnosisPASS 1.00

? The Question

Review the sequences. Per sequence: sent, open rate, reply rate, meeting rate, weakest step. Identify tracking errors (opened above sent), audience overlap between sequences, and the failure mode for anything under 2 percent reply. One change per weak sequence and which to fix first. Under 350 words.
cost $0.0492 · 149s · in 39,126 / out 28,696 tok
✓ tracking_error✓ overlap_pair✓ weakest_step✓ fix_priority

✓ Correct Answer (ground truth)

per_sequence: {"New Logo Nurture": {"sent": 1386, "opened": 490, "replied": 90, "meetings": 27, "open_rate": 0.3535, "reply_rate": 0.0649, "meeting_rate": 0.0195, "weakest_step": 3}, "Expansion Nurture": {"sent": 8
tracking_error: sequence=Expansion Nurture; step=2; sent=300; opened=340; issue=opened exceeds sent
audience_overlap_pair: Cold Outbound - HR Leaders, Cold Outbound - People Ops
all_pairwise_overlaps: ['Cold Outbound - HR Leaders', 'Cold Outbound - People Ops'], ['Expansion Nurture', 'New Logo Nurture']
under_2pct_reply: Cold Outbound - HR Leaders
failure_mode: Cold Outbound - HR Leaders=opens healthy (~37%) but replies near zero from step 1 -> body/CTA not landing, not a deliverability or subject-line problem
fix_priority_first: Cold Outbound - HR Leaders
graded checks: tracking_error · overlap_pair · weakest_step · fix_priority

Model's actual answer

show the model's full answer
**Sequences - totals and per-step rates**

*New Logo Nurture*
Sent 500+458+428=1,386. Opened 210+160+120=490 → 490/1,386=35.4%. Replied 42+30+18=90 → 90/1,386=6.5%. Meetings 12+9+6=27 → 27/1,386=1.9%.
Step 1: sent 500 open 210=42.0% reply 42=8.4% meet 12=2.4%
Step 2: sent 458 open 160=34.9% reply 30=6.6% meet 9=2.0%
Step 3: sent 428 open 120=28.0% reply 18=4.2% meet 6=1.4%
Weakest step: 3, reply 4.2%.

*Expansion Nurture*
Sent 300+300+275=875. Opened 130+340+95=565 → 565/875=64.6%. Replied 22+25+12=59 → 59/875=6.7%. Meetings 5+4+3=12 → 12/875=1.4%.
Step 1: sent 300 open 130=43.3% reply 22=7.3% meet 5=1.7%
Step 2: sent 300 open 340=113.3% reply 25=8.3% meet 4=1.3%
Step 3: sent 275 open 95=34.5% reply 12=4.4% meet 3=1.1%
Weakest step: 3, reply 4.4%.
Tracking error: Step 2 opened 340 > sent 300.

*Cold Outbound - HR Leaders*
Sent 600+595+590=1,785. Opened 240+175+130=545 → 545/1,785=30.5%. Replied 5+2+1=8 → 8/1,785=0.4%. Meetings 0+0+0=0 → 0%.
Step 1: sent 600 open 240=40.0% reply 5=0.8% meet 0=0%
Step 2: sent 595 open 175=29.4% reply 2=0.3% meet 0=0%
Step 3: sent 590 open 130=22.0% reply 1=0.2% meet 0=0%
Weakest step: 3, reply 0.2%.

*Cold Outbound - People Ops*
Sent 400+386+377=1,163. Opened 150+110+80=340 → 340/1,163=29.2%. Replied 14+9+6=29 → 29/1,163=2.5%. Meetings 3+2+1=6 → 6/1,163=0.5%.
Step 1: sent 400 open 150=37.5% reply 14=3.5% meet 3=0.8%
Step 2: sent 386 open 110=28.5% reply 9=2.3% meet 2=0.5%
Step 3: sent 377 open 80=21.2% reply 6=1.6% meet 1=0.3%
Weakest step: 3, reply 1.6%.

**Tracking errors**
Expansion Nurture step 2 opened 340 > sent 300.

**Audience overlap**
23 contacts appear in >1 sequence.
Pairs:
Cold Outbound - HR Leaders / Cold Outbound - People Ops: 21 contacts e.g. CT-001217, CT-001159, CT-001153, CT-001227
Expansion Nurture / New Logo Nurture: 2 contacts e.g. CT-000301, CT-000624
No overlap data provided for other pairs.

**<2% reply failure mode**
Cold Outbound - HR Leaders step1 0.8%, step2 0.3%, step3 0.2% with meeting rate 0% across all steps.
Cold Outbound - People Ops step3 1.6% reply.
Data shows declining open and reply with step and zero meetings for HR Leaders.

**One change per weak sequence**
New Logo Nurture: review step 3 content/delivery; weakest reply 4.2%.
Expansion Nurture: fix tracking for step 2 opened > sent before interpreting performance.
Cold Outbound - HR Leaders: pause or audit targeting/dedupe; 0 meetings and reply <1%.
Cold Outbound - People Ops: review step 3; reply 1.6%.

Fix first: Cold Outbound - HR Leaders. Zero meetings and reply <2% on all steps with 21-contact overlap to Cold Outbound - People Ops.
weekly-marketing-goals-rollupPASS 1.00

? The Question

Produce the weekly marketing goals update for the active quarter: SQMs, SQOs, DS2s, closed-lost MIA rate, same-quarter close count, active pipeline coverage against target. For each: QTD actual, target, delta, pace (on/behind/ahead given days elapsed). One paragraph on what moved this week.
cost $0.0055 · 18s · in 3,891 / out 2,890 tok
✓ metrics_present✓ deltas✓ coverage_ratio✓ pace✓ paragraph

✓ Correct Answer (ground truth)

days_elapsed: 66
days_in_quarter: 92
elapsed_fraction: 0.7174
metrics: {"SQMs": {"qtd_actual": 230, "target": 300, "delta": -70, "expected_at_day_elapsed": 215.22, "pace": "ahead"}, "SQOs": {"qtd_actual": 84, "target": 120, "delta": -36, "expected_at_day_elapsed": 86.09,
coverage_ratio_active_pipeline: 0.75
pace_rule: expected = target * days_elapsed/days_in_quarter; within +/-5% of expected = on, above = ahead, below = behind; lower_better metrics invert.
this_week_paragraph_required: True
graded checks: metrics_present · deltas · coverage_ratio · pace · paragraph

Model's actual answer

show the model's full answer
Q3-2026 active quarter
Days elapsed 66 / 92 = 0.7174 = 71.7% of quarter

SQMs
QTD actual 230, target 300, delta -70
Expected at pace = 300 * 66/92 = 215.22
230 >= 215.22 → pace ahead

SQOs
QTD actual 84, target 120, delta -36
Expected at pace = 120 * 66/92 = 86.09
84 < 86.09 → pace behind

DS2s
QTD actual 40, target 75, delta -35
Expected at pace = 75 * 66/92 = 53.80
40 < 53.80 → pace behind

closed_lost_mia_rate
closed_lost_mia 5, closed_lost_total 25
Actual rate = 5 / 25 = 0.20
Target 0.10, direction lower_better
Delta = 0.20 - 0.10 = +0.10
Actual > target → pace behind

same_quarter_closes
QTD actual 10, target 20, delta -10
Expected at pace = 20 * 66/92 = 14.35
10 < 14.35 → pace behind

active_pipeline
QTD actual 3,000,000, target 4,000,000, delta -1,000,000
Coverage vs target = 3,000,000 / 4,000,000 = 75.0%
Expected at pace = 4,000,000 * 66/92 = 2,869,565.22
3,000,000 >= 2,869,565.22 → pace ahead

What moved this week
Weekly change data is missing. The files provided contain only QTD actuals, targets, and quarter meta with days elapsed. No prior-week values or week-over-week deltas are included, so what moved this week cannot be determined from the data provided.
communication
ceo-slack-compressionPASS 1.00

? The Question

Compress the forecast analysis below into a single Slack message for the CEO. Rules: at most three sentences; under 80 words; lead with the decision or the so-what; keep every distinct number that matters and the one caveat that changes the conclusion; cut any sentence that does not carry a claim, a number, or a directive. No em dashes, no throat-clearing phrases.
cost $0.0047 · 15s · in 3,662 / out 2,854 tok
✓ max_3_sentences✓ under_80_words✓ weighted_present✓ caveat_present✓ no_em_dash

✓ Correct Answer (ground truth)

graded checks: max_3_sentences · under_80_words · weighted_present · caveat_present · no_em_dash

Model's actual answer

show the model's full answer
Do not approve Q3 forecast: weighted forecast is $115,976.75 from $44,729 COMMIT on 7 in-quarter deals plus 35% of $203,565 BEST_CASE on 24 deals out of 54 in-quarter deals of 86. 32 deals totaling $227,575 excluded post-9/30 including COMMIT Deal-D348E1 $13,770 dated 2026-10-15. Why-buys empty on 71 of 86 deals including all 7 COMMIT deals and owner blank on 85 of 86 so forecast cannot be validated.
stalled-deal-followup-emailPASS 1.00

? The Question

Write a follow-up email for this stalled deal. Under 120 words. Reference one specific prior touch from the engagement history, make one clear ask, and match the tone of the prior thread. No invented details.
cost $0.0033 · 16s · in 3,492 / out 1,616 tok
✓ word_cap✓ references_touch✓ single_ask

✓ Correct Answer (ground truth)

deal_alias: Deal-0D2F7A
amount: 5100
stage: DS3
owner: Alex Franklin
days_since_last_engagement: 31
word_cap: 120
must_reference: engagement_id=ENG-00002; date=2026-07-28; type=meeting; why=most substantive prior touch; concrete prospect reactions to milestone awards and points catalog
single_cta: True
tone: warm, concise, professional (matches prior thread)
allowed_facts: 14 items — e.g. Deal-0D2F7A, $5,100, DS3, Alex Franklin
allowed_new: ENG-00001, ENG-00002, ENG-00003, ENG-00004, ENG-00005
graded checks: word_cap · references_touch · single_ask

Model's actual answer

show the model's full answer
Subject: Following up on July 28 demo recap

Hi,

Following up on the recap I emailed on August 5 after our July 28 demo with pricing for 150 seats. No reply yet.

You had a strong reaction to automated milestone awards and the points catalog. Would you be open to a 15-minute check-in this week to confirm next steps?

Best,
Alex Franklin
funnel-digest-paragraphPASS 1.00

? The Question

Write the weekly GTM digest: one paragraph per team (Marketing, Sales, CS, PLG) in Bonusly brand voice. Every number must trace to the extracts. No customer contact names, no employee performance commentary, no compensation. Four paragraphs, under 400 words total.
cost $0.0024 · 9s · in 3,349 / out 759 tok
✓ four_paragraphs✓ word_cap✓ teams

✓ Correct Answer (ground truth)

required_paragraphs: Marketing, Sales, CS, PLG
allowed_numbers: 16 items — e.g. 2, 3, 9, 08
word_cap: 400
banned: customer contact names, employee performance commentary, compensation
graded checks: four_paragraphs · word_cap · teams

Model's actual answer

show the model's full answer
Marketing kept the momentum human and measurable this week. SQMs came in at 46 against a target of 52, with the webinar channel delivering 18 of those SQMs. Cost per SQM on paid search held at $150.

Sales stayed focused on quality conversations and pipeline. The team converted 14 SQOs and set 9 DS2 meetings, creating $310,000 in new pipeline. Same-quarter close count for the week was 3.

CS protected the relationship foundation. Renewals saved this week: 2. Team NPS moved to 61. There are 3 open red-flag accounts heading into next week.

PLG grew the top of funnel with product-led signals. PLG added 412 new signups with activation at 31 percent. 38 companies reached the aha moment of 10 recognition gives.
partner-digest-section0.67

? The Question

Write the partner digest for the period. Every partner in the roster appears once: active plays get status with numbers, quiet partners get one line. Partner-sourced pipeline count and amount must match the deal data. Nothing omitted, nothing added.
cost $0.0036 · 10s · in 3,542 / out 1,833 tok
✓ all_partners✗ pipeline_amount✓ quiet_one_line

✓ Correct Answer (ground truth)

roster: partner_name=Apex Rewards Co; status=active | partner_name=HRCloud Partners; status=active | partner_name=CultureBridge; status=active | partner_name=WorkWell Group; status=active | partner_name=Recogniq; status=quiet | partner_name=KudosWave; status=quiet … (+2 more)
per_partner_pipeline: {"Apex Rewards Co": {"count": 2, "amount": 275000}, "HRCloud Partners": {"count": 1, "amount": 140000}, "CultureBridge": {"count": 2, "amount": 135000}}
partner_sourced_pipeline: count=5; amount=550000
quiet_partners: Recogniq, KudosWave, PeopleFirst Advisors, TotalPerk
active_no_deals: WorkWell Group
rule: every roster partner named exactly once; quiet partners get one line; no partners outside roster.
graded checks: all_partners · pipeline_amount · quiet_one_line

Model's actual answer

show the model's full answer
Partner Digest: 2026-08-24 to 2026-09-04

Apex Rewards Co – active
Status: co-webinar locked for 09-15. Two sourced opps this week both in DS1; confirmed logged with UTM Source = Partner.
Partner-sourced pipeline: 2 opps, $275,000
- Deal-DDAAF2 $180,000
- Deal-2CF33E $95,000
Arithmetic: 180,000 + 95,000 = 275,000

HRCloud Partners – active
Status: integration referral closed security review; opp moved to DS2. One sourced opp this period.
Partner-sourced pipeline: 1 opp, $140,000
- Deal-F1CDA5 $140,000

CultureBridge – active
Status: lunch-and-learn produced two sourced opps; both early stage.
Partner-sourced pipeline: 2 opps, $135,000
- Deal-096E1D $60,000
- Deal-067213 $75,000
Arithmetic: 60,000 + 75,000 = 135,000

WorkWell Group – active
Status: wants to restart joint playbook in Q4; no sourced deals this period; planning call booked for 09-09.
Partner-sourced pipeline: 0 opps, $0

Recogniq – quiet

KudosWave – quiet

PeopleFirst Advisors – quiet

TotalPerk – quiet
rfp-answer-from-policyPASS 1.00

? The Question

Answer the 10 questionnaire items from the policy excerpts only. Cite the excerpt id for every answer. Where the excerpts do not answer the question, mark it unanswerable and say what document would; do not guess. Tone: vendor response, direct.
cost $0.0029 · 8s · in 3,540 / out 1,225 tok
✓ answered_count✓ unanswerable_flagged✓ citations✓ no_guess

✓ Correct Answer (ground truth)

answered_count: 7
answer_map: {"Q1": ["P1"], "Q2": ["P1"], "Q3": ["P2"], "Q4": ["P3"], "Q5": ["P4"], "Q6": ["P5"], "Q7": ["P6"]}
unanswerable: Q8, Q9, Q10
valid_citation_ids: P1, P2, P3, P4, P5, P6, P7, P8
rule: unanswerable items must be flagged and the document that would answer them named; do not guess.
graded checks: answered_count · unanswerable_flagged · citations · no_guess

Model's actual answer

show the model's full answer
Q1 Is customer data encrypted at rest? If so, how?
Yes. All customer data is encrypted at rest using AES-256. [P1]

Q2 Is data encrypted in transit?
Yes. Data in transit is encrypted with TLS 1.2 or higher. [P1]

Q3 Do you hold a SOC 2 report, and can we obtain a copy?
Yes. Bonusly maintains a SOC 2 Type II report, renewed annually. The report is available to prospects under NDA. [P2]

Q4 How long is customer data retained after contract termination?
Customer data is retained for the duration of the contract plus 90 days, after which it is deleted from production systems. [P3]

Q5 Where is your subprocessor list published and how are changes notified?
Current list of subprocessors is maintained at trust.bonusly.com. Customers receive 30 days notice before a new subprocessor is added. [P4]

Q6 Do you support SAML single sign-on?
Yes. SAML 2.0 single sign-on is available on the Enterprise plan. [P5]

Q7 What is your production backup schedule and retention?
Production databases are backed up daily. Backups are encrypted and retained for 35 days. [P6]

Q8 Do you hold an ISO 27001 certification?
Unanswerable from provided policy excerpts. The excerpts do not address ISO 27001 certification.

Q9 What is your contractual uptime SLA percentage?
Unanswerable from provided policy excerpts. The excerpts do not address contractual uptime SLA.

Q10 Will you sign a HIPAA Business Associate Agreement?
Unanswerable from provided policy excerpts. The excerpts do not address HIPAA Business Associate Agreement.
ops-maintenance
skill-file-reconciliationPASS 1.00

? The Question

Reconcile the skill set below against its manifest. Find and report (with severity CRITICAL/WARNING/INFO and an action type TRIM_DESC, UPDATE_BODY, DELETE_SKILL, MERGE, or REVIEW): (1) any two skills whose ALWAYS-trigger phrases overlap/duplicate, (2) any circular delegation chain between skills (name it), (3) any delegation target that does not exist (dangling, name it), (4) any version conflict and which skill should survive, (5) how many manifest descriptions exceed 1,024 characters, (6) any hardcoded page ids, dates, or person names in skill bodies, (7) manifest drift in BOTH directions (files with no manifest row, and manifest rows with no file). Make one proposal per finding; rewrite nothing.
cost $0.0299 · 76s · in 67,503 / out 8,286 tok
✓ duplicate_trigger_found✓ circular_chain✓ dangling_target✓ oversized_desc_count✓ hardcoded_values✓ manifest_drift✓ severity_and_action

✓ Correct Answer (ground truth)

graded checks: duplicate_trigger_found · circular_chain · dangling_target · oversized_desc_count · hardcoded_values · manifest_drift · severity_and_action

Model's actual answer

show the model's full answer
Reconciliation against skill_manifest.csv and the 14 SKILL.md files supplied.

**1. Overlapping ALWAYS-trigger phrases**

* Finding: `comms-drafter` and `email-drafter` duplicate ALWAYS triggers.  
Both trigger on: “write me an email”, “draft a follow-up”, “help me reply”, “what should I say”, “bump email”, “contract nudge”, and on paste-and-review requests. `email-drafter` is a subset of `comms-drafter`’s external-communication scope.  
Severity: CRITICAL  
Action: MERGE — consolidate email drafting into `comms-drafter` and retire `email-drafter`, or UPDATE_BODY to carve a non-overlapping boundary.

* Finding: `pipeline-intelligence-report` and `weekly-pipeline-report` overlap.  
`pipeline-intelligence-report` ALWAYS triggers for: “run the pipeline report”, “pipeline review”, “pipeline intelligence”, “score the pipeline”, “full pipeline”, “pipeline update”, “what’s the pipeline look like”.  
`weekly-pipeline-report` ALWAYS triggers for: “run the pipeline update”, “weekly pipeline report”, “pipeline summary”, “generate the pipeline report”, “what does pipeline look like”, “update the pipeline”.  
Severity: WARNING  
Action: UPDATE_BODY — disambiguate triggers; reserve “score/tiered/8-signal” language for `pipeline-intelligence-report` and “weekly/Ben Lavin/HTML KPI strip” language for `weekly-pipeline-report`.

**2. Circular delegation chain**

* Finding: `email-drafter` ↔ `deal-strategy-coach` loop.  
`email-drafter` Lane marker: “If the user needs strategic deal coaching … point them to the `deal-strategy-coach` skill.”  
`deal-strategy-coach` Manager-to-prospect email frameworks: “When drafting manager-to-prospect emails, use the `email-drafter` skill which automatically retrieves your Gmail signature…”.  
Severity: WARNING  
Action: UPDATE_BODY — break the loop by making `deal-strategy-coach` own strategy only and delegate drafting unconditionally to `email-drafter`/`comms-drafter` without a return reference.

**3. Dangling delegation targets**

* Finding: `bonusly-brand` is required but absent.  
`comms-drafter` Step 0: “Before drafting any communication, apply the `bonusly-brand` skill.”  
`email-drafter`: “Before drafting, apply the bonusly-brand org skill…”.  
`sales-forecast`: “Always reference `bonusly-brand` skill for full brand guidance.”  
`weekly-pipeline-report`: “Read the SignalForge design system … Also reference `bonusly-brand`.”  
`bonusly-brand` is not in the manifest.  
Severity: CRITICAL  
Action: REVIEW — add `bonusly-brand` to manifest or UPDATE_BODY to remove the hard dependency.

* Finding: `prospect-research-multithreading` referenced but not in manifest.  
`comms-drafter`: “If researching a new partner contact, invoke `prospect-research-multithreading` first.” / “invoke `prospect-research-multithreading` in Contact Lookup mode first.”  
`deal-strategy-coach`: “Invoke **prospect-research-multithreading** whenever the diagnosis surfaces…”.  
`email-drafter`: “invoke **prospect-research-multithreading** in Contact Lookup mode first.”  
Severity: WARNING  
Action: REVIEW — add skill to manifest or UPDATE_BODY to remove references.

* Finding: `signalforge-reports` referenced but not in manifest.  
`pipeline-intelligence-report` Phase 5: “Read `/mnt/skills/organization/signalforge-reports/SKILL.md` … Read `/mnt/skills/organization/signalforge-reports/DESIGN-SYSTEM.md` … Embed `/mnt/skills/organization/signalforge-reports/signalforge.css`”.  
Severity: WARNING  
Action: REVIEW — add `signalforge-reports` to manifest or UPDATE_BODY to reference an existing skill.

**4. Version conflict**

No conflicting version declarations found across the 14 skills.  
`analysis-validator` declares Version: 3.6. `pipeline-intelligence-report` declares version: v6 · May 2026. No duplicate name with differing version.  
Severity: INFO  
Action: REVIEW — none required.

**5. Manifest description length > 1,024 characters**

Manifest `description_chars` values:
656, 897, 996, 792, 965, 676, 945, 1004, 1006, 962, 1006, 708, 762, 656  
Maximum = 1,006. 1,006 < 1,024.  
Count exceeding 1,024: 0 / 14.  
Severity: INFO  
Action: TRIM_DESC — not required.

**6. Hardcoded page IDs, dates, or person names in skill bodies**

* `partner-digest`: Hardcoded Confluence Cloud ID `73fe98de-a4a3-4869-9f8a-bb1eeed4cf7f`, Space ID `1958248479`, Partnerships Digest folder ID `2286616609`, page URLs with IDs.  
* `signalforge-feedback`: Hardcoded Page ID `2295136266`, Space ID `2232811524`, Cloud ID `73fe98de-a4a3-4869-9f8a-bb1eeed4cf7f`, Parent page ID `2234417154`.  
* `pipeline-intelligence-report`: Hardcoded HubSpot org ID `1973303`, AE owner IDs with person names Bryce Harmon 119337721, Dana Mercer 83155923, Cole Ingram 83155924, Alex Franklin 84342457, Gavin Porter 1520255671.  
* `analysis-validator`: Hardcoded GTM roster with person names and HubSpot Owner IDs — Alaina Loori 82535637, Bryce Harmon 119337721, Hugo Lindqvist 77260721, Dana Mercer 83155923, Alex Franklin 84342457, Cole Ingram 83155924, Gavin Porter 1520255671, Colleen Perry 77938470, etc.; hardcoded dates April 26, 2026, May 9, 2026, May 4, 2026.  
* `sales-forecast`: Hardcoded Google Spreadsheet IDs `1CLZeOsElVDF_LF0ZG_t2nfwvhnZ6bpwqM_nX3WEYzcw` and `1ENuaEcCuLjdKhMvp8FK3Ys1ek5Aw9ZuOZhsHJJFoB_k`.  
Severity: WARNING  
Action: UPDATE_BODY — externalize IDs/dates/person names to config/references.

**7. Manifest drift**

Files present: 14 SKILL.md files  
Manifest rows: 14  
file names match declared_name 1:1:
analysis-validator, closed-lost-analysis, comms-drafter, deal-strategy-coach, email-drafter, model-selection, next-to-close, partner-digest, pipeline-intelligence-report, sales-forecast, signalforge-claim-compressor, signalforge-feedback, stale-pipeline-report, weekly-pipeline-report

Files with no manifest row: 0  
Manifest rows with no file: 0  
Severity: INFO  
Action: REVIEW — no drift detected.
runbook-from-slack-thread0.75

? The Question

Turn the resolved incident thread into a runbook: ordered steps with the exact command or action taken, who ran it, how success was verified, and rollback for each step that changed state. Every step must trace to a message in the thread. Add nothing the thread does not contain; where the thread is unclear, mark the step as needing confirmation.
cost $0.0039 · 14s · in 3,704 / out 2,092 tok
✗ ordered_steps✓ rollback✓ needs_confirmation✓ sourced

✓ Correct Answer (ground truth)

steps: order=1; msg=M01; actor=Bryce Harmon; action=acknowledge PagerDuty alert, take IC; state_change=False; verified_by=M01 | order=2; msg=M02; actor=Farid Osman; action=check queue depth; command=bundle exec rake sidekiq:queue_depth; state_change=False; verified_by=M02 (48,213 pending) | order=3; msg=M04; actor=Farid Osman; action=pause auto-enqueue via feature flag; command=bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'; state_change=True; rollback=bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'; verified_by=M07 (queue depth falling) | order=4; msg=M05; actor=Elena Sinclair; action=cleared the dead set; state_change=True; needs_confirmation=thread gives no command or method for clearing the dead set | order=5; msg=M06; actor=Bryce Harmon; action=scale reward-worker to 6 replicas; command=kubectl scale deployment/reward-worker --replicas=6; state_change=True; rollback=kubectl scale deployment/reward-worker --replicas=3; verified_by=M07/M08 | order=6; msg=M08; actor=Cole Ingram; action=verify queue drained and error rate baseline; command=bundle exec rake sidekiq:queue_depth; state_change=False; verified_by=M08 (returns 0) … (+2 more)
ambiguous_step: order=4; msg=M05; mark=needs-confirmation
rule: no steps beyond what the thread contains; every step traces to a message id.
graded checks: ordered_steps · rollback · needs_confirmation · sourced

Model's actual answer

show the model's full answer
Runbook — #incident-2026-08-29-reward-queue

Step 1 — Acknowledge / take ownership
Who: Bryce Harmon
Action taken: PagerDuty alert fired for reward-worker queue depth > 10k. Acknowledging, taking IC.
Message: M01
Success verified: Not stated in thread.
Rollback: N/A — no state change.

Step 2 — Assess queue depth
Who: Farid Osman
Command: `bundle exec rake sidekiq:queue_depth`
Result observed: reward queue at 48,213 pending jobs. Normal is under 500.
Message: M02
Success verified: Output of command reported.
Rollback: N/A — read-only.

Step 3 — Check dead set
Who: Farid Osman
Action taken: Observed dead set
Result observed: Dead set has 112 jobs, all Redis::TimeoutError from around 13:58.
Message: M03
Success verified: Observation reported.
Rollback: N/A — read-only.

Step 4 — Pause enqueue to stop bleed
Who: Farid Osman
Command: `bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'`
Message: M04
Rollback provided in thread: `bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'`
Success verified: Not stated in thread for this step — needs confirmation.

Step 5 — Clear dead set
Who: Elena Sinclair
Action taken: Cleared out the dead set while in console.
Message: M05
Exact command / action: Not provided in thread — needs confirmation.
Success verified: Not stated in thread — needs confirmation.
Rollback: Not provided in thread — needs confirmation. Step changed state.

Step 6 — Scale workers up
Who: Bryce Harmon
Command: `kubectl scale deployment/reward-worker --replicas=6` — was 3
Message: M06
Rollback provided in thread: `kubectl scale deployment/reward-worker --replicas=3`
Success verified: Not stated in thread for this step — needs confirmation.

Step 7 — Monitor queue decline
Who: Farid Osman
Observation: Queue depth down to 9,400 and falling ~1,200/min.
Message: M07
Success verified: Observation reported.
Rollback: N/A — read-only.

Step 8 — Verify queue clear and error baseline
Who: Cole Ingram
Command: `bundle exec rake sidekiq:queue_depth`
Result observed: now returns 0. Error rate in Datadog back to baseline.
Message: M08
Success verified: Command output and Datadog error rate reported.
Rollback: N/A — read-only.

Step 9 — Re-enable enqueue
Who: Bryce Harmon
Command: `bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'`
Message: M09
Rollback provided in thread: Not provided — needs confirmation.
Success verified: Verified 40 new jobs processed cleanly in the next 3 minutes.
Message: M09

Step 10 — Scale workers back down
Who: Bryce Harmon
Command: `kubectl scale deployment/reward-worker --replicas=3`
Message: M10
Rollback provided in thread: Not provided — needs confirmation.
Success verified: Queue stable at 0. Incident resolved.
Message: M10
datadog-incident-timelinePASS 1.00

? The Question

From the log slice, sequence the failure: first error with timestamp, the cascade in order, the service and job involved, and the Datadog query that would confirm the first error. State what the logs do not show.
cost $0.0052 · 16s · in 4,732 / out 2,958 tok
✓ first_error_ts✓ service_named✓ cascade_ordered✓ query✓ gaps_stated

✓ Correct Answer (ground truth)

first_error: timestamp=2026-09-03T14:01:12Z; service=reward-service; message=Redis::TimeoutError: Connection to Redis at redis-primary:6379 timed out after 5s
cascade_order: service=reward-service; first_error_at=2026-09-03T14:01:12Z | service=sidekiq; first_error_at=2026-09-03T14:01:40Z | service=api-gateway; first_error_at=2026-09-03T14:03:05Z | service=web-app; first_error_at=2026-09-03T14:03:30Z
job_class_involved: RewardGiveJob
confirming_query: service:reward-service status:error "@message:Redis::TimeoutError"
logs_do_not_show: root cause of the Redis timeout, redis-server's own logs, infrastructure metrics (CPU/memory/network), any deploy or config change preceding the incident, who applied mitigation or when (no actor fields in logs)
clean_services: postgres
graded checks: first_error_ts · service_named · cascade_ordered · query · gaps_stated

Model's actual answer

show the model's full answer
**First error**

* 2026-09-03T14:01:12Z  service: reward-service  level: ERROR  
`Redis::TimeoutError: Connection to Redis at redis-primary:6379 timed out after 5s`

**Cascade in order from datadog_logs.csv**

1. 2026-09-03T14:01:12Z  reward-service  ERROR  Redis::TimeoutError: Connection to Redis at redis-primary:6379 timed out after 5s
2. 2026-09-03T14:01:20Z  reward-service  ERROR  Redis::TimeoutError: retry exhausted for RewardGiveJob
3. 2026-09-03T14:01:30Z  reward-service  ERROR  Redis::TimeoutError: retry exhausted for RewardGiveJob
4. 2026-09-03T14:01:40Z  reward-service  ERROR  Redis::TimeoutError: retry exhausted for RewardGiveJob
5. 2026-09-03T14:01:40Z  sidekiq  ERROR  RewardGiveJob failed: Redis::TimeoutError; retrying in 60s
6. 2026-09-03T14:02:28Z  sidekiq  ERROR  RewardGiveJob failed: Redis::TimeoutError; retrying
7. 2026-09-03T14:02:30Z  sidekiq  WARN  "Queue reward depth above 10,000"
8. 2026-09-03T14:03:05Z  api-gateway  ERROR  502 upstream timeout calling reward-service /gives
9. 2026-09-03T14:03:30Z  web-app  ERROR  Give form submission failed: upstream 502 from api-gateway
10. 2026-09-03T14:03:31Z  sidekiq  ERROR  RewardGiveJob failed: Redis::TimeoutError; retrying
11. 2026-09-03T14:03:48Z  api-gateway  ERROR  502 upstream timeout calling reward-service
12. 2026-09-03T14:04:13Z  api-gateway  ERROR  502 upstream timeout calling reward-service
13. 2026-09-03T14:04:22Z  sidekiq  ERROR  RewardGiveJob failed: Redis::TimeoutError; retrying
14. 2026-09-03T14:04:45Z  web-app  ERROR  Give form submission failed: upstream 502
15. 2026-09-03T14:05:16Z  api-gateway  ERROR  502 upstream timeout calling reward-service
16. 2026-09-03T14:05:26Z  sidekiq  ERROR  RewardGiveJob failed: Redis::TimeoutError; retrying
17. 2026-09-03T14:05:42Z  web-app  ERROR  Give form submission failed: upstream 502
18. 2026-09-03T14:06:47Z  sidekiq  ERROR  RewardGiveJob failed: Redis::TimeoutError; retrying
19. 2026-09-03T14:06:49Z  web-app  ERROR  Give form submission failed: upstream 502
20. 2026-09-03T14:06:52Z  api-gateway  ERROR  502 upstream timeout calling reward-service

Recovery signals:
* 2026-09-03T14:22:10Z  reward-service  INFO  Redis connection restored; resuming job processing
* 2026-09-03T14:24:45Z  sidekiq  INFO  Queue reward depth below 500

Time arithmetic:
* First error to first 502: 2026-09-03T14:03:05Z - 2026-09-03T14:01:12Z = 1 min 53 s
* First error to Redis restored: 2026-09-03T14:22:10Z - 2026-09-03T14:01:12Z = 20 min 58 s

Service and job involved
* Service: reward-service
* Job class first seen in errors: RewardGiveJob
* Sidekiq failures logged for RewardGiveJob and, per sidekiq_jobs.csv, also RecognitionDigestJob:
  - RewardGiveJob failed_at 2026-09-03T14:01:46Z to 2026-09-03T14:02:57Z
  - RecognitionDigestJob failed_at 2026-09-03T14:02:36Z, 2026-09-03T14:03:15Z, 2026-09-03T14:04:55Z, 2026-09-03T14:05:50Z
All with error Redis::TimeoutError.

Datadog query to confirm first error
`service:reward-service level:error "Redis::TimeoutError: Connection to Redis at redis-primary:6379 timed out after 5s"`
Equivalent: `service:reward-service level:error @timestamp:2026-09-03T14:01:12Z "Redis::TimeoutError"`

What the logs do not show
* No logs from service redis-primary; no Redis server metrics, network, CPU/memory, or restart events.
* No root cause for the Redis connection timeout; no action taken to restore connection is logged.
* No request IDs linking api-gateway 502s to specific RewardGiveJob executions.
* No count of affected users, requests, or failed submissions.
* No datadog_logs.csv entries for RecognitionDigestJob errors; sidekiq_jobs.csv shows RecognitionDigestJob failures but they are absent from the log slice.
* No logs explaining why Queue reward depth exceeded 10,000 or how it returned below 500.
* No logs for web-app job enqueue at 2026-09-03T13:58:49Z and reward-service job enqueue at 2026-09-03T13:59:30Z linking to the later failures.
feature-flag-state-summaryPASS 1.00

? The Question

Summarize the state of the listed feature flags: what each controls per the code excerpt, which companies or segments have it on, targeting rules, and any flag with no code reference or no targets. Company counts must match the export. Do not describe a flag that is not in the export.
cost $0.0037 · 10s · in 3,575 / out 1,930 tok
✓ orphans✓ no_targets✓ counts_match✓ no_invented

✓ Correct Answer (ground truth)

flags: {"recognition_streaks_v2": {"state": "on", "targeting_rules": "segment:beta_companies", "company_count": 42}, "points_budget_guardrails": {"state": "on", "targeting_rules": "all_companies", "company_c
code_referenced: recognition_streaks_v2, points_budget_guardrails, slack_dm_nudges, redeem_flow_redesign, analytics_dashboard_v3, ms_teams_app_v2
orphan_flags_no_code_reference: legacy_give_modal, survey_boosters_q3, paused_offboard_cleanup
flags_with_no_targets: paused_offboard_cleanup
rule: company counts must equal the export; do not describe flags absent from the export.
graded checks: orphans · no_targets · counts_match · no_invented

Model's actual answer

show the model's full answer
**flags_export.csv summary — state, control per code excerpt, targeting, company count**

### Flags with code reference

**recognition_streaks_v2**
- state: on
- targeting_rules: segment:beta_companies
- company_count: 42
- controls per code: `if FeatureFlags.enabled?("recognition_streaks_v2", company: company)` → `StreakTracker.record(give)` in `app/models/recognition.rb`
- On for: segment:beta_companies, 42 companies per export

**points_budget_guardrails**
- state: on
- targeting_rules: all_companies
- company_count: 220
- controls per code: `if FeatureFlags.enabled?("points_budget_guardrails", company: company)` → `BudgetService.new(company).enforce!(giver, points)` in `app/services/budget_service.rb`
- On for: all_companies, 220 companies per export

**slack_dm_nudges**
- state: on
- targeting_rules: segment:region_na
- company_count: 87
- controls per code: `return unless FeatureFlags.enabled?("slack_dm_nudges", company: company)` → `SlackDm.send_nudge(user)` in `app/jobs/nudge_job.rb`
- On for: segment:region_na, 87 companies per export

**redeem_flow_redesign**
- state: off
- targeting_rules: targeted_list
- company_count: 12
- controls per code: `if FeatureFlags.enabled?("redeem_flow_redesign", company: company)` → `render RedeemV2Component.new(company)` else `render RedeemV1Component.new(company)` in `app/controllers/redeem_controller.rb`
- Currently off. Targeting rules = targeted_list, 12 companies per export would be eligible when on.

**analytics_dashboard_v3**
- state: on
- targeting_rules: segment:tier_three
- company_count: 65
- controls per code: `if FeatureFlags.enabled?("analytics_dashboard_v3", company: company)` → `@dashboard = AnalyticsV3.new(company)` in `app/controllers/analytics_controller.rb`
- On for: segment:tier_three, 65 companies per export

**ms_teams_app_v2**
- state: off
- targeting_rules: targeted_list
- company_count: 9
- controls per code: `if FeatureFlags.enabled?("ms_teams_app_v2", company: company)` → `TeamsAppV2.install(company)` in `app/services/teams_installer.rb`
- Currently off. Targeting rules = targeted_list, 9 companies per export would be eligible when on.

### Flags with no code reference in flag_code.md

**legacy_give_modal**
- state: off
- targeting_rules: segment:legacy_plan
- company_count: 14
- controls per code: No code reference provided
- On/off status: off

**survey_boosters_q3**
- state: on
- targeting_rules: segment:legacy_plan
- company_count: 7
- controls per code: No code reference provided
- On/off status: on

**paused_offboard_cleanup**
- state: off
- targeting_rules: [empty]
- company_count: 0
- controls per code: No code reference provided
- No targets: targeting_rules is empty and company_count is 0

Company counts are as in export. No company-level names provided in the export; only segments/targeting_rules are given.
nda-triagePASS 1.00

? The Question

Triage each NDA as GREEN (standard approval), YELLOW (counsel review), or RED (full legal review). Cite the clause that drives each classification by quoting it. Catch embedded non-solicits, non-competes, missing carve-outs, and unusual term or governing law. One line of reasoning per NDA.
cost $0.0037 · 11s · in 3,628 / out 1,943 tok
✓ green✓ yellow✓ red✓ non_solicit_caught✓ clause_quoted✓ correct_labels

✓ Correct Answer (ground truth)

classifications: {"NDA-1": {"classification": "GREEN", "driving_reason": "standard mutual NDA: 2-year term, standard carve-outs, Delaware law", "driving_clause_quote": "Carve-outs: information that (a) is or becomes p
non_solicit_in: NDA-2
checks: classification_per_nda, clause_quotes_exist_in_document, non_solicit_caught, carveouts_checked
graded checks: green · yellow · red · non_solicit_caught · clause_quoted · correct_labels

Model's actual answer

show the model's full answer
NDA-1 — GREEN — Standard mutual NDA with standard carve-outs and Delaware law; no non-solicit / non-compete. Clause: "Carve-outs: information that (a) is or becomes publicly available through no breach, (b) was known prior to disclosure, (c) is received from a third party without duty of confidence, (d) is independently developed, or (e) must be disclosed by law or court order." Term exposure = 2 years term + 3 years survival = 5 years total.

NDA-2 — YELLOW — Counsel review due to embedded non-solicitation. Clause: "During the term of this Agreement and for eighteen (18) months thereafter, neither party shall solicit for employment or hire any employee of the other party with whom it came into contact under this Agreement."

NDA-3 — RED — Full legal review due to one-way confidentiality, non-compete, missing carve-outs, 5-year term and non-US governing law/jurisdiction. Clause: "For a period of three (3) years following the Effective Date, Recipient shall not, directly or indirectly, engage in or provide services to any business that competes with Discloser's business." Additional drivers: "One-way confidentiality in favor of [PARTY C]. Term: five (5) years."; "No confidentiality carve-outs are stated in this Agreement."; "This Agreement shall be governed by and construed in accordance with the laws of the Republic of Ireland, and the parties submit to the exclusive jurisdiction of its courts."