home · ← back to all models
openai

openai/gpt-6-astra-pro

Mean 0.941 · 32/40 perfect tests · $43.54 total · each card: the question → the correct answer (gold) → this model's actual answer

data-crm
stage-hygiene-auditPASS 1.00

? The Question

Audit stage hygiene on the open pipeline below. Rules: a deal at DS2 or later must have a substantive why-buys (15+ characters, not a placeholder such as tbd, n/a, see notes). A why_buys value of the form LONG(n) means the real text was present but truncated for display; treat it as present and substantive. Every deal should have a DS1 entry timestamp. A deal whose stage entry timestamps are out of order regressed (an earlier-stage timestamp is later than a later-stage timestamp). Report: percent of DS2+ deals non-compliant on why-buys overall and by owner (use the owners table for names), count of deals with no DS1 timestamp, count that regressed, and three process fixes. State definitions before numbers.
cost $1.3345 · 61s · in 94,138 / out 4,514 tok
✓ noncomp_pct✓ no_ds1_0✓ regressed_0✓ per_owner_breakdown✓ definitions_first✓ three_fixes

✓ Correct Answer (ground truth)

noncomp_pct: 81.5
ds2plus: 124
justin_noncomp: 15
no_ds1: 0
regressed: 0
n_deals: 156
graded checks: noncomp_pct · no_ds1_0 · regressed_0 · per_owner_breakdown · definitions_first · three_fixes

Model's actual answer

show the model's full answer
Definitions:
- DS2+ means deals currently at DS2–DS5. Why-buys is non-compliant if blank, fewer than 15 trimmed characters, or a placeholder (e.g., tbd, n/a, see notes). LONG(n) is substantive.
- Non-compliance rate = non-compliant DS2+ deals ÷ all DS2+ deals in the same group × 100.
- Missing DS1 means t_ds1 is blank, across all open deals.
- Regression means any populated earlier-stage timestamp is strictly later than a populated later-stage timestamp. Equal timestamps are valid; missing timestamps alone do not establish regression.

Why-buys non-compliance:
- Overall: 101 ÷ 124 × 100 = 81.45%.
- Bryce Harmon: 15 ÷ 18 × 100 = 83.33%.
- Dana Mercer: 18 ÷ 22 × 100 = 81.82%.
- Alex Franklin: 48 ÷ 61 × 100 = 78.69%.
- Cole Ingram: 19 ÷ 22 × 100 = 86.36%.
- Farid Osman: 1 ÷ 1 × 100 = 100%.
- Elena Sinclair: N/A—no DS2+ deals.
- Gavin Porter and Hugo Lindqvist: N/A—no open deals.

Timestamp hygiene:
- No DS1 timestamp: 0 of 156 deals.
- Regressed: 0 of 156 deals under the defined timestamp test.

Three process fixes:
1. Gate DS2+ advancement on substantive why-buys; reject blanks, short text, and placeholders. Assign existing gaps to owners.
2. Automatically capture DS1 entry timestamps; audit missing values and backfill only from verified history.
3. Preserve immutable stage-transition history and flag timestamp inversions for review rather than overwriting evidence of regression.
owner-resolutionPASS 1.00

? The Question

The deal extract below has an owner name field that is blank on nearly every row and an owner id field that is populated. Using the owners table, resolve each open deal to an owner name. Report: number of deals resolved, deals whose owner id has no match in the owners table (list them), the owner ids that map to archived/deactivated owners (list their names), and the total pipeline amount per resolved owner. Do not guess a name for any unmatched id.
cost $1.4079 · 64s · in 63,780 / out 4,498 tok
✓ resolved_count✓ unmatched_none✓ deactivated_flagged✓ jake_amount✓ justin_amount✓ no_invented

✓ Correct Answer (ground truth)

resolved: 156
jake_amount: 1054144.0
justin_amount: 624310.0
per_owner: Bryce Harmon=1054144.0; Alex Franklin=624310.0; Dana Mercer=341195.0; Cole Ingram=288161.43; Farid Osman=4134.0; Elena Sinclair=2100.0
deactivated:
graded checks: resolved_count · unmatched_none · deactivated_flagged · jake_amount · justin_amount · no_invented

Model's actual answer

show the model's full answer
Resolved: 156 of 156 open deals, joining deals_open.owner_id to owners.owner_id.
Arithmetic: 35 + 24 + 67 + 22 + 7 + 1 = 156.

Unmatched owner IDs/deals: None.

Archived owners in owners.csv:
- 1520255671 — Gavin Porter
- 77260721 — Hugo Lindqvist

Neither archived owner has any deals in the provided open-deal extract.

Total pipeline per resolved owner (sum of amount across all their open deals; currency not specified):

Owner ID    Owner name           Deals    Pipeline amount
119337721   Bryce Harmon              35       1,054,144.00
83155923    Dana Mercer              24         341,195.00
84342457    Alex Franklin           67         624,310.00
83155924    Cole Ingram      22         288,161.43
716654662   Farid Osman              7           4,134.00
701163055   Elena Sinclair            1           2,100.00

Pipeline reconciliation:
1,054,144.00 + 341,195.00 + 624,310.00 + 288,161.43 + 4,134.00 + 2,100.00 = 2,314,044.43.
gong-hubspot-join-integrityPASS 1.00

? The Question

Check the integrity of the call-to-deal mapping below. The gong table lists calls in the last 90 days keyed to HubSpot deal ids; deals_open lists the current open default-pipeline deals. Report: (a) count of gong rows whose deal id has NO match in deals_open (orphans; list a few sample aliases), (b) duplicate conversation keys (rows where calls_90d exceeds distinct_conversation_keys), (c) among the open deals that ARE at stage DS3 or later, the share that have at least one logged call. Give sample aliases for the orphan finding, all taken from the data. Note: gong_owner_id is a Gong user id, not a HubSpot owner id.
cost $1.5110 · 61s · in 107,135 / out 5,542 tok
✓ orphan_count✓ dup_keys_0✓ ds3_coverage✓ sample_ids_present

✓ Correct Answer (ground truth)

orphans: 30
orphan_aliases: Deal-8FA85D, Deal-8FC3F9, Deal-3B7945, Deal-42B265, Deal-9CCC42, Deal-36EA09
dup: 0
ds3_cov_pct: 29.4
graded checks: orphan_count · dup_keys_0 · ds3_coverage · sample_ids_present

Model's actual answer

show the model's full answer
(a) Orphans: 67 Gong rows − 37 matched to deals_open = 30 orphan rows, joining hs_deal_id to deal_id.
Sample aliases: Deal-8FA85D, Deal-8FC3F9, Deal-3B7945, Deal-42B265, Deal-9CCC42. These are unmatched to the supplied open-deal table, not necessarily invalid deals.

(b) Duplicate conversation keys: 0 rows where calls_90d > distinct_conversation_keys. All 67 rows have equal counts; total excess = 0. Individual conversation keys were not provided, so cross-deal duplication cannot be checked.

(c) Current DS3+ call coverage:
- Eligible open deals: 61 DS3 + 14 DS4 + 10 DS5 = 85.
- With at least one logged call: 12 DS3 + 6 DS4 + 7 DS5 = 25.
- Share: 25 ÷ 85 × 100 = 29.41%.
snowflake-sql-trapped-catalog0.50

? The Question

Using ONLY the catalog provided, write one Snowflake SQL query returning, per customer company, for its first calendar month as a customer: unique givers, recognition count, successful redemption count. Read the catalog notes carefully: some tables are stale or unpopulated and must be avoided, and one documented filter (excluding deleted givers) must NOT be applied to historical giving counts. Put a comment above each FROM or JOIN explaining the table choice. Return only fenced SQL.
cost $0.3047 · 20s · in 24,824 / out 811 tok
✓ avoids_stale_deals✓ avoids_unpopulated_live✗ uses_cohort_summary✗ succeeded_filter✓ no_giver_deletion_filter✗ returns_sql

✓ Correct Answer (ground truth)

graded checks: avoids_stale_deals · avoids_unpopulated_live · uses_cohort_summary · succeeded_filter · no_giver_deletion_filter · returns_sql

Model's actual answer

show the model's full answer
```sql
-- Cannot construct the requested query from this catalog alone.
-- Missing:
--   1. A customer-company source with a company identifier and the date
--      the company became a customer.
--   2. A recognition-event source with company ID, giver ID, event timestamp,
--      and an event identifier or documented row grain.
--   3. Company ID and event timestamp columns, plus an event identifier or
--      documented row grain, for
--      PRODUCTION.DEPRECATED_RECOGNITION.REDEMPTION_RECORDS_V2.
--
-- PRODUCTION.PLG.COMPANY_COHORT_SUMMARY covers only self-serve companies;
-- M1_* is not documented as the first calendar month as a customer,
-- and no unique-giver metric or company identifier is documented.
--
-- Once the missing schema is provided:
--   Count distinct givers without excluding deleted givers.
--   Count recognition events separately from redemption events to avoid fanout.
--   Use the documented redemption source with STATE = 'succeeded' only.
--   Bound events to the calendar month containing the customer-start date.
--   Avoid all sources marked stale, unpopulated, or retired.
```
crm-completeness-enrichmentPASS 1.00

? The Question

Audit the CRM extract for completeness and propose fixes. Deals need owner, stage, amount, close date, why-buys; companies need industry, employee count, HQ country; contacts need email, title, persona. Report completeness percent per field. Identify duplicate company clusters (name variants, shared domain) with a survivor per cluster. List invalid emails and domain mismatches. Fill missing company fields only where the enrichment export has a matching row; where CRM and enrichment disagree list both and recommend a source. Never invent a value. End with the 10 fixes with the most pipeline amount at stake.
cost $1.2957 · 166s · in 43,979 / out 10,072 tok
✓ dup_clusters✓ invalid_emails✓ survivor_language✓ completeness_pct✓ no_invent

✓ Correct Answer (ground truth)

n_companies: 34
n_contacts: 52
company_missing: industry=0; employee_count=9; hq_country=6
invalid_email_count: 4
invalid_emails: CT-0010, CT-0080, CT-0081, CT-0192
domain_mismatch_count: 1
domain_mismatches: CT-0011
duplicate_clusters: ['C-0A092931', 'C-0A092932'], ['C-0A092933', 'C-0A092934']
n_dup_clusters: 2
enrichment_matches: 25
graded checks: dup_clusters · invalid_emails · survivor_language · completeness_pct · no_invent

Model's actual answer

show the model's full answer
Completeness

Calculated on the original extract, before enrichment or deduplication. Complete means nonblank after trimming whitespace; validity is assessed separately.

Field                       Arithmetic                 Completeness
Deals: owner                No deals extract            Not calculable
Deals: stage                No deals extract            Not calculable
Deals: amount               No deals extract            Not calculable
Deals: close date           No deals extract            Not calculable
Deals: why-buys             No deals extract            Not calculable
Companies: industry         34 ÷ 34 × 100               100.00%
Companies: employee count   25 ÷ 34 × 100                73.53%
Companies: HQ country       28 ÷ 34 × 100                82.35%
Contacts: email             52 ÷ 52 × 100               100.00%
Contacts: title             39 ÷ 52 × 100                75.00%
Contacts: persona           37 ÷ 52 × 100                71.15%

Email syntax validity: (52 − 4) ÷ 52 × 100 = 92.31%. This does not establish deliverability.

Duplicate company clusters

Company names are not provided, so name-variant detection is unavailable. Shared domains identify these candidate duplicate clusters:

Domain         Members                    Proposed survivor
acme-corp.com  C-0A092931, C-0A092932       C-0A092931
globex.io      C-0A092933, C-0A092934       C-0A092933

Survivors use the lowest alias as a deterministic tie-breaker, not evidence of better data. Confirm they represent the same entity before merging; preserve associations and conflicting values.

• C-0A092931 versus C-0A092932: industry “Technology” versus “tech”; employee count 500 versus 510; HQ country “US” versus “USA”. Industry/country differences can be normalized; employee count requires verification.
• C-0A092933 versus C-0A092934: industry “SaaS” versus “Technology”; employee count 200 and HQ country “US” agree. Preserve the industry conflict pending taxonomy review.

Neither domain has an enrichment row. No supplied source resolves their substantive conflicts.

Invalid emails and domain mismatches

Contact   Company    Invalid email   Problem
CT-0010   C-66D1FC   user0@          Missing domain
CT-0080   C-92D97D   user0@          Missing domain
CT-0081   C-92D97D   user1@          Missing domain
CT-0192   C-425E2A   user2@          Missing domain

Obtain verified replacements; do not construct addresses by appending company domains.

CT-0011, C-66D1FC: email “user1@other-domain.com” conflicts with both the contact’s domain field and company domain, “66d1fc.com”. Verify the email and company association before changing either.

All supplied contact domain fields match their linked company domains. The four missing email domains are reported as invalid, not additional valid-domain mismatches.

Supported company fills

Using exact domain matches in zoominfo_enrichment.csv, fill employee_count = 400 for:

C-EC3025, C-96039F, C-44EA29, C-D04904, C-B23205, C-60C75F, C-7BBDFA, C-50D386.

These are eight supported fills. Employee-count completeness would become:
(25 + 8) ÷ 34 × 100 = 97.06%, before deduplication.

Leave these fields blank:

• C-93C8BF: employee count; no matching enrichment row.
• C-2D1F1B, C-D73B89, C-44EA29, C-D04904, C-2C60E5: HQ country; matching enrichment rows also have blank country.
• C-EE9FFB: HQ country; no matching enrichment row.

No HQ-country fill is supported; completeness remains 82.35%.

CRM versus enrichment differences

Industry — every listed enrichment value is “Computer Software”:

Companies                                      CRM value
C-66D1FC, C-44EA29, C-60C75F                     “tech”
C-EC3025, C-92D97D, C-D04904, C-77A95A,
C-AA8DDA, C-B25F40                              “Technology”
C-425E2A                                       “Tech ” (trailing space)

Recommendation: use the matching enrichment value for a standardized industry field if its taxonomy is adopted, while retaining the raw CRM value and source. “Computer Software” is more specific than “Technology”; the data does not establish which source is fresher, so do not silently treat them as equivalent.

HQ country — every listed enrichment value is “United States”:

Companies                                      CRM value
C-66D1FC, C-950043, C-77A95A, C-B23205,
C-D0662E                                       “US”
C-EC3025, C-96039F, C-E51FB7, C-425E2A,
C-2D7423                                       “USA”

Recommendation: use enrichment’s “United States” as the canonical label. These are formatting differences, not geographic disagreements.

There are no conflicting nonblank employee counts between CRM and matching enrichment rows.

Elsewhere, normalize “tech”/“Tech ” to “Technology” and “health care” to “Healthcare” under an explicit CRM taxonomy; do not infer “Computer Software” for companies without supporting enrichment.

Missing contact fields

Titles — obtain verified values for:
CT-0000, CT-0022, CT-0072, CT-0080, CT-0081, CT-0092, CT-0120, CT-0121, CT-0122, CT-0132, CT-0141, CT-0162, CT-0170.

Personas — obtain confirmed roles for:
CT-0000, CT-0022, CT-0041, CT-0060, CT-0070, CT-0081, CT-0082, CT-0092, CT-0110, CT-0132, CT-0162, CT-0171, CT-0172, CT-0180, CT-0181.

No contact enrichment was supplied. Do not infer persona from title.

10 fixes — pipeline ranking unavailable

No deals, amounts, or deal-to-company/contact associations were supplied. Therefore, the ten highest-pipeline fixes cannot be identified or valued. The following are unranked remediation candidates; pipeline amount at stake is unknown for every item, not zero.

1. Supply the linked deals extract; audit owner, stage, amount, close date, and substantive why-buys, then enforce required-field validation.
2. Resolve the acme-corp.com duplicate cluster into proposed survivor C-0A092931, preserving the 500-versus-510 employee-count conflict.
3. Resolve the globex.io duplicate cluster into proposed survivor C-0A092933, preserving the “SaaS”-versus-“Technology” industry conflict.
4. Replace the four invalid emails with verified addresses and add email-syntax validation.
5. Verify CT-0011’s email and association with C-66D1FC.
6. Apply the eight supported employee_count = 400 enrichment fills.
7. Obtain evidence for C-93C8BF’s missing employee count and the six unresolved HQ countries.
8. Resolve industry taxonomy differences and normalize country labels, retaining original values and provenance.
9. Obtain the 13 missing contact titles.
10. Obtain the 15 missing contact personas from confirmed stakeholder-role information.
deal-intelligence
closed-lost-classification0.80

? The Question

Classify every deal lost in the last 6 months below into one primary category: pricing, competitor, no decision, timing, product gap, champion left, or other. Also assign a side (Bonusly, buyer, unknown). Then summarize: category counts, the side split, the number of deals where the structured closed_lost_tag clearly disagrees with the free-text reason, and the two patterns most worth acting on. Base everything only on the tag and free-text provided.
cost $1.9998 · 202s · in 95,985 / out 15,206 tok
✓ total_90✗ competitor_top✓ category_counts_present✓ side_split✓ two_patterns

✓ Correct Answer (ground truth)

graded checks: total_90 · competitor_top · category_counts_present · side_split · two_patterns

Model's actual answer

show the model's full answer
Classification rules: specific reasons take precedence; tags supply a provisional category when text is vague. “No decision” includes ghosting, deprioritization, and choosing not to buy R&R. “Pricing” includes budget constraints. Side identifies the stated barrier—not blame: Bonusly for explicit offering/value limitations, buyer for internal constraints or preferences, unknown when attribution is unsupported. A competitor tag alone does not establish a Bonusly-side problem.

### Every supplied deal

| Deal alias | Primary category | Side | Basis |
|---|---|---|---|
| Deal-DB0AAC | timing | buyer | Paused; reconnect timeline pending |
| Deal-F7F635 | competitor | unknown | Tag; “another direction” is nonspecific |
| Deal-AC944F | no decision | unknown | Unresponsive |
| Deal-214060 | no decision | unknown | Unresponsive |
| Deal-91A056 | timing | buyer | Reconnect early in 2027 |
| Deal-29326C | timing | buyer | Timing |
| Deal-5DB9B0 | other | unknown | Spam |
| Deal-831B7B | timing | buyer | Revisit in the new year |
| Deal-F97C37 | competitor | Bonusly | Other vendor’s broader offerings |
| Deal-13E9CF | no decision | buyer | R&R deprioritized; explicitly not budget |
| Deal-39E25C | timing | buyer | Reconnect next year |
| Deal-7ED004 | pricing | buyer | Budget approval denied |
| Deal-21B045 | no decision | unknown | MIA |
| Deal-B3ABED | timing | buyer | Future revisit and budget planning |
| Deal-422BA6 | competitor | Bonusly | Other vendor’s preferred ADP partnership and benefits |
| Deal-ED9AE7 | other | buyer | Timing, budget, authority; no dominant reason stated |
| Deal-988493 | no decision | unknown | MIA |
| Deal-381C8C | competitor | unknown | Tag; text only confirms rejection |
| Deal-F308CA | no decision | unknown | No contact; ignored outreach |
| Deal-F1E8A6 | competitor | unknown | Tag; text only confirms rejection |
| Deal-B6AC09 | timing | buyer | Revisiting in 2027 |
| Deal-70F704 | no decision | unknown | MIA; anniversary-only interest does not establish a gap |
| Deal-E6E80A | timing | buyer | Pushed to early 2027 |
| Deal-B038F0 | timing | buyer | Pushed to early 2027 |
| Deal-4664E1 | no decision | unknown | No contact; ignored outreach |
| Deal-175756 | timing | buyer | Other priorities; hold until 2027 |
| Deal-E74A73 | no decision | buyer | Test manually before buying a platform |
| Deal-DDAB52 | competitor | Bonusly | Rippl’s breadth, same cost, easier currency budgeting |
| Deal-ACE061 | competitor | unknown | Tag; HeyTaco is the rep’s suspicion, not confirmation |
| Deal-BB78F3 | timing | buyer | Leadership wants survey action items first |
| Deal-D48E0B | no decision | unknown | MIA |
| Deal-15DA99 | timing | buyer | Revisit early 2027 |
| Deal-F4AF5D | timing | buyer | Revisit early next year |
| Deal-79B7A1 | timing | buyer | Timing |
| Deal-583ADB | no decision | unknown | MIA |
| Deal-8E27DA | no decision | buyer | Bought swag only; did not want R&R currently |
| Deal-2D2F8D | competitor | unknown | Tag; “different direction” is nonspecific |
| Deal-E0441F | no decision | unknown | No contact; departed rep was seller-side |
| Deal-7CB44D | no decision | unknown | No meaningful contact; ignored outreach |
| Deal-0F96AA | competitor | unknown | Tag; RFP elimination without a stated reason |
| Deal-1BCA50 | pricing | buyer | “Mostly” budget and gift-card details; another vendor secondary |
| Deal-7CC678 | competitor | unknown | Tag only; nothing specific provided |
| Deal-FAC17C | no decision | buyer | Executive IT Director’s final approval missing |
| Deal-242273 | competitor | Bonusly | Other vendors supported internal currency and onsite spending |
| Deal-50E5D8 | no decision | buyer | Leadership pause without a restart commitment |
| Deal-A2C349 | competitor | Bonusly | Stayed with Awardco and added its surveys |
| Deal-9F176A | timing | buyer | Paused until closer to year-end |
| Deal-7B2236 | pricing | Bonusly | Wanted simpler and cheaper |
| Deal-AFA56C | no decision | unknown | Unresponsive |
| Deal-C7156E | competitor | unknown | Selected another vendor; reason absent |
| Deal-C33D91 | pricing | buyer | Budget cuts prevented approval |
| Deal-9048EB | product gap | Bonusly | Explicit multiple feature gaps; also no contact |
| Deal-5E64CE | timing | buyer | Waiting for Nectar contract end; exit fee inhibits switching |
| Deal-8A0992 | competitor | unknown | Canadian provider “more closely aligns”; specifics absent |
| Deal-D0C698 | competitor | buyer | Client prefers returning to Kudos |
| Deal-69CF3D | timing | buyer | On hold; timing tag |
| Deal-ECBF89 | timing | buyer | On hold; timing tag |
| Deal-3618CC | other | unknown | Wanted surveys; no explicit gap or decision-maker departure |
| Deal-EECC02 | competitor | unknown | Tag; “another direction” is nonspecific |
| Deal-5AD03E | competitor | unknown | Tag; “budget access” is too vague to attribute |
| Deal-D1A623 | timing | buyer | Timing |
| Deal-413C56 | no decision | buyer | Back-to-school priority; CEO not ready |
| Deal-47F1A1 | competitor | buyer | Staying with WorkTango |
| Deal-BF2A98 | competitor | buyer | Already deployed HiThrive |
| Deal-2A292B | no decision | buyer | Building internally instead of buying |
| Deal-D1AABF | no decision | unknown | No response |
| Deal-FEDBCB | timing | buyer | Reconnect closer to year-end |
| Deal-1E7DA9 | competitor | unknown | Selected another platform; reason absent |
| Deal-2BBA21 | no decision | unknown | No contact; ignored nudges |
| Deal-286F9C | competitor | unknown | Another platform; unspecified fit issue |
| Deal-7FBAC6 | no decision | buyer | Leadership paused again |
| Deal-369281 | competitor | buyer | Using existing Paylocity offering |
| Deal-386F6E | no decision | unknown | No response |
| Deal-9FCD0D | competitor | buyer | CEO preferred a Canadian company; no shortfall stated |
| Deal-55867E | timing | unknown | Tag only; text supplies no timing explanation |
| Deal-DAFB82 | pricing | buyer | Budget allocated elsewhere; not expected until 2028 |
| Deal-2FEDDB | timing | buyer | Uncertain timing |
| Deal-64B19A | competitor | unknown | Likely Motivosity; unconfirmed |
| Deal-3F86A0 | no decision | unknown | Unresponsive |
| Deal-096750 | no decision | unknown | No meaningful contact; ignored outreach |
| Deal-F325A5 | no decision | buyer | Layoffs/leadership change made this a nonpriority |
| Deal-ABD14C | no decision | buyer | Not interested in signing up |
| Deal-79E61A | no decision | unknown | Unresponsive |
| Deal-8A119B | pricing | buyer | Budget/price tag; approval denied, type unspecified |
| Deal-AE7C4E | no decision | unknown | Unresponsive |
| Deal-DAB4F1 | no decision | unknown | Unresponsive |
| Deal-B4B50F | no decision | unknown | Unresponsive |
| Deal-981AD4 | product gap | Bonusly | UI and UK-focus mismatch |
| Deal-DC77FE | competitor | Bonusly | Competitor’s customization; explicitly not price |
| Deal-5885B9 | no decision | unknown | MIA |

### Category counts

| Category | Deals |
|---|---:|
| pricing | 6 |
| competitor | 25 |
| no decision | 32 |
| timing | 22 |
| product gap | 2 |
| champion left | 0 |
| other | 3 |
| **Total** | **90** |

Arithmetic: 6 + 25 + 32 + 22 + 2 + 0 + 3 = 90.

No text explicitly confirms that a buyer-side champion left. “Lost DM” alone is insufficient; leadership change does not necessarily mean champion departure.

### Side split

| Side | Deals | Share |
|---|---:|---:|
| Bonusly | 9 | 9 ÷ 90 × 100 = 10.0% |
| buyer | 42 | 42 ÷ 90 × 100 = 46.7% |
| unknown | 39 | 39 ÷ 90 × 100 = 43.3% |

Arithmetic: 9 + 42 + 39 = 90.

### Clear tag–reason disagreements

**2 deals have a clear primary-reason coding mismatch:**

- **Deal-8E27DA:** “Feature Request” versus explicitly not wanting R&R currently and choosing swag only. This describes a buying-scope decision, not a stated missing feature.
- **Deal-9048EB:** “MIA” versus an explicit bad fit and multiple feature gaps. Ghosting also occurred, but the text supplies a substantive product-loss reason that the tag obscures.

Arithmetic: 1 + 1 = 2.

This counts primary-reason mismatches, not every unsupported tag. Vague competitor reasons, “Wanted Surveys” under “Lost DM,” and approval-related notes lack enough detail to prove disagreement. Competitor selection and pricing/product shortcomings can coexist. The compound “Doing nothing/Not a priority/Cost” tag is not contradicted merely because one component does not apply.

### Two patterns most worth acting on

1. **Prioritization, stalled engagement, and deferral dominate.**  
   No decision + timing = 32 + 22 = **54 of 90 deals, or 60.0%**. Use explicit restart milestones and approval paths for deferred opportunities; distinguish them from unresponsive opportunities. Examples: Deal-175756 has a stated hold until 2027, Deal-BB78F3 has prerequisite survey actions, Deal-FAC17C lacks final approval, and Deal-F308CA has ignored outreach rather than a confirmed restart.

2. **Specific competitive losses point to breadth, integrations, and customization—not blanket discounting.**  
   Deal-F97C37 cites broader offerings; Deal-422BA6 cites ADP partnership benefits; Deal-242273 cites internal currency and onsite spending; Deal-A2C349 cites Awardco surveys; Deal-DC77FE cites customization and explicitly excludes price. Tighten qualification and competitive positioning around these requirements, and validate whether each is a capability gap or a positioning issue before committing roadmap changes or discounts.
pipeline-tieringPASS 1.00

? The Question

Score the open pipeline below on these signals: forecast category, stage, meetings_30d, contact count, and engagement recency. Assign exactly one tier per deal: LOCK, ACTION, BUILD, REVIVE, WATCH, or RISKY (RISKY = the forecast category disagrees with the engagement evidence). Never tier a deal LOCK if it has zero meetings_30d. Do NOT list every deal. Return ONLY a compact JSON object with exactly these keys: - "tier_counts": object mapping each tier to the number of deals in it (must sum to the total deal count), - "tier_examples": object mapping each tier to up to 3 example deal aliases from the data, - "risky_deals": array of the deal aliases you tiered RISKY, - "lock_violations": number of deals tiered LOCK that have zero meetings_30d (should be 0), - "pipeline_shape": one short paragraph on the overall pipeline shape. Note: inbound_emails_30d is 0 for every row due to a data defect, so treat meetings_30d as the inbound signal.
cost $1.7062 · 105s · in 116,101 / out 6,441 tok
✓ json_returned✓ tier_vocabulary✓ tier_counts_sum✓ risky_deals_listed✓ no_lock_violation_claimed

✓ Correct Answer (ground truth)

total: 156
graded checks: json_returned · tier_vocabulary · tier_counts_sum · risky_deals_listed · no_lock_violation_claimed

Model's actual answer

show the model's full answer
{"tier_counts":{"LOCK":3,"ACTION":17,"BUILD":34,"REVIVE":11,"WATCH":55,"RISKY":36},"tier_examples":{"LOCK":["Deal-D348E1","Deal-C26D20","Deal-403845"],"ACTION":["Deal-25F752","Deal-3974EB","Deal-C6FE92"],"BUILD":["Deal-A5E80A","Deal-1FC049","Deal-499BF6"],"REVIVE":["Deal-2D1F1B","Deal-66D1FC","Deal-950043"],"WATCH":["Deal-6787C2","Deal-C9C286","Deal-332637"],"RISKY":["Deal-E53952","Deal-5408B0","Deal-9AAE5F"]},"risky_deals":["Deal-E53952","Deal-5408B0","Deal-9AAE5F","Deal-547B2B","Deal-944310","Deal-B7EBD1","Deal-A2B47C","Deal-2465CE","Deal-C61CF7","Deal-62D607","Deal-584EE5","Deal-C6D97A","Deal-7B3B0F","Deal-F9A08A","Deal-0660B4","Deal-FD9F4E","Deal-BA571A","Deal-FC22A3","Deal-7BBDFA","Deal-60C2C2","Deal-4A13AD","Deal-8AD4A5","Deal-15D24F","Deal-9D0060","Deal-690476","Deal-635B8E","Deal-ED725A","Deal-55164C","Deal-3BA5EA","Deal-5FDCE4","Deal-F336B6","Deal-5EED42","Deal-BA3DDC","Deal-7599B8","Deal-F9A3C1","Deal-FA32A0"],"lock_violations":0,"pipeline_shape":"Totals: 3+17+34+11+55+36=156; 99 have zero meetings, 55 have positive meetings, and 2 lack engagement rows (Deal-3EED2C and Deal-57FF13), so 99+55+2=156. Pipeline is meeting-light, with limited closing support. Snapshot date is missing; scoring assumes 2026-09-04 and excludes later engagement dates from recency while retaining supplied meeting counts. Analyst rubric, applied in order: missing evidence → WATCH; COMMIT/BEST_CASE with zero meetings or engagement older than 30 days → RISKY; DS5 COMMIT with meetings, at least 3 contacts and engagement within 14 days → LOCK; other DS3+ deals meeting those engagement/contact thresholds → ACTION; remaining meeting-positive deals with at least 2 contacts → BUILD; remaining deals inactive over 14 days → REVIVE; otherwise WATCH. Recency uses the latest nonfuture email, call or meeting; defective inbound-email counts are ignored."}
call-transcript-extractionPASS 1.00

? The Question

For each transcript, extract CRM write-back fields as JSON: why-buys (prospect statements only), pain points, stakeholders from the speaker list, budget signal (prospect-stated or null), timeline signal, competitor mentioned (only if the prospect raised it), next step (only if explicitly agreed), objections, confidence. Never fill a field from what the rep said or hoped.
cost $0.6211 · 43s · in 38,955 / out 3,614 tok
✓ json_emitted✓ budget_signal✓ null_discipline✓ stakeholders✓ competitor_prospect_only

✓ Correct Answer (ground truth)

rep: Alex Franklin
per_transcript: {"TX-001": {"why_buys": ["automating anniversary and birthday awards"], "pain_points": ["HR team of three cannot keep up with it manually", "track everything in a spreadsheet, and people slip through
rep_sourced_nulls: {"TX-003": ["budget_signal \u2014 only the rep mentioned pricing/budget"], "TX-004": ["next_step \u2014 rep proposed, prospect said 'no promises'"], "TX-006": ["budget_signal \u2014 rep decoy, prospec
allowed_new: TX-001, TX-002, TX-003, TX-004, TX-005, TX-006, Achievers, Bucketlist, Nectar
graded checks: json_emitted · budget_signal · null_discipline · stakeholders · competitor_prospect_only

Model's actual answer

show the model's full answer
[
  {
    "transcript_id": "TX-001",
    "deal_alias": "Deal-CFE7F4",
    "why_buys": [
      "The big win for us would be automating anniversary and birthday awards — our HR team of three cannot keep up with it manually."
    ],
    "pain_points": [
      "Our HR team of three cannot keep up with it manually.",
      "Right now we track everything in a spreadsheet, and people slip through the cracks."
    ],
    "stakeholders": [
      "Prospect (VP People)",
      "Prospect (HR Admin)"
    ],
    "budget_signal": "We have about $40k earmarked for engagement tools this fiscal year.",
    "timeline_signal": "Ideally we would have this live before open enrollment in November.",
    "competitor_mentioned": [
      {
        "name": "Achievers",
        "context": "We looked at Achievers last year, but it was too heavy for a team our size."
      }
    ],
    "next_step": "Yes — let's do the security review on September 12.",
    "objections": [
      "One concern: we need SSO and audit logs for IT to sign off."
    ],
    "confidence": {
      "extraction": "high",
      "basis": "Direct prospect statements and speaker labels; confidence is in extraction, not deal outcome."
    }
  },
  {
    "transcript_id": "TX-002",
    "deal_alias": "Deal-70BB30",
    "why_buys": [
      "We want to tie recognition to retention for our hourly workforce — regretted turnover there is over 30%."
    ],
    "pain_points": [
      "Regretted turnover there is over 30%."
    ],
    "stakeholders": [
      "Prospect (Head of Total Rewards)",
      "Prospect (CFO)"
    ],
    "budget_signal": "Finance has approved a $25k pilot budget for this quarter.",
    "timeline_signal": "We want a decision by end of September.",
    "competitor_mentioned": null,
    "next_step": "Yes — send the pilot agreement and we'll route it to legal this week.",
    "objections": [
      "Integration with Workday has to be rock solid — that's my one condition."
    ],
    "confidence": {
      "extraction": "high",
      "basis": "Direct prospect statements and speaker labels. No competitor named; Workday is an integration requirement, not a competitor. Confidence is in extraction, not deal outcome."
    }
  },
  {
    "transcript_id": "TX-003",
    "deal_alias": "Deal-530B50",
    "why_buys": [
      "We need to make recognition visible across our 12 retail locations."
    ],
    "pain_points": [
      "Store managers have zero budget autonomy for on-the-spot recognition today."
    ],
    "stakeholders": [
      "Prospect (People Ops Manager)"
    ],
    "budget_signal": null,
    "timeline_signal": "Honestly there's no rush on our side until Q1.",
    "competitor_mentioned": [
      {
        "name": "Bucketlist",
        "context": "My CEO used Bucketlist at her last company and liked it."
      }
    ],
    "next_step": "Yes, let's schedule a call with our CEO — I'll send two times.",
    "objections": [
      "Honestly there's no rush on our side until Q1.",
      "The CEO has to be sold first — she decides anything people-related."
    ],
    "confidence": {
      "extraction": "high",
      "basis": "Direct prospect statements and speaker labels. No prospect-stated purchase budget; rep pricing excluded. CEO is mentioned but is not a speaker. Confidence is in extraction, not deal outcome."
    }
  },
  {
    "transcript_id": "TX-004",
    "deal_alias": "Deal-180D02",
    "why_buys": [
      "We want to consolidate three separate recognition tools into one."
    ],
    "pain_points": [
      "We're paying for three tools and none of them talk to our HRIS."
    ],
    "stakeholders": [
      "Prospect (VP People)",
      "Prospect (IT Security Lead)"
    ],
    "budget_signal": "If it's under $15k annually, I can approve it without going to the board.",
    "timeline_signal": "Our procurement cycle runs six to eight weeks minimum.",
    "competitor_mentioned": null,
    "next_step": null,
    "objections": [
      "The security review took three months for our last vendor — that's my hesitation."
    ],
    "confidence": {
      "extraction": "high",
      "basis": "Direct prospect statements and speaker labels. Budget signal is an approval threshold, not a committed budget. Competitors are unnamed. CFO follow-up was not agreed: \"Maybe — I need to check her calendar, no promises.\" Confidence is in extraction, not deal outcome."
    }
  },
  {
    "transcript_id": "TX-005",
    "deal_alias": "Deal-F8767A",
    "why_buys": [
      "Two things: automate service milestones, and give us analytics on recognition equity across departments."
    ],
    "pain_points": [
      "Our night-shift teams feel invisible — their engagement scores run 20 points lower."
    ],
    "stakeholders": [
      "Prospect (HR Director)",
      "Prospect (People Ops Coordinator)"
    ],
    "budget_signal": "We have $12k approved under our engagement line.",
    "timeline_signal": "We need this running before our January all-hands.",
    "competitor_mentioned": [
      {
        "name": "Nectar",
        "context": "We're mid-pilot with Nectar right now, so you'd need to beat that experience."
      }
    ],
    "next_step": "Yes — come present to our exec team on October 2.",
    "objections": [
      "We're mid-pilot with Nectar right now, so you'd need to beat that experience.",
      "Our exec team is skeptical after a failed rollout two years ago."
    ],
    "confidence": {
      "extraction": "high",
      "basis": "Direct prospect statements and speaker labels; confidence is in extraction, not deal outcome."
    }
  },
  {
    "transcript_id": "TX-006",
    "deal_alias": "Deal-EE195F",
    "why_buys": [
      "I'd love to cut the admin time on service awards."
    ],
    "pain_points": [
      "I personally spend five hours a month ordering and shipping plaques.",
      "Budget isn't the issue — time is."
    ],
    "stakeholders": [
      "Prospect (HR Manager)"
    ],
    "budget_signal": "Budget isn't the issue — time is.",
    "timeline_signal": "Our fiscal year resets in February, so a Q1 start is realistic.",
    "competitor_mentioned": null,
    "next_step": "Yes — send the one-page overview and I'll forward it to our COO this week.",
    "objections": [
      "Nobody else — we're comparing this against just doing it internally.",
      "Fair warning, our COO usually prefers building things in-house."
    ],
    "confidence": {
      "extraction": "high",
      "basis": "Direct prospect statements and speaker labels. No budget amount stated. No external competitor named; internal delivery is the alternative. COO is mentioned but is not a speaker. Confidence is in extraction, not deal outcome."
    }
  }
]
next-to-close0.50

? The Question

Which 3 deals are closest to signature right now? Use the CRM data and the Slack thread; Slack often has the real status before the CRM does. For each: deal, amount, why it is close, what is left. Under 250 words.
cost $0.4717 · 23s · in 37,086 / out 987 tok
✓ exactly_three✗ top3_named✓ slack_first✗ blocked_excluded

✓ Correct Answer (ground truth)

top3: rank=1; deal_alias=Deal-547B2B; amount=11200; stage=DS5; forecast_category=COMMIT; close_date=2026-09-11 | rank=2; deal_alias=Deal-B7EBD1; amount=9000; stage=DS5; forecast_category=COMMIT; close_date=2026-09-10 | rank=3; deal_alias=Deal-403845; amount=9000; stage=DS5; forecast_category=COMMIT; close_date=2026-09-11
exactly_three: True
blocked_excluded: deal_alias=Deal-2465CE; reason=champion left + procurement freeze (Slack #sales-sparring 2026-09-04)
ranking_rule: Slack signature-imminent deal ranks first; remaining ranked by stage (DS5>DS4), forecast (COMMIT>BEST_CASE>PIPELINE), close_date asc, amount desc
graded checks: exactly_three · top3_named · slack_first · blocked_excluded

Model's actual answer

show the model's full answer
Based on the CRM and September 4 Slack thread—the latest status provided:

1. Deal-547B2B — $11,200
   Why close: CRM shows DS5/COMMIT. Slack says redlines came back clean, the signing page is out, and VP People said they would sign “tomorrow” (September 5). Strongest explicit signature signal.
   What is left: Customer signature; no subsequent confirmation of signing is provided.
   Sources: next_close_deals.csv; Slack, Alex Franklin, 09:12.

2. Deal-403845 — $9,000
   Why close: CRM shows DS5/COMMIT. Slack confirms the order form is with their finance team and the deal is “moving fine.”
   What is left: Finance processing/review and signature; exact finance requirements and signing timing are not provided.
   Sources: next_close_deals.csv; Slack, Dana Mercer, 10:02.

3. Deal-A2B47C — $6,360
   Why close: CRM shows DS5/COMMIT with a September 11 close date. Slack says it remains warm, progressing at normal legal-review pace. Provisionally ahead of Deal-D348E1, which has the same Slack status but an October 15 CRM close date.
   What is left: Complete legal review and obtain signature; no signing commitment is provided.
   Sources: next_close_deals.csv; Slack, Alex Franklin, 09:44.

Deal-2465CE is excluded despite CRM COMMIT: Slack reports champion departure, a procurement freeze, and a likely Q4 delay. No newer updates establish the actual status right now.
product-gap-detectionPASS 1.00

? The Question

Surface true product gaps raised by prospects in active deals. For each candidate: quote the prospect line, classify as real gap, plan gate, or rollout/enablement issue using the product docs excerpt, and name the deal and amount. Only real gaps go in the summary. Rep language does not count as prospect voice.
cost $0.4707 · 25s · in 36,045 / out 1,200 tok
✓ real_gaps✓ plan_gate_classified✓ enablement_classified✓ gap_summary

✓ Correct Answer (ground truth)

real_gaps: transcript_id=TG-001; deal_alias=Deal-EC3025; amount=62000; feature=SCIM user provisioning; quote=We need SCIM user provisioning — our security team will not approve any tool that requires manual account management.; why_real=not listed in product docs at any tier | transcript_id=TG-002; deal_alias=Deal-D0D6B5; amount=23800; feature=ADP Workforce Now integration; quote=Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us.; why_real=supported HRIS list is Workday/BambooHR/Gusto/Rippling only
plan_gates: transcript_id=TG-003; deal_alias=Deal-CFE7F4; amount=32000; feature=Custom report builder; tier_required=Enterprise; quote=I tried to build a custom report combining recognition frequency with turnover and couldn't — is that possible on our current plan?
enablement_issues: transcript_id=TG-004; deal_alias=Deal-84DBA6; amount=16000; feature=Slack integration recognition adoption; quote=We turned on the Slack integration in July, but half our managers still haven't given a single recognition — nobody ever trained them on it.; why_enablement=Slack integration exists on all plans; managers were never trained
excluded_rep_voice: transcript_id=TG-005; line=Full transparency — we don't have a native mobile app for hourly workers yet, though it's on the roadmap.; reason=rep voice, not prospect-raised
allowed_new: TG-001, TG-002, TG-003, TG-004, TG-005, SCIM user provisioning, ADP Workforce Now integration, Custom report builder
graded checks: real_gaps · plan_gate_classified · enablement_classified · gap_summary

Model's actual answer

show the model's full answer
Deal amounts and active-deal status are not provided. Classifications below use only transcripts_gaps.csv and product_docs.md; active status cannot be verified.

Candidate review

• Deal-EC3025 — Real gap: SCIM user provisioning. Amount: not provided.
  Prospect: “We need SCIM user provisioning — our security team will not approve any tool that requires manual account management.”
  Basis: Product docs explicitly say SCIM is not currently listed as supported. (TG-001; product_docs.md, Supported integrations)

• Deal-D0D6B5 — Real gap: ADP Workforce Now integration. Amount: not provided.
  Prospect: “Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us.”
  Basis: Product docs explicitly say ADP Workforce Now is not currently listed as supported. (TG-002; product_docs.md, Supported integrations)

• Deal-CFE7F4 — Plan gate. Amount: not provided.
  Prospect: “I tried to build a custom report combining recognition frequency with turnover and couldn't — is that possible on our current plan?”
  Basis: Custom report builder is Enterprise-only. The prospect’s current plan and support for this specific recognition/turnover combination are not documented. The excerpt supports a plan-gate classification for builder access, not a confirmed gap in reporting functionality. (TG-003; product_docs.md, Plans and tiers)

• Deal-84DBA6 — Rollout/enablement issue. Amount: not provided.
  Prospect: “We turned on the Slack integration in July, but half our managers still haven't given a single recognition — nobody ever trained them on it.”
  Basis: Slack integration is available on all plans; the prospect describes missing training, not missing functionality. (TG-004; product_docs.md, Plans and tiers)

• Deal-36C33F — Excluded: no prospect-raised gap. Amount: not provided.
  Prospect: “Good to know. The web version should be fine for our office staff for now.”
  Basis: The mobile-app claim comes from Alex Franklin, not the prospect. Product docs do not address native mobile availability, so no gap/plan-gate/enablement classification is supported. (TG-005)

Real-gap summary

• SCIM user provisioning — Deal-EC3025: explicit security-approval blocker. Amount not provided.
• ADP Workforce Now integration — Deal-D0D6B5: explicit dealbreaker. Amount not provided.

Affected amount = amount for Deal-EC3025 + amount for Deal-D0D6B5 = unknown + unknown; cannot calculate. Neither deal’s active status is supplied.
rep-performance
stale-pipeline-by-rep0.67

? The Question

The snapshot date is 2026-09-05. Using the open deals and the engagements-by-deal table, list every open deal with no logged email, call, or meeting in the last 7 days. Compute recency from the engagements table's last_email/last_call/last_meeting fields, NOT from any date field on the deal (the deal's last_contacted_field is unreliable). For each stale deal give: deal alias, owner name, stage, amount, days since last contact. Group by owner, order by amount descending within each owner. Then state, per owner, how many stale deals they carry and the total stale amount. (Do not write outreach text.)
cost $2.2147 · 124s · in 128,240 / out 14,474 tok
✓ stale_recall_ge_80✓ grouped_by_owner✓ jake_top_amount✓ kevin_stale_amt✗ jenn_stale_amt✗ stale_count

✓ Correct Answer (ground truth)

stale_count: 65
jake_amt: 626243.0
kevin_amt: 252905.03
jenn_amt: 261645.0
top_alias: Deal-2D1F1B
top_amount: 240000.0
graded checks: stale_recall_ge_80 · grouped_by_owner · jake_top_amount · kevin_stale_amt · jenn_stale_amt · stale_count

Model's actual answer

show the model's full answer
Recency = 2026-09-05 − latest nonblank last_email, last_call, or last_meeting dated on or before the snapshot. Stale means latest contact before 2026-08-29 (more than 7 days ago). Future-dated engagements do not count as past contact; last_contacted_field is not used. Amount currency was not provided.

In the tables, “Last contact → days” shows the date subtracted from 2026-09-05; for example, 2026-09-05 − 2026-08-28 = 8 days.

Bryce Harmon

Deal alias  | Stage | Amount | Last contact → days
---|---|---:|---
Deal-2D1F1B | DS1 | 240,000 | 2026-06-16 → 81
Deal-66D1FC | DS1 | 99,000 | 2026-08-20 → 16
Deal-950043 | DS1 | 70,000 | 2026-08-17 → 19
Deal-B23205 | DS1 | 45,000 | 2026-08-20 → 16
Deal-7BBDFA | DS3 | 37,440 | 2026-07-21 → 46
Deal-332637 | DS2 | 36,000 | 2026-08-27 → 9
Deal-1BEEBF | DS1 | 31,500 | 2026-08-17 → 19
Deal-A414F6 | DS1 | 25,200 | 2026-08-17 → 19
Deal-C5658B | DS1 | 23,400 | 2026-08-20 → 16
Deal-40522D | DS3 | 21,000 | 2026-08-17 → 19
Deal-C1FA6D | DS1 | 18,000 | 2026-08-20 → 16
Deal-01E193 | DS1 | 12,600 | 2026-08-28 → 8
Deal-F0EBBB | DS3 | 11,400 | 2026-08-12 → 24
Deal-927338 | DS1 | 10,920 | 2026-08-18 → 18
Deal-E25A09 | DS1 | 6,000 | 2026-08-27 → 9
Deal-C9C286 | DS2 | 5,502 | 2026-08-27 → 9
Deal-012CB1 | DS1 | 1 | 2026-08-13 → 23
Deal-3795AD | DS2 | 1 | 2026-08-28 → 8

Dana Mercer

Deal alias | Stage | Amount | Last contact → days
---|---|---:|---
Deal-44EA29 | DS2 | 60,000 | 2026-08-26 → 10
Deal-E51FB7 | DS2 | 43,875 | 2026-08-24 → 12
Deal-B42F46 | DS1 | 27,000 | 2026-08-17 → 19
Deal-BA3DDC | DS3 | 23,400 | 2026-08-21 → 15
Deal-9DDE86 | DS2 | 20,000 | 2026-08-21 → 15
Deal-215CCA | DS3 | 18,900 | 2026-08-19 → 17
Deal-5EED42 | DS3 | 16,250 | 2026-08-25 → 11
Deal-57887A | DS2 | 15,000 | 2026-08-28 → 8
Deal-944310 | DS4 | 10,500 | 2026-08-03 → 33
Deal-B7EBD1 | DS5 | 9,000 | 2026-08-20 → 16
Deal-3974EB | DS4 | 9,000 | 2026-08-28 → 8
Deal-F40F04 | DS2 | 8,100 | 2026-08-21 → 15
Deal-7599B8 | DS3 | 7,350 | 2026-08-18 → 18
Deal-87DDD1 | DS1 | 5,000 | 2026-08-17 → 19
Deal-F336B6 | DS3 | 4,200 | 2026-08-21 → 15
Deal-0660B4 | DS4 | 1,920 | 2026-08-20 → 16

Alex Franklin

Deal alias | Stage | Amount | Last contact → days
---|---|---:|---
Deal-CC08D1 | DS1 | 24,000 | 2026-08-20 → 16
Deal-E73427 | DS3 | 18,000 | 2026-08-26 → 10
Deal-885F45 | DS2 | 9,300 | 2026-08-24 → 12
Deal-C2FF3C | DS1 | 8,316 | 2026-08-26 → 10
Deal-0D2F7A | DS3 | 5,100 | 2026-08-24 → 12
Deal-6C60D4 | DS3 | 4,800 | 2026-08-24 → 12
Deal-13FEBD | DS2 | 4,680 | 2026-08-24 → 12
Deal-819506 | DS1 | 4,400 | 2026-08-28 → 8
Deal-9D0060 | DS3 | 3,840 | 2026-08-24 → 12
Deal-690476 | DS2 | 3,600 | 2026-08-18 → 18
Deal-C6D97A | DS4 | 3,240 | 2026-08-28 → 8
Deal-EE195F | DS3 | 3,120 | 2026-08-28 → 8
Deal-278DEC | DS3 | 2,700 | 2026-08-28 → 8
Deal-635B8E | DS3 | 2,600 | 2026-08-18 → 18
Deal-6883F3 | DS1 | 2,400 | 2026-08-20 → 16
Deal-4A13AD | DS3 | 2,160 | 2026-08-10 → 26
Deal-F67D31 | DS2 | 1,800 | 2026-08-28 → 8
Deal-5FDCE4 | DS3 | 1,600 | 2026-08-24 → 12
Deal-BA571A | DS4 | 1,080 | 2026-08-18 → 18

Cole Ingram

Deal alias | Stage | Amount | Last contact → days
---|---|---:|---
Deal-D04904 | DS2 | 58,529.25 | 2026-08-25 → 11
Deal-B25F40 | DS3 | 40,000 | 2026-08-28 → 8
Deal-813836 | DS2 | 32,175 | 2026-08-25 → 11
Deal-1BA595 | DS2 | 31,750 | 2026-08-25 → 11
Deal-CFE1E8 | DS3 | 18,000 | 2026-08-25 → 11
Deal-CD47A6 | DS2 | 12,168 | 2026-08-25 → 11
Deal-627646 | DS3 | 11,193 | 2026-08-25 → 11
Deal-FF809F | DS2 | 7,781.20 | 2026-08-25 → 11
Deal-AF932D | DS2 | 7,225.40 | 2026-08-25 → 11
Deal-A71728 | DS2 | 6,947.50 | 2026-08-25 → 11
Deal-8BC9F5 | DS2 | 5,616 | 2026-08-26 → 10
Deal-175395 | DS3 | 4,779.88 | 2026-08-25 → 11
Deal-481E24 | DS3 | 4,140 | 2026-08-26 → 10
Deal-C7F9BF | DS2 | 3,360 | 2026-08-25 → 11
Deal-2F3A66 | DS3 | 3,334.80 | 2026-08-25 → 11
Deal-342E96 | DS2 | 2,700 | 2026-08-12 → 24
Deal-E568D5 | DS3 | 1,875 | 2026-08-25 → 11
Deal-FD9F4E | DS5 | 1,330 | 2026-08-26 → 10

Farid Osman

Deal alias | Stage | Amount | Last contact → days
---|---|---:|---
Deal-8BA24E | DS1 | 2,880 | 2026-08-28 → 8
Deal-8FDCD2 | DS1 | 1 | 2026-08-21 → 15

Missing engagement data

These deals have no row in engagements_by_deal_90d.csv. No recent contact is documented in the supplied table, but days since last contact and stale status cannot be established. They are excluded from confirmed-stale totals.

Owner | Deal alias | Stage | Amount | Days since last contact
---|---|---|---:|---
Alex Franklin | Deal-3EED2C | DS2 | 7,200 | Unknown
Elena Sinclair | Deal-57FF13 | DS1 | 2,100 | Unknown

Per-owner confirmed-stale totals

Owner | Stale deals | Total stale amount
---|---:|---:
Bryce Harmon | 18 | 692,964.00
Dana Mercer | 16 | 279,495.00
Alex Franklin | 19 | 106,736.00
Cole Ingram | 18 | 252,905.03
Farid Osman | 2 | 2,881.00
Elena Sinclair | 0 confirmed; 1 unknown | 0.00 confirmed

Amount arithmetic, summing each owner’s listed deals:

Bryce Harmon:
240,000 + 99,000 + 70,000 + 45,000 + 37,440 + 36,000 + 31,500 + 25,200 + 23,400 + 21,000 + 18,000 + 12,600 + 11,400 + 10,920 + 6,000 + 5,502 + 1 + 1 = 692,964.00

Dana Mercer:
60,000 + 43,875 + 27,000 + 23,400 + 20,000 + 18,900 + 16,250 + 15,000 + 10,500 + 9,000 + 9,000 + 8,100 + 7,350 + 5,000 + 4,200 + 1,920 = 279,495.00

Alex Franklin:
24,000 + 18,000 + 9,300 + 8,316 + 5,100 + 4,800 + 4,680 + 4,400 + 3,840 + 3,600 + 3,240 + 3,120 + 2,700 + 2,600 + 2,400 + 2,160 + 1,800 + 1,600 + 1,080 = 106,736.00

Cole Ingram:
58,529.25 + 40,000 + 32,175 + 31,750 + 18,000 + 12,168 + 11,193 + 7,781.20 + 7,225.40 + 6,947.50 + 5,616 + 4,779.88 + 4,140 + 3,360 + 3,334.80 + 2,700 + 1,875 + 1,330 = 252,905.03

Farid Osman:
2,880 + 1 = 2,881.00
activity-mix-vs-outcomePASS 1.00

? The Question

The snapshot date is 2026-09-05. For each sales rep (owner), compute the last-30-day totals of emails, calls, and meetings from the engagements table, and the number of deals that entered DS2 in the last 30 days (t_ds2 within 30 days of the snapshot). Report per rep: the activity mix as percentages (emails/calls/meetings share of that rep's total activities), total activities per DS2 entry, and rank the reps by that efficiency ratio (lowest activities per DS2 entry = most efficient). Name the most efficient rep and the highest-volume rep, and note if they differ. Do not attribute cause.
cost $1.8240 · 86s · in 117,578 / out 8,525 tok
✓ justin_ds2✓ justin_volume_high✓ efficient_named✓ mix_present✓ no_causal_claim

✓ Correct Answer (ground truth)

justin_ds2: 4
graded checks: justin_ds2 · justin_volume_high · efficient_named · mix_present · no_causal_claim

Model's actual answer

show the model's full answer
Window: 2026-08-06 through 2026-09-05, inclusive (snapshot minus 30 days). Activities use the supplied emails_30d, calls_30d, and meetings_30d fields, joined to deal owners by deal_id. DS2 entries use t_ds2, regardless of current stage.

Activity share = activity count ÷ total activities × 100.
Efficiency = total activities ÷ DS2 entries; lower ranks better.

| Rank | Rep | Emails | Calls | Meetings | Total activities: arithmetic | Mix: emails / calls / meetings | DS2 entries | Activities per DS2: arithmetic |
|---:|---|---:|---:|---:|---|---|---:|---|
| 1 | Alex Franklin | 307 | 36 | 41 | 307 + 36 + 41 = 384 | 79.95% / 9.38% / 10.68% | 18 | 384 ÷ 18 = 21.33 |
| 2 | Bryce Harmon | 162 | 0 | 43 | 162 + 0 + 43 = 205 | 79.02% / 0.00% / 20.98% | 4 | 205 ÷ 4 = 51.25 |
| 3 | Cole Ingram | 96 | 14 | 1 | 96 + 14 + 1 = 111 | 86.49% / 12.61% / 0.90% | 2 | 111 ÷ 2 = 55.50 |
| 4 | Farid Osman | 38 | 0 | 34 | 38 + 0 + 34 = 72 | 52.78% / 0.00% / 47.22% | 1 | 72 ÷ 1 = 72.00 |
| 5 | Dana Mercer | 84 | 18 | 11 | 84 + 18 + 11 = 113 | 74.34% / 15.93% / 9.73% | 1 | 113 ÷ 1 = 113.00 |
| — | Elena Sinclair | Missing | Missing | Missing | Missing | N/A | 0 | Undefined |
| — | Gavin Porter (archived) | Missing | Missing | Missing | Missing | N/A | 0 | Undefined |
| — | Hugo Lindqvist (archived) | Missing | Missing | Missing | Missing | N/A | 0 | Undefined |

Most efficient and highest-volume rep: Alex Franklin, at 21.33 recorded activities per DS2 entry and 384 recorded activities. They do not differ.

Data limitations:
- Alex Franklin’s Deal-3EED2C has no engagement row but does qualify as a DS2 entry. His activity totals and ranking therefore reflect available engagement records, not confirmed complete activity coverage.
- Elena Sinclair’s Deal-57FF13 has no engagement row. Neither archived owner has deals or associated engagement records in the supplied data. Missing activity data is not treated as zero.
- The supplied last-30-day counts are used as given. Some last_meeting dates fall after the snapshot; individual activity timestamps are unavailable to independently validate or adjust those counts.
- Percentages are rounded and may not sum to exactly 100%.
ae-qtd-scorecardPASS 1.00

? The Question

Build a QTD scorecard for the named AE as of 2026-09-05: bookings vs quota with attainment percent, new vs expansion split, active pipeline by stage with amount, rolling 90-day DS2-to-won rate, win and loss counts with top loss reason, activity volume by type in the last 30 days. Exclude any closed-won deal dated before the quarter. Close with three coaching observations grounded in the numbers.
cost $1.8569 · 120s · in 103,631 / out 11,785 tok
✓ bookings✓ quota✓ attainment✓ new_expansion_split✓ top_loss_reason✓ coaching_obs

✓ Correct Answer (ground truth)

owner: Alex Franklin
quarter: 2026-Q3
snapshot_date: 2026-09-05
bookings: 150000
quota: 200000
attainment_pct: 75.0
new_bookings: 113500
new_count: 5
expansion_bookings: 36500
expansion_count: 3
pipeline_by_stage: {"DS1": {"count": 20, "amount": 284621}, "DS2": {"count": 28, "amount": 353760}, "DS3": {"count": 67, "amount": 552705}, "DS4": {"count": 5, "amount": 23574}, "DS5": {"count": 5, "amount": 45730}}
open_deals_count: 125
ds2_to_won: window_start=2026-06-07; won_in_window=8; closed_in_window=35; rate=0.2286
wins_qtd: 8
losses_qtd: 27
top_loss_reason: reason=Lost- Timing (1 year or more); count=13
activity_30d_totals: emails_30d=807; calls_30d=112; meetings_30d=128; notes_30d=50
prior_quarter_excluded: deal_alias=Deal-B3E6F1; close_date=2026-06-20; amount=24000
allowed_new: 18 items — e.g. Deal-A1C3E5, Deal-B7D2F4, Deal-C9E1A6, Deal-D4B8C2
graded checks: bookings · quota · attainment · new_expansion_split · top_loss_reason · coaching_obs

Model's actual answer

show the model's full answer
Alex Franklin — QTD scorecard as of 2026-09-05

QTD: 2026-07-01–2026-09-05. Amounts use the supplied units; currency is not specified. Deal-B3E6F1, closed-won on 2026-06-20 for 24,000, is excluded throughout.

Bookings vs quota
Source: ae_deals.csv and quota.csv.

- New: 40,000 + 35,000 + 21,000 + 11,000 + 6,500 = 113,500 across 5 wins.
- Expansion: 20,000 + 12,000 + 4,500 = 36,500 across 3 wins.
- Total bookings: 113,500 + 36,500 = 150,000.
- Quarterly quota: 200,000.
- Attainment: 150,000 ÷ 200,000 × 100 = 75.0%.
- Remaining quota: 200,000 − 150,000 = 50,000.
- Booking mix: new 113,500 ÷ 150,000 = 75.7%; expansion 36,500 ÷ 150,000 = 24.3%.

Active pipeline
Source: ae_deals.csv. All supplied open deals, regardless of expected close date; amounts are unweighted.

Stage   Deals       Amount
DS1        20      284,621
DS2        28      353,760
DS3        67      552,705
DS4         5       23,574
DS5         5       45,730
Total     125    1,260,390

Total amount = 284,621 + 353,760 + 552,705 + 23,574 + 45,730 = 1,260,390.

Of this, 22 open deals totaling 109,363 have expected close dates within Q3, including the overdue Deal-7A2454.

Conversion and outcomes
Source: ae_deals.csv.

- Rolling 90-day window: 2026-06-08–2026-09-05, inclusive.
- DS2-to-won rate, defined by DS2-entry cohort: 111 deals entered DS2 during the window; 8 won, 27 lost, and 76 remain open.
- Conversion to date: 8 ÷ (8 + 27 + 76) × 100 = 7.2%. This is not a final cohort outcome because 76 deals remain open.
- QTD outcomes: 8 wins and 27 losses.
- Closed-deal win rate, distinct from cohort conversion: 8 ÷ (8 + 27) × 100 = 22.9%.
- Top loss reason by count: “Lost- Timing (1 year or more)” — 13 ÷ 27 = 48.1% of losses, totaling 184,681.

Last-30-day activity
Source: ae_engagements.csv, joined to the included deals. The supplied `_30d` fields are treated as covering 2026-08-07–2026-09-05; individual timestamps are unavailable.

Type       Lost deals + Open deals + QTD wins = Total
Emails            109 + 599 + 89             = 797
Calls              25 +  54 + 31             = 110
Meetings           13 +  90 + 23             = 126
Notes              25 +   1 + 21             = 47

Total recorded activity: 797 + 110 + 126 + 47 = 1,080.

Three coaching observations

1. Focus quarter-end execution on the 50,000 quota gap. Q3-dated DS4/DS5 pipeline totals only 34,204, so even winning all of it would leave 50,000 − 34,204 = 15,796 to source from earlier stages or other opportunities.

2. Tighten timing qualification before DS2. Timing accounts for 13 of 27 losses and 184,681 in lost amount. Require a buyer-confirmed decision date and compelling event; Deal-B3ABED alone contributed 40,001 to this loss category.

3. Re-engage near-term late-stage deals beyond email. Deal-547B2B, Deal-A2B47C, and Deal-F9A08A total 11,200 + 6,360 + 2,484 = 20,044, with September close dates but zero recorded calls or meetings in the last 30 days. Confirm live buyer next steps before relying on these closes.
multithreading-gapPASS 1.00

? The Question

Find every open deal that is single-threaded (fewer than 2 active contacts) or under-threaded (fewer than 3, or all contacts in one persona). Active means engaged in the last 60 days and not marked former. Personas: economic buyer, champion, HR admin, IT security, finance. For each flagged deal: amount, stage, active contact count, personas present, personas missing, the most valuable persona to add given the stage, and any on-file unengaged contact at that company who fits; otherwise say none on file.
cost $0.8976 · 51s · in 34,353 / out 4,595 tok
✓ flagged_recall✓ persona_language✓ single_thread_term✓ on_file_fit

✓ Correct Answer (ground truth)

active_window_start: 2026-07-07
persona_universe: economic buyer, champion, HR admin, IT security, finance
flag_rule: flag if active_contacts < 2 (single-threaded) or active_contacts < 3 or all active contacts share one persona (under-threaded); active = engaged within 60 days and not former
best_persona_rule: DS2/DS3: economic buyer > IT security > finance > HR admin > champion; DS4/DS5: economic buyer > finance > IT security > HR admin > champion; first persona not present among active contacts
flagged: deal_alias=Deal-36C33F; amount=15000; stage=DS2; active_count=1; reason=single-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-50D386; amount=36000; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-5408B0; amount=14850; stage=DS4; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-5BFE3B; amount=18000; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-885F45; amount=9300; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=IT security | deal_alias=Deal-92D97D; amount=60000; stage=DS2; active_count=1; reason=single-threaded; best_persona_to_add=economic buyer … (+5 more)
healthy_not_flagged: Deal-4B0BEB, Deal-84DBA6, Deal-D348E1
allowed_new: 59 items — e.g. C-077A0E, C-0D15DF, C-1BDFE0, C-2AE3AA
graded checks: flagged_recall · persona_language · single_thread_term · on_file_fit

Model's actual answer

show the model's full answer
Open/closed status, amount, and stage are missing for every deal. Therefore, I cannot confirm which deals are open or determine the most valuable persona to add given the stage. Below are all threading-flagged deals in the supplied data; these are not confirmed open deals.

Assuming an as-of date of 2026-09-13: 2026-09-13 − 60 days = 2026-07-15. Active contacts must have engaged on or after that cutoff and not be former. Personas present/missing refer only to active contacts.

11 flagged deals = 5 single-threaded + 6 additional under-threaded. Single-threaded deals also meet the under-threaded definition.

Amount, stage, and stage-specific priority: unavailable for every row. Listed contacts fit missing personas, but are not stage-prioritized.

| Deal / company | Flag | Active count arithmetic¹ | Personas present | Personas missing | On-file unengaged contacts fitting missing personas |
|---|---|---|---|---|---|
| Deal-EC3025 / C-FDD0C7 | Single-threaded | 2 − 1 − 0 = 1 | champion | economic buyer, HR admin, IT security, finance | CT-6827DB — economic buyer |
| Deal-92D97D / C-E23238 | Single-threaded | 2 − 0 − 1 = 1 | HR admin | economic buyer, champion, IT security, finance | CT-A902AE — champion; last engaged 2026-06-01 |
| Deal-50D386 / C-EB10E4 | Under-threaded: fewer than 3 | 2 − 0 − 0 = 2 | champion, HR admin | economic buyer, IT security, finance | CT-A1C4B3 — economic buyer |
| Deal-D0D6B5 / C-32918E | Under-threaded: one persona | 3 − 0 − 0 = 3 | champion | economic buyer, HR admin, IT security, finance | CT-1FA4DB — economic buyer |
| Deal-5BFE3B / C-535D36 | Under-threaded: fewer than 3; one persona | 2 − 0 − 0 = 2 | champion | economic buyer, HR admin, IT security, finance | none on file |
| Deal-36C33F / C-077A0E | Single-threaded | 3 − 2 − 0 = 1 | IT security | economic buyer, champion, HR admin, finance | CT-1DB73E — economic buyer |
| Deal-885F45 / C-5E8EFB | Under-threaded: fewer than 3 | 2 − 0 − 0 = 2 | economic buyer, champion | HR admin, IT security, finance | CT-B3F25D — IT security |
| Deal-FCBE5B / C-737030 | Single-threaded | 1 − 0 − 0 = 1 | champion | economic buyer, HR admin, IT security, finance | none on file |
| Deal-5408B0 / C-2AE3AA | Under-threaded: fewer than 3 | 2 − 0 − 0 = 2 | champion, HR admin | economic buyer, IT security, finance | CT-07FA76 — economic buyer |
| Deal-C6D97A / C-5A8FC2 | Under-threaded: one persona | 3 − 0 − 0 = 3 | champion | economic buyer, HR admin, IT security, finance | none on file |
| Deal-F9A08A / C-0D15DF | Single-threaded | 2 − 0 − 1 = 1 | champion | economic buyer, HR admin, IT security, finance | CT-697541 — economic buyer; also CT-913581 — economic buyer, last engaged 2026-06-20 |

¹ Arithmetic: total listed contacts − former contacts − non-former contacts outside the engagement window = active contacts.

Sources: deal_contacts.csv for activity, former status, personas, and lapsed contacts; unengaged_contacts.csv for other on-file candidates. Former contacts are excluded from candidates.
talk-track-from-transcriptsPASS 1.00

? The Question

From this rep's last 10 calls: what they lead with in the first five minutes, how they handle the three most common objections, how often a concrete next step is agreed (rate), and every competitor a prospect raised. Quote at most one line per finding and only lines that appear in the transcripts. End with two coaching notes.
cost $0.9292 · 60s · in 64,303 / out 3,033 tok
✓ objection_count✓ next_step_rate✓ competitors✓ coaching✓ quote_used

✓ Correct Answer (ground truth)

rep: Alex Franklin
n_transcripts: 10
opener_theme: theme=opens with a customer recognition win story (400-person retailer, 18% turnover reduction); count=8; total=10; quote=Quick one to kick off: a 400-person retailer we work with cut regretted turnover 18% in two quarters after automating milestone awards, and I wanted to walk you through exactly how they did it.; transcript_id=TT-001
objections: objection=budget; count=4; quote=Honestly, budget is locked until next fiscal year — I can't add a new line item right now.; transcript_id=TT-001 | objection=timing; count=3; quote=This is good, but can we revisit it next quarter? Open enrollment starts in October and we're underwater.; transcript_id=TT-002 | objection=in-house; count=3; quote=We already do recognition with a spreadsheet and quarterly gift cards — why would we change?; transcript_id=TT-004
objection_count: 3
next_step: agreed=7; total=10; rate=0.7
competitors_prospect_raised: Awardco, Kudos
competitor_quotes: Awardco=We're also in late talks with Awardco — their rewards catalog looks bigger than yours.; Kudos=How are you different from Kudos? Our CEO used them at her last company.
rep_raised_competitors_excluded: Workhuman
allowed_new: 13 items — e.g. TT-001, TT-002, TT-003, TT-004
graded checks: objection_count · next_step_rate · competitors · coaching · quote_used

Model's actual answer

show the model's full answer
Based on the 10 supplied call excerpts:

First five minutes
- Alex Franklin leads with the same customer-outcome story in 8 ÷ 10 × 100 = 80% of calls: a retailer’s claimed turnover reduction through automated milestone awards. These are the rep’s claims, not independently verified results. Calls: Deal-D348E1, Deal-5408B0, Deal-547B2B, Deal-C61CF7, Deal-60C2C2, Deal-EDC141, Deal-D9A12F, Deal-84DBA6.
- Exceptions: Deal-403845 opens with a security-review/pricing agenda; Deal-1E2498 starts with pricing at the prospect’s prior request.
- Early competitive positioning also appears: Alex introduces Workhuman in Deal-C61CF7 at minute 2 and responds to prospect-raised Awardco and Kudos at minute 5 in Deal-547B2B and Deal-EDC141, respectively.

Three most common objections
1. Budget/approval: 5 distinct calls—4 locked-budget objections plus Deal-403845’s budget-committee dependency. The locked-budget calls are Deal-D348E1, Deal-547B2B, Deal-60C2C2, and Deal-84DBA6. Alex reframes cost around avoided turnover/backfills:
   “Totally fair. Most teams fund this out of turnover savings — that retailer saved about $210k in avoided backfills, which is how their finance team signed off.” — Deal-D348E1, minute 8.
   When committee approval blocks Deal-403845 and Deal-84DBA6, he accepts the delay without securing a concrete follow-up.

2. Timing/workload: 3 calls—Deal-5408B0, Deal-C61CF7, Deal-D9A12F. He proposes a smaller pilot to generate evidence before planning:
   “Makes sense. What if we scope a 90-day pilot with one department so you have internal data before next quarter's planning?” — Deal-5408B0, minute 8.
   All three agree to a working session; the excerpts do not establish agreement to the pilot itself.

3. Status quo/manual recognition: 3 calls—Deal-403845, Deal-EDC141, Deal-1E2498. He emphasizes automation and visibility:
   “Spreadsheets work until they scale — the difference is automation: milestones fire without HR lifting a finger, and you get analytics on who is being recognized.” — Deal-403845, minute 8.
   Only Deal-1E2498 agrees to a concrete next step; Deal-403845 stalls on approval and Deal-EDC141 on lack of urgency.

Counts above are distinct calls per objection category; categories can overlap.

Concrete next-step agreement rate
A concrete agreement requires prospect acceptance of a specific action and time—not just the rep proposing one.

7 ÷ 10 × 100 = 70%.

- Agreed: Deal-D348E1, Deal-5408B0, Deal-547B2B, Deal-C61CF7, Deal-60C2C2, Deal-D9A12F, Deal-1E2498.
- Each accepts a Thursday, 2pm working session and commits to bringing the HRIS manager:
  “Yes, Thursday at 2pm works — send the invite and I'll bring our HRIS manager.” — Deal-D348E1, minute 15.
- No concrete agreement recorded: Deal-403845, Deal-EDC141, Deal-84DBA6. Invite delivery and meeting attendance are not provided.

Every prospect-raised competitor
- Awardco — Deal-547B2B, minute 4: “We're also in late talks with Awardco — their rewards catalog looks bigger than yours.”
- Kudos — Deal-EDC141, minute 4: “How are you different from Kudos? Our CEO used them at her last company.”

Workhuman is excluded: Alex, not the prospect, raises it in Deal-C61CF7.

Two coaching notes
1. Make the outcome story a bridge to discovery: ask prospects to quantify their own turnover costs or manual-recognition burden before moving into the standard automation/ROI response.
2. Strengthen stalled-call closes: for Deal-403845 and Deal-84DBA6, clarify the committee owner and decision date and seek a scheduled follow-up; for Deal-EDC141, explore what would create urgency before accepting an open-ended delay.
reporting-analytics
q3-weighted-forecastPASS 1.00

? The Question

Below are the open deals in the default sales pipeline with a close date between 2026-07-01 and 2026-10-15, pulled from the CRM warehouse on 2026-09-05. Stages: DS1 (discovery) through DS5 (contract). Build the Q3 2026 forecast (quarter = 2026-07-01 to 2026-09-30). Weighting rule: 100 percent of COMMIT plus 35 percent of BEST_CASE. PIPELINE deals count zero. Only deals with a close date inside the quarter count. Report: COMMIT total, BEST_CASE total, weighted forecast, count of deals in each category inside the quarter, the deals you excluded for being outside the quarter (count and total amount), and the top 5 BEST_CASE deals by amount inside the quarter. Show arithmetic. Then, in a final '## Data quality' section of no more than four sentences, name the data quality problems in this extract that would make the forecast unreliable if it were run unattended.
cost $0.8297 · 61s · in 43,654 / out 5,957 tok
✓ commit_total_44729✓ best_case_total_203565✓ weighted_115976.75✓ commit_count_7✓ best_case_count_24✓ excluded_count_32✓ excluded_amount_227575✓ top5_aliases✓ flags_blank_owner✓ flags_missing_why_buys

✓ Correct Answer (ground truth)

graded checks: commit_total_44729 · best_case_total_203565 · weighted_115976.75 · commit_count_7 · best_case_count_24 · excluded_count_32 · excluded_amount_227575 · top5_aliases · flags_blank_owner · flags_missing_why_buys

Model's actual answer

show the model's full answer
## Q3 2026 forecast

Close-date filter: 2026-07-01 through 2026-09-30, inclusive. Amounts are in the extract’s unspecified currency.

| Category | In-quarter deal count | Total amount | Weight | Forecast contribution |
|---|---:|---:|---:|---:|
| COMMIT | 7 | 44,729.00 | 100% | 44,729.00 |
| BEST_CASE | 24 | 203,565.00 | 35% | 71,247.75 |
| PIPELINE | 23 | 201,637.40 | 0% | 0.00 |
| **Weighted forecast** | **54** | | | **115,976.75** |

Arithmetic:

- COMMIT = 11,200 + 9,000 + 9,000 + 6,360 + 5,400 + 2,520 + 1,249 = **44,729**
- BEST_CASE = 38,935 + 24,000 + 19,656 + 16,250 + 11,116 + 10,800 + 10,500 + 9,890 + 9,720 + 9,000 + 7,200 + 3,840 + 3,780 + 3,600 + 3,240 + 3,150 + 3,120 + 3,060 + 2,916 + 2,760 + 2,484 + 2,100 + 1,920 + 528 = **203,565**
- Weighted forecast = (100% × 44,729) + (35% × 203,565) + (0% × 201,637.40) = 44,729 + 71,247.75 + 0 = **115,976.75**
- In-quarter count = 7 + 24 + 23 = **54 deals**

## Excluded: close dates outside Q3

| Deal alias | Close date | Amount |
|---|---|---:|
| Deal-E51FB7 | 2026-10-01 | 43,875 |
| Deal-B936FE | 2026-10-09 | 18,000 |
| Deal-D9A12F | 2026-10-15 | 17,000 |
| Deal-D348E1 | 2026-10-15 | 13,770 |
| Deal-4062CF | 2026-10-15 | 10,800 |
| Deal-293AF3 | 2026-10-09 | 9,000 |
| Deal-034D49 | 2026-10-15 | 9,000 |
| Deal-E0ADD8 | 2026-10-15 | 7,920 |
| Deal-9F2E43 | 2026-10-08 | 7,690 |
| Deal-FCBE5B | 2026-10-07 | 7,500 |
| Deal-712010 | 2026-10-15 | 7,200 |
| Deal-6691E0 | 2026-10-15 | 5,700 |
| Deal-C61CF7 | 2026-10-09 | 5,400 |
| Deal-600CD9 | 2026-10-02 | 5,400 |
| Deal-A92065 | 2026-10-15 | 5,400 |
| Deal-1D532E | 2026-10-15 | 5,400 |
| Deal-48B656 | 2026-10-15 | 5,160 |
| Deal-E531A6 | 2026-10-15 | 4,800 |
| Deal-D1E6C2 | 2026-10-09 | 4,400 |
| Deal-D9E112 | 2026-10-09 | 4,300 |
| Deal-5AD94B | 2026-10-15 | 4,000 |
| Deal-901332 | 2026-10-15 | 3,600 |
| Deal-47AE31 | 2026-10-09 | 3,600 |
| Deal-15D24F | 2026-10-09 | 3,600 |
| Deal-766C74 | 2026-10-14 | 3,300 |
| Deal-ED725A | 2026-10-08 | 2,400 |
| Deal-8AD4A5 | 2026-10-07 | 1,800 |
| Deal-D7E999 | 2026-10-15 | 1,800 |
| Deal-ED13B0 | 2026-10-09 | 1,680 |
| Deal-5FDCE4 | 2026-10-01 | 1,600 |
| Deal-7FA0C3 | 2026-10-01 | 1,400 |
| Deal-F5A622 | 2026-10-08 | 1,080 |

Excluded count = 1 COMMIT + 9 BEST_CASE + 22 PIPELINE = **32 deals**.  
Excluded amount = 13,770 COMMIT + 28,240 BEST_CASE + 185,565 PIPELINE = **227,575**.

## Top 5 in-quarter BEST_CASE deals

| Rank | Deal alias | Amount | Close date |
|---|---|---:|---|
| 1 | Deal-2D7423 | 38,935 | 2026-09-30 |
| 2 | Deal-25F752 | 24,000 | 2026-09-25 |
| 3 | Deal-E53952 | 19,656 | 2026-09-30 |
| 4 | Deal-5EED42 | 16,250 | 2026-09-30 |
| 5 | Deal-FA32A0 | 11,116 | 2026-09-25 |

## Data quality

Owners are blank on 85 of 86 deals; only Deal-C9C286 has an owner, limiting accountability and owner-level validation.  
`why_buys_chars` is zero on 71 deals, including 67 at DS2–DS5, leaving qualification rationale absent in this extract.  
Deal-A5E80A is COMMIT at DS1 and Deal-499BF6 is COMMIT at DS2, creating stage/category anomalies that warrant validation before unattended 100% weighting.  
Deal-31AD2C, Deal-333EBB, Deal-57FF13, and Deal-7A2454 remain open despite close dates before the 2026-09-05 extraction date, indicating potentially stale close dates or statuses.
aha-moment-2x20.83

? The Question

Test the activation hypothesis on the company cohort below: companies with 5+ unique givers (m1_users >= 5) AND 1+ successful redemption (m1_redemptions >= 1) in their first calendar month retain better at 24 months than those with one signal or neither. Every company in this extract is already 25+ months old, so current_status = 'active' means retained at 24 months. Report the full 2x2 (cohort size and 24-month retention rate for: both signals, givers-only, redemption-only, neither), state how many companies were excluded from the denominator and why (if none, say so), name the single signal with the largest retention lift, and state what this does and does not prove.
cost $1.4967 · 85s · in 100,780 / out 5,460 tok
✓ both_66✓ givers_47✓ redemp_31✓ neither_40✓ excluded_0✗ largest_lift_givers

✓ Correct Answer (ground truth)

graded checks: both_66 · givers_47 · redemp_31 · neither_40 · excluded_0 · largest_lift_givers

Model's actual answer

show the model's full answer
Using m1_users ≥ 5 and m1_redemptions ≥ 1 as the signals, and only current_status = 'active' as retained:

| First-month cohort | Companies | Retained at 24 months | Retention arithmetic |
|---|---:|---:|---:|
| Both signals | 47 | 31 | 31 ÷ 47 = 65.96% |
| Givers-only | 49 | 23 | 23 ÷ 49 = 46.94% |
| Redemption-only | 29 | 9 | 9 ÷ 29 = 31.03% |
| Neither | 95 | 38 | 38 ÷ 95 = 40.00% |

Denominator: 47 + 49 + 29 + 95 = 220 companies. Excluded: 0. All companies meet the supplied age requirement and have the required signal and status fields. Zero activity is included, not treated as missing. The non_renewing companies—C-0B2078FB, C-0A96134F, C-0BEAF685—remain in the denominator but are not counted as retained.

Hypothesis: supported descriptively. Both-signal retention exceeds:
- Givers-only by 65.96% − 46.94% = 19.02 percentage points.
- Redemption-only by 65.96% − 31.03% = 34.92 percentage points.
- Neither by 65.96% − 40.00% = 25.96 percentage points.

Strongest single signal: 5+ unique givers.
- Givers-only versus neither: 46.94% − 40.00% = +6.94 percentage points.
- Redemption-only versus neither: 31.03% − 40.00% = −8.97 percentage points.

The ranking also holds when measuring each signal across the entire cohort, including companies with both:
- Givers present versus absent: (54 ÷ 96 − 47 ÷ 124) × 100 = +18.35 percentage points.
- Redemption present versus absent: (40 ÷ 76 − 61 ÷ 144) × 100 = +10.27 percentage points.

What this establishes: an observed association between both first-month signals and higher retention in this extract.

What it does not prove: that causing these actions improves retention, that these thresholds are optimal, or that the relationship generalizes beyond this cohort. Confounding is not controlled, and statistical significance has not been tested.
arr-rollforward-reconciliationPASS 1.00

? The Question

Reconcile ARR between CRM company records and billing subscriptions as of 2026-09-05. Billing ARR is MRR x 12. Report both totals, the variance, and a decomposition into buckets (status mismatch, rounding, missing records, other) that sums exactly to the variance. List each mismatched account with a suggested owner. Business rule: any subscription with a term other than 12 months must have cf_agreement_end_date populated; list violations.
cost $0.9048 · 59s · in 50,897 / out 5,489 tok
✓ billing_total✓ crm_total✓ variance✓ buckets_sum✓ violations_listed

✓ Correct Answer (ground truth)

as_of: 2026-09-05
billing_arr_total: 604739.28
crm_arr_total: 603581.76
variance: 1157.52
variance_definition: billing_arr_total - crm_arr_total (billing ARR = sum of MRR x 12 over status=active subscriptions)
buckets: {"status_mismatch": {"amount": "-13158.48", "accounts": [{"company_alias": "C-0C8323BF", "billing_arr": "0.00", "hubspot_arr": "4905.24", "diff": "-4905.24", "suggested_owner": "Dana Mercer"}, {"compa
bucket_sum_equals_variance: True
n_mismatched_accounts: 7
rule: term_months != 12 requires cf_agreement_end_date
violations: subscription_id=SUB-0002; company_alias=C-1794A52C; term_months=24 | subscription_id=SUB-0019; company_alias=C-22170CA1; term_months=36
n_companies: 40
graded checks: billing_total · crm_total · variance · buckets_sum · violations_listed

Model's actual answer

show the model's full answer
ARR reconciliation — 2026-09-05

Basis: Treat supplied statuses as the as-of snapshot; no status-effective dates were provided to verify historical status. Active billing ARR = MRR × 12; cancelled subscriptions contribute zero. Currency was not specified. Variance = CRM − billing.

Totals
- CRM ARR: Σ hubspot_arr = 603,581.76
- Billing ARR: Σ active MRR × 12 = 50,394.94 × 12 = 604,739.28
- Variance: 603,581.76 − 604,739.28 = −1,157.52

Variance decomposition

| Bucket | Arithmetic | CRM − billing |
|---|---|---:|
| Status mismatch: cancelled billing, positive CRM ARR | 4,905.24 + 8,253.24 | +13,158.48 |
| Rounding, provisional* | 16.00 + 20.00 | +36.00 |
| Missing records | 16,497.24 − 28,449.24 | −11,952.00 |
| Other: unexplained ARR difference | 24,396.00 − 26,796.00 | −2,400.00 |
| Total | 13,158.48 + 36.00 − 11,952.00 − 2,400.00 | −1,157.52 |

*The two small differences are consistent with rounding billing ARR to the nearest hundred, but no rounding policy was provided. Classification is provisional, not a confirmed cause.

Mismatched accounts

Missing-side amounts are treated as zero solely for the reconciliation; a missing record does not establish zero actual ARR. Owners below are suggested functions; named owners were not provided.

| Company alias | CRM ARR | Billing ARR calculation | Variance | Bucket / suggested owner |
|---|---:|---|---:|---|
| C-0C8323BF | 4,905.24 | 0; cancelled | +4,905.24 | Status mismatch — RevOps; confirm cancellation with Billing |
| C-0DC4FB8C | 8,253.24 | 0; cancelled | +8,253.24 | Status mismatch — RevOps; confirm cancellation with Billing |
| C-0D66DF9E | 23,200.00 | 1,932.00 × 12 = 23,184.00 | +16.00 | Provisional rounding — RevOps; validate precision policy |
| C-14D70CE0 | 18,200.00 | 1,515.00 × 12 = 18,180.00 | +20.00 | Provisional rounding — RevOps; validate precision policy |
| C-0D5BBE3A | 16,497.24 | Missing subscription | +16,497.24 | Missing billing record — Billing Operations; investigate coverage/mapping |
| C-21629AA4 | Missing | 2,370.77 × 12 = 28,449.24 | −28,449.24 | Missing CRM record — RevOps; investigate coverage/mapping |
| C-0F7269D7 | 24,396.00 | 2,233.00 × 12 = 26,796.00 | −2,400.00 | Other; cause unknown — RevOps + Billing Operations |

Agreement-end-date violations

| Subscription | Company alias | Term months | cf_agreement_end_date |
|---|---|---:|---|
| SUB-0002 | C-1794A52C | 24 | Missing |
| SUB-0019 | C-22170CA1 | 36 | Missing |

Both violate the requirement that subscriptions with terms other than 12 months have cf_agreement_end_date populated.
kvm-month-over-monthPASS 1.00

? The Question

Report the four core KVMs for 2026-08 against 2026-07: giving rate, redemptions per user, 1:1 meetings engagement, pulse check engagement. For each: value, prior value, absolute and relative change, direction. Name the KVM with the largest relative move and the segment (plan_tier or size_band) driving it if the data supports that; otherwise say the data does not. Under 200 words plus one table.
cost $1.0114 · 50s · in 45,107 / out 3,652 tok
✓ four_kvms✓ biggest_mover_named✓ direction_correct✓ segment_named✓ word_cap

✓ Correct Answer (ground truth)

aggregation: simple mean of company-level values per month
kvms: {"giving_rate": {"2026-07": 0.6023, "2026-08": 0.6027, "abs_change": 0.0004, "rel_change_pct": 0.07, "direction": "up"}, "redemptions_per_user": {"2026-07": 1.73, "2026-08": 1.7302, "abs_change": 0.00
biggest_mover: pulse_engagement
driving_segment: size_band=enterprise; 2026-07=0.55; 2026-08=0.2743; rel_change_pct=-50.13
segment_detail: {"smb": {"2026-07": 0.6588, "2026-08": 0.6573, "rel_change_pct": -0.23}, "mid_market": {"2026-07": 0.593, "2026-08": 0.5942, "rel_change_pct": 0.2}, "enterprise": {"2026-07": 0.55, "2026-08": 0.2743,
n_company_rows_per_month: 30
graded checks: four_kvms · biggest_mover_named · direction_correct · segment_named · word_cap

Model's actual answer

show the model's full answer
Unweighted means across 30 companies; user counts are missing, so user-weighted KVMs cannot be calculated.

| KVM | 2026-08 | 2026-07 | Absolute change | Relative change | Direction |
|---|---:|---:|---:|---:|---|
| Giving rate | 60.2713% | 60.2297% | +0.0417 pp | +0.0692% | Up |
| Redemptions per user | 1.730163 | 1.729983 | +0.000180 | +0.0104% | Up |
| 1:1 meetings engagement | 44.7177% | 44.6887% | +0.0290 pp | +0.0649% | Up |
| Pulse check engagement | 50.8610% | 60.0587% | −9.1977 pp | −15.3145% | Down |

Arithmetic: monthly mean = sum ÷ 30; absolute change = August − July; relative change = absolute change ÷ July × 100. August/July sums:
- Giving: 18.0814 / 18.0689.
- Redemptions: 51.9049 / 51.8995.
- 1:1: 13.4153 / 13.4066.
- Pulse: 15.2583 / 18.0176.

Pulse check engagement had the largest relative move. The enterprise size_band drives the observed decline: 54.998% → 27.428%, or −27.570 pp (−50.1291%). Its contribution is −27.570 × 10/30 = −9.190 pp, accounting for 99.9166% of the net decline. This is descriptive attribution, not evidence of cause.

All companies are tier_three; the data does not support comparisons across plan_tier.
rewards-ytd-sectionPASS 1.00

? The Question

Produce the redemption section of the weekly rewards report through the last completed month (2026-08): redemption count, spend, unique redeemers, redemptions per redeemer, provider mix as percent of spend, top 5 countries by redemptions. Say which month is the last completed month. Provider shares must sum to 100.
cost $2.6015 · 94s · in 153,702 / out 7,053 tok
✓ count✓ spend✓ unique_redeemers✓ month_named✓ provider_mix✓ top_country

✓ Correct Answer (ground truth)

redemption_count: 378
total_spend_usd: 27846.00
unique_redeemers: 235
redemptions_per_redeemer: 1.61
provider_mix_pct_of_spend: TangoCard=11.6; Tremendous=30.54; custom=39.05; Snappy=18.81
top5_countries_by_count: country=US; count=244 | country=CA; count=24 | country=AU; count=21 | country=GB; count=17 | country=NL; count=17
graded checks: count · spend · unique_redeemers · month_named · provider_mix · top_country

Model's actual answer

show the model's full answer
Redemptions — YTD through August 2026

Last completed month: 2026-08 (August). Reporting period: January 1–August 31, 2026, using only the supplied redemptions_ytd.csv rows.

- Redemption count: 90 + 192 + 37 + 59 = 378
- Spend: $3,230 + $8,505 + $10,873 + $5,238 = $27,846 USD
- Unique redeemers: 235 distinct user_key values across all providers
- Redemptions per redeemer: 378 ÷ 235 = 1.61

Provider mix (% of spend):
- TangoCard: $3,230 ÷ $27,846 × 100 = 11.60%
- Tremendous: $8,505 ÷ $27,846 × 100 = 30.54%
- custom: $10,873 ÷ $27,846 × 100 = 39.05%
- Snappy: $5,238 ÷ $27,846 × 100 = 18.81%
- Total: 11.60% + 30.54% + 39.05% + 18.81% = 100.00%

Top 5 countries by redemption count:
1. US — 244
2. CA — 24
3. AU — 21
4. GB — 17 (tied)
4. NL — 17 (tied)
customer-success
churn-save-eligibilityPASS 1.00

? The Question

Which at-risk accounts qualify for a churn-save offer under the documented eligibility rules, what amount is at stake per account and in total, and which play fits each (usage revival, executive touch, commercial concession)? Cite the signal that justifies each play. List accounts that look at risk but do not qualify and why.
cost $0.7287 · 50s · in 45,270 / out 2,979 tok
✓ eligible_set✓ total_at_stake✓ plays_cited✓ noneligible_named✓ rules_applied

✓ Correct Answer (ground truth)

snapshot_date: 2026-09-05
rules: health_score < 60, churn_save_eligible_amount > 0, renewal within 120 days of snapshot
eligible: account_alias=C-0F6C0F34; amount_at_stake=49707.00; play=executive touch; justifying_signal=champion_active is false - no executive sponsor engaged | account_alias=C-0B827671; amount_at_stake=25365.00; play=usage revival; justifying_signal=usage_trend_3m=declining over the last 3 months | account_alias=C-0B360C78; amount_at_stake=35748.00; play=commercial concession; justifying_signal=usage stable/growing with seat utilization 75% - risk is commercial, not adoption | account_alias=C-0B0F1BAB; amount_at_stake=5494.00; play=executive touch; justifying_signal=champion_active is false - no executive sponsor engaged | account_alias=C-0CA21961; amount_at_stake=16829.00; play=usage revival; justifying_signal=seat utilization 26% is below 50% | account_alias=C-0E9C27D1; amount_at_stake=41235.00; play=commercial concession; justifying_signal=usage stable/growing with seat utilization 85% - risk is commercial, not adoption … (+2 more)
total_amount_at_stake: 224601.00
non_eligible_at_risk: account_alias=C-0BC71BDD; health_score=55 | account_alias=C-0BA71F12; health_score=52 | account_alias=C-0F6694C3; health_score=43 | account_alias=C-0BE96399; health_score=54 | account_alias=C-0F876796; health_score=47 | account_alias=C-0FCCD2DF; health_score=43 … (+1 more)
n_accounts: 30
graded checks: eligible_set · total_at_stake · plays_cited · noneligible_named · rules_applied

Model's actual answer

show the model's full answer
8 accounts qualify, with a total churn-save eligible amount of 224,601.00.

Eligibility requires all three rules in eligibility_rules.csv:
- R1: health_score < 60.
- R2: churn_save_eligible_amount > 0.
- R3: renewal within 120 days of 2026-09-05: cutoff = 2027-01-03.

Amounts below use churn_save_eligible_amount, not full ARR. Currency is not specified.

### Qualifying accounts

Account signals and amounts are from churnzero_accounts.csv. Days are measured from the documented snapshot, not today. Play recommendations are analytical judgments; the supplied rules do not define play selection.

| Account | Health | Renewal (days away) | Eligible amount | Play and supporting signal |
|---|---:|---|---:|---|
| C-0F6C0F34 | 51 | 2026-10-03 (28) | 49,707.00 | Executive touch: champion_active=false despite growing usage. |
| C-0B827671 | 56 | 2026-11-14 (70) | 25,365.00 | Usage revival: declining usage; seat utilization = 113 ÷ 202 × 100 = 55.9%. |
| C-0B360C78 | 57 | 2026-10-28 (53) | 35,748.00 | Undetermined: usage growing, champion active, utilization = 246 ÷ 327 × 100 = 75.2%. No commercial concern is documented to justify a concession. |
| C-0B0F1BAB | 38 | 2026-09-23 (18) | 5,494.00 | Executive touch: champion_active=false, health 38, renewal only 18 days away. |
| C-0CA21961 | 58 | 2026-12-28 (114) | 16,829.00 | Usage revival: flat usage with utilization = 84 ÷ 325 × 100 = 25.8%. |
| C-0E9C27D1 | 39 | 2026-09-24 (19) | 41,235.00 | Undetermined: health 39 and imminent renewal establish urgency, but champion is active and utilization = 134 ÷ 157 × 100 = 85.4%. No commercial concern is documented to justify a concession. |
| C-0CEF69FD | 53 | 2026-11-21 (77) | 32,621.00 | Executive touch: champion_active=false despite growing usage. |
| C-0D3278C7 | 54 | 2026-11-12 (68) | 17,602.00 | Usage revival: declining usage; utilization = 126 ÷ 380 × 100 = 33.2%. |

Total arithmetic:
49,707 + 25,365 + 35,748 + 5,494 + 16,829 + 41,235 + 32,621 + 17,602 = 224,601.00.

Commercial concession: no account has a supplied price, budget, or contract-friction signal. Eligibility alone does not establish that a concession is the right play, and the eligible amount is not a documented discount amount.

### At risk but not eligible

All accounts below pass R1 but fail R2 and/or R3.

| Account | Health | Eligible amount field | Renewal (days away) | Why excluded |
|---|---:|---:|---|---|
| C-0BC71BDD | 55 | 0.00 | 2026-10-27 (52) | Fails R2: amount is zero. |
| C-0BA71F12 | 52 | 6,824.00 | 2027-04-11 (218) | Fails R3: 218 > 120 days. |
| C-0F6694C3 | 43 | 0.00 | 2027-03-21 (197) | Fails R2 and R3: zero amount; 197 > 120 days. |
| C-0BE96399 | 54 | 0.00 | 2026-10-29 (54) | Fails R2: amount is zero. |
| C-0F876796 | 47 | 19,958.00 | 2027-02-06 (154) | Fails R3: 154 > 120 days. |
| C-0FCCD2DF | 43 | 0.00 | 2027-04-23 (230) | Fails R2 and R3: zero amount; 230 > 120 days. |
| C-10A56B0F | 54 | 0.00 | 2026-12-12 (98) | Fails R2: amount is zero. |
expansion-kitPASS 1.00

? The Question

A customer replied to outreach (reply included). Build the expansion kit: seat coverage (licensed over headcount), two lines on usage health, headroom in seats and ARR at the current per-seat rate, who replied and whether they can buy, the right buyer from our contacts, and a reply email under 150 words that moves toward the right person without being pushy and cites one real usage fact.
cost $0.5687 · 35s · in 39,942 / out 1,730 tok
✓ coverage_pct✓ headroom_arr✓ right_buyer✓ replier_not_buyer✓ usage_fact✓ email_under_150

✓ Correct Answer (ground truth)

company_alias: C-0DDFC9A7
coverage_pct: 37.5
headroom_seats: 250
per_seat_annual_rate: 60.00
headroom_arr: 15000.00
replier: Maria S.
replier_can_buy: False
right_buyer: name=Dana R.; title=VP People
usage_facts: active_users_2026_08=126; active_users_2026_03=88; growth_pct_6m=43.2; licensed_seat_utilization_pct=84.0
usage_health_lines: Monthly active users grew from 88 (2026-03) to 126 (2026-08), +43.2% over 6 months., Last month 126 of 150 licensed seats were active (84.0% utilization), approaching the seat cap.
graded checks: coverage_pct · headroom_arr · right_buyer · replier_not_buyer · usage_fact · email_under_150

Model's actual answer

show the model's full answer
Expansion kit: C-0DDFC9A7

Seat coverage: 150 licensed / 400 headcount × 100 = 37.5%.

Usage health:
- Monthly active users increased every month, from 88 in March to 126 in August: +38 users; 38 / 88 × 100 = 43.2% growth.
- August utilization: 126 / 150 licensed seats × 100 = 84.0%. Maria S. reports the team loves Bonusly and the recognition feed is busy weekly.

Expansion headroom:
- Seats: 400 − 150 = 250.
- Current annual per-seat rate: $9,000 / 150 = $60.
- Potential incremental ARR: 250 × $60 = $15,000, assuming full-headcount coverage at the current rate—not confirmed demand.

Who replied: Maria S., People Operations Coordinator, last engaged 2026-09-02. She explicitly cannot make purchasing decisions and offered an introduction.

Right buyer: Dana R., VP People, last engaged 2026-05-18. Maria S. identifies her as responsible for budget and seat expansion and says she has been asking about usage numbers.

Reply email

Subject: RE: Growing your team's recognition program

Hi Maria,

Thanks for the context—glad the team is enjoying Bonusly! Monthly active users reached 126 in August, up from 88 in March.

An introduction to Dana would be helpful, if you're still happy to make one. I can share a short usage summary so she has the numbers she's been asking about, and we can explore whether broader access would be useful. No need to make any seat decisions now.

Thanks for your help,
Cole
onboarding-statusPASS 1.00

? The Question

Prep the CSM for the mid-onboarding call: what is complete (integrations connected, allowance set, admins added, first recognition given, first redemption), what is not, early engagement signals, and the three things to cover on the call. Mark nothing complete without a data field that shows it.
cost $0.4556 · 30s · in 16,475 / out 1,056 tok
✓ complete_items✓ incomplete_items✓ three_items✓ grounded

✓ Correct Answer (ground truth)

company_alias: C-0D284E42
days_live: 25
checklist: {"slack_integration_connected": {"complete": true, "field": "integration_slack", "value": "2026-08-12"}, "hris_integration_connected": {"complete": false, "field": "integration_hris", "value": ""}, "a
engagement_signals: avg_active_givers_week1=4.3; avg_active_givers_last_week=13.0; latest_active_givers=15; days_with_activity=25
call_items: Connect the HRIS integration (integration_hris is empty) to automate user provisioning and offboarding., Get to first redemption (first_redemption_at is empty): walk admins through the rewards catalog and confirm allowance funding., Build the giving habit: daily active givers averaged 4.3 in week 1 vs 13.0 last week - agree on manager nudges and a recognition cadence.
graded checks: complete_items · incomplete_items · three_items · grounded

Model's actual answer

show the model's full answer
C-0D284E42 — Mid-onboarding call prep

Complete, supported by fields:
- Slack connected: integration_slack = 2026-08-12.
- Allowance set: allowance_set = 2026-08-13.
- Admins added: admins_added = 2.
- First recognition given: first_recognition_at = 2026-08-15 14:22.

Not confirmed complete:
- HRIS connected: integration_hris is blank.
- First redemption: first_redemption_at is blank.
Blank fields indicate missing evidence, not proof these actions never happened.

Early engagement signals:
- Daily active givers increased from 3 on 2026-08-11 to 15 on 2026-09-04: 15 − 3 = 12 more; 12 ÷ 3 × 100 = 400% growth.
- Growth is not uninterrupted: active givers fell from 7 to 5 on 2026-08-18, for example.
- Usage records active givers before the recorded first recognition on 2026-08-15; clarify the metric definition or timestamp discrepancy.
- Employee count and giver identities are missing, so participation rate and repeat-giver retention cannot be calculated.

Three things to cover on the call:
1. Confirm HRIS connection status, identify any blocker, and agree on an owner and next step if incomplete.
2. Confirm whether any redemption has occurred; if not, walk through the redemption process and identify barriers.
3. Review growing giver activity with the two admins, clarify the pre-recognition usage discrepancy, and agree on how to sustain participation.
renewal-risk-conflicting-datesPASS 1.00

? The Question

Produce a 90-day renewal risk brief. Two systems hold renewal dates and disagree on some accounts; decide which to trust per account and say why (multi-year contracts are known to be wrong in ChurnZero). For every renewal: company, CSM, ARR, date used, seat utilization, 3-month usage trend, risk rating with one sentence of evidence. Flag every disagreement. Close with total ARR renewing and ARR at risk.
cost $1.1391 · 76s · in 65,287 / out 6,701 tok
✓ total_renewing✓ arr_at_risk✓ disagreements_flagged✓ trust_rule

✓ Correct Answer (ground truth)

snapshot_date: 2026-09-05
window: 2026-09-05 to 2026-12-04
trust_rule: multi-year contracts: Chargebee is authoritative (ChurnZero known wrong); otherwise systems agree or Chargebee wins
accounts: 20 items — e.g. account_alias=C-0B144C78; csm=Cole Ingram; arr=30899.00; trusted_renewal_date=2026-11-02; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=75.4; usage_3m_ratio=1.03; risk=low; evidence=3-month usage ratio 1.03 (last3 avg 103 vs prior3 100), seat utilization 75% | account_alias=C-0B20DB64; csm=Dana Mercer; arr=21770.00; trusted_renewal_date=2026-10-07; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=56.6; usage_3m_ratio=1.0; risk=medium; evidence=3-month usage ratio 1.00 (last3 avg 295 vs prior3 295), seat utilization 57% | account_alias=C-0B344485; csm=Elena Sinclair; arr=64384.00; trusted_renewal_date=2026-11-16; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=78.0; usage_3m_ratio=1.04; risk=low; evidence=3-month usage ratio 1.04 (last3 avg 241 vs prior3 231), seat utilization 78% | account_alias=C-0B369871; csm=Dana Mercer; arr=85128.00; trusted_renewal_date=2026-10-29; trusted_source_why=systems agree (annual term); in_90d_window=True; dates_disagree=False; seat_utilization_pct=75.1; usage_3m_ratio=1.04; risk=low; evidence=3-month usage ratio 1.04 (last3 avg 330 vs prior3 316), seat utilization 75%
disagreements: account_alias=C-0B7D2C30; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-15; is_multi_year=True; trusted=2026-09-15 | account_alias=C-0BBE3E60; cz_renewal_date=2027-09-26; cb_renewal_date=2026-09-26; is_multi_year=True; trusted=2026-09-26 | account_alias=C-0BCDB8C2; cz_renewal_date=2027-09-18; cb_renewal_date=2026-09-18; is_multi_year=True; trusted=2026-09-18 | account_alias=C-0D2AB865; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-22; is_multi_year=True; trusted=2026-09-22 | account_alias=C-0F5D2323; cz_renewal_date=2026-09-10; cb_renewal_date=2026-09-29; is_multi_year=True; trusted=2026-09-29
n_disagreements: 5
total_arr_renewing_90d: 1048715.00
arr_at_risk_90d: 359409.00
n_accounts: 20
graded checks: total_renewing · arr_at_risk · disagreements_flagged · trust_rule

Model's actual answer

show the model's full answer
90-day renewal risk brief

Scope: No as-of date was provided. Assuming September 1–November 29, 2026, the 90 days following the latest usage month, all 20 accounts renew within the window.

Method:
- Seat utilization = seats_used ÷ seats × 100, from churnzero_renewals.csv.
- Three-month trend uses June → July → August active_users from usage_12m.csv; percentage change = (August − June) ÷ June × 100. Example: C-0B7D2C30 = (84 − 97) ÷ 97 × 100 = −13.4%.
- Analyst-defined risk rules, not supplied company policy: High = utilization below 50% or usage decline of at least 10%; Medium = utilization below 70% unless High; Low = otherwise.
- ARR at risk includes High + Medium, representing exposed ARR—not predicted losses.

Date reconciliation

Trust Chargebee for the five disagreements below because each account has a multi-year contract, which is known to be wrong in ChurnZero. For every other account, use the shared date: both systems agree and Chargebee reports a 12-month, non-multi-year term.

| Company | ChurnZero date | Chargebee date used | Reason |
|---|---|---|---|
| C-0B7D2C30 | 2026-09-10 | 2026-09-15 | Disagreement; 36-month multi-year contract. |
| C-0BCDB8C2 | 2027-09-18 | 2026-09-18 | Disagreement; 36-month multi-year contract. |
| C-0D2AB865 | 2026-09-10 | 2026-09-22 | Disagreement; 24-month multi-year contract. |
| C-0BBE3E60 | 2027-09-26 | 2026-09-26 | Disagreement; 24-month multi-year contract. |
| C-0F5D2323 | 2026-09-10 | 2026-09-29 | Disagreement; 24-month multi-year contract. |

Renewal-level assessment

“CB override” references the disagreement above; “Agreed” means both systems support the date. ARR, CSM and seat counts come from churnzero_renewals.csv.

| Company | CSM | ARR | Date used / basis | Seat utilization | Jun → Jul → Aug users; change | Risk — evidence |
|---|---|---:|---|---|---|---|
| C-0B7D2C30 | Dana Mercer | $65,901 | 2026-09-15 / CB override | 274/476 = 57.6% | 97 → 94 → 84; −13.4% | High — Usage fell 13.4% with only 57.6% of seats used. |
| C-0BCDB8C2 | Cole Ingram | $54,427 | 2026-09-18 / CB override | 232/424 = 54.7% | 127 → 118 → 110; −13.4% | High — Usage declined each month and utilization is 54.7%. |
| C-0D2AB865 | Elena Sinclair | $38,022 | 2026-09-22 / CB override | 250/407 = 61.4% | 125 → 117 → 109; −12.8% | High — Usage fell 12.8%, exceeding the decline threshold. |
| C-0BBE3E60 | Dana Mercer | $30,993 | 2026-09-26 / CB override | 74/114 = 64.9% | 39 → 35 → 33; −15.4% | High — Usage fell 15.4% despite 64.9% seat utilization. |
| C-0F5D2323 | Cole Ingram | $90,647 | 2026-09-29 / CB override | 111/390 = 28.5% | 20 → 21 → 18; −10.0% | High — Utilization is only 28.5% and usage fell 10.0%. |
| C-0EC6999D | Elena Sinclair | $79,419 | 2026-10-03 / Agreed | 31/112 = 27.7% | 17 → 16 → 15; −11.8% | High — Utilization is only 27.7% and usage declined each month. |
| C-0B20DB64 | Dana Mercer | $21,770 | 2026-10-07 / Agreed | 214/378 = 56.6% | 294 → 298 → 294; 0.0% | Medium — Usage is flat, but seat utilization is only 56.6%. |
| C-0BBC4E7A | Cole Ingram | $56,374 | 2026-10-10 / Agreed | 228/337 = 67.7% | 142 → 141 → 139; −2.1% | Medium — Utilization is below 70% and usage declined slightly. |
| C-0FD551AB | Elena Sinclair | $48,815 | 2026-10-14 / Agreed | 210/376 = 55.9% | 123 → 122 → 126; +2.4% | Medium — Modest usage growth does not remove the 55.9% utilization concern. |
| C-0F9F8F13 | Dana Mercer | $46,230 | 2026-10-18 / Agreed | 199/352 = 56.5% | 185 → 185 → 182; −1.6% | Medium — Utilization is only 56.5% with slightly declining usage. |
| C-0BC34584 | Cole Ingram | $16,740 | 2026-10-22 / Agreed | 327/494 = 66.2% | 104 → 104 → 106; +1.9% | Medium — Usage grew slightly, but utilization remains below 70%. |
| C-0B7A7546 | Elena Sinclair | $35,062 | 2026-10-25 / Agreed | 182/205 = 88.8% | 64 → 65 → 63; −1.6% | Low — Utilization is 88.8% and the usage decline is limited to 1.6%. |
| C-0B369871 | Dana Mercer | $85,128 | 2026-10-29 / Agreed | 317/422 = 75.1% | 326 → 330 → 333; +2.1% | Low — Utilization exceeds 70% and usage increased each month. |
| C-0B144C78 | Cole Ingram | $30,899 | 2026-11-02 / Agreed | 169/224 = 75.4% | 101 → 101 → 106; +5.0% | Low — Utilization is 75.4% and usage grew 5.0%. |
| C-0FC4DBB8 | Elena Sinclair | $94,732 | 2026-11-05 / Agreed | 356/464 = 76.7% | 189 → 191 → 193; +2.1% | Low — Utilization exceeds 70% and usage increased each month. |
| C-0D5BBE3A | Dana Mercer | $39,740 | 2026-11-09 / Agreed | 85/102 = 83.3% | 88 → 90 → 91; +3.4% | Low — Utilization is 83.3% and usage increased each month. |
| C-0FB9D5AF | Cole Ingram | $63,158 | 2026-11-13 / Agreed | 144/199 = 72.4% | 173 → 173 → 176; +1.7% | Low — Utilization exceeds 70% and usage grew 1.7%. |
| C-0B344485 | Elena Sinclair | $64,384 | 2026-11-16 / Agreed | 224/287 = 78.0% | 238 → 240 → 244; +2.5% | Low — Utilization is 78.0% and usage increased each month. |
| C-0CB2C1B4 | Dana Mercer | $40,628 | 2026-11-20 / Agreed | 386/473 = 81.6% | 47 → 48 → 49; +4.3% | Low — Reported seat utilization is 81.6% and usage is growing, although August activity is just 49 users. |
| C-22170CA1 | Cole Ingram | $45,646 | 2026-11-24 / Agreed | 251/294 = 85.4% | 143 → 148 → 146; +2.1% | Low — Utilization is 85.4% and August usage remains above June despite July’s higher reading. |

Data limitation: The seat snapshot has no timestamp, and the relationship between seats_used and active_users is undefined. August active_users exceeds seats_used for C-0B20DB64, C-0B369871, C-0D5BBE3A, C-0FB9D5AF and C-0B344485; the metrics are therefore kept separate rather than reconciled by assumption. Contract cancellation rights and customer renewal intent are not provided, so ratings are adoption-based signals only.

ARR roll-up

- High: $65,901 + $54,427 + $38,022 + $30,993 + $90,647 + $79,419 = $359,409.
- Medium: $21,770 + $56,374 + $48,815 + $46,230 + $16,740 = $189,929.
- Low: $35,062 + $85,128 + $30,899 + $94,732 + $39,740 + $63,158 + $64,384 + $40,628 + $45,646 = $499,377.

Total ARR renewing: $359,409 + $189,929 + $499,377 = $1,048,715.
ARR at risk: $359,409 + $189,929 = $549,338, or $549,338 ÷ $1,048,715 × 100 = 52.4%.
ticket-theme-synthesis0.80

? The Question

Synthesize the quarter's support tickets into 4 to 7 themes. Do not trust existing tags; read the text. Per theme: name, count, share, distinct accounts, ARR affected, two ticket ids, one-line recommendation. Separate broad patterns from single-account noise and rank by ARR exposure, not volume.
cost $0.9160 · 54s · in 57,127 / out 3,991 tok
✗ theme_count_range✓ counts_verifiable✓ single_account_flagged✓ arr_ranked✓ ticket_ids

✓ Correct Answer (ground truth)

quarter_window: 2026-06-01 through 2026-08-31
total_tickets: 80
themes: {"points_not_posting": {"count": 20, "share_pct": 25.0, "distinct_accounts": 9, "arr_affected": "31100.00", "example_ticket_ids": ["IC-460001", "IC-460002"], "single_account": false}, "slack_sync": {"
ranking_by_arr_exposure: hris_provisioning, redemption_failures, billing_dispute, points_not_posting, slack_sync
single_account_noise_theme: billing_dispute
note: existing tags are noisy/unreliable; classify by text
sample_ticket_ids: 80 items — e.g. IC-460004, IC-460016, IC-460001, IC-460041
graded checks: theme_count_range · counts_verifiable · single_account_flagged · arr_ranked · ticket_ids

Model's actual answer

show the model's full answer
Scope: all 80 supplied tickets, June–August 2026. Themes come from body_text, not tags. Ranked by affected ARR, counting each account’s ARR once per theme—not once per ticket. ARR exposure is not confirmed revenue loss; resolution and churn outcomes are missing.

1. HRIS provisioning failures — broad pattern
   Count/share: 12 tickets; 12 ÷ 80 × 100 = 15.0%.
   Distinct accounts: 3.
   ARR affected: C-0B2213A9 ($36,000) + C-0F6C0F34 ($30,000) + C-0DDFC9A7 ($48,000) = $114,000.
   Evidence: IC-460059 reports new-hire accounts not being created; IC-460062 reports skipped hires without logged errors.
   Recommendation: Reconcile HRIS rosters against provisioned accounts, recover missing hires, and alert on silent sync failures.

2. Redemption and gift-card failures — broad pattern
   Count/share: 18 tickets; 18 ÷ 80 × 100 = 22.5%.
   Distinct accounts: 7.
   ARR affected: C-0CEF69FD ($8,900) + C-0B827671 ($10,700) + C-0FCCD2DF ($9,600) + C-0F876796 ($8,700) + C-14264ABD ($11,000) + C-0D9CA315 ($9,600) + C-0B0F1BAB ($10,300) = $68,800.
   Evidence: IC-460025 reports checkout hanging and failing; IC-460024 reports points deducted despite a gift-card order error.
   Recommendation: Trace checkout through fulfillment and reconcile failed orders with deducted points or missing gift cards.

3. Invoice seat-count and renewal-pricing errors — single-account concentration
   Count/share: 16 tickets; 16 ÷ 80 × 100 = 20.0%.
   Distinct accounts: 1.
   ARR affected: C-0E9C27D1 ($52,000) counted once = $52,000.
   Evidence: IC-460069 reports billing for 200 seats versus 150 licensed; IC-460078 reports the wrong annual-renewal tier price.
   Recommendation: Assign one billing owner to reconcile licensed seats and tier pricing, correct discrepancies, and validate the next invoice.
   Interpretation: Material account-specific escalation, not evidence of a broad billing problem; repeated tickets must not inflate ARR exposure.

4. Recognition points not posting — broad pattern
   Count/share: 20 tickets; 20 ÷ 80 × 100 = 25.0%.
   Distinct accounts: 9.
   ARR affected: C-0D3278C7 ($3,500) + C-0BF20542 ($4,500) + C-0D0B047C ($4,500) + C-0BE96399 ($2,700) + C-0D284E42 ($3,400) + C-0D6CC8E3 ($4,200) + C-21FEBCBB ($2,900) + C-0DD0626C ($2,500) + C-0B2895EF ($2,900) = $31,100.
   Evidence: IC-460004 reports delivered recognitions without points; IC-460016 reports team-wide points not posting.
   Recommendation: Reconcile delivered recognitions against point balances, repair missing credits, and monitor posting delays.

5. Slack integration failures — broad pattern
   Count/share: 14 tickets; 14 ÷ 80 × 100 = 17.5%.
   Distinct accounts: 4.
   ARR affected: C-0B843542 ($4,400) + C-10A56B0F ($5,400) + C-0BA71F12 ($3,900) + C-8C2E8F00 ($5,200) = $18,900.
   Evidence: IC-460047 reports recognition sync stopping and its toggle resetting; IC-460051 reports team-wide slash-command errors.
   Recommendation: Test sync-setting persistence, reauthorization, and slash commands end to end, then add integration-health alerts.

Check: 12 + 18 + 16 + 20 + 14 = 80 tickets; shares total 100%. The themes cover 24 distinct accounts with no cross-theme account overlap: $114,000 + $68,800 + $52,000 + $31,100 + $18,900 = $284,800 affected ARR.
marketing
case-study-matchPASS 1.00

? The Question

Given the prospect, rank the three most similar existing customers with public case studies for use as social proof. Similarity on industry, size band, use case, region. Cite the fields that drove each match. Do not name any customer without a case study and never include billing data or contact names.
cost $0.4487 · 42s · in 32,658 / out 1,390 tok
✓ top3✓ decoys_excluded✓ cites_fields✓ no_billing

✓ Correct Answer (ground truth)

top3: company_alias=C-11C31562; similarity_score=3 | company_alias=C-64171065; similarity_score=3 | company_alias=C-A13C193D; similarity_score=3
decoys_excluded: company_alias=C-D6217CAA; similarity_score=4; reason=no_case_study | company_alias=C-C153868F; similarity_score=4; reason=no_case_study
constraints: no_billing_data, no_contact_names, only_customers_with_has_case_study_true_named
prospect_fields_used: industry, size_band, use_case, region
prospect_alias: C-82AF3719
graded checks: top3 · decoys_excluded · cites_fields · no_billing

Model's actual answer

show the model's full answer
For prospect C-82AF3719, scoring gives 1 point per exact match on industry, size_band, use_case, and region. All three customers have has_case_study=true and tie at 3/4; ranking breaks ties by use_case, then industry.

1. C-64171065 — 1 + 1 + 1 + 0 = 3/4
   Matches: industry=Technology; size_band=Mid-Market; use_case=employee_recognition.
   Differs: region=NA-East versus prospect’s NA-West.

2. C-11C31562 — 0 + 1 + 1 + 1 = 3/4
   Matches: size_band=Mid-Market; use_case=employee_recognition; region=NA-West.
   Differs: industry=Manufacturing versus prospect’s Technology.

3. C-A13C193D — 1 + 1 + 0 + 1 = 3/4
   Matches: industry=Technology; size_band=Mid-Market; region=NA-West.
   Differs: use_case=retention versus prospect’s employee_recognition.

Case-study URLs and content were not provided; eligibility is based only on has_case_study=true.
channel-attribution-mathPASS 1.00

? The Question

Compute channel performance for the trailing 6 months. Paid: spend, SQMs, SQOs, cost per SQM, cost per SQO, SQM-to-SQO rate, pipeline amount, pipeline per dollar. Organic: volume, SQO rate, pipeline. Report a channel with spend and zero SQMs as undefined, not zero. Flag rows where SQO date precedes SQM date. Finish with a reallocation recommendation and your confidence given sample sizes.
cost $0.9101 · 72s · in 57,395 / out 4,748 tok
✓ undefined_channel✓ undefined_not_zero✓ flagged_rows✓ best_channel✓ reallocation

✓ Correct Answer (ground truth)

months: 2026-03, 2026-04, 2026-05, 2026-06, 2026-07, 2026-08
per_channel: {"linkedin_ads": {"spend_usd": 24000, "sqms": 25, "sqos": 8, "pipeline_amount": 96000, "cost_per_sqm": 960.0, "cost_per_sqo": 3000.0, "sqm_to_sqo_rate": 0.32, "pipeline_per_dollar": 4.0}, "organic_sea
undefined_channels: paid_social
flagged_rows_sqo_before_sqm: CT-000041, CT-000044
best_pipeline_per_dollar_channel: paid_search
reallocation_recommendation: from=paid_social; to=paid_search; rationale=paid_social has spend but zero SQMs (cost per SQM undefined, not zero); paid_search has the highest pipeline per dollar.
confidence_note: webinars and referral have small SQM samples (<20); treat rates as directional.
graded checks: undefined_channel · undefined_not_zero · flagged_rows · best_channel · reallocation

Model's actual answer

show the model's full answer
Period: March–August 2026, the six months supplied. Volume means contacts with an SQM date; SQOs are contacts with a populated SQO date. Results below retain flagged records as reported.

Paid performance

| Channel | Spend | SQMs | SQOs | Cost/SQM | Cost/SQO | SQM→SQO | Pipeline | Pipeline/$ |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| paid_search | $36,000 | 40 | 18 | $900 | $2,000 | 45.00% | $720,000 | 20.00 |
| linkedin_ads | $24,000 | 25 | 8 | $960 | $3,000 | 32.00% | $96,000 | 4.00 |
| paid_social | $18,000 | 0 | 0 | Undefined | Undefined | Undefined | $0 | 0.00 |
| webinars | $9,000 | 12 | 5 | $750 | $1,800 | 41.67% | $60,000 | 6.67 |

Arithmetic:
- paid_search: spend = 6 × $6,000 = $36,000; cost/SQM = $36,000 ÷ 40 = $900; cost/SQO = $36,000 ÷ 18 = $2,000; conversion = 18 ÷ 40 = 45%; pipeline = 18 × $40,000 = $720,000; pipeline/$ = $720,000 ÷ $36,000 = 20.
- linkedin_ads: spend = 6 × $4,000 = $24,000; cost/SQM = $24,000 ÷ 25 = $960; cost/SQO = $24,000 ÷ 8 = $3,000; conversion = 8 ÷ 25 = 32%; pipeline = 8 × $12,000 = $96,000; pipeline/$ = $96,000 ÷ $24,000 = 4.
- paid_social: spend = 6 × $3,000 = $18,000; cost/SQM and cost/SQO = $18,000 ÷ 0, undefined; conversion = 0 ÷ 0, undefined. No contact rows are supplied for this channel; recorded pipeline = $0, and pipeline/$ = $0 ÷ $18,000 = 0.
- webinars: spend = 6 × $1,500 = $9,000; cost/SQM = $9,000 ÷ 12 = $750; cost/SQO = $9,000 ÷ 5 = $1,800; conversion = 5 ÷ 12 = 41.67%; pipeline = 5 × $12,000 = $60,000; pipeline/$ = $60,000 ÷ $9,000 = 6.67.

Organic performance

| Channel | Volume (SQMs) | SQOs | SQO rate | Pipeline |
|---|---:|---:|---:|---:|
| organic_search | 30 | 10 | 10 ÷ 30 = 33.33% | 10 × $9,000 = $90,000 |
| referral | 15 | 6 | 6 ÷ 15 = 40.00% | 6 × $8,000 = $48,000 |

Organic spend and pre-SQM lead/traffic volume are not provided.

Date-quality flags

- CT-000044, linkedin_ads: SQO date 2026-07-18 precedes SQM date 2026-07-23 by 5 days.
- CT-000041, linkedin_ads: SQO date 2026-06-09 precedes SQM date 2026-06-14 by 5 days.

If those two SQOs and their pipeline are withheld pending validation, while retaining all 25 SQMs, linkedin_ads becomes: 8 − 2 = 6 SQOs; 6 ÷ 25 = 24% conversion; $24,000 ÷ 6 = $4,000/SQO; 6 × $12,000 = $72,000 pipeline; $72,000 ÷ $24,000 = 3 pipeline/$.

Reallocation recommendation and confidence

Pause paid_social pending attribution/completeness checks. If its zero recorded SQMs are confirmed, redirect its budget primarily toward a staged paid_search expansion, with a smaller webinars test. Consider reducing linkedin_ads after resolving its date errors: its pipeline/$ trails both alternatives even before adjustment.

Confidence: moderate in prioritizing paid_search, with the largest paid sample at 40 SQMs/18 SQOs and 20 pipeline/$. Low in scaling webinars aggressively from only 12 SQMs/5 SQOs, or declaring referral superior from 15 SQMs/6 SQOs. Confidence in an exact allocation is low: marginal returns, closed-won revenue, attribution completeness, and subsequent conversion of recent SQMs are not provided. Pipeline/$ is not realized revenue ROI.
competitor-battlecard-updatePASS 1.00

? The Question

Update the battlecard for the competitor. Sections: one-line positioning, pricing with source and date (newer source wins, note the conflict), where they win, where we win, objections and responses, recent changes, our 12-month win/loss record against them. Cite a snippet id for every factual claim. Rep opinion on a call is not a fact about the competitor. Anything from the old card you cannot re-source gets marked unverified.
cost $0.8402 · 72s · in 29,552 / out 4,013 tok
✓ competitor_named✓ pricing✓ win_loss✓ unverified_marked✓ citations

✓ Correct Answer (ground truth)

competitor: Rivally
pricing: value_per_user_per_month=7; source_snippet=S17; source_date=2026-08-12
win_loss_12mo: wins=13; losses=7
unverified_old_claims: Rivally lacks a Slack integration., Rivally was acquired by WorkHuman in 2025.
rep_opinion_snippet_ids_not_facts: S09, S21
citation_ids_available: 25 items — e.g. S01, S02, S03, S04
graded checks: competitor_named · pricing · win_loss · unverified_marked · citations

Model's actual answer

show the model's full answer
Battlecard: Rivally

1. One-line positioning

Points-based recognition with a reviewer-praised feed and multi-language support for distributed EU teams; also offers the Rivally Pulse engagement-survey add-on. [S02, S12, S06]

Old-card “for mid-market” positioning: unverified as a target-market claim; a mid-market reviewer’s experience does not establish vendor positioning. [S04]

2. Pricing — source, date, and conflict

- Current published price: Recognition Starter is $7/user/month, annual billing required. Source: pricing_page, 2026-08-12. [S17]
- Conflict: the old card’s $5/user/month reflected the 2026-01-20 pricing page; the 2026-04-01 page also showed $5 for Recognition Starter. The newer pricing page wins. Increase: $7 − $5 = $2/user/month; $2 ÷ $5 × 100 = 40%. [S03, S08, S17]
- Deal-specific evidence: call notes recorded a $6.50/user/month quote for a 500-seat prospect on an annual term, dated 2026-06-02. This is not current published pricing. [S13]
- On 2026-08-14, a prospect reported $7/user/month list with 15% off for a three-year term: $7 × (1 − 0.15) = $5.95/user/month. Treat as a reported offer, not a generally available discount. [S18]
- Rivally Pulse is separately priced, not bundled; the supplied announcement gives no add-on price. Source: press, 2026-09-01. [S23]

3. Where they win

Evidence-backed strengths, not documented reasons for our losses:

- EU requirements: EU data residency announced generally available; an EU enterprise reviewer praised distributed-team fit and multi-language support. [S15, S12]
- Setup and Slack: one mid-market reviewer reported setup under a week and Slack integration working out of the box. [S04]
- Recognition experience: reviewers praised the points-based feed and described it as engaging. [S02, S16]
- Support: one reviewer praised response times under four hours; this is not a verified SLA. [S22]

4. Where we win

- Analytics depth: call notes report an 800-seat prospect choosing Bonusly over Rivally for this reason. No deal alias is supplied, so this cannot be linked to a named deal. [S25]
- Discovery opportunities, not proven Bonusly advantages: reviewers described limited/basic analytics, missing SCIM, CSV-only analytics exports that complicated migration, and missing bulk recognition editing. Validate current functionality and demonstrate our capabilities rather than asserting unsupported superiority. [S02, S07, S10] [S20, S24]
- EMEA rewards coverage: one reviewer described Rivally’s EMEA catalog as thinner than its US catalog. This does not establish that Bonusly’s catalog is broader. [S14]

5. Objections and responses

- “Rivally is cheaper.” Response: “Its latest published Starter price is $7/user/month, billed annually. Let’s compare equivalent scope and terms.” Bonusly pricing is not provided, so relative cost cannot be established. [S17]
- “We need EU support.” Response: “Rivally announced generally available EU data residency, and a reviewer praised multi-language support. Let’s validate your residency and language requirements.” Do not dismiss these strengths. [S15, S12]
- “Their analytics are sufficient.” Response: “Let’s test your required dashboards and export workflows.” Reviews describe basic reporting and CSV-only analytics exports; one prospect reportedly chose Bonusly for analytics depth. [S07, S20, S25]
- “Rivally has no Slack integration.” Response: reject this old-card assertion. Mark it unverified and contradicted by a reviewer reporting a working Slack integration. [S04]
- “Their UI is clunky / they discount aggressively.” Response: neither is an established competitor fact; these are explicitly unverified AE opinions. [S09, S21]

6. Recent changes

- 2026-03-05: launched Rivally Pulse, a lightweight engagement-survey add-on. [S06]
- 2026-05-09: hired an ex-Workday VP EMEA to lead European expansion. [S11]
- 2026-07-01: opened a Dublin office and announced EU data residency generally available. [S15]
- 2026-08-12: published Starter pricing increased to $7/user/month. [S17]
- 2026-08-20: announced Microsoft Teams app v2 in public preview—not general availability. [S19]
- 2026-09-01: Pulse exited beta and remained a separately priced add-on. [S23]

Old-card acquisition claim: “Rivally was acquired by WorkHuman in 2025” is unverified; no supplied snippet supports it. S01 reports a $40M Series C led by Northgate Ventures, not an acquisition; it does not conclusively disprove the claim. [S01]

7. Our 12-month win/loss record

Period: September 2025–August 2026, the supplied 12-month dataset.

Source: deals_with_competitor.csv. These records have no snippet IDs; none can truthfully be supplied.

- Wins: 13
- Losses: 7
- Total: 13 + 7 = 20
- Win rate: 13 ÷ 20 × 100 = 65%
- Loss rate: 7 ÷ 20 × 100 = 35%

Wins: Deal-A9FD43, Deal-7AA785, Deal-44C524, Deal-0D0CD6, Deal-D5B790, Deal-5C636E, Deal-67BE14, Deal-1B6969, Deal-F03E7B, Deal-072E31, Deal-F65C8F, Deal-E46EAB, Deal-1D2392.

Losses: Deal-7767F5, Deal-5645A5, Deal-C6FFAA, Deal-D263E0, Deal-935746, Deal-9066A6, Deal-72A02F.

Deal values and outcome reasons are missing from the deal file. The September 2026 anecdote is outside this reporting window and is not added to the record. [S25]
nurture-sequence-diagnosisPASS 1.00

? The Question

Review the sequences. Per sequence: sent, open rate, reply rate, meeting rate, weakest step. Identify tracking errors (opened above sent), audience overlap between sequences, and the failure mode for anything under 2 percent reply. One change per weak sequence and which to fix first. Under 350 words.
cost $1.5648 · 93s · in 81,008 / out 3,561 tok
✓ tracking_error✓ overlap_pair✓ weakest_step✓ fix_priority

✓ Correct Answer (ground truth)

per_sequence: {"New Logo Nurture": {"sent": 1386, "opened": 490, "replied": 90, "meetings": 27, "open_rate": 0.3535, "reply_rate": 0.0649, "meeting_rate": 0.0195, "weakest_step": 3}, "Expansion Nurture": {"sent": 8
tracking_error: sequence=Expansion Nurture; step=2; sent=300; opened=340; issue=opened exceeds sent
audience_overlap_pair: Cold Outbound - HR Leaders, Cold Outbound - People Ops
all_pairwise_overlaps: ['Cold Outbound - HR Leaders', 'Cold Outbound - People Ops'], ['Expansion Nurture', 'New Logo Nurture']
under_2pct_reply: Cold Outbound - HR Leaders
failure_mode: Cold Outbound - HR Leaders=opens healthy (~37%) but replies near zero from step 1 -> body/CTA not landing, not a deliverability or subject-line problem
fix_priority_first: Cold Outbound - HR Leaders
graded checks: tracking_error · overlap_pair · weakest_step · fix_priority

Model's actual answer

show the model's full answer
Rates = summed events ÷ summed sends × 100; these are per-send rates, not unique-contact conversion rates. “Weakest” means lowest reply rate.

• New Logo Nurture: sent = 500+458+428 = 1,386. Open: 490/1,386 = 35.35%; reply: 90/1,386 = 6.49%; meeting: 27/1,386 = 1.95%. Weakest: step 3, replies 18/428 = 4.21%.

• Expansion Nurture: sent = 300+300+275 = 875. Reported open: 565/875 = 64.57%—unreliable. Reply: 59/875 = 6.74%; meeting: 12/875 = 1.37%. Weakest: step 3, replies 12/275 = 4.36%.

• Cold Outbound - HR Leaders: sent = 600+595+590 = 1,785. Open: 545/1,785 = 30.53%; reply: 8/1,785 = 0.45%; meeting: 0/1,785 = 0%. Weakest: step 3, replies 1/590 = 0.17%; all steps generated zero meetings.

• Cold Outbound - People Ops: sent = 400+386+377 = 1,163. Open: 340/1,163 = 29.23%; reply: 29/1,163 = 2.49%; meeting: 6/1,163 = 0.52%. Weakest: step 3, replies 6/377 = 1.59%.

Tracking error: Expansion Nurture step 2 has 340 opened versus 300 sent: 113.33%, exceeding sends by 40. Change: reconcile opening-event deduplication before using open rates.

Audience overlap:
• New Logo Nurture ↔ Expansion Nurture: 2 contacts, CT-000301 and CT-000624.
• Cold Outbound - HR Leaders ↔ Cold Outbound - People Ops: 21 contacts, including CT-000849, CT-000884 and CT-001159.
No other cross-sequence overlaps appear in the supplied rows. Overlap creates duplicate-outreach risk; timing is unavailable.

Under-2% failure modes and changes:
• Cold Outbound - HR Leaders: all steps fail—5/600 = 0.83%, 2/595 = 0.34%, 1/590 = 0.17%. Recorded opens do not translate into replies. Change: test a lower-friction reply CTA.
• Cold Outbound - People Ops: step 3 alone fails; late-step response deterioration. Change: replace step 3 with a new value proposition.

Fix Cold Outbound - HR Leaders first for performance. Actual causes remain unproven: copy, delivery diagnostics and enrollment timing are missing.
weekly-marketing-goals-rollupPASS 1.00

? The Question

Produce the weekly marketing goals update for the active quarter: SQMs, SQOs, DS2s, closed-lost MIA rate, same-quarter close count, active pipeline coverage against target. For each: QTD actual, target, delta, pace (on/behind/ahead given days elapsed). One paragraph on what moved this week.
cost $0.5836 · 46s · in 42,324 / out 1,602 tok
✓ metrics_present✓ deltas✓ coverage_ratio✓ pace✓ paragraph

✓ Correct Answer (ground truth)

days_elapsed: 66
days_in_quarter: 92
elapsed_fraction: 0.7174
metrics: {"SQMs": {"qtd_actual": 230, "target": 300, "delta": -70, "expected_at_day_elapsed": 215.22, "pace": "ahead"}, "SQOs": {"qtd_actual": 84, "target": 120, "delta": -36, "expected_at_day_elapsed": 86.09,
coverage_ratio_active_pipeline: 0.75
pace_rule: expected = target * days_elapsed/days_in_quarter; within +/-5% of expected = on, above = ahead, below = behind; lower_better metrics invert.
this_week_paragraph_required: True
graded checks: metrics_present · deltas · coverage_ratio · pace · paragraph

Model's actual answer

show the model's full answer
Q3-2026 weekly marketing goals update
July 1–September 30; day 66 of 92 = 71.74% elapsed.

Pace assumes linear progress: expected QTD = target × 66/92. Delta = actual − full-quarter target. MIA rate is compared directly with its ceiling, not prorated.

| Metric | QTD actual | Target | Delta | Pace |
|---|---:|---:|---:|---|
| SQMs | 230 | 300 | −70 | Ahead: 230 vs. 300 × 66/92 = 215.22 |
| SQOs | 84 | 120 | −36 | Behind: 84 vs. 120 × 66/92 = 86.09 |
| DS2s | 40 | 75 | −35 | Behind: 40 vs. 75 × 66/92 = 53.80 |
| Closed-lost MIA rate | 5/25 = 20% | ≤10% | +10 percentage points | Behind: 20% exceeds 10% ceiling |
| Same-quarter close count | 10 | 20 | −10 | Behind: 10 vs. 20 × 66/92 = 14.35 |
| Active pipeline coverage against target | 3,000,000 / 4,000,000 = 75% | 4,000,000 (100%) | −1,000,000; −25 percentage points | Ahead on linear proxy: 3,000,000 vs. 4,000,000 × 66/92 = 2,869,565.22 |

Active pipeline is a point-in-time balance; its pace above is a linear proxy, not evidence of pipeline growth. Currency is not specified.

What moved this week: Weekly movement cannot be determined because no prior-week snapshot or weekly activity data was provided. The supplied QTD snapshot shows SQMs ahead of linear pace, SQOs, DS2s and same-quarter closes behind, and the closed-lost MIA rate above its ceiling. Active pipeline covers 75% of target, ahead of the 71.74% elapsed-time proxy but still 1,000,000 below the full target.
communication
ceo-slack-compression0.80

? The Question

Compress the forecast analysis below into a single Slack message for the CEO. Rules: at most three sentences; under 80 words; lead with the decision or the so-what; keep every distinct number that matters and the one caveat that changes the conclusion; cut any sentence that does not carry a claim, a number, or a directive. No em dashes, no throat-clearing phrases.
cost $0.6898 · 35s · in 40,249 / out 1,510 tok
✓ max_3_sentences✓ under_80_words✓ weighted_present✗ caveat_present✓ no_em_dash

✓ Correct Answer (ground truth)

graded checks: max_3_sentences · under_80_words · weighted_present · caveat_present · no_em_dash

Model's actual answer

show the model's full answer
Treat Q3 2026’s 115,976.75 forecast as provisional: all 7 in-quarter COMMIT deals lack documented why-buys. For 54/86 in-quarter deals, 100% × 44,729 COMMIT + 35% × 203,565 BEST_CASE (24 deals) = 115,976.75; 23 PIPELINE deals contribute 0. Exclude 32 October deals totaling 227,575, including COMMIT Deal-D348E1 (13,770, 2026-10-15) and 9 BEST_CASE totaling 28,240; the remaining 22 PIPELINE deals carry zero weight.
stalled-deal-followup-emailPASS 1.00

? The Question

Write a follow-up email for this stalled deal. Under 120 words. Reference one specific prior touch from the engagement history, make one clear ask, and match the tone of the prior thread. No invented details.
cost $0.5081 · 33s · in 23,990 / out 443 tok
✓ word_cap✓ references_touch✓ single_ask

✓ Correct Answer (ground truth)

deal_alias: Deal-0D2F7A
amount: 5100
stage: DS3
owner: Alex Franklin
days_since_last_engagement: 31
word_cap: 120
must_reference: engagement_id=ENG-00002; date=2026-07-28; type=meeting; why=most substantive prior touch; concrete prospect reactions to milestone awards and points catalog
single_cta: True
tone: warm, concise, professional (matches prior thread)
allowed_facts: 14 items — e.g. Deal-0D2F7A, $5,100, DS3, Alex Franklin
allowed_new: ENG-00001, ENG-00002, ENG-00003, ENG-00004, ENG-00005
graded checks: word_cap · references_touch · single_ask

Model's actual answer

show the model's full answer
Subject: Deal-0D2F7A — following up on pricing

Hi team,

Following up on my August 5 email with the July 28 demo recap and pricing for 150 seats.

Could you share whether that pricing fits the recognition budget?

Thanks,
Alex Franklin
funnel-digest-paragraphPASS 1.00

? The Question

Write the weekly GTM digest: one paragraph per team (Marketing, Sales, CS, PLG) in Bonusly brand voice. Every number must trace to the extracts. No customer contact names, no employee performance commentary, no compensation. Four paragraphs, under 400 words total.
cost $0.6366 · 57s · in 38,622 / out 872 tok
✓ four_paragraphs✓ word_cap✓ teams

✓ Correct Answer (ground truth)

required_paragraphs: Marketing, Sales, CS, PLG
allowed_numbers: 16 items — e.g. 2, 3, 9, 08
word_cap: 400
banned: customer contact names, employee performance commentary, compensation
graded checks: four_paragraphs · word_cap · teams

Model's actual answer

show the model's full answer
Marketing: For the week of 2026-08-31, Marketing brought in 46 SQMs against a target of 52, a gap of 6 (52 − 46 = 6). Webinars contributed 18 of those SQMs, and paid search cost per SQM held at $150. There’s progress to celebrate and a target still to reach.

Sales: Sales converted 14 SQOs, set 9 DS2 meetings, and created $310,000 in new pipeline. The week also brought 3 same-quarter closes. Let’s celebrate those wins while keeping new pipeline distinct from closed revenue.

CS: A win worth recognizing: 2 renewals saved this week. Team NPS moved to 61; the extract doesn’t provide the prior score, so the direction and size of the change aren’t available. Heading into next week, 3 red-flag accounts remain open and need attention.

PLG: Welcome to the 412 new signups! Activation was 31%, and 38 companies reached the aha moment of 10 recognition gives. That’s a milestone worth celebrating: more workplaces putting recognition into practice.
partner-digest-sectionPASS 1.00

? The Question

Write the partner digest for the period. Every partner in the roster appears once: active plays get status with numbers, quiet partners get one line. Partner-sourced pipeline count and amount must match the deal data. Nothing omitted, nothing added.
cost $0.4526 · 52s · in 16,609 / out 955 tok
✓ all_partners✓ pipeline_amount✓ quiet_one_line

✓ Correct Answer (ground truth)

roster: partner_name=Apex Rewards Co; status=active | partner_name=HRCloud Partners; status=active | partner_name=CultureBridge; status=active | partner_name=WorkWell Group; status=active | partner_name=Recogniq; status=quiet | partner_name=KudosWave; status=quiet … (+2 more)
per_partner_pipeline: {"Apex Rewards Co": {"count": 2, "amount": 275000}, "HRCloud Partners": {"count": 1, "amount": 140000}, "CultureBridge": {"count": 2, "amount": 135000}}
partner_sourced_pipeline: count=5; amount=550000
quiet_partners: Recogniq, KudosWave, PeopleFirst Advisors, TotalPerk
active_no_deals: WorkWell Group
rule: every roster partner named exactly once; quiet partners get one line; no partners outside roster.
graded checks: all_partners · pipeline_amount · quiet_one_line

Model's actual answer

show the model's full answer
Partner digest | 2026-08-24 to 2026-09-04

Partner-sourced pipeline: 2 + 1 + 2 + 0 = 5 deals; 275,000 + 140,000 + 135,000 + 0 = 550,000. All five deals have utm_source = Partner. Currency not provided.

- Apex Rewards Co — Co-webinar confirmed for 09-15. Two sourced opportunities, both in DS1: Deal-DDAAF2 (180,000) + Deal-2CF33E (95,000) = 275,000.
- HRCloud Partners — Integration referral completed security review and moved to DS2. One sourced opportunity: Deal-F1CDA5 (140,000) = 140,000.
- CultureBridge — Lunch-and-learn produced two sourced opportunities, both early stage: Deal-096E1D (60,000) + Deal-067213 (75,000) = 135,000.
- WorkWell Group — Wants to restart the joint playbook in Q4; planning call booked for 09-09. Zero sourced deals; pipeline amount 0.
- Recogniq — Quiet; no updates or sourced deals in the provided data.
- KudosWave — Quiet; no updates or sourced deals in the provided data.
- PeopleFirst Advisors — Quiet; no updates or sourced deals in the provided data.
- TotalPerk — Quiet; no updates or sourced deals in the provided data.
rfp-answer-from-policyPASS 1.00

? The Question

Answer the 10 questionnaire items from the policy excerpts only. Cite the excerpt id for every answer. Where the excerpts do not answer the question, mark it unanswerable and say what document would; do not guess. Tone: vendor response, direct.
cost $0.4463 · 54s · in 31,535 / out 821 tok
✓ answered_count✓ unanswerable_flagged✓ citations✓ no_guess

✓ Correct Answer (ground truth)

answered_count: 7
answer_map: {"Q1": ["P1"], "Q2": ["P1"], "Q3": ["P2"], "Q4": ["P3"], "Q5": ["P4"], "Q6": ["P5"], "Q7": ["P6"]}
unanswerable: Q8, Q9, Q10
valid_citation_ids: P1, P2, P3, P4, P5, P6, P7, P8
rule: unanswerable items must be flagged and the document that would answer them named; do not guess.
graded checks: answered_count · unanswerable_flagged · citations · no_guess

Model's actual answer

show the model's full answer
Q1. Yes. All customer data is encrypted at rest using AES-256. [P1]

Q2. Yes. Data in transit is encrypted with TLS 1.2 or higher. [P1]

Q3. Yes. Bonusly maintains an annually renewed SOC 2 Type II report, available to prospects under NDA. [P2]

Q4. Customer data is retained for 90 days after the contract ends, then deleted from production systems. [P3]

Q5. Our subprocessor list is published at trust.bonusly.com. Customers receive 30 days’ notice before a new subprocessor is added. The notification method and notice for other types of changes are unanswerable from the excerpt; our subprocessor notification policy would be needed. [P4]

Q6. Yes. SAML 2.0 single sign-on is available on the Enterprise plan. [P5]

Q7. Production databases are backed up daily. Backups are encrypted and retained for 35 days. [P6]

Q8. Unanswerable. The excerpts do not state whether Bonusly holds ISO 27001 certification. An ISO 27001 certificate or formal certification-status statement would be needed. [P1–P8 reviewed; no supporting excerpt]

Q9. Unanswerable. The excerpts do not specify a contractual uptime SLA percentage. The applicable contractual Service Level Agreement would be needed. [P1–P8 reviewed; no supporting excerpt]

Q10. Unanswerable. The excerpts do not state whether Bonusly will sign a HIPAA Business Associate Agreement. A formal BAA eligibility and execution policy would be needed. [P1–P8 reviewed; no supporting excerpt]
ops-maintenance
skill-file-reconciliationPASS 1.00

? The Question

Reconcile the skill set below against its manifest. Find and report (with severity CRITICAL/WARNING/INFO and an action type TRIM_DESC, UPDATE_BODY, DELETE_SKILL, MERGE, or REVIEW): (1) any two skills whose ALWAYS-trigger phrases overlap/duplicate, (2) any circular delegation chain between skills (name it), (3) any delegation target that does not exist (dangling, name it), (4) any version conflict and which skill should survive, (5) how many manifest descriptions exceed 1,024 characters, (6) any hardcoded page ids, dates, or person names in skill bodies, (7) manifest drift in BOTH directions (files with no manifest row, and manifest rows with no file). Make one proposal per finding; rewrite nothing.
cost $5.1596 · 100s · in 314,706 / out 7,598 tok
✓ duplicate_trigger_found✓ circular_chain✓ dangling_target✓ oversized_desc_count✓ hardcoded_values✓ manifest_drift✓ severity_and_action

✓ Correct Answer (ground truth)

graded checks: duplicate_trigger_found · circular_chain · dangling_target · oversized_desc_count · hardcoded_values · manifest_drift · severity_and_action

Model's actual answer

show the model's full answer
Scope: supplied files and manifest only. “Dangling” means absent from this supplied set, not proven absent from any external installation. Proposals only; nothing rewritten.

1. Overlapping triggers

| Severity | Action | Finding | Proposal |
|---|---|---|---|
| WARNING | TRIM_DESC | `pipeline-intelligence-report` and `weekly-pipeline-report` overlap on “pipeline update,” “run the pipeline update,” pipeline-report requests, and “what's the pipeline look like” / “what does pipeline look like.” Both use ALWAYS triggers. | Reserve scored/tiered pipeline requests for `pipeline-intelligence-report`; reserve weekly performance, funnel, targets, and bookings updates for `weekly-pipeline-report`. |
| WARNING | MERGE | `comms-drafter` and `email-drafter` duplicate “write me an email,” “draft a follow-up,” “what should I say,” “bump email,” and “contract nudge.” These are unconditional routing descriptions, although not literally labeled ALWAYS. | Merge email-specific capabilities, including signature retrieval, into `comms-drafter` and redirect callers there. |
| WARNING | TRIM_DESC | `deal-strategy-coach` claims requests asking which deals are likely to close; `next-to-close` ALWAYS claims “which deals are most likely to close.” | Give shortlist/ranking requests to `next-to-close`; retain strategic diagnosis in `deal-strategy-coach`. |
| WARNING | TRIM_DESC | `deal-strategy-coach` claims “draft a manager email,” overlapping the customer-facing email scopes of both `email-drafter` and `comms-drafter`. | Route draft-only requests to the drafting skill; invoke coaching only when strategy is requested or needed. |
| WARNING | TRIM_DESC | `pipeline-intelligence-report` claims VP requests for “forecast context” and all pipeline questions; `sales-forecast` ALWAYS claims “pipeline forecast,” current-quarter outlook, and deal-level confidence. | Reserve quarter forecast requests for `sales-forecast`, rather than allowing the master-pipeline catch-all to capture them. |
| WARNING | TRIM_DESC | `pipeline-intelligence-report` says never answer pipeline questions without running it, encompassing `stale-pipeline-report`’s ALWAYS triggers for “stale pipeline” and “pipeline hygiene.” | Exclude stale-contact/hygiene requests from the master-pipeline catch-all. |
| INFO | REVIEW | `analysis-validator`, `signalforge-claim-compressor`, and `signalforge-feedback` intentionally overlap on analytical/report outputs. `model-selection` covers every task. Shared triggers alone do not make these duplicate skills. | Retain distinct gate roles but document one execution order, including whether compression requires revalidation. |

2. Circular delegation

| Severity | Action | Finding | Proposal |
|---|---|---|---|
| WARNING | UPDATE_BODY | Conditional reciprocal handoff: `deal-strategy-coach` → `email-drafter` → `deal-strategy-coach`. The coach explicitly invokes the drafter for manager-to-prospect emails; the drafter points strategic requests back to the coach. This is a potential routing loop, not an unconditional recursive call. | Make the drafter return its draft to the invoking coach without redispatching strategy already handled by that coach. |

No other closed delegation chain is established by the supplied bodies. In particular, `next-to-close` → `pipeline-intelligence-report` → `closed-lost-analysis` has no supplied return edge.

3. Dangling delegation/dependency targets

Each row is a separate finding and proposal.

| Severity | Action | Missing target | Supplied caller(s) | Proposal |
|---|---|---|---|---|
| CRITICAL | UPDATE_BODY | `bonusly-data-questions` | `analysis-validator` §12.4 | Replace the unresolved delegation with a verified available specialist target. |
| CRITICAL | UPDATE_BODY | `bonusly-product-questions` | `analysis-validator` §12.4 | Replace the unresolved delegation with a verified available specialist target. |
| CRITICAL | UPDATE_BODY | `bonusly-business-reporting-questions` | `analysis-validator` §12.4 | Replace the unresolved delegation with a verified available specialist target. |
| CRITICAL | UPDATE_BODY | `bonusly-rewards-questions` | `analysis-validator` §12.4 | Replace the unresolved delegation with a verified available specialist target. |
| CRITICAL | UPDATE_BODY | `bonusly-ppp-questions` | `analysis-validator` §12.4 | Replace the unresolved delegation with a verified available specialist target. |
| CRITICAL | UPDATE_BODY | `bonusly-feature-flag-questions` | `analysis-validator` §12.4 | Replace the unresolved delegation with a verified available specialist target. |
| CRITICAL | UPDATE_BODY | `bonusly-deal-desk-questions` | `analysis-validator` §12.4 | Replace the unresolved delegation with a verified available specialist target. |
| CRITICAL | UPDATE_BODY | `bonusly-datadog-questions` | `analysis-validator` §12.4 | Replace the unresolved delegation with a verified available specialist target. |
| CRITICAL | UPDATE_BODY | `bonusly-brand` | `comms-drafter`, `email-drafter`, `sales-forecast`; also routed to by `signalforge-claim-compressor` | Resolve the brand prerequisite to an available skill. |
| CRITICAL | UPDATE_BODY | `prospect-research-multithreading` | `comms-drafter`, `email-drafter`, `deal-strategy-coach` | Resolve the contact-research handoff to an available skill. |
| CRITICAL | UPDATE_BODY | `signalforge-reports` | `pipeline-intelligence-report`, `weekly-pipeline-report` | Resolve the mandatory design-skill dependency and its referenced assets. |
| WARNING | REVIEW | `caveman` | Optional combined use in `signalforge-claim-compressor` | Verify availability before retaining the optional integration. |
| WARNING | REVIEW | `skill-orchestrator` | Sync reference in `analysis-validator`; registration requirement in `signalforge-feedback` | Verify this missing integration target; it is not an explicit runtime delegation in the supplied text. |

Referenced supporting files such as `references/data-sources.md` were not provided. Their existence cannot be verified from this set.

4. Version conflict

| Severity | Action | Finding | Proposal |
|---|---|---|---|
| WARNING | UPDATE_BODY | `analysis-validator` declares v3.6 in its header, changelog, and footer, but its validation-trail template identifies `analysis-validator v3.2`. Full Mode also retains older check ranges, and the decision tree omits G2-F. | Preserve `analysis-validator` v3.6 and align its operational summaries and trail template to that version. |

No separate competing version files were supplied, so no version-based `DELETE_SKILL` proposal is supported. `pipeline-intelligence-report` v6 using a v4 design system is not, by itself, a version conflict.

5. Manifest descriptions exceeding 1,024 characters

INFO | REVIEW — None.

Arithmetic: count(description_chars > 1,024) = 0 of 14. Maximum = 1,006; 1,024 − 1,006 = 18 characters below the limit.

Proposal: retain the descriptions against this length criterion; no length-based `TRIM_DESC` is warranted.

6. Hardcoded page IDs, dates, and person names in bodies

Historical changelog dates and explicitly labeled examples are distinguished from operational constants below.

| Severity | Action | Skill and evidence | Proposal |
|---|---|---|---|
| WARNING | UPDATE_BODY | `analysis-validator`: operational escalation names `Manish` and `Amani`; dated GTM roster containing `Alaina Loori`, `Shealagh Coughlin`, `Bryce Harmon`, `Hugo Lindqvist`, `Dana Mercer`, `Alex Franklin`, `Cole Ingram`, `Gavin Porter`, `Colleen Perry`, `Ellie Barton`, `Ashley Reyer`, `Megan Franz`, `Elena Sinclair`, `Youssef Elkhateeb`, `Amanda Czenkus`, `Ben Castelli`, `Amani Phipps`, `John Thomas`, `Yasmin Wahid`; deprecated spelling references `Ashley Le` and `Tracy`. Dates include April 26, 2026; May 4, 2026; May 9, 2026; March 28, 2023; May 2026 calibration anchors; Q1/Q2 2026 examples. | Resolve operational people and calibration data dynamically; retain historical dates only as provenance. |
| WARNING | UPDATE_BODY | `closed-lost-analysis`: May 2026 observations and field confirmations; MinIO vacation dates May 4–12; “demo on 4/13”; June 3 outreach example. | Separate historical evidence/examples from reusable live instructions and require current evidence before applying them. |
| WARNING | UPDATE_BODY | `deal-strategy-coach`: playbook page ID `2257879045`, April 2026 playbook title, “Pricing — 2026,” and routing names `Perseus` and `Farid`. | Externalize the playbook, pricing version, and routing owners into verified configuration. |
| INFO | REVIEW | `model-selection`: `last_checked: 2026-05-19`, dated changelog, April 14, 2026 deprecation claim, knowledge-cutoff dates, and date-stamped model ID. | Keep dates as registry metadata subject to its refresh procedure, not timeless model facts. |
| WARNING | UPDATE_BODY | `partner-digest`: folder ID `2286616609`; page IDs `2286321666`, `2265382925`, `2236940297`, `2237825028`, `2239365136`, `2238283777`; names `Amani`, `Amani Phipps`, `Kelli`, `Jen Lee`, `Hani`, `Bryce`, `Sara`; May 16, May 19, June 2, 2026 examples; Q2/Q3 2026 page title; changelog `2026-05-17`. | Configure destinations and contacts, and distinguish historical issues from dynamically generated reporting periods. |
| WARNING | UPDATE_BODY | `pipeline-intelligence-report`: May 2026 version/owner verification; hardcoded owners `Bryce Harmon`, `Dana Mercer`, `Cole Ingram`, `Alex Franklin`, `Gavin Porter`; March 2023 stale-table reference. | Resolve owners live and retain dated system claims only as verification history. |
| WARNING | UPDATE_BODY | `sales-forecast`: parent page ID `2232582148`; `Alaina` in Manager Forecast; `Elena` and `Alaina` in changelog; April 27, 2026 and July 9, 2026 dates; residual “Open Q2 Deals” and “Q2 Narrative” despite current-quarter instructions. | Parameterize operational destination, leader, and quarter labels while preserving labeled history/examples. |
| INFO | REVIEW | `signalforge-claim-compressor`: changelog date `2026-05-09` and attribution `JuliusBrussee/caveman`. | Retain these as historical provenance, not operational personalization. |
| WARNING | UPDATE_BODY | `signalforge-feedback`: target page `2295136266`, parent `2234417154`, but activation checklist names Build Log page `2247295002`; example report title uses `Gavin Porter` and “Q2 Pipeline Review.” | Establish one configured feedback destination and align the activation checklist with it. |
| INFO | REVIEW | `stale-pipeline-report`: example dates `5/15`, `5/19`, `5/7`; changelog `2026-06-10`. | Keep these explicitly illustrative/historical so they cannot become live outreach facts. |
| WARNING | UPDATE_BODY | `weekly-pipeline-report`: `Ben Lavin` in title and `Ben` throughout workflow; hardcoded Q2 window “April 1 – June 30, 2026,” Q2/Q3+ buckets, and Q1 2026 static context. | Parameterize recipient and reporting quarter; retain prior-quarter context only when relevant and labeled. |

No literal page IDs, calendar dates, or person names were found in the supplied bodies of `comms-drafter`, `email-drafter`, or `next-to-close`. The HubSpot `1973303` value is an organization/portal ID, not a page ID.

7. Manifest drift in both directions

| Severity | Action | Finding and arithmetic | Proposal |
|---|---|---|---|
| INFO | REVIEW | Files with no manifest row: 14 supplied skill files − 14 matched files = 0. | No manifest additions needed for the supplied files. |
| INFO | REVIEW | Manifest rows with no file: 14 manifest rows − 14 matched files = 0. | No manifest-row removals needed for the supplied set. |

All supplied filename stems also match their manifest `declared_name` and skill frontmatter `name`. Missing referenced dependencies above are not manifest drift: neither their files nor their manifest rows were supplied.
runbook-from-slack-thread0.75

? The Question

Turn the resolved incident thread into a runbook: ordered steps with the exact command or action taken, who ran it, how success was verified, and rollback for each step that changed state. Every step must trace to a message in the thread. Add nothing the thread does not contain; where the thread is unclear, mark the step as needing confirmation.
cost $0.4995 · 66s · in 17,221 / out 1,481 tok
✓ ordered_steps✓ rollback✓ needs_confirmation✗ sourced

✓ Correct Answer (ground truth)

steps: order=1; msg=M01; actor=Bryce Harmon; action=acknowledge PagerDuty alert, take IC; state_change=False; verified_by=M01 | order=2; msg=M02; actor=Farid Osman; action=check queue depth; command=bundle exec rake sidekiq:queue_depth; state_change=False; verified_by=M02 (48,213 pending) | order=3; msg=M04; actor=Farid Osman; action=pause auto-enqueue via feature flag; command=bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'; state_change=True; rollback=bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'; verified_by=M07 (queue depth falling) | order=4; msg=M05; actor=Elena Sinclair; action=cleared the dead set; state_change=True; needs_confirmation=thread gives no command or method for clearing the dead set | order=5; msg=M06; actor=Bryce Harmon; action=scale reward-worker to 6 replicas; command=kubectl scale deployment/reward-worker --replicas=6; state_change=True; rollback=kubectl scale deployment/reward-worker --replicas=3; verified_by=M07/M08 | order=6; msg=M08; actor=Cole Ingram; action=verify queue drained and error rate baseline; command=bundle exec rake sidekiq:queue_depth; state_change=False; verified_by=M08 (returns 0) … (+2 more)
ambiguous_step: order=4; msg=M05; mark=needs-confirmation
rule: no steps beyond what the thread contains; every step traces to a message id.
graded checks: ordered_steps · rollback · needs_confirmation · sourced

Model's actual answer

show the model's full answer
1. Acknowledge alert and take incident command. [M01]
   - Who: Bryce Harmon.
   - Action: Acknowledged the PagerDuty alert for reward-worker queue depth > 10k; took IC.
   - Verification: Not documented — needs confirmation.
   - Rollback: Not documented — needs confirmation.

2. Check queue depth. [M02]
   - Who: Farid Osman.
   - Command: `bundle exec rake sidekiq:queue_depth`
   - Verification: Returned 48,213 pending reward jobs; normal is under 500.

3. Inspect the dead set. [M03]
   - Who: Farid Osman.
   - Action: Reported 112 dead jobs, all Redis::TimeoutError from around 13:58. Exact inspection command not documented — needs confirmation.
   - Verification: Reported findings only; verification method not documented — needs confirmation.

4. Pause enqueue. [M04]
   - Who: Farid Osman.
   - Command: `bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'`
   - Verification: No direct confirmation that enqueue was disabled — needs confirmation.
   - Rollback: `bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'`

5. Clear the dead set — execution details need confirmation. [M05]
   - Who: Elena Sinclair.
   - Action: Cleared the dead set in the console. Exact command not documented — needs confirmation.
   - Verification: Completion was reported; no verification documented — needs confirmation.
   - Rollback: Not documented — needs confirmation.

6. Scale workers from 3 to 6. [M06]
   - Who: Bryce Harmon.
   - Command: `kubectl scale deployment/reward-worker --replicas=6`
   - Verification: No direct replica-count verification documented — needs confirmation. Subsequent queue observations appear in steps 7–8.
   - Rollback: `kubectl scale deployment/reward-worker --replicas=3`

7. Check queue-drain progress. [M07]
   - Who: Farid Osman.
   - Action: Reported queue depth down to 9,400 and falling ~1,200/min. Measurement command not documented — needs confirmation.
   - Verification: Reported observations only; measurement method needs confirmation.

8. Verify queue drained and error rate recovered. [M08]
   - Who: Cole Ingram.
   - Command: `bundle exec rake sidekiq:queue_depth`
   - Action: Checked error rate in Datadog.
   - Verification: Queue-depth command returned 0; Datadog error rate was back to baseline.

9. Re-enable enqueue. [M09]
   - Who: Bryce Harmon.
   - Command: `bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'`
   - Verification: Verified 40 new jobs processed cleanly in the next 3 minutes.
   - Rollback: Not documented for this step — needs confirmation.

10. Scale workers back to 3 and resolve the incident. [M10]
    - Who: Bryce Harmon.
    - Command: `kubectl scale deployment/reward-worker --replicas=3`
    - Action: Declared the incident resolved.
    - Verification: Queue stable at 0. Direct replica-count verification not documented — needs confirmation.
    - Rollback: Not documented for this step — needs confirmation.
datadog-incident-timelinePASS 1.00

? The Question

From the log slice, sequence the failure: first error with timestamp, the cascade in order, the service and job involved, and the Datadog query that would confirm the first error. State what the logs do not show.
cost $0.5414 · 71s · in 19,384 / out 1,570 tok
✓ first_error_ts✓ service_named✓ cascade_ordered✓ query✓ gaps_stated

✓ Correct Answer (ground truth)

first_error: timestamp=2026-09-03T14:01:12Z; service=reward-service; message=Redis::TimeoutError: Connection to Redis at redis-primary:6379 timed out after 5s
cascade_order: service=reward-service; first_error_at=2026-09-03T14:01:12Z | service=sidekiq; first_error_at=2026-09-03T14:01:40Z | service=api-gateway; first_error_at=2026-09-03T14:03:05Z | service=web-app; first_error_at=2026-09-03T14:03:30Z
job_class_involved: RewardGiveJob
confirming_query: service:reward-service status:error "@message:Redis::TimeoutError"
logs_do_not_show: root cause of the Redis timeout, redis-server's own logs, infrastructure metrics (CPU/memory/network), any deploy or config change preceding the incident, who applied mitigation or when (no actor fields in logs)
clean_services: postgres
graded checks: first_error_ts · service_named · cascade_ordered · query · gaps_stated

Model's actual answer

show the model's full answer
First error — 2026-09-03T14:01:12Z, reward-service:
“Redis::TimeoutError: Connection to Redis at redis-primary:6379 timed out after 5s”

Observed cascade (all timestamps UTC, 2026-09-03):
1. 14:01:20, 14:01:30, 14:01:40 — reward-service reports retries exhausted for RewardGiveJob.
2. 14:01:40 — sidekiq reports RewardGiveJob failed with Redis::TimeoutError; retrying in 60s. This shares a timestamp with the last retry-exhaustion entry; finer ordering is unavailable.
3. 14:01:46 — earliest failure in sidekiq_jobs.csv: J-00005, RewardGiveJob, Redis::TimeoutError.
4. 14:02:28 — sidekiq reports another RewardGiveJob failure/retry.
5. 14:02:30 — sidekiq warns: “Queue reward depth above 10,000”.
6. 14:02:36 — J-00013, RecognitionDigestJob, also fails with Redis::TimeoutError, showing another job class is affected.
7. 14:03:05 — api-gateway reports “502 upstream timeout calling reward-service /gives”.
8. 14:03:30 — web-app reports “Give form submission failed: upstream 502 from api-gateway”.
9. Through 14:06:52 — further job failures/retries, gateway timeouts, and form-submission errors appear.
10. 14:22:10 — reward-service reports Redis connection restored and job processing resuming.
11. 14:24:45 — sidekiq reports reward queue depth below 500.

Service/job involved: The first error is in reward-service, connecting to redis-primary:6379. Subsequent errors name RewardGiveJob, executed by sidekiq. The jobs file additionally records RecognitionDigestJob failures.

Datadog query to confirm the first error:
```text
service:reward-service status:error *:"Connection to Redis" *:"timed out after 5s"
```
Set the time range to 2026-09-03T13:55:00Z–2026-09-03T14:02:00Z and sort oldest first. This query is proposed, not executed.

The logs do not show:
- Why Redis timed out, or what restored connectivity.
- A job ID or job class attached to the first timeout; it cannot be attributed to J-00005.
- Trace/request IDs proving causality across services; the cascade is an observed chronological sequence.
- Exact queue depths, customer impact, lost/duplicated rewards, or successful completion of every failed job.
- Explicit recovery of /gives or the web form. Later queue improvement does not establish full end-to-end recovery.
feature-flag-state-summaryPASS 1.00

? The Question

Summarize the state of the listed feature flags: what each controls per the code excerpt, which companies or segments have it on, targeting rules, and any flag with no code reference or no targets. Company counts must match the export. Do not describe a flag that is not in the export.
cost $0.3203 · 91s · in 8,313 / out 573 tok
✓ orphans✓ no_targets✓ counts_match✓ no_invented

✓ Correct Answer (ground truth)

flags: {"recognition_streaks_v2": {"state": "on", "targeting_rules": "segment:beta_companies", "company_count": 42}, "points_budget_guardrails": {"state": "on", "targeting_rules": "all_companies", "company_c
code_referenced: recognition_streaks_v2, points_budget_guardrails, slack_dm_nudges, redeem_flow_redesign, analytics_dashboard_v3, ms_teams_app_v2
orphan_flags_no_code_reference: legacy_give_modal, survey_boosters_q3, paused_offboard_cleanup
flags_with_no_targets: paused_offboard_cleanup
rule: company counts must equal the export; do not describe flags absent from the export.
graded checks: orphans · no_targets · counts_match · no_invented

Model's actual answer

show the model's full answer
Counts below are taken directly from the export; no calculation is needed. Individual company aliases are not provided. For flags marked off, listed targets do not mean the feature is enabled.

| Flag | State | Controls per code excerpt | Targeting rule | Company count |
|---|---|---|---|---:|
| recognition_streaks_v2 | on | Records a give via `StreakTracker.record(give)`. | `segment:beta_companies` | 42 |
| points_budget_guardrails | on | Enforces the company’s points budget for the giver and points amount. | `all_companies` | 220 |
| slack_dm_nudges | on | Permits sending a Slack DM nudge; the job returns without sending if disabled. | `segment:region_na` | 87 |
| redeem_flow_redesign | off | Selects `RedeemV2Component` when enabled; otherwise renders `RedeemV1Component`. | `targeted_list`; company identities not provided | 12 |
| analytics_dashboard_v3 | on | Initializes `AnalyticsV3` for the company. | `segment:tier_three` | 65 |
| ms_teams_app_v2 | off | Permits `TeamsAppV2.install(company)`. | `targeted_list`; company identities not provided | 9 |
| legacy_give_modal | off | No code reference in the provided excerpt; behavior unknown. | `segment:legacy_plan` | 14 |
| survey_boosters_q3 | on | No code reference in the provided excerpt; behavior unknown. | `segment:legacy_plan` | 7 |
| paused_offboard_cleanup | off | No code reference in the provided excerpt; behavior unknown. | No targeting rule provided; no targets | 0 |

`legacy_give_modal` and `survey_boosters_q3` share the targeting rule `segment:legacy_plan` but have different exported company counts. The provided data does not explain the difference.
nda-triagePASS 1.00

? The Question

Triage each NDA as GREEN (standard approval), YELLOW (counsel review), or RED (full legal review). Cite the clause that drives each classification by quoting it. Catch embedded non-solicits, non-competes, missing carve-outs, and unusual term or governing law. One line of reasoning per NDA.
cost $0.4384 · 92s · in 35,025 / out 769 tok
✓ green✓ yellow✓ red✓ non_solicit_caught✓ clause_quoted✓ correct_labels

✓ Correct Answer (ground truth)

classifications: {"NDA-1": {"classification": "GREEN", "driving_reason": "standard mutual NDA: 2-year term, standard carve-outs, Delaware law", "driving_clause_quote": "Carve-outs: information that (a) is or becomes p
non_solicit_in: NDA-2
checks: classification_per_nda, clause_quotes_exist_in_document, non_solicit_caught, carveouts_checked
graded checks: green · yellow · red · non_solicit_caught · clause_quoted · correct_labels

Model's actual answer

show the model's full answer
- NDA-1 — [PARTY A] and Bonusly: GREEN (standard approval) — Mutual confidentiality with all five carve-outs, Delaware law, and bounded duration: clause 2 states “Term: two (2) years from the Effective Date; obligations survive for three (3) years after termination.”
- NDA-2 — [PARTY B] and Bonusly: YELLOW (counsel review) — Clause 4 embeds a non-solicit and no-hire restriction: “During the term of this Agreement and for eighteen (18) months thereafter, neither party shall solicit for employment or hire any employee of the other party with whom it came into contact under this Agreement.”
- NDA-3 — [PARTY C] and Bonusly: RED (full legal review) — Clause 2 imposes a broad non-compete: “For a period of three (3) years following the Effective Date, Recipient shall not, directly or indirectly, engage in or provide services to any business that competes with Discloser's business”; compounded by clause 4, “No confidentiality carve-outs are stated in this Agreement,” the unusually long clause 1 “Term: five (5) years,” and clause 3’s non-Delaware forum: “the laws of the Republic of Ireland, and the parties submit to the exclusive jurisdiction of its courts.”