z-ai

z-ai/glm-5-turbo

Model's answer

not yet run — check back after the runners complete

snowflake-sql-trapped-catalogPASS 1.00

? The Question

Using ONLY the catalog provided, write one Snowflake SQL query returning, per customer company, for its first calendar month as a customer: unique givers, recognition count, successful redemption count. Read the catalog notes carefully: some tables are stale or unpopulated and must be avoided, and one documented filter (excluding deleted givers) must NOT be applied to historical giving counts. Put a comment above each FROM or JOIN explaining the table choice. Return only fenced SQL.

cost $0.0256 · 154s · in 967 / out 5,169 tok

✓ avoids_stale_deals✓ avoids_unpopulated_live✓ uses_cohort_summary✓ succeeded_filter✓ no_giver_deletion_filter✓ returns_sql

✓ Correct Answer (ground truth)

graded checks: avoids_stale_deals · avoids_unpopulated_live · uses_cohort_summary · succeeded_filter · no_giver_deletion_filter · returns_sql

Model's actual answer

show the model's full answer

```sql
SELECT
    ccs.COMPANY_ID,
    ccs.M1_USERS                AS unique_givers,
    ccs.M1_GIVES                AS recognition_count,
    ccs.M1_REDEMPTIONS          AS successful_redemption_count
FROM
    -- COMPANY_COHORT_SUMMARY: one row per self-serve company with pre-aggregated
    -- month-1 metrics aligned to the first calendar month as a customer
    -- (anchored on FIRST_SUB_PAYMENT_DATE).  Chosen over raw tables because:
    --   • No raw recognition/giving-events table appears in the catalog,
    --     so unique givers and recognition count must come from these
    --     pre-aggregated M1_ columns.  M1_USERS is the best available
    --     proxy for unique givers (may include non-giver active users;
    --     exact unique-giver count is unavailable from catalog data).
    --   • M1_REDEMPTIONS is assumed to already count only STATE='succeeded'
    --     per the catalog business rule ("Redemptions: STATE = 'succeeded' only").
    --   • Using the pre-aggregated cohort summary satisfies the rule that
    --     the deleted-giver filter must NOT be applied to historical giving
    --     counts — the M1_GIVES and M1_USERS figures are historical aggregates
    --     and are not subject to that runtime filter.
    --   • REDEMPTION_RECORDS_V2 is the documented source for redemptions but
    --     its full column schema (company key, date key) is not listed in the
    --     catalog; M1_REDEMPTIONS from the cohort summary is used instead.
    PRODUCTION.PLG.COMPANY_COHORT_SUMMARY AS ccs
WHERE
    ccs.FIRST_SUB_PAYMENT_DATE IS NOT NULL   -- restrict to customer companies
;
```

crm-completeness-enrichmentPASS 1.00

? The Question

Audit the CRM extract for completeness and propose fixes. Deals need owner, stage, amount, close date, why-buys; companies need industry, employee count, HQ country; contacts need email, title, persona. Report completeness percent per field. Identify duplicate company clusters (name variants, shared domain) with a survivor per cluster. List invalid emails and domain mismatches. Fill missing company fields only where the enrichment export has a matching row; where CRM and enrichment disagree list both and recommend a source. Never invent a value. End with the 10 fixes with the most pipeline amount at stake.

cost $0.2214 · 667s · in 30,281 / out 25,392 tok

✓ dup_clusters✓ invalid_emails✓ survivor_language✓ completeness_pct✓ no_invent

✓ Correct Answer (ground truth)

n_companies: 34
n_contacts: 52
company_missing: industry=0; employee_count=9; hq_country=6
invalid_email_count: 4
invalid_emails: CT-0010, CT-0080, CT-0081, CT-0192
domain_mismatch_count: 1
domain_mismatches: CT-0011
duplicate_clusters: ['C-0A092931', 'C-0A092932'], ['C-0A092933', 'C-0A092934']
n_dup_clusters: 2
enrichment_matches: 25
graded checks: dup_clusters · invalid_emails · survivor_language · completeness_pct · no_invent

Model's actual answer

show the model's full answer

Here is the full CRM data audit.

---

## 1. FIELD COMPLETENESS

### Companies (34 rows)

| Field           | Filled | Total | %     |
|-----------------|--------|-------|-------|
| domain          | 34     | 34    | 100.0 |
| industry        | 34     | 34    | 100.0 |
| employee_count  | 25     | 34    | 73.5  |
| hq_country      | 28     | 34    | 82.4  |

### Contacts (52 rows)

| Field           | Filled | Total | %     | Notes |
|-----------------|--------|-------|-------|-------|
| company_alias   | 52     | 52    | 100.0 |       |
| email           | 52     | 52    | 100.0 | 4 filled but invalid (see §3) |
| title           | 39     | 52    | 75.0  | 13 empty |
| persona         | 37     | 52    | 71.2  | 15 empty |
| domain          | 52     | 52    | 100.0 |       |

### Deals

**No deals.csv provided.** Completeness unknown for: owner, stage, amount, close_date, why_buys. All five fields cannot be assessed.

---

## 2. DUPLICATE COMPANY CLUSTERS (shared domain)

### Cluster: acme-corp.com

| Alias        | industry    | employee_count | hq_country |
|--------------|-------------|----------------|------------|
| C-0A092931   | Technology  | 500            | US         |
| C-0A092932   | tech        | 510            | USA        |

- **SURVIVOR: C-0A092931** (most-normalized values; both have equal field count)
- **MERGE OUT: C-0A092932** — reassign all contacts/deals to C-0A092931
- **CONFLICTS:** industry `Technology` vs `tech` (same meaning, normalize); employee_count 500 vs 510 (real conflict — recommend ZoomInfo tiebreak); hq_country `US` vs `USA` (same meaning, normalize)

### Cluster: globex.io

| Alias        | industry    | employee_count | hq_country |
|--------------|-------------|----------------|------------|
| C-0A092933   | SaaS        | 200            | US         |
| C-0A092934   | Technology  | 200            | US         |

- **SURVIVOR: C-0A092933** (`SaaS` is more specific than `Technology`)
- **MERGE OUT: C-0A092934** — reassign all contacts/deals to C-0A092933
- **CONFLICT:** industry `SaaS` vs `Technology` — `SaaS` is the more specific taxonomy; retain C-0A092933's value

---

## 3. INVALID EMAILS & DOMAIN MISMATCHES

### Invalid emails

| Contact  | Email              | Issue                         |
|----------|--------------------|-------------------------------|
| CT-0010  | `user0@`           | Malformed: no domain after @  |
| CT-0080  | `user0@`           | Malformed: no domain after @  |
| CT-0081  | `user1@`           | Malformed: no domain after @  |
| CT-0192  | `user2@`           | Malformed: no domain after @  |

### Domain mismatches (email domain ≠ contact domain)

| Contact  | Email                       | Email domain       | Contact domain  |
|----------|-----------------------------|--------------------|-----------------|
| CT-0011  | user1@other-domain.com      | other-domain.com   | 66d1fc.com      |

CT-0011's email is at `other-domain.com` but the contact record says `66d1fc.com` — verify whether this is a personal email or a data-entry error.

---

## 4. ENRICHMENT: FILL MISSING & DISAGREEMENTS

### Missing company fields fillable from enrichment

| Alias      | Field           | CRM value | ZoomInfo value |
|------------|-----------------|-----------|----------------|
| C-EC3025   | employee_count  | (empty)   | 400            |
| C-96039F   | employee_count  | (empty)   | 400            |
| C-44EA29   | employee_count  | (empty)   | 400            |
| C-D04904   | employee_count  | (empty)   | 400            |
| C-B23205   | employee_count  | (empty)   | (empty)        |
| C-60C75F   | employee_count  | (empty)   | 400            |
| C-7BBDFA   | employee_count  | (empty)   | 400            |
| C-50D386   | employee_count  | (empty)   | 400            |

(8 rows gain employee_count from ZoomInfo. C-B23205 listed in enrichment with emp=400 — the enrichment file does include b23205.com with zi_employee_count=400, so C-B23205 is also fillable. Correcting below.)

**Re-verified:** C-B23205 enrichment has zi_employee_count=400. All 8 fills confirmed.

### CRM vs ZoomInfo disagreements

No true semantic disagreements (different numeric or categorical values for the same fact). All differences are **label normalizations** — see §5.

### Companies with NO enrichment match — cannot auto-fill

| Alias      | Domain        | Missing fields (unfillable)           |
|------------|---------------|---------------------------------------|
| C-93C8BF   | 93c8bf.com    | employee_count                        |
| C-EE9FFB   | ee9ffb.com    | hq_country                            |

Additionally, C-2D1F1B (2d1f1b.com) and C-D73B89 (d73b89.com) have empty hq_country where enrichment also has empty hq_country — cannot fill.

---

## 5. INDUSTRY & COUNTRY NORMALIZATION

### Industry — 4 raw variants mapping to 2 canonical values

| Raw value    | Rows | Canonical    | Aliases                                                        |
|--------------|------|--------------|----------------------------------------------------------------|
| tech         | 4    | Technology   | C-66D1FC, C-44EA29, C-60C75F, C-0A092932                      |
| Tech         | 4    | Technology   | C-425E2A, C-BA969B, C-93C8BF, C-C9BB20                        |
| Technology   | 9    | Technology   | C-EC3025, C-92D97D, C-D04904, C-77A95A, C-AA8DDA, C-B25F40, C-332637, C-0A092931, C-0A092934 |
| health care  | 2    | Healthcare   | C-7BBDFA, C-50D386                                            |

`tech`, `Tech ` (with trailing space), and `Technology` all resolve to **Technology**. `health care` resolves to **Healthcare**.

### Country — 3 raw variants for United States

| Raw value      | Rows | Canonical |
|----------------|------|-----------|
| US             | 9    | US        |
| USA            | 6    | US        |
| United States  | 2    | US        |

### ZoomInfo label differences (not disagreements after normalization)

ZoomInfo uses `Computer Software` where CRM uses `Technology`/`tech` — these map to the same canonical value `Technology`. ZoomInfo uses `United States` where CRM uses `US`/`USA` — same canonical `US`. **Recommendation:** adopt CRM canonicals (Technology, US) and treat ZoomInfo values as equivalent, not as conflicts.

---

## 6. TOP 10 FIXES BY PIPELINE AMOUNT AT STAKE

No deals.csv provided — ranking by **employee_count** as proxy for pipeline at stake (larger company = higher potential deal value). Where CRM employee_count is empty, the ZoomInfo enrichment fill value is used.

| Rank | Emp proxy | Fix |
|------|-----------|-----|
| 1    | 1,500     | NORM: C-E51FB7 hq_country `USA` → `US` |
| 2    |   900     | NORM: C-66D1FC industry `tech` → `Technology` |
| 3    |   900     | NORM: C-C9BB20 industry `Tech ` → `Technology` |
| 4    |   900     | NORM: C-2D7423 hq_country `USA` → `US` |
| 5    |   900     | FIX: CT-0010 email `user0@` malformed for C-66D1FC — obtain valid email |
| 6    |   900     | FIX: CT-0011 email `user1@other-domain.com` domain ≠ `66d1fc.com` for C-66D1FC — verify email |
| 7    |   510     | MERGE: C-0A092932 into C-0A092931 (acme-corp.com) — resolve emp_count 500 vs 510, normalize industry/country |
| 8    |   400     | ENRICH: C-EC3025 employee_count: (empty) → 400 from ZoomInfo |
| 9    |   400     | ENRICH: C-96039F employee_count: (empty) → 400 from ZoomInfo |
| 10   |   400     | ENRICH: C-44EA29 employee_count: (empty) → 400 from ZoomInfo |

---

**Summary of data quality issues:**
- 9 employee_count gaps (8 fillable from ZoomInfo, 1 unfillable)
- 6 hq_country gaps (0 fillable from ZoomInfo, 3 unfillable where enrichment is also empty, 3 with no enrichment row at all)
- 2 duplicate-domain clusters requiring merge
- 4 malformed emails, 1 domain-mismatched email
- 10 industry-label normalizations, 8 country-label normalizations
- 0 true semantic disagreements between CRM and ZoomInfo
- Deals file missing — 5 required deal fields (owner, stage, amount, close_date, why_buys) cannot be audited

deal-intelligence

closed-lost-classification0.80

? The Question

Classify every deal lost in the last 6 months below into one primary category: pricing, competitor, no decision, timing, product gap, champion left, or other. Also assign a side (Bonusly, buyer, unknown). Then summarize: category counts, the side split, the number of deals where the structured closed_lost_tag clearly disagrees with the free-text reason, and the two patterns most worth acting on. Base everything only on the tag and free-text provided.

cost $0.0829 · 431s · in 5,081 / out 18,268 tok

✓ total_90✗ competitor_top✓ category_counts_present✓ side_split✓ two_patterns

✓ Correct Answer (ground truth)

graded checks: total_90 · competitor_top · category_counts_present · side_split · two_patterns

Model's actual answer

show the model's full answer

## Closed-Lost Classification (90 deals)

### Deal-by-Deal Classification

| Deal | Tag | My Category | Side | Notes |
|---|---|---|---|---|
| Deal-DB0AAC | Lost- Timing | **timing** | buyer | paused, reconnect later |
| Deal-F7F635 | Competitor | **competitor** | unknown | went another direction, no detail |
| Deal-AC944F | MIA | **no decision** | buyer | unresponsive |
| Deal-214060 | MIA | **no decision** | buyer | unresponsive |
| Deal-91A056 | Lost- Timing | **timing** | buyer | reconnect early 2027 |
| Deal-29326C | Lost- Timing | **timing** | unknown | just "Timing" |
| Deal-5DB9B0 | Lost- Does not fit ICP | **other** | unknown | Spam |
| Deal-831B7B | Lost- Timing | **timing** | buyer | look again in new year |
| Deal-F97C37 | Competitor | **competitor** | Bonusly | other vendor more diversified offerings |
| Deal-13E9CF | Doing nothing/Not a priority/Cost | **no decision** | buyer | ⚠ TAG says "Cost" but reason says "Not a budget issue" |
| Deal-39E25C | Lost- Timing | **timing** | unknown | Timing, reconnect next year |
| Deal-7ED004 | Lost- Budget/Price | **pricing** | buyer | no budget approval |
| Deal-21B045 | MIA | **no decision** | unknown | MIA |
| Deal-B3ABED | Lost- Timing | **timing** | buyer | revisit Q2 next year for 2028 budget |
| Deal-422BA6 | Competitor | **competitor** | buyer | ADP TotalSource PEO preferred partner |
| Deal-ED9AE7 | Lost DM | **no decision** | buyer | timing + budget + authority — no decision made |
| Deal-988493 | MIA | **no decision** | unknown | MIA |
| Deal-381C8C | Competitor | **competitor** | unknown | not moving forward, no detail |
| Deal-F308CA | MIA | **no decision** | buyer | ignored all outreach since April |
| Deal-F1E8A6 | Competitor | **competitor** | unknown | not moving forward, no detail |
| Deal-B6AC09 | Lost- Timing | **timing** | buyer | revisiting 2027 |
| Deal-70F704 | Lost DM | **product gap** | Bonusly | ⚠ TAG "Lost DM" but reason = only wanted anniversary automation (narrow use case) |
| Deal-E6E80A | Lost- Timing | **timing** | unknown | pushed to early 2027 |
| Deal-B038F0 | Lost- Timing | **timing** | unknown | pushed to early 2027 |
| Deal-4664E1 | MIA | **no decision** | buyer | ignored all outreach after intro |
| Deal-175756 | Lost- Timing | **timing** | buyer | hold until 2027 |
| Deal-E74A73 | Doing nothing/Not a priority/Cost | **no decision** | buyer | want to test manually first |
| Deal-DDAB52 | Competitor | **competitor** | Bonusly | Rippl: more features at same cost + no FX issues |
| Deal-ACE061 | Competitor | **competitor** | unknown | likely HeyTaco (rep guessing) |
| Deal-BB78F3 | Lost- Timing | **timing** | buyer | rolling out other survey items first |
| Deal-D48E0B | MIA | **no decision** | unknown | MIA |
| Deal-15DA99 | Lost- Timing | **timing** | buyer | early 2027 |
| Deal-F4AF5D | Lost- Timing | **timing** | unknown | early next year |
| Deal-79B7A1 | Lost- Timing | **timing** | unknown | Timing |
| Deal-583ADB | MIA | **no decision** | unknown | MIA |
| Deal-8E27DA | Feature Request | **product gap** | Bonusly | ⚠ TAG "Feature Request" but they wanted swag, not R&R — wrong product category |
| Deal-2D2F8D | Competitor | **competitor** | unknown | different direction, no detail |
| Deal-E0441F | MIA | **no decision** | Bonusly | stale inherited from departed rep |
| Deal-7CB44D | MIA | **no decision** | buyer | ignored outreach after demo |
| Deal-0F96AA | Competitor | **competitor** | Bonusly | eliminated from RFP finalist stage |
| Deal-1BCA50 | Competitor | **competitor** | buyer | budget + other stakeholder already with another vendor |
| Deal-7CC678 | Competitor | **competitor** | unknown | nothing specific provided |
| Deal-FAC17C | Lost DM | **no decision** | buyer | ⚠ TAG "Lost DM" but DM/champion still there — couldn't get exec approval |
| Deal-242273 | Competitor | **competitor** | Bonusly | other vendor could digitize internal points + onsite spend |
| Deal-50E5D8 | Doing nothing/Not a priority/Cost | **no decision** | buyer | leadership paused |
| Deal-A2C349 | Competitor | **competitor** | Bonusly | staying with Awardco, adding their surveys |
| Deal-9F176A | Lost- Timing | **timing** | buyer | pause until end of year |
| Deal-7B2236 | Doing nothing/Not a priority/Cost | **pricing** | Bonusly | wanted simpler and cheaper |
| Deal-AFA56C | MIA | **no decision** | buyer | unresponsive |
| Deal-C7156E | Competitor | **competitor** | unknown | selected another vendor |
| Deal-C33D91 | Lost- Budget/Price | **pricing** | buyer | budget cuts |
| Deal-9048EB | MIA | **product gap** | Bonusly | ⚠ TAG "MIA" but reason = bad fit + multiple feature gaps |
| Deal-5E64CE | Doing nothing/Not a priority/Cost | **pricing** | buyer | ⚠ TAG "Not a priority/Cost" but reason = Nectar exit fee + contract lock-in |
| Deal-8A0992 | Competitor | **competitor** | buyer | Canadian provider (geography preference) |
| Deal-D0C698 | Competitor | **competitor** | buyer | past Kudos user, wants Kudos again |
| Deal-69CF3D | Lost- Timing | **timing** | unknown | On Hold |
| Deal-ECBF89 | Lost- Timing | **timing** | unknown | On Hold |
| Deal-3618CC | Lost DM | **product gap** | Bonusly | ⚠ TAG "Lost DM" but reason = wanted surveys |
| Deal-EECC02 | Competitor | **competitor** | unknown | went another direction |
| Deal-5AD03E | Competitor | **product gap** | Bonusly | ⚠ TAG "Competitor" but reason = wanted more defined budget access (feature) |
| Deal-D1A623 | Lost- Timing | **timing** | unknown | timing |
| Deal-413C56 | Doing nothing/Not a priority/Cost | **timing** | buyer | ⚠ TAG includes "Cost" but reason = back-to-school priority, CEO not ready |
| Deal-47F1A1 | Competitor | **competitor** | buyer | staying with WorkTango (incumbent) |
| Deal-BF2A98 | Competitor | **competitor** | buyer | HiThrive already deployed |
| Deal-2A292B | Doing nothing/Not a priority/Cost | **no decision** | buyer | building internally |
| Deal-D1AABF | MIA | **no decision** | unknown | no response |
| Deal-FEDBCB | Doing nothing/Not a priority/Cost | **timing** | buyer | ⚠ TAG includes "Cost" but reason = reconnect end of year |
| Deal-1E7DA9 | Competitor | **competitor** | unknown | selected another platform |
| Deal-2BBA21 | MIA | **no decision** | buyer | ignored outreach since intro |
| Deal-286F9C | Competitor | **competitor** | Bonusly | not a good fit |
| Deal-7FBAC6 | Doing nothing/Not a priority/Cost | **no decision** | buyer | leadership paused (again) |
| Deal-369281 | Competitor | **competitor** | buyer | using what's in Paylocity (incumbent) |
| Deal-386F6E | MIA | **no decision** | unknown | no response |
| Deal-9FCD0D | Competitor | **competitor** | buyer | Canadian company (CEO preference) |
| Deal-55867E | Lost- Timing | **no decision** | unknown | ⚠ TAG "Timing (1+ year)" but reason = careful consideration, not moving forward — definitive no, not a delay |
| Deal-DAFB82 | Lost- Budget/Price | **pricing** | buyer | other priorities consuming budget, not budgeted until 2028 |
| Deal-2FEDDB | Doing nothing/Not a priority/Cost | **timing** | buyer | ⚠ TAG includes "Cost" but reason = unsure on timing |
| Deal-64B19A | Competitor | **competitor** | buyer | stayed with Motivosity (incumbent) |
| Deal-3F86A0 | MIA | **no decision** | buyer | unresponsive |
| Deal-096750 | MIA | **no decision** | buyer | ignored four attempts after intro |
| Deal-F325A5 | Lost DM | **champion left** | buyer | layoffs + change in leadership |
| Deal-ABD14C | Doing nothing/Not a priority/Cost | **no decision** | buyer | not interested |
| Deal-79E61A | MIA | **no decision** | unknown | unresponsive |
| Deal-8A119B | Lost- Budget/Price | **pricing** | buyer | didn't get approval |
| Deal-AE7C4E | MIA | **no decision** | unknown | unresponsive |
| Deal-DAB4F1 | MIA | **no decision** | unknown | unresponsive |
| Deal-B4B50F | MIA | **no decision** | unknown | unresponsive |
| Deal-981AD4 | Feature Request | **product gap** | Bonusly | UI doesn't fit + not UK focused |
| Deal-DC77FE | Competitor | **competitor** | Bonusly | other system offered more customization (label points as dollars) |
| Deal-5885B9 | MIA | **no decision** | unknown | MIA |

---

### Summary

**Category counts** (90 deals total):

| Category | Count | % |
|---|---|---|
| no decision | 30 | 33% |
| competitor | 25 | 28% |
| timing | 21 | 23% |
| pricing | 6 | 7% |
| product gap | 6 | 7% |
| champion left | 1 | 1% |
| other | 1 | 1% |

**Side split:**

| Side | Count | % |
|---|---|---|
| buyer | 43 | 48% |
| unknown | 32 | 36% |
| Bonusly | 15 | 17% |

Arithmetic check: 30 + 25 + 21 + 6 + 6 + 1 + 1 = 90. 43 + 32 + 15 = 90.

---

### Tag vs. Reason Disagreements: **12 deals**

| Deal | Tag | Why it disagrees |
|---|---|---|
| Deal-13E9CF | Doing nothing/Not a priority/**Cost** | Reason: "Not a budget issue" |
| Deal-70F704 | **Lost DM** | Reason: only wanted anniversary automation (product gap), not DM access |
| Deal-8E27DA | **Feature Request** | Reason: wanted swag provider, not R&R — wrong product category entirely |
| Deal-FAC17C | **Lost DM** | Reason: couldn't get exec IT Director approval (DM still there) |
| Deal-9048EB | **MIA** | Reason: "bad fit" + "multiple feature gaps" |
| Deal-5E64CE | Doing nothing/Not a priority/Cost | Reason: competitor exit fee / contract lock-in, not deprioritization |
| Deal-3618CC | **Lost DM** | Reason: wanted surveys (product gap) |
| Deal-5AD03E | **Competitor** | Reason: wanted more defined budget access (feature need) |
| Deal-413C56 | Doing nothing/Not a priority/**Cost** | Reason: back-to-school priority + CEO not ready (timing, not cost) |
| Deal-FEDBCB | Doing nothing/Not a priority/**Cost** | Reason: reconnect end of year (timing, not cost) |
| Deal-55867E | **Lost- Timing (1 year or more)** | Reason: "after careful consideration, not moving forward" — definitive no, not a delay |
| Deal-2FEDDB | Doing nothing/Not a priority/**Cost** | Reason: unsure on timing (no cost mention) |

---

### Two Patterns Most Worth Acting On

**1. "Lost DM" tag is masking product-gap signal (3 deals: Deal-70F704, Deal-3618CC, Deal-FAC17C)**

Three deals tagged "Lost DM" actually lost because Bonusly lacked features the buyer needed (anniversary-only automation, surveys, exec-approval workflow). When the team tags these as a decision-maker access problem, the product team never sees the real driver. Fix: require a structured close-lost sub-reason when "Lost DM" is selected — specifically a "was this actually a feature/capability gap?" checkpoint. The 3 deals total $23,116 in pipeline value that was misrouted.

**2. Product capability gaps are the engine behind competitor losses (at least 6 deals, ~$207K pipeline)**

Deal-242273 ($60K) — competitor could digitize internal points currency + onsite spend. Deal-A2C349 ($21.6K) — Awardco has surveys. Deal-F97C37 ($4.3K) — competitor had more diversified offerings. Deal-DDAB52 ($4K) — Rippl offered more at same cost + FX handling. Deal-DC77FE ($8K) — competitor offered customization (label points as dollars). Deal-5AD03E ($24K) — wanted more defined budget access. These are all tagged "Competitor" but the root cause is Bonusly's product not matching a stated capability need. If Bonusly closed even 2–3 of these gaps (surveys, internal points/onsite spend, point labeling/customization, international FX), a meaningful share of these losses becomes winnable. This is the single highest-leverage product investment signal in the dataset.

pipeline-tieringdiscarded

? The Question

Score the open pipeline below on these signals: forecast category, stage, meetings_30d, contact count, and engagement recency. Assign exactly one tier per deal: LOCK, ACTION, BUILD, REVIVE, WATCH, or RISKY (RISKY = the forecast category disagrees with the engagement evidence). Never tier a deal LOCK if it has zero meetings_30d. Do NOT list every deal. Return ONLY a compact JSON object with exactly these keys: - "tier_counts": object mapping each tier to the number of deals in it (must sum to the total deal count), - "tier_examples": object mapping each tier to up to 3 example deal aliases from the data, - "risky_deals": array of the deal aliases you tiered RISKY, - "lock_violations": number of deals tiered LOCK that have zero meetings_30d (should be 0), - "pipeline_shape": one short paragraph on the overall pipeline shape. Note: inbound_emails_30d is 0 for every row due to a data defect, so treat meetings_30d as the inbound signal.

✓ Correct Answer (ground truth)

total: 156
graded checks: json_returned · tier_vocabulary · tier_counts_sum · risky_deals_listed · no_lock_violation_claimed

Model's answer

## Analysis

### TG-001 — Deal-EC3025

> "We need SCIM user provisioning — our security team will not approve any tool that requires manual account management."

**Classification: Real gap.** Product docs explicitly state SCIM is NOT a supported capability. Prospect frames it as a hard security-approval gate, not a nice-to-have.

### TG-002 — Deal-D0D6B5

> "Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us."

**Classification: Real gap.** Product docs confirm ADP Workforce Now is NOT a supported integration. Prospect calls it a dealbreaker.

### TG-003 — Deal-CFE7F4

> "I tried to build a custom report combining recognition frequency with turnover and couldn't — is that possible on our current plan?"

**Classification: Plan gate.** The custom report builder exists — it ships on Enterprise tier per the product docs. The prospect simply doesn't have the right plan. This is a packaging/up-sell issue, not a missing capability.

### TG-004 — Deal-84DBA6

> "We turned on the Slack integration in July, but half our managers still haven't given a single recognition — nobody ever trained them on it."

**Classification: Rollout/enablement issue.** The Slack integration works (supported on all plans per docs). The problem is training and adoption, not a product limitation.

### TG-005 — Deal-36C33F

> (Alex Franklin, rep): "Full transparency — we don't have a native mobile app for hourly workers yet, though it's on the roadmap."

**Classification: Excluded — rep language, not prospect voice.** The mobile-app gap was raised by the rep, not the prospect. The prospect's own response was "The web version should be fine for our office staff for now," which signals acceptance, not a blocking objection. Per instructions, rep language does not count as prospect voice.

---

## Summary — Real Product Gaps

| Deal | Prospect Quote | Gap | Docs Evidence |
|---|---|---|---|
| Deal-EC3025 | "We need SCIM user provisioning — our security team will not approve any tool that requires manual account management." | No SCIM user provisioning | Docs: "SCIM user provisioning … NOT currently listed as supported" |
| Deal-D0D6B5 | "Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us." | No ADP Workforce Now integration | Docs: "ADP Workforce Now integrations are NOT currently listed as supported" |

**Note on deal amounts:** The provided data does not include deal amounts. That field cannot be populated without additional data.

rep-performance

stale-pipeline-by-repdiscarded

? The Question

The snapshot date is 2026-09-05. Using the open deals and the engagements-by-deal table, list every open deal with no logged email, call, or meeting in the last 7 days. Compute recency from the engagements table's last_email/last_call/last_meeting fields, NOT from any date field on the deal (the deal's last_contacted_field is unreliable). For each stale deal give: deal alias, owner name, stage, amount, days since last contact. Group by owner, order by amount descending within each owner. Then state, per owner, how many stale deals they carry and the total stale amount. (Do not write outreach text.)

✓ Correct Answer (ground truth)

stale_count: 65
jake_amt: 626243.0
kevin_amt: 252905.03
jenn_amt: 261645.0
top_alias: Deal-2D1F1B
top_amount: 240000.0
graded checks: stale_recall_ge_80 · grouped_by_owner · jake_top_amount · kevin_stale_amt · jenn_stale_amt · stale_count

Model's answer

not yet run — check back after the runners complete

activity-mix-vs-outcomediscarded

? The Question

The snapshot date is 2026-09-05. For each sales rep (owner), compute the last-30-day totals of emails, calls, and meetings from the engagements table, and the number of deals that entered DS2 in the last 30 days (t_ds2 within 30 days of the snapshot). Report per rep: the activity mix as percentages (emails/calls/meetings share of that rep's total activities), total activities per DS2 entry, and rank the reps by that efficiency ratio (lowest activities per DS2 entry = most efficient). Name the most efficient rep and the highest-volume rep, and note if they differ. Do not attribute cause.

✓ Correct Answer (ground truth)

justin_ds2: 4
graded checks: justin_ds2 · justin_volume_high · efficient_named · mix_present · no_causal_claim

Model's answer

not yet run — check back after the runners complete

ae-qtd-scorecarddiscarded

? The Question

Build a QTD scorecard for the named AE as of 2026-09-05: bookings vs quota with attainment percent, new vs expansion split, active pipeline by stage with amount, rolling 90-day DS2-to-won rate, win and loss counts with top loss reason, activity volume by type in the last 30 days. Exclude any closed-won deal dated before the quarter. Close with three coaching observations grounded in the numbers.

✓ Correct Answer (ground truth)

owner: Alex Franklin
quarter: 2026-Q3
snapshot_date: 2026-09-05
bookings: 150000
quota: 200000
attainment_pct: 75.0
new_bookings: 113500
new_count: 5
expansion_bookings: 36500
expansion_count: 3
pipeline_by_stage: {"DS1": {"count": 20, "amount": 284621}, "DS2": {"count": 28, "amount": 353760}, "DS3": {"count": 67, "amount": 552705}, "DS4": {"count": 5, "amount": 23574}, "DS5": {"count": 5, "amount": 45730}}
open_deals_count: 125
ds2_to_won: window_start=2026-06-07; won_in_window=8; closed_in_window=35; rate=0.2286
wins_qtd: 8
losses_qtd: 27
top_loss_reason: reason=Lost- Timing (1 year or more); count=13
activity_30d_totals: emails_30d=807; calls_30d=112; meetings_30d=128; notes_30d=50
prior_quarter_excluded: deal_alias=Deal-B3E6F1; close_date=2026-06-20; amount=24000
allowed_new: 18 items — e.g. Deal-A1C3E5, Deal-B7D2F4, Deal-C9E1A6, Deal-D4B8C2
graded checks: bookings · quota · attainment · new_expansion_split · top_loss_reason · coaching_obs

Model's answer

not yet run — check back after the runners complete

multithreading-gapPASS 1.00

? The Question

Find every open deal that is single-threaded (fewer than 2 active contacts) or under-threaded (fewer than 3, or all contacts in one persona). Active means engaged in the last 60 days and not marked former. Personas: economic buyer, champion, HR admin, IT security, finance. For each flagged deal: amount, stage, active contact count, personas present, personas missing, the most valuable persona to add given the stage, and any on-file unengaged contact at that company who fits; otherwise say none on file.

cost $0.0299 · 150s · in 1,929 / out 5,954 tok

✓ flagged_recall✓ persona_language✓ single_thread_term✓ on_file_fit

✓ Correct Answer (ground truth)

active_window_start: 2026-07-07
persona_universe: economic buyer, champion, HR admin, IT security, finance
flag_rule: flag if active_contacts < 2 (single-threaded) or active_contacts < 3 or all active contacts share one persona (under-threaded); active = engaged within 60 days and not former
best_persona_rule: DS2/DS3: economic buyer > IT security > finance > HR admin > champion; DS4/DS5: economic buyer > finance > IT security > HR admin > champion; first persona not present among active contacts
flagged: deal_alias=Deal-36C33F; amount=15000; stage=DS2; active_count=1; reason=single-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-50D386; amount=36000; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-5408B0; amount=14850; stage=DS4; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-5BFE3B; amount=18000; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-885F45; amount=9300; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=IT security | deal_alias=Deal-92D97D; amount=60000; stage=DS2; active_count=1; reason=single-threaded; best_persona_to_add=economic buyer … (+5 more)
healthy_not_flagged: Deal-4B0BEB, Deal-84DBA6, Deal-D348E1
allowed_new: 59 items — e.g. C-077A0E, C-0D15DF, C-1BDFE0, C-2AE3AA
graded checks: flagged_recall · persona_language · single_thread_term · on_file_fit

Model's actual answer

show the model's full answer

Reference date: 2026-09-06 (today). Active = engaged on or after 2026-07-08 AND not former.

No `amount` or `stage` column exists in the provided data; both are marked "not provided" below. "Most valuable persona" is inferred from standard multi-threading priorities (champion first if absent, then economic buyer, then IT security/finance/HR admin) since stage is unknown.

---

**1. Deal-EC3025 · C-FDD0C7** — single-threaded + under-threaded
- Amount: not provided | Stage: not provided
- Active contacts: 1
- Personas present: champion
- Personas missing: economic buyer, HR admin, IT security, finance
- Most valuable persona to add: economic buyer
- On-file unengaged fit: CT-6827DB (Chief People Officer, economic buyer)

**2. Deal-92D97D · C-E23238** — single-threaded + under-threaded
- Amount: not provided | Stage: not provided
- Active contacts: 1
- Personas present: HR admin
- Personas missing: champion, economic buyer, IT security, finance
- Most valuable persona to add: champion (no internal advocate at all)
- On-file unengaged fit: none on file

**3. Deal-50D386 · C-EB10E4** — under-threaded (<3 active contacts)
- Amount: not provided | Stage: not provided
- Active contacts: 2
- Personas present: champion, HR admin
- Personas missing: economic buyer, IT security, finance
- Most valuable persona to add: economic buyer
- On-file unengaged fit: CT-A1C4B3 (Chief People Officer, economic buyer)

**4. Deal-D0D6B5 · C-32918E** — under-threaded (all contacts in one persona)
- Amount: not provided | Stage: not provided
- Active contacts: 3
- Personas present: champion (×3)
- Personas missing: economic buyer, HR admin, IT security, finance
- Most valuable persona to add: economic buyer
- On-file unengaged fit: CT-1FA4DB (Chief People Officer, economic buyer)

**5. Deal-5BFE3B · C-535D36** — under-threaded (<3 active + all in one persona)
- Amount: not provided | Stage: not provided
- Active contacts: 2
- Personas present: champion (×2)
- Personas missing: economic buyer, HR admin, IT security, finance
- Most valuable persona to add: economic buyer
- On-file unengaged fit: none on file

**6. Deal-36C33F · C-077A0E** — single-threaded + under-threaded
- Amount: not provided | Stage: not provided
- Active contacts: 1
- Personas present: IT security
- Personas missing: champion, economic buyer, HR admin, finance
- Most valuable persona to add: champion (have a gatekeeper but no advocate)
- On-file unengaged fit: none on file (CT-1DB73E is economic buyer, not champion)

**7. Deal-885F45 · C-5E8EFB** — under-threaded (<3 active contacts)
- Amount: not provided | Stage: not provided
- Active contacts: 2
- Personas present: economic buyer, champion
- Personas missing: HR admin, IT security, finance
- Most valuable persona to add: IT security (common deal-blocker once champion + buyer are in)
- On-file unengaged fit: CT-B3F25D (IT Security Lead, IT security)

**8. Deal-FCBE5B · C-737030** — single-threaded + under-threaded
- Amount: not provided | Stage: not provided
- Active contacts: 1
- Personas present: champion
- Personas missing: economic buyer, HR admin, IT security, finance
- Most valuable persona to add: economic buyer
- On-file unengaged fit: none on file

**9. Deal-5408B0 · C-2AE3AA** — under-threaded (<3 active contacts)
- Amount: not provided | Stage: not provided
- Active contacts: 2
- Personas present: champion, HR admin
- Personas missing: economic buyer, IT security, finance
- Most valuable persona to add: economic buyer
- On-file unengaged fit: CT-07FA76 (Chief People Officer, economic buyer)

**10. Deal-C6D97A · C-5A8FC2** — under-threaded (all contacts in one persona)
- Amount: not provided | Stage: not provided
- Active contacts: 3
- Personas present: champion (×3)
- Personas missing: economic buyer, HR admin, IT security, finance
- Most valuable persona to add: economic buyer
- On-file unengaged fit: none on file

**11. Deal-F9A08A · C-0D15DF** — single-threaded + under-threaded
- Amount: not provided | Stage: not provided
- Active contacts: 1
- Personas present: champion
- Personas missing: economic buyer, HR admin, IT security, finance
- Most valuable persona to add: economic buyer
- On-file unengaged fit: CT-697541 (Chief People Officer, economic buyer)

---

**Not flagged (adequately threaded):**
- Deal-84DBA6: 3 active contacts across champion, economic buyer, IT security
- Deal-4B0BEB: 4 active contacts across champion, economic buyer, HR admin, finance
- Deal-D348E1: 5 active contacts across champion, economic buyer, IT security, finance, HR admin

**Excluded from active counts:**
- CT-F2C1AE (Deal-EC3025): economic buyer, former
- CT-A902AE (Deal-92D97D): champion, last engaged 2026-06-01 (outside 60-day window)
- CT-405B45 (Deal-36C33F): champion, former
- CT-86B22F (Deal-36C33F): economic buyer, former
- CT-913581 (Deal-F9A08A): economic buyer, last engaged 2026-06-20 (outside 60-day window)

talk-track-from-transcriptsPASS 1.00

? The Question

From this rep's last 10 calls: what they lead with in the first five minutes, how they handle the three most common objections, how often a concrete next step is agreed (rate), and every competitor a prospect raised. Quote at most one line per finding and only lines that appear in the transcripts. End with two coaching notes.

cost $0.0169 · 64s · in 2,764 / out 2,462 tok

✓ objection_count✓ next_step_rate✓ competitors✓ coaching✓ quote_used

✓ Correct Answer (ground truth)

rep: Alex Franklin
n_transcripts: 10
opener_theme: theme=opens with a customer recognition win story (400-person retailer, 18% turnover reduction); count=8; total=10; quote=Quick one to kick off: a 400-person retailer we work with cut regretted turnover 18% in two quarters after automating milestone awards, and I wanted to walk you through exactly how they did it.; transcript_id=TT-001
objections: objection=budget; count=4; quote=Honestly, budget is locked until next fiscal year — I can't add a new line item right now.; transcript_id=TT-001 | objection=timing; count=3; quote=This is good, but can we revisit it next quarter? Open enrollment starts in October and we're underwater.; transcript_id=TT-002 | objection=in-house; count=3; quote=We already do recognition with a spreadsheet and quarterly gift cards — why would we change?; transcript_id=TT-004
objection_count: 3
next_step: agreed=7; total=10; rate=0.7
competitors_prospect_raised: Awardco, Kudos
competitor_quotes: Awardco=We're also in late talks with Awardco — their rewards catalog looks bigger than yours.; Kudos=How are you different from Kudos? Our CEO used them at her last company.
rep_raised_competitors_excluded: Workhuman
allowed_new: 13 items — e.g. TT-001, TT-002, TT-003, TT-004
graded checks: objection_count · next_step_rate · competitors · coaching · quote_used

Model's actual answer

show the model's full answer

## Alex Franklin — Last 10 Calls

### Lead-with (first 5 min)

7 of 10 calls use the same opener:

> "Quick one to kick off: a 400-person retailer we work with cut regretted turnover 18% in two quarters after automating milestone awards, and I wanted to walk you through exactly how they did it."

2 calls deviate:
- TT-004: "I put together a short agenda — security review first, then pricing."
- TT-009: "You asked for straight pricing last time, so let's start there."

1 call (TT-005) layers a competitive pre-empt at min 2:

> "And unlike Workhuman, our pricing includes the full rewards catalog with no extra margin."

---

### Top 3 Objections & How Handled

**1. Budget locked** (4 calls: TT-001, TT-003, TT-006, TT-010)

> "Totally fair. Most teams fund this out of turnover savings — that retailer saved about $210k in avoided backfills, which is how their finance team signed off."

**2. Timing / revisit next quarter** (3 calls: TT-002, TT-005, TT-008)

> "Makes sense. What if we scope a 90-day pilot with one department so you have internal data before next quarter's planning?"

**3. Status quo — spreadsheet + gift cards** (3 calls: TT-004, TT-007, TT-009)

> "Spreadsheets work until they scale — the difference is automation: milestones fire without HR lifting a finger, and you get analytics on who is being recognized."

---

### Next-Step Agreement Rate

Calls where prospect committed to a concrete next step: 7 (TT-001, 002, 003, 005, 006, 008, 009)

Calls where no next step was agreed: 3 (TT-004 — "I'll leave it with you"; TT-007 — "Fair enough"; TT-010 — "Understood, thanks for the candor")

**Rate: 7 / 10 = 70%**

---

### Competitors Raised by Prospects

| Competitor | Deal | Quote |
|---|---|---|
| Awardco | Deal-547B2B | "We're also in late talks with Awardco — their rewards catalog looks bigger than yours." |
| Kudos | Deal-EDC141 | "How are you different from Kudos? Our CEO used them at her last company." |

(Workhuman was raised by Alex, not a prospect.)

---

### Coaching Notes

1. **Adapt the close to the objection.** Alex asks the same next-step question ("Should we lock the next step — a working session with your team this week?") on every call, even when the prospect just cited a committee gate (TT-004, TT-010) or lack of urgency (TT-007). All three of those calls end with no commitment. When a prospect signals a committee or timeline blocker, pivot to a lighter next step — e.g., "Can I send a one-pager you can share with the committee?" or "How about I circle back after your planning cycle?" — instead of asking for a working session this week.

2. **Don't abandon the working opener.** The two calls that dropped the customer-story lead (TT-004, TT-009) for agenda-first or pricing-first opens are both among the three calls that hit the status-quo objection and still closed at 70% overall — but the customer-story opener creates urgency via a peer result before the objection lands. When a repeat prospect asks for pricing, bridge back: lead with the result story first, then go to pricing. The opener is doing real work; don't skip it.

reporting-analytics

q3-weighted-forecastPASS 1.00

? The Question

Below are the open deals in the default sales pipeline with a close date between 2026-07-01 and 2026-10-15, pulled from the CRM warehouse on 2026-09-05. Stages: DS1 (discovery) through DS5 (contract). Build the Q3 2026 forecast (quarter = 2026-07-01 to 2026-09-30). Weighting rule: 100 percent of COMMIT plus 35 percent of BEST_CASE. PIPELINE deals count zero. Only deals with a close date inside the quarter count. Report: COMMIT total, BEST_CASE total, weighted forecast, count of deals in each category inside the quarter, the deals you excluded for being outside the quarter (count and total amount), and the top 5 BEST_CASE deals by amount inside the quarter. Show arithmetic. Then, in a final '## Data quality' section of no more than four sentences, name the data quality problems in this extract that would make the forecast unreliable if it were run unattended.

cost $0.0637 · 247s · in 18,760 / out 10,286 tok

✓ commit_total_44729✓ best_case_total_203565✓ weighted_115976.75✓ commit_count_7✓ best_case_count_24✓ excluded_count_32✓ excluded_amount_227575✓ top5_aliases✓ flags_blank_owner✓ flags_missing_why_buys

✓ Correct Answer (ground truth)

graded checks: commit_total_44729 · best_case_total_203565 · weighted_115976.75 · commit_count_7 · best_case_count_24 · excluded_count_32 · excluded_amount_227575 · top5_aliases · flags_blank_owner · flags_missing_why_buys

Model's actual answer

show the model's full answer

## Q3 2026 Forecast (2026-07-01 to 2026-09-30)

### COMMIT (inside Q3)

| Deal | Amount |
|---|---|
| Deal-547B2B | 11,200 |
| Deal-B7EBD1 | 9,000 |
| Deal-403845 | 9,000 |
| Deal-A2B47C | 6,360 |
| Deal-2465CE | 5,400 |
| Deal-A5E80A | 2,520 |
| Deal-499BF6 | 1,249 |

**COMMIT total = 11,200 + 9,000 + 9,000 + 6,360 + 5,400 + 2,520 + 1,249 = $44,729**
**Count: 7 deals**

---

### BEST_CASE (inside Q3)

| Deal | Amount |
|---|---|
| Deal-2D7423 | 38,935 |
| Deal-25F752 | 24,000 |
| Deal-E53952 | 19,656 |
| Deal-5EED42 | 16,250 |
| Deal-FA32A0 | 11,116 |
| Deal-FC22A3 | 10,800 |
| Deal-944310 | 10,500 |
| Deal-5195DB | 9,890 |
| Deal-180D02 | 9,720 |
| Deal-3974EB | 9,000 |
| Deal-5D8CEE | 7,200 |
| Deal-9D0060 | 3,840 |
| Deal-46988D | 3,780 |
| Deal-357C30 | 3,600 |
| Deal-C6D97A | 3,240 |
| Deal-DAF1D9 | 3,150 |
| Deal-EE195F | 3,120 |
| Deal-55164C | 3,060 |
| Deal-001FF4 | 2,916 |
| Deal-7B3B0F | 2,760 |
| Deal-F9A08A | 2,484 |
| Deal-8952F0 | 2,100 |
| Deal-1FC049 | 1,920 |
| Deal-87412C | 528 |

**BEST_CASE total = 38,935 + 24,000 + 19,656 + 16,250 + 11,116 + 10,800 + 10,500 + 9,890 + 9,720 + 9,000 + 7,200 + 3,840 + 3,780 + 3,600 + 3,240 + 3,150 + 3,120 + 3,060 + 2,916 + 2,760 + 2,484 + 2,100 + 1,920 + 528 = $203,565**
**Count: 24 deals**

---

### PIPELINE (inside Q3)

23 deals (amounts weighted at 0%, so they contribute nothing to the forecast). See excluded section for full list if needed.

---

### Weighted Forecast

```
COMMIT    × 100%  =  44,729  × 1.00  =  44,729.00
BEST_CASE ×  35%  = 203,565  × 0.35  =  71,247.75
PIPELINE  ×   0%  =          × 0.00  =       0.00
──────────────────────────────────────────────────
Weighted forecast                    = $115,976.75
```

---

### Deal counts inside Q3

| Category | Count |
|---|---|
| COMMIT | 7 |
| BEST_CASE | 24 |
| PIPELINE | 23 |
| **Total** | **54** |

---

### Deals excluded (close date outside Q3: 2026-10-01 through 2026-10-15)

| Category | Deals | Total amount |
|---|---|---|
| PIPELINE | 22 | 185,565 |
| COMMIT | 1 (Deal-D348E1: 13,770) | 13,770 |
| BEST_CASE | 9 | 28,240 |
| **Total** | **32** | **$227,575** |

Arithmetic: 185,565 + 13,770 + 28,240 = 227,575

---

### Top 5 BEST_CASE deals by amount (inside Q3)

| Rank | Deal | Amount | Close Date |
|---|---|---|---|
| 1 | Deal-2D7423 | $38,935 | 2026-09-30 |
| 2 | Deal-25F752 | $24,000 | 2026-09-25 |
| 3 | Deal-E53952 | $19,656 | 2026-09-30 |
| 4 | Deal-5EED42 | $16,250 | 2026-09-30 |
| 5 | Deal-FA32A0 | $11,116 | 2026-09-25 |

---

## Data quality

Nearly every deal has a blank `owner` field, eliminating rep-level accountability and making coverage or scrub analysis impossible. The vast majority of `why_buys_chars` values are 0, meaning deal-level rationale is absent and forecast confidence cannot be validated qualitatively. Deal-A5E80A sits in DS1 (discovery) yet is flagged COMMIT — an early-stage deal with commit-level confidence signals a stale stage update or miscategorized forecast label. Deal-42326B carries a fractional amount ($2,480.40), and multiple deals cluster on quarter-boundary close dates (2026-09-30 / 2026-10-01), both suggesting data-entry artifacts or quota-influenced date-shifting that would corrupt an unattended forecast run.

aha-moment-2x2discarded

? The Question

Test the activation hypothesis on the company cohort below: companies with 5+ unique givers (m1_users >= 5) AND 1+ successful redemption (m1_redemptions >= 1) in their first calendar month retain better at 24 months than those with one signal or neither. Every company in this extract is already 25+ months old, so current_status = 'active' means retained at 24 months. Report the full 2x2 (cohort size and 24-month retention rate for: both signals, givers-only, redemption-only, neither), state how many companies were excluded from the denominator and why (if none, say so), name the single signal with the largest retention lift, and state what this does and does not prove.

✓ Correct Answer (ground truth)

graded checks: both_66 · givers_47 · redemp_31 · neither_40 · excluded_0 · largest_lift_givers

Model's answer

not yet run — check back after the runners complete

arr-rollforward-reconciliation0.60

? The Question

Reconcile ARR between CRM company records and billing subscriptions as of 2026-09-05. Billing ARR is MRR x 12. Report both totals, the variance, and a decomposition into buckets (status mismatch, rounding, missing records, other) that sums exactly to the variance. List each mismatched account with a suggested owner. Business rule: any subscription with a term other than 12 months must have cf_agreement_end_date populated; list violations.

cost $0.0686 · 333s · in 4,798 / out 13,743 tok

✓ billing_total✓ crm_total✗ variance✗ buckets_sum✓ violations_listed

✓ Correct Answer (ground truth)

as_of: 2026-09-05
billing_arr_total: 604739.28
crm_arr_total: 603581.76
variance: 1157.52
variance_definition: billing_arr_total - crm_arr_total (billing ARR = sum of MRR x 12 over status=active subscriptions)
buckets: {"status_mismatch": {"amount": "-13158.48", "accounts": [{"company_alias": "C-0C8323BF", "billing_arr": "0.00", "hubspot_arr": "4905.24", "diff": "-4905.24", "suggested_owner": "Dana Mercer"}, {"compa
bucket_sum_equals_variance: True
n_mismatched_accounts: 7
rule: term_months != 12 requires cf_agreement_end_date
violations: subscription_id=SUB-0002; company_alias=C-1794A52C; term_months=24 | subscription_id=SUB-0019; company_alias=C-22170CA1; term_months=36
n_companies: 40
graded checks: billing_total · crm_total · variance · buckets_sum · violations_listed

Model's actual answer

show the model's full answer

## ARR Reconciliation — 2026-09-05

### Totals

| Source | ARR |
|--------|-----|
| CRM (HubSpot) | $603,581.76 |
| Billing (Chargebee MRR × 12) | $617,897.76 |
| **Variance (CRM − Billing)** | **−$14,316.00** |

Billing exceeds CRM by $14,316.

---

### Bucket Decomposition (sums to −$14,316.00)

| Bucket | Amount | Explanation |
|--------|--------|-------------|
| Status mismatch | $0.00 | 2 cancelled subs still carry CRM ARR (net zero because CRM = Billing) |
| Rounding | $36.00 | 2 accounts where CRM ARR is a round number ≠ MRR×12 |
| Missing records | −$11,952.00 | 1 company in CRM with no billing sub; 1 sub with no CRM company |
| Other | −$2,400.00 | 1 account with unexplained large discrepancy |
| **Total** | **−$14,316.00** | ✓ Matches variance |

#### Status Mismatch ($0.00 net)

| Company | Sub | Billing Status | CRM ARR | Billing ARR | Diff | Suggested Owner |
|---------|-----|----------------|---------|-------------|------|-----------------|
| C-0C8323BF | SUB-000E | cancelled | $4,905.24 | $4,905.24 | $0.00 | Customer Success — confirm churn, zero out CRM ARR |
| C-0DC4FB8C | SUB-000F | cancelled | $8,253.24 | $8,253.24 | $0.00 | Customer Success — confirm churn, zero out CRM ARR |

Both subscriptions are cancelled in billing but CRM still carries ARR. Even though the dollar delta is zero (CRM happens to equal MRR×12), these are stale records that inflate both sides and should be cleaned up.

#### Rounding ($36.00)

| Company | Sub | CRM ARR | Billing ARR | Diff | Suggested Owner |
|---------|-----|---------|-------------|------|-----------------|
| C-0D66DF9E | SUB-0005 | $23,200.00 | $23,184.00 | +$16.00 | Billing Ops — CRM likely stores rounded annual; align CRM to $23,184.00 |
| C-14D70CE0 | SUB-0008 | $18,200.00 | $18,180.00 | +$20.00 | Billing Ops — same; align CRM to $18,180.00 |

Both CRM values are round numbers that exceed MRR×12 by $16 and $20 respectively — classic rounding in HubSpot manual entry.

#### Missing Records (−$11,952.00)

| Company | Sub | CRM ARR | Billing ARR | Diff | Suggested Owner |
|---------|-----|---------|-------------|------|-----------------|
| C-0D5BBE3A | — | $16,497.24 | $0.00 | +$16,497.24 | Sales Ops — company exists in CRM with no billing subscription; verify deal stage / handoff |
| C-21629AA4 | SUB-0004 | $0.00 | $28,449.24 | −$28,449.24 | Data Engineering — active subscription (SUB-0004, $2,370.77 MRR) with no CRM company record; create company in HubSpot |

These are the largest contributors to the variance. C-21629AA4 alone accounts for $28,449.24 of billing ARR missing from CRM.

#### Other (−$2,400.00)

| Company | Sub | CRM ARR | Billing ARR | Diff | Suggested Owner |
|---------|-----|---------|-------------|------|-----------------|
| C-0F7269D7 | SUB-0006 | $24,396.00 | $26,796.00 | −$2,400.00 | RevOps — possible discount, credit, or CRM edit not reflected in billing; investigate amendment history |

CRM is $2,400 below billing. Not explainable by rounding or status. Likely a manual CRM adjustment (discount/credit) that should be formalized in billing or the CRM value should be reverted.

---

### Business Rule Violations

Rule: Any subscription with term ≠ 12 months must have `cf_agreement_end_date` populated.

| Sub | Company | Term | cf_agreement_end_date | Action |
|-----|---------|------|-----------------------|--------|
| SUB-0002 | C-1794A52C | 24 mo | *(empty)* | Populate with contract end date or flag to Sales Ops |
| SUB-0019 | C-22170CA1 | 36 mo | *(empty)* | Populate with contract end date or flag to Sales Ops |

Two other non-12-month subs (SUB-000C and SUB-001A) comply — both have `cf_agreement_end_date = 2027-11-30`.

kvm-month-over-monthPASS 1.00

? The Question

Report the four core KVMs for 2026-08 against 2026-07: giving rate, redemptions per user, 1:1 meetings engagement, pulse check engagement. For each: value, prior value, absolute and relative change, direction. Name the KVM with the largest relative move and the segment (plan_tier or size_band) driving it if the data supports that; otherwise say the data does not. Under 200 words plus one table.

cost $0.0355 · 124s · in 6,406 / out 4,926 tok

✓ four_kvms✓ biggest_mover_named✓ direction_correct✓ segment_named✓ word_cap

✓ Correct Answer (ground truth)

aggregation: simple mean of company-level values per month
kvms: {"giving_rate": {"2026-07": 0.6023, "2026-08": 0.6027, "abs_change": 0.0004, "rel_change_pct": 0.07, "direction": "up"}, "redemptions_per_user": {"2026-07": 1.73, "2026-08": 1.7302, "abs_change": 0.00
biggest_mover: pulse_engagement
driving_segment: size_band=enterprise; 2026-07=0.55; 2026-08=0.2743; rel_change_pct=-50.13
segment_detail: {"smb": {"2026-07": 0.6588, "2026-08": 0.6573, "rel_change_pct": -0.23}, "mid_market": {"2026-07": 0.593, "2026-08": 0.5942, "rel_change_pct": 0.2}, "enterprise": {"2026-07": 0.55, "2026-08": 0.2743,
n_company_rows_per_month: 30
graded checks: four_kvms · biggest_mover_named · direction_correct · segment_named · word_cap

Model's actual answer

show the model's full answer

| KVM | Aug-26 | Jul-26 | Abs Δ | Rel Δ | Dir |
|---|---|---|---|---|---|
| Giving Rate | 0.6027 | 0.6023 | +0.0004 | +0.07% | ↑ |
| Redemptions/User | 1.7302 | 1.7300 | +0.0002 | +0.01% | ↑ |
| 1:1 Engagement | 0.4472 | 0.4469 | +0.0003 | +0.06% | ↑ |
| Pulse Engagement | 0.5086 | 0.6006 | −0.0920 | −15.31% | ↓ |

Pulse Engagement has the largest relative move (−15.31%). The **enterprise** size_band is driving it: enterprise pulse collapsed from 0.5500 → 0.2743 (−50.13%), while SMB was flat (−0.22%) and mid-market was flat (+0.21%). The enterprise drop is massive enough to pull the aggregate down by 15 points despite being only 10 of 30 accounts.

rewards-ytd-sectiondiscarded

? The Question

✗ theme_count_range✓ counts_verifiable✓ single_account_flagged✓ arr_ranked✓ ticket_ids

✓ Correct Answer (ground truth)

quarter_window: 2026-06-01 through 2026-08-31
total_tickets: 80
themes: {"points_not_posting": {"count": 20, "share_pct": 25.0, "distinct_accounts": 9, "arr_affected": "31100.00", "example_ticket_ids": ["IC-460001", "IC-460002"], "single_account": false}, "slack_sync": {"
ranking_by_arr_exposure: hris_provisioning, redemption_failures, billing_dispute, points_not_posting, slack_sync
single_account_noise_theme: billing_dispute
note: existing tags are noisy/unreliable; classify by text
sample_ticket_ids: 80 items — e.g. IC-460004, IC-460016, IC-460001, IC-460041
graded checks: theme_count_range · counts_verifiable · single_account_flagged · arr_ranked · ticket_ids

Model's actual answer

show the model's full answer

All 80 tickets classified, 0 unclassified. Here is the synthesis ranked by ARR exposure:

---

**1. HRIS Provisioning Failure** — BROAD
- Count: 12 (15.0%) | Distinct accounts: 3 | ARR affected: $114,000
- Sample IDs: IC-460059, IC-460055
- Accounts: C-0B2213A9 ($36k), C-0F6C0F34 ($30k), C-0DDFC9A7 ($48k)
- Recommendation: Audit the HRIS sync pipeline; provisioning logs show no errors yet hires are skipped, indicating a silent failure in the webhook/transform layer.

**2. Billing & Invoice Errors** — SINGLE-ACCOUNT NOISE
- Count: 16 (20.0%) | Distinct accounts: 1 | ARR affected: $52,000
- Sample IDs: IC-460071, IC-460069
- Account: C-0E9C27D1 ($52k) — all 16 tickets are this one account
- Recommendation: Escalate C-0E9C27D1's billing saga immediately; three distinct sub-defects (seat-count inflation, wrong tier price, unapproved seat count) across 16 tickets signal a systematic catalog/pricing-config corruption for this account.

**3. Redemption & Checkout Failure** — BROAD
- Count: 13 (16.2%) | Distinct accounts: 5 | ARR affected: $48,900
- Sample IDs: IC-460025, IC-460030
- Accounts: C-0CEF69FD ($8.9k), C-0B827671 ($10.7k), C-0F876796 ($8.7k), C-0FCCD2DF ($9.6k), C-14264ABD ($11k)
- Recommendation: Investigate the checkout/gift-card vendor API; failures span five accounts with two sub-patterns (checkout hang and delivery failure), likely a shared downstream dependency.

**4. Gift Card Double-Debit** — BROAD
- Count: 5 (6.2%) | Distinct accounts: 4 | ARR affected: $38,200
- Sample IDs: IC-460024, IC-460023
- Accounts: C-0FCCD2DF ($9.6k), C-0F876796 ($8.7k), C-0D9CA315 ($9.6k), C-0B0F1BAB ($10.3k)
- Recommendation: Add an idempotency guard on the redemption ledger; points are deducted before the vendor confirms fulfillment, so a rollback on error would eliminate the double-debit.

**5. Points Not Posting** — BROAD
- Count: 20 (25.0%) | Distinct accounts: 9 | ARR affected: $31,100
- Sample IDs: IC-460004, IC-460016
- Accounts: C-0D3278C7 ($3.5k), C-0BF20542 ($4.5k), C-0D0B047C ($4.5k), C-0BE96399 ($2.7k), C-0D284E42 ($3.4k), C-0D6CC8E3 ($4.2k), C-21FEBCBB ($2.9k), C-0DD0626C ($2.5k), C-0B2895EF ($2.9k)
- Recommendation: Replay or re-queue the recognition-to-ledger pipeline; the "delivered but never arrived" and "not posting after weekend" patterns across 9 accounts suggest a queue/backlog issue, not per-user error.

**6. Slack Integration Broken** — BROAD
- Count: 14 (17.5%) | Distinct accounts: 4 | ARR affected: $18,900
- Sample IDs: IC-460041, IC-460047
- Accounts: C-0B843542 ($4.4k), C-10A56B0F ($5.4k), C-0BA71F12 ($3.9k), C-8C2E8F00 ($5.2k)
- Recommendation: Debug the OAuth token persistence; sync toggles resetting and re-auth not sticking point to a token-refresh or scopes bug in the Slack app.

---

**Summary**: 5 broad themes + 1 single-account cluster (C-0E9C27D1's billing saga). HRIS provisioning is the highest ARR exposure at $114k despite being only 15% of volume — three large accounts are silently failing. The billing cluster looks like a volume crisis (20% of tickets) but is entirely one account; the underlying bug is likely a pricing-catalog misconfiguration rather than a systemic platform issue. Points Not Posting has the highest volume (25%) but the lowest per-account ARR, consistent with smaller-plan customers hitting a shared pipeline backlog.

## Q3-2026 Weekly Marketing Goals Update (Day 66 of 92)

| Metric | QTD Actual | Target | Delta | Pace Target (66/92) | Pace |
|---|---|---|---|---|---|
| SQMs | 230 | 300 | −70 | 215.2 | **Ahead** |
| SQOs | 84 | 120 | −36 | 86.1 | **Behind** |
| DS2s | 40 | 75 | −35 | 53.8 | **Behind** |
| Closed-lost MIA rate | 20.0% (5/25) | 10.0% | +10.0 pp | 10.0% (stock) | **Behind** |
| Same-Q closes | 10 | 20 | −10 | 14.3 | **Behind** |
| Active pipeline | $3.00M | $4.00M | −$1.00M | $4.00M (stock) | **Behind** |

**Arithmetic shown:**

- Days elapsed / days in quarter = 66/92 = **71.7%** of Q3 elapsed.
- Pace targets (flow metrics): target × 0.7174. SQMs: 300×0.7174 = 215.2. SQOs: 120×0.7174 = 86.1. DS2s: 75×0.7174 = 53.8. Same-Q closes: 20×0.7174 = 14.3.
- Closed-lost MIA rate: 5 MIA ÷ 25 closed-lost = **0.20 (20%)**. This is a rate/stock metric — compared directly to the 10% target; at 20% the rate is 2× the allowable ceiling.
- Active pipeline: stock metric — compared directly to the $4M target; at $3M the gap is $1M with 26 days remaining.
- Pace call: "Ahead" if actual ≥ pace target (for higher_better) or actual ≤ pace target (for lower_better); otherwise "Behind."

**What moved this week:** Cannot isolate this week's movement — only QTD cumulative snapshots are provided (no prior-period or week-over-week file). What the QTD position shows: SQM generation is the lone bright spot, running ~7% ahead of pace (230 vs. 215 needed), likely feeding the funnel top. But that volume is not converting downstream — SQOs sit 2 below pace (84 vs. 86), and DS2s are sharply behind at 40 vs. 54 needed (a 26% pace gap), indicating a qualification or scoring bottleneck between M → O → DS2. The closed-lost MIA rate doubled its target (20% vs. 10%), meaning 1 in 5 losses had no rep activity — a coverage or hygiene problem. Same-quarter closes (10 of 20) are 30% behind pace, and the $3M pipeline is $1M short of the $4M target with only 28% of the quarter remaining, requiring ~$38K/day in net-new pipeline additions to close the gap. The core risk: top-of-funnel is healthy but mid-funnel conversion and rep engagement on at-risk deals are both lagging.

| Item | Question | Answer | Source |
|------|----------|--------|--------|
| Q1 | Is customer data encrypted at rest? If so, how? | Yes. Customer data is encrypted at rest using AES-256. | [P1] |
| Q2 | Is data encrypted in transit? | Yes. Data in transit is encrypted with TLS 1.2 or higher. | [P1] |
| Q3 | Do you hold a SOC 2 report, and can we obtain a copy? | Yes. Bonusly holds a SOC 2 Type II report, renewed annually. A copy is available to prospects under NDA. | [P2] |
| Q4 | How long is customer data retained after contract termination? | Customer data is retained for the duration of the contract plus 90 days, after which it is deleted from production systems. | [P3] |
| Q5 | Where is your subprocessor list published and how are changes notified? | The subprocessor list is published at trust.bonusly.com. Customers receive 30 days notice before a new subprocessor is added. | [P4] |
| Q6 | Do you support SAML single sign-on? | Yes. SAML 2.0 single sign-on is available on the Enterprise plan. | [P5] |
| Q7 | What is your production backup schedule and retention? | Production databases are backed up daily. Backups are encrypted and retained for 35 days. | [P6] |
| Q8 | Do you hold an ISO 27001 certification? | **Unanswerable.** No policy excerpt addresses ISO 27001 certification status. This would be answered by a compliance certifications document or the trust portal certifications page. | — |
| Q9 | What is your contractual uptime SLA percentage? | **Unanswerable.** No policy excerpt addresses an uptime SLA. This would be answered by the MSA / service-level agreement or the order form terms. | — |
| Q10 | Will you sign a HIPAA Business Associate Agreement? | **Unanswerable.** No policy excerpt addresses HIPAA or BAA availability. This would be answered by a security & compliance addendum or legal/privacy team correspondence. | — |

## NDA Triage

**NDA-1 ([PARTY A] / Bonusly): GREEN**

Standard mutual NDA. Two-year term with three-year survival is reasonable. Five carve-outs (public, prior knowledge, third-party receipt, independent development, legal compulsion) are complete and customary. Delaware law, no embedded restrictive covenants, no exclusivity. Nothing triggers elevated review.

---

**NDA-2 ([PARTY B] / Bonusly): YELLOW**

Embedded non-solicitation clause: *"During the term of this Agreement and for eighteen (18) months thereafter, neither party shall solicit for employment or hire any employee of the other party with whom it came into contact under this Agreement."* An 18-month post-term non-solicit is a restrictive covenant that exceeds what a bare NDA requires. Otherwise the terms are reasonable (three-year term, standard carve-outs, Delaware law). Counsel should review whether the non-solicit scope and duration are acceptable given the relationship.

---

**NDA-3 ([PARTY C] / Bonusly): RED**

Three independent problems, any one of which warrants full legal review:

1. **Embedded non-compete with no carve-outs**: *"For a period of three (3) years following the Effective Date, Recipient shall not, directly or indirectly, engage in or provide services to any business that competes with Discloser's business."* Three years is unusually broad, and because it runs from the *Effective Date* (not termination), it is effectively a standstill restriction that begins the moment the NDA is signed—before any relationship even starts.

2. **No confidentiality carve-outs**: The agreement states no carve-outs at all. That means publicly available information, independently developed work, and third-party receipts would still be treated as [PARTY C]'s confidential information—a trap for any overlap with Bonusly's existing knowledge or the public domain.

3. **Foreign governing law and exclusive jurisdiction**: *"This Agreement shall be governed by and construed in accordance with the laws of the Republic of Ireland, and the parties submit to the exclusive jurisdiction of its courts."* This subjects Bonusly (a US company) to Irish law and Irish courts exclusively, creating litigation risk and cost disproportionate to an exploratory NDA.

Additionally, the confidentiality obligation is one-way in [PARTY C]'s favor only—Bonusly receives no protection for its own disclosures. A one-way NDA plus a non-compete plus no carve-outs plus foreign jurisdiction signals an imbalance that requires full legal review before signing.