Mean 0.929 · 10/14 perfect tests · $0.05 total · 14/40 tests completed — remaining tests discarded from study (endpoint timeouts after dedicated retries) · each card: the question → the correct answer (gold) → this model's actual answer
data-crm
stage-hygiene-auditdiscarded
? The Question
Audit stage hygiene on the open pipeline below. Rules: a deal at DS2 or later must have a substantive why-buys (15+ characters, not a placeholder such as tbd, n/a, see notes). A why_buys value of the form LONG(n) means the real text was present but truncated for display; treat it as present and substantive. Every deal should have a DS1 entry timestamp. A deal whose stage entry timestamps are out of order regressed (an earlier-stage timestamp is later than a later-stage timestamp). Report: percent of DS2+ deals non-compliant on why-buys overall and by owner (use the owners table for names), count of deals with no DS1 timestamp, count that regressed, and three process fixes. State definitions before numbers.
not yet run — check back after the runners complete
owner-resolutiondiscarded
? The Question
The deal extract below has an owner name field that is blank on nearly every row and an owner id field that is populated. Using the owners table, resolve each open deal to an owner name. Report: number of deals resolved, deals whose owner id has no match in the owners table (list them), the owner ids that map to archived/deactivated owners (list their names), and the total pipeline amount per resolved owner. Do not guess a name for any unmatched id.
not yet run — check back after the runners complete
gong-hubspot-join-integritydiscarded
? The Question
Check the integrity of the call-to-deal mapping below. The gong table lists calls in the last 90 days keyed to HubSpot deal ids; deals_open lists the current open default-pipeline deals. Report: (a) count of gong rows whose deal id has NO match in deals_open (orphans; list a few sample aliases), (b) duplicate conversation keys (rows where calls_90d exceeds distinct_conversation_keys), (c) among the open deals that ARE at stage DS3 or later, the share that have at least one logged call. Give sample aliases for the orphan finding, all taken from the data. Note: gong_owner_id is a Gong user id, not a HubSpot owner id.
not yet run — check back after the runners complete
snowflake-sql-trapped-catalog0.83
? The Question
Using ONLY the catalog provided, write one Snowflake SQL query returning, per customer company, for its first calendar month as a customer: unique givers, recognition count, successful redemption count. Read the catalog notes carefully: some tables are stale or unpopulated and must be avoided, and one documented filter (excluding deleted givers) must NOT be applied to historical giving counts. Put a comment above each FROM or JOIN explaining the table choice. Return only fenced SQL.
```sql
WITH customer_first_month AS (
SELECT
CUSTOMER_ID AS customer_company,
DATE_TRUNC('MONTH', TO_DATE(FIRST_SUB_PAYMENT_DATE)) AS first_calendar_month_as_customer
FROM PRODUCTION.PLG.COMPANY_COHORT_SUMMARY
-- FROM COMPANY_COHORT_SUMMARY: documented self-serve company source with first payment and M1 recognition metrics.
WHERE FIRST_SUB_PAYMENT_DATE IS NOT NULL
),
giving AS (
SELECT
customer_company,
COUNT(DISTINCT giver_id) AS unique_givers,
COUNT(*) AS recognition_count
FROM PRODUCTION.PLG.COMPANY_COHORT_SUMMARY
-- FROM COMPANY_COHORT_SUMMARY: documented source for M1_GIVES; no deleted-giver exclusion is applied because catalog says that filter must not be used for historical giving counts.
GROUP BY 1
),
redemptions AS (
SELECT
customer_company,
COUNT(*) AS successful_redemption_count
FROM PRODUCTION.PLG.COMPANY_COHORT_SUMMARY
-- FROM COMPANY_COHORT_SUMMARY: documented source for M1_REDEMPTIONS; PRODUCTION.DEPRECATED_RECOGNITION.REDEMPTION_RECORDS_V2 is documented as requiring confirmation for finance-grade counts, so this query uses the cataloged cohort summary.
GROUP BY 1
)
SELECT
c.customer_company,
c.first_calendar_month_as_customer,
COALESCE(g.unique_givers, 0) AS unique_givers,
COALESCE(g.recognition_count, 0) AS recognition_count,
COALESCE(r.successful_redemption_count, 0) AS successful_redemption_count
FROM customer_first_month c
-- FROM customer_first_month: derives each company's first calendar month as a customer from FIRST_SUB_PAYMENT_DATE.
LEFT JOIN giving g
-- JOIN giving: attaches first-month giving counts without excluding deleted givers.
ON g.customer_company = c.customer_company
LEFT JOIN redemptions r
-- JOIN redemptions: attaches first-month redemption counts.
ON r.customer_company = c.customer_company
ORDER BY
c.customer_company;
```
crm-completeness-enrichmentdiscarded
? The Question
Audit the CRM extract for completeness and propose fixes. Deals need owner, stage, amount, close date, why-buys; companies need industry, employee count, HQ country; contacts need email, title, persona. Report completeness percent per field. Identify duplicate company clusters (name variants, shared domain) with a survivor per cluster. List invalid emails and domain mismatches. Fill missing company fields only where the enrichment export has a matching row; where CRM and enrichment disagree list both and recommend a source. Never invent a value. End with the 10 fixes with the most pipeline amount at stake.
not yet run — check back after the runners complete
deal-intelligence
closed-lost-classificationdiscarded
? The Question
Classify every deal lost in the last 6 months below into one primary category: pricing, competitor, no decision, timing, product gap, champion left, or other. Also assign a side (Bonusly, buyer, unknown). Then summarize: category counts, the side split, the number of deals where the structured closed_lost_tag clearly disagrees with the free-text reason, and the two patterns most worth acting on. Base everything only on the tag and free-text provided.
not yet run — check back after the runners complete
pipeline-tieringdiscarded
? The Question
Score the open pipeline below on these signals: forecast category, stage, meetings_30d, contact count, and engagement recency. Assign exactly one tier per deal: LOCK, ACTION, BUILD, REVIVE, WATCH, or RISKY (RISKY = the forecast category disagrees with the engagement evidence). Never tier a deal LOCK if it has zero meetings_30d.
Do NOT list every deal. Return ONLY a compact JSON object with exactly these keys:
- "tier_counts": object mapping each tier to the number of deals in it (must sum to the total deal count),
- "tier_examples": object mapping each tier to up to 3 example deal aliases from the data,
- "risky_deals": array of the deal aliases you tiered RISKY,
- "lock_violations": number of deals tiered LOCK that have zero meetings_30d (should be 0),
- "pipeline_shape": one short paragraph on the overall pipeline shape.
Note: inbound_emails_30d is 0 for every row due to a data defect, so treat meetings_30d as the inbound signal.
not yet run — check back after the runners complete
call-transcript-extractiondiscarded
? The Question
For each transcript, extract CRM write-back fields as JSON: why-buys (prospect statements only), pain points, stakeholders from the speaker list, budget signal (prospect-stated or null), timeline signal, competitor mentioned (only if the prospect raised it), next step (only if explicitly agreed), objections, confidence. Never fill a field from what the rep said or hoped.
✓ Correct Answer (ground truth)
rep: Alex Franklin per_transcript: {"TX-001": {"why_buys": ["automating anniversary and birthday awards"], "pain_points": ["HR team of three cannot keep up with it manually", "track everything in a spreadsheet, and people slip through rep_sourced_nulls: {"TX-003": ["budget_signal \u2014 only the rep mentioned pricing/budget"], "TX-004": ["next_step \u2014 rep proposed, prospect said 'no promises'"], "TX-006": ["budget_signal \u2014 rep decoy, prospec allowed_new: TX-001, TX-002, TX-003, TX-004, TX-005, TX-006, Achievers, Bucketlist, Nectar graded checks: json_emitted · budget_signal · null_discipline · stakeholders · competitor_prospect_only
Model's answer
not yet run — check back after the runners complete
next-to-close0.75
? The Question
Which 3 deals are closest to signature right now? Use the CRM data and the Slack thread; Slack often has the real status before the CRM does. For each: deal, amount, why it is close, what is left. Under 250 words.
Closest to signature:
1. Deal-547B2B — $11,200
Why close: Slack says redlines came back clean, signing page is out, and VP People said they are signing tomorrow. Arithmetic: thread date 2026-09-04; “tomorrow” = 1 day. CRM DS5/COMMIT close date 2026-09-11 = 7 days.
Left: signature / contract execution.
2. Deal-403845 — $9,000
Why close: Slack says the order form is with finance; CRM is DS5/COMMIT. Arithmetic: 2026-09-11 - 2026-09-04 = 7 days.
Left: finance approval and signature.
3. Deal-A2B47C — $6,360
Why close: Slack says it is still warm with normal legal-review pace; CRM is DS5/COMMIT. Arithmetic: 2026-09-11 - 2026-09-04 = 7 days.
Left: complete legal review and signature.
Excluded: Deal-2465CE — CRM shows $5,400 and 2026-09-10, but Slack says the champion left, procurement froze new vendors, and it is blocked until they re-staff / Q4.
product-gap-detectionPASS 1.00
? The Question
Surface true product gaps raised by prospects in active deals. For each candidate: quote the prospect line, classify as real gap, plan gate, or rollout/enablement issue using the product docs excerpt, and name the deal and amount. Only real gaps go in the summary. Rep language does not count as prospect voice.
real_gaps: transcript_id=TG-001; deal_alias=Deal-EC3025; amount=62000; feature=SCIM user provisioning; quote=We need SCIM user provisioning — our security team will not approve any tool that requires manual account management.; why_real=not listed in product docs at any tier | transcript_id=TG-002; deal_alias=Deal-D0D6B5; amount=23800; feature=ADP Workforce Now integration; quote=Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us.; why_real=supported HRIS list is Workday/BambooHR/Gusto/Rippling only plan_gates: transcript_id=TG-003; deal_alias=Deal-CFE7F4; amount=32000; feature=Custom report builder; tier_required=Enterprise; quote=I tried to build a custom report combining recognition frequency with turnover and couldn't — is that possible on our current plan? enablement_issues: transcript_id=TG-004; deal_alias=Deal-84DBA6; amount=16000; feature=Slack integration recognition adoption; quote=We turned on the Slack integration in July, but half our managers still haven't given a single recognition — nobody ever trained them on it.; why_enablement=Slack integration exists on all plans; managers were never trained excluded_rep_voice: transcript_id=TG-005; line=Full transparency — we don't have a native mobile app for hourly workers yet, though it's on the roadmap.; reason=rep voice, not prospect-raised allowed_new: TG-001, TG-002, TG-003, TG-004, TG-005, SCIM user provisioning, ADP Workforce Now integration, Custom report builder graded checks: real_gaps · plan_gate_classified · enablement_classified · gap_summary
Model's actual answer
show the model's full answer
Candidate review:
| Transcript | Deal | Amount | Prospect line | Classification | Evidence from product docs |
|---|---:|---:|---|---|---|
| TG-001 | Deal-EC3025 | Not provided | “We need SCIM user provisioning — our security team will not approve any tool that requires manual account management.” | Real gap | Docs state: “SCIM user provisioning … are NOT currently listed as supported capabilities.” |
| TG-002 | Deal-D0D6B5 | Not provided | “Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us.” | Real gap | Docs state: “ADP Workforce Now integrations are NOT currently listed as supported capabilities.” |
| TG-003 | Deal-CFE7F4 | Not provided | “I tried to build a custom report combining recognition frequency with turnover and couldn't — is that possible on our current plan?” | Plan gate | Docs show “Custom report builder” is Enterprise only. |
| TG-004 | Deal-84DBA6 | Not provided | “We turned on the Slack integration in July, but half our managers still haven't given a single recognition — nobody ever trained them on it.” | Rollout/enablement issue | Docs show “Slack and Microsoft Teams integration” is available on all plans. |
| TG-005 | Deal-36C33F | Not provided | “Good to know. The web version should be fine for our office staff for now.” | Not a prospect-raised gap | Prospect accepted the web version; no gap raised by prospect. Rep-only mobile-app statement is excluded. |
Arithmetic:
- Prospect-raised candidate lines reviewed: 4
- Real product gaps: 2
- Plan gates: 1
- Rollout/enablement issues: 1
- Prospect lines not treated as gaps: 1
- Real-gap deal amount total: not calculable because amount is not provided for any deal.
Summary of true product gaps:
1. Deal-EC3025 — SCIM user provisioning
Prospect quote: “We need SCIM user provisioning — our security team will not approve any tool that requires manual account management.”
Amount: Not provided.
2. Deal-D0D6B5 — ADP Workforce Now integration
Prospect quote: “Our payroll runs on ADP Workforce Now and I don't see ADP anywhere in your integration list — that's a dealbreaker for us.”
Amount: Not provided.
rep-performance
stale-pipeline-by-repdiscarded
? The Question
The snapshot date is 2026-09-05. Using the open deals and the engagements-by-deal table, list every open deal with no logged email, call, or meeting in the last 7 days. Compute recency from the engagements table's last_email/last_call/last_meeting fields, NOT from any date field on the deal (the deal's last_contacted_field is unreliable). For each stale deal give: deal alias, owner name, stage, amount, days since last contact. Group by owner, order by amount descending within each owner. Then state, per owner, how many stale deals they carry and the total stale amount. (Do not write outreach text.)
not yet run — check back after the runners complete
activity-mix-vs-outcomediscarded
? The Question
The snapshot date is 2026-09-05. For each sales rep (owner), compute the last-30-day totals of emails, calls, and meetings from the engagements table, and the number of deals that entered DS2 in the last 30 days (t_ds2 within 30 days of the snapshot). Report per rep: the activity mix as percentages (emails/calls/meetings share of that rep's total activities), total activities per DS2 entry, and rank the reps by that efficiency ratio (lowest activities per DS2 entry = most efficient). Name the most efficient rep and the highest-volume rep, and note if they differ. Do not attribute cause.
not yet run — check back after the runners complete
ae-qtd-scorecarddiscarded
? The Question
Build a QTD scorecard for the named AE as of 2026-09-05: bookings vs quota with attainment percent, new vs expansion split, active pipeline by stage with amount, rolling 90-day DS2-to-won rate, win and loss counts with top loss reason, activity volume by type in the last 30 days. Exclude any closed-won deal dated before the quarter. Close with three coaching observations grounded in the numbers.
not yet run — check back after the runners complete
multithreading-gapdiscarded
? The Question
Find every open deal that is single-threaded (fewer than 2 active contacts) or under-threaded (fewer than 3, or all contacts in one persona). Active means engaged in the last 60 days and not marked former. Personas: economic buyer, champion, HR admin, IT security, finance. For each flagged deal: amount, stage, active contact count, personas present, personas missing, the most valuable persona to add given the stage, and any on-file unengaged contact at that company who fits; otherwise say none on file.
✓ Correct Answer (ground truth)
active_window_start: 2026-07-07 persona_universe: economic buyer, champion, HR admin, IT security, finance flag_rule: flag if active_contacts < 2 (single-threaded) or active_contacts < 3 or all active contacts share one persona (under-threaded); active = engaged within 60 days and not former best_persona_rule: DS2/DS3: economic buyer > IT security > finance > HR admin > champion; DS4/DS5: economic buyer > finance > IT security > HR admin > champion; first persona not present among active contacts flagged: deal_alias=Deal-36C33F; amount=15000; stage=DS2; active_count=1; reason=single-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-50D386; amount=36000; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-5408B0; amount=14850; stage=DS4; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-5BFE3B; amount=18000; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=economic buyer | deal_alias=Deal-885F45; amount=9300; stage=DS2; active_count=2; reason=under-threaded; best_persona_to_add=IT security | deal_alias=Deal-92D97D; amount=60000; stage=DS2; active_count=1; reason=single-threaded; best_persona_to_add=economic buyer … (+5 more) healthy_not_flagged: Deal-4B0BEB, Deal-84DBA6, Deal-D348E1 allowed_new: 59 items — e.g. C-077A0E, C-0D15DF, C-1BDFE0, C-2AE3AA graded checks: flagged_recall · persona_language · single_thread_term · on_file_fit
Model's answer
not yet run — check back after the runners complete
talk-track-from-transcriptsdiscarded
? The Question
From this rep's last 10 calls: what they lead with in the first five minutes, how they handle the three most common objections, how often a concrete next step is agreed (rate), and every competitor a prospect raised. Quote at most one line per finding and only lines that appear in the transcripts. End with two coaching notes.
✓ Correct Answer (ground truth)
rep: Alex Franklin n_transcripts: 10 opener_theme: theme=opens with a customer recognition win story (400-person retailer, 18% turnover reduction); count=8; total=10; quote=Quick one to kick off: a 400-person retailer we work with cut regretted turnover 18% in two quarters after automating milestone awards, and I wanted to walk you through exactly how they did it.; transcript_id=TT-001 objections: objection=budget; count=4; quote=Honestly, budget is locked until next fiscal year — I can't add a new line item right now.; transcript_id=TT-001 | objection=timing; count=3; quote=This is good, but can we revisit it next quarter? Open enrollment starts in October and we're underwater.; transcript_id=TT-002 | objection=in-house; count=3; quote=We already do recognition with a spreadsheet and quarterly gift cards — why would we change?; transcript_id=TT-004 objection_count: 3 next_step: agreed=7; total=10; rate=0.7 competitors_prospect_raised: Awardco, Kudos competitor_quotes: Awardco=We're also in late talks with Awardco — their rewards catalog looks bigger than yours.; Kudos=How are you different from Kudos? Our CEO used them at her last company. rep_raised_competitors_excluded: Workhuman allowed_new: 13 items — e.g. TT-001, TT-002, TT-003, TT-004 graded checks: objection_count · next_step_rate · competitors · coaching · quote_used
Model's answer
not yet run — check back after the runners complete
reporting-analytics
q3-weighted-forecastdiscarded
? The Question
Below are the open deals in the default sales pipeline with a close date between 2026-07-01 and 2026-10-15, pulled from the CRM warehouse on 2026-09-05. Stages: DS1 (discovery) through DS5 (contract).
Build the Q3 2026 forecast (quarter = 2026-07-01 to 2026-09-30). Weighting rule: 100 percent of COMMIT plus 35 percent of BEST_CASE. PIPELINE deals count zero. Only deals with a close date inside the quarter count.
Report: COMMIT total, BEST_CASE total, weighted forecast, count of deals in each category inside the quarter, the deals you excluded for being outside the quarter (count and total amount), and the top 5 BEST_CASE deals by amount inside the quarter. Show arithmetic. Then, in a final '## Data quality' section of no more than four sentences, name the data quality problems in this extract that would make the forecast unreliable if it were run unattended.
not yet run — check back after the runners complete
aha-moment-2x2discarded
? The Question
Test the activation hypothesis on the company cohort below: companies with 5+ unique givers (m1_users >= 5) AND 1+ successful redemption (m1_redemptions >= 1) in their first calendar month retain better at 24 months than those with one signal or neither. Every company in this extract is already 25+ months old, so current_status = 'active' means retained at 24 months. Report the full 2x2 (cohort size and 24-month retention rate for: both signals, givers-only, redemption-only, neither), state how many companies were excluded from the denominator and why (if none, say so), name the single signal with the largest retention lift, and state what this does and does not prove.
not yet run — check back after the runners complete
arr-rollforward-reconciliationdiscarded
? The Question
Reconcile ARR between CRM company records and billing subscriptions as of 2026-09-05. Billing ARR is MRR x 12. Report both totals, the variance, and a decomposition into buckets (status mismatch, rounding, missing records, other) that sums exactly to the variance. List each mismatched account with a suggested owner. Business rule: any subscription with a term other than 12 months must have cf_agreement_end_date populated; list violations.
not yet run — check back after the runners complete
kvm-month-over-monthdiscarded
? The Question
Report the four core KVMs for 2026-08 against 2026-07: giving rate, redemptions per user, 1:1 meetings engagement, pulse check engagement. For each: value, prior value, absolute and relative change, direction. Name the KVM with the largest relative move and the segment (plan_tier or size_band) driving it if the data supports that; otherwise say the data does not. Under 200 words plus one table.
not yet run — check back after the runners complete
rewards-ytd-sectiondiscarded
? The Question
Produce the redemption section of the weekly rewards report through the last completed month (2026-08): redemption count, spend, unique redeemers, redemptions per redeemer, provider mix as percent of spend, top 5 countries by redemptions. Say which month is the last completed month. Provider shares must sum to 100.
not yet run — check back after the runners complete
customer-success
churn-save-eligibilitydiscarded
? The Question
Which at-risk accounts qualify for a churn-save offer under the documented eligibility rules, what amount is at stake per account and in total, and which play fits each (usage revival, executive touch, commercial concession)? Cite the signal that justifies each play. List accounts that look at risk but do not qualify and why.
✓ Correct Answer (ground truth)
snapshot_date: 2026-09-05 rules: health_score < 60, churn_save_eligible_amount > 0, renewal within 120 days of snapshot eligible: account_alias=C-0F6C0F34; amount_at_stake=49707.00; play=executive touch; justifying_signal=champion_active is false - no executive sponsor engaged | account_alias=C-0B827671; amount_at_stake=25365.00; play=usage revival; justifying_signal=usage_trend_3m=declining over the last 3 months | account_alias=C-0B360C78; amount_at_stake=35748.00; play=commercial concession; justifying_signal=usage stable/growing with seat utilization 75% - risk is commercial, not adoption | account_alias=C-0B0F1BAB; amount_at_stake=5494.00; play=executive touch; justifying_signal=champion_active is false - no executive sponsor engaged | account_alias=C-0CA21961; amount_at_stake=16829.00; play=usage revival; justifying_signal=seat utilization 26% is below 50% | account_alias=C-0E9C27D1; amount_at_stake=41235.00; play=commercial concession; justifying_signal=usage stable/growing with seat utilization 85% - risk is commercial, not adoption … (+2 more) total_amount_at_stake: 224601.00 non_eligible_at_risk: account_alias=C-0BC71BDD; health_score=55 | account_alias=C-0BA71F12; health_score=52 | account_alias=C-0F6694C3; health_score=43 | account_alias=C-0BE96399; health_score=54 | account_alias=C-0F876796; health_score=47 | account_alias=C-0FCCD2DF; health_score=43 … (+1 more) n_accounts: 30 graded checks: eligible_set · total_at_stake · plays_cited · noneligible_named · rules_applied
Model's answer
not yet run — check back after the runners complete
expansion-kitPASS 1.00
? The Question
A customer replied to outreach (reply included). Build the expansion kit: seat coverage (licensed over headcount), two lines on usage health, headroom in seats and ARR at the current per-seat rate, who replied and whether they can buy, the right buyer from our contacts, and a reply email under 150 words that moves toward the right person without being pushy and cites one real usage fact.
company_alias: C-0DDFC9A7 coverage_pct: 37.5 headroom_seats: 250 per_seat_annual_rate: 60.00 headroom_arr: 15000.00 replier: Maria S. replier_can_buy: False right_buyer: name=Dana R.; title=VP People usage_facts: active_users_2026_08=126; active_users_2026_03=88; growth_pct_6m=43.2; licensed_seat_utilization_pct=84.0 usage_health_lines: Monthly active users grew from 88 (2026-03) to 126 (2026-08), +43.2% over 6 months., Last month 126 of 150 licensed seats were active (84.0% utilization), approaching the seat cap. graded checks: coverage_pct · headroom_arr · right_buyer · replier_not_buyer · usage_fact · email_under_150
Model's actual answer
show the model's full answer
Company alias: C-0DDFC9A7
Deal alias: not provided
Seat coverage
- Licensed seats / headcount = 150 / 400 = 37.5%
- Headroom in seats = 400 - 150 = 250 seats
- Current per-seat rate = $9,000 / 150 = $60/seat
- ARR headroom = 250 seats × $60/seat = $15,000
Usage health
- MAUs grew every month: 88 → 95 → 102 → 110 → 118 → 126; Mar-to-Aug increase = 126 - 88 = +38 users.
- Aug usage = 126 / 150 licensed seats = 84.0% licensed-seat utilization.
Who replied and whether they can buy
- Maria S., People Operations Coordinator, replied on 2026-09-02.
- She said she is not the right person for purchasing decisions, so she cannot buy.
Right buyer from our contacts
- Dana R., VP People, last engaged 2026-05-18.
- Maria said budget and seat expansion sit with Dana R.
Reply email under 150 words
Hi Maria,
Thanks for sharing — glad to hear the team loves Bonusly. Your usage has grown from 88 MAUs in Mar to 126 in Aug, so it may be helpful to compare that trend with Dana’s seat-expansion planning.
If you’re comfortable, could you introduce me to Dana? I’ll keep it brief and share the usage snapshot she’s been asking about.
Best,
Cole
onboarding-statusPASS 1.00
? The Question
Prep the CSM for the mid-onboarding call: what is complete (integrations connected, allowance set, admins added, first recognition given, first redemption), what is not, early engagement signals, and the three things to cover on the call. Mark nothing complete without a data field that shows it.
company_alias: C-0D284E42 days_live: 25 checklist: {"slack_integration_connected": {"complete": true, "field": "integration_slack", "value": "2026-08-12"}, "hris_integration_connected": {"complete": false, "field": "integration_hris", "value": ""}, "a engagement_signals: avg_active_givers_week1=4.3; avg_active_givers_last_week=13.0; latest_active_givers=15; days_with_activity=25 call_items: Connect the HRIS integration (integration_hris is empty) to automate user provisioning and offboarding., Get to first redemption (first_redemption_at is empty): walk admins through the rewards catalog and confirm allowance funding., Build the giving habit: daily active givers averaged 4.3 in week 1 vs 13.0 last week - agree on manager nudges and a recognition cadence. graded checks: complete_items · incomplete_items · three_items · grounded
Model's actual answer
show the model's full answer
Company: C-0D284E42
## Complete
- Slack integration connected: `integration_slack = 2026-08-12`
- Allowance set: `allowance_set = 2026-08-13`
- Admins added: `admins_added = 2`
- First recognition given: `first_recognition_at = 2026-08-15 14:22`
## Not complete / missing
- HRIS integration not connected: `integration_hris` is blank.
- First redemption not complete: `first_redemption_at` is blank.
## Early engagement signals
Usage data covers 25 days: 2026-08-11 through 2026-09-04.
- Total active-giver-days: `214`
- Average active givers/day: `214 / 25 = 8.56`
- First 7 days: `30 / 7 = 4.29` active givers/day
- Last 7 days: `91 / 7 = 13.00` active givers/day
- First-to-latest active givers: `15 - 3 = +12`
- Growth multiple: `15 / 3 = 5.0x`
- Peak active givers: `15` on `2026-09-03` and `2026-09-04`
## Three things to cover on the call
1. HRIS integration is still missing; confirm owner and next step for `integration_hris`.
2. First redemption is still missing despite recognition happening on `2026-08-15 14:22`; identify why no `first_redemption_at` exists yet.
3. Reinforce the active-giver growth trend: active givers rose from `3` to `15`, with the last 7-day average at `13.00/day` versus the first 7-day average at `4.29/day`.
renewal-risk-conflicting-datesdiscarded
? The Question
Produce a 90-day renewal risk brief. Two systems hold renewal dates and disagree on some accounts; decide which to trust per account and say why (multi-year contracts are known to be wrong in ChurnZero). For every renewal: company, CSM, ARR, date used, seat utilization, 3-month usage trend, risk rating with one sentence of evidence. Flag every disagreement. Close with total ARR renewing and ARR at risk.
not yet run — check back after the runners complete
ticket-theme-synthesisdiscarded
? The Question
Synthesize the quarter's support tickets into 4 to 7 themes. Do not trust existing tags; read the text. Per theme: name, count, share, distinct accounts, ARR affected, two ticket ids, one-line recommendation. Separate broad patterns from single-account noise and rank by ARR exposure, not volume.
not yet run — check back after the runners complete
marketing
case-study-matchPASS 1.00
? The Question
Given the prospect, rank the three most similar existing customers with public case studies for use as social proof. Similarity on industry, size band, use case, region. Cite the fields that drove each match. Do not name any customer without a case study and never include billing data or contact names.
Filtered to customers with `has_case_study=true`.
Scoring: 1 point per exact match on `industry`, `size_band`, `use_case`, `region`; max = 4/4.
Prospect `C-82AF3719`: Technology / Mid-Market / employee_recognition / NA-West.
1. `C-64171065` — 3/4
- Matched fields: `industry=Technology`, `size_band=Mid-Market`, `use_case=employee_recognition`
- Mismatch: `region=NA-East` vs `NA-West`
- Arithmetic: 1 + 1 + 1 + 0 = 3/4
2. `C-A13C193D` — 3/4
- Matched fields: `industry=Technology`, `size_band=Mid-Market`, `region=NA-West`
- Mismatch: `use_case=retention` vs `employee_recognition`
- Arithmetic: 1 + 1 + 0 + 1 = 3/4
3. `C-11C31562` — 3/4
- Matched fields: `size_band=Mid-Market`, `use_case=employee_recognition`, `region=NA-West`
- Mismatch: `industry=Manufacturing` vs `Technology`
- Arithmetic: 0 + 1 + 1 + 1 = 3/4
No public-case-study customer matched all four fields.
channel-attribution-mathPASS 1.00
? The Question
Compute channel performance for the trailing 6 months. Paid: spend, SQMs, SQOs, cost per SQM, cost per SQO, SQM-to-SQO rate, pipeline amount, pipeline per dollar. Organic: volume, SQO rate, pipeline. Report a channel with spend and zero SQMs as undefined, not zero. Flag rows where SQO date precedes SQM date. Finish with a reallocation recommendation and your confidence given sample sizes.
months: 2026-03, 2026-04, 2026-05, 2026-06, 2026-07, 2026-08 per_channel: {"linkedin_ads": {"spend_usd": 24000, "sqms": 25, "sqos": 8, "pipeline_amount": 96000, "cost_per_sqm": 960.0, "cost_per_sqo": 3000.0, "sqm_to_sqo_rate": 0.32, "pipeline_per_dollar": 4.0}, "organic_sea undefined_channels: paid_social flagged_rows_sqo_before_sqm: CT-000041, CT-000044 best_pipeline_per_dollar_channel: paid_search reallocation_recommendation: from=paid_social; to=paid_search; rationale=paid_social has spend but zero SQMs (cost per SQM undefined, not zero); paid_search has the highest pipeline per dollar. confidence_note: webinars and referral have small SQM samples (<20); treat rates as directional. graded checks: undefined_channel · undefined_not_zero · flagged_rows · best_channel · reallocation
Model's actual answer
show the model's full answer
Trailing 6 months: 2026-03 through 2026-08.
### Paid channels
| Channel | Spend | SQMs | SQOs | Cost/SQM | Cost/SQO | SQM→SQO rate | Pipeline | Pipeline/$ |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| paid_search | $36,000 | 40 | 18 | $900 | $2,000 | 45.0% | $720,000 | $20.00 |
| linkedin_ads | $24,000 | 25 | 8 | $960 | $3,000 | 32.0% | $96,000 | $4.00 |
| paid_social | $18,000 | 0 | 0 | undefined | undefined | undefined | $0 | $0.00 |
| webinars | $9,000 | 12 | 5 | $750 | $1,800 | 41.7% | $60,000 | $6.67 |
### Organic channels
| Channel | Volume / SQMs | SQOs | SQO rate | Pipeline |
|---|---:|---:|---:|---:|
| organic_search | 30 | 10 | 33.3% | $90,000 |
| referral | 15 | 6 | 40.0% | $48,000 |
### SQO date precedes SQM date
Flagged rows:
- CT-000044, linkedin_ads: SQM date = 2026-07-23; SQO date = 2026-07-18; pipeline_amount = $12,000.
- CT-000041, linkedin_ads: SQM date = 2026-06-14; SQO date = 2026-06-09; pipeline_amount = $12,000.
### Reallocation recommendation
Shift paid budget toward paid_search and away from paid_social and linkedin_ads.
Reason:
- paid_search has the best pipeline/$ at $20.00, strongest SQM→SQO rate at 45.0%, and $720,000 pipeline on $36,000 spend.
- webinars has $6.67 pipeline/$ and 41.7% SQM→SQO rate, but lower scale.
- linkedin_ads has $4.00 pipeline/$, 32.0% SQM→SQO rate, and 2 data-quality flags.
- paid_social has spend of $18,000 and zero SQMs, so cost/SQM, cost/SQO, and SQM→SQO rate are undefined.
Confidence: Medium-low. Sample sizes are small: paid_search has 40 SQMs, linkedin_ads has 25 SQMs, paid_social has 0 SQMs, webinars has 12 SQMs, organic_search has 30 SQMs, and referral has 15 SQMs. The recommendation is directionally clear from the provided data, but paid_social and webinars are especially noisy due to low/no SQM counts.
competitor-battlecard-updatePASS 1.00
? The Question
Update the battlecard for the competitor. Sections: one-line positioning, pricing with source and date (newer source wins, note the conflict), where they win, where we win, objections and responses, recent changes, our 12-month win/loss record against them. Cite a snippet id for every factual claim. Rep opinion on a call is not a fact about the competitor. Anything from the old card you cannot re-source gets marked unverified.
competitor: Rivally pricing: value_per_user_per_month=7; source_snippet=S17; source_date=2026-08-12 win_loss_12mo: wins=13; losses=7 unverified_old_claims: Rivally lacks a Slack integration., Rivally was acquired by WorkHuman in 2025. rep_opinion_snippet_ids_not_facts: S09, S21 citation_ids_available: 25 items — e.g. S01, S02, S03, S04 graded checks: competitor_named · pricing · win_loss · unverified_marked · citations
Model's actual answer
show the model's full answer
# Battlecard: Rivally
## One-line positioning
Rivally is a recognition platform centered on a points-based recognition feed, with mid-market setup evidence and EU expansion/data-residency messaging (S02, S04, S05, S11, S12, S15).
## Pricing — newer source wins; note conflicts
- Current public pricing to use: Recognition Starter is $7/user/month, annual billing required (S17, 2026-08-12 pricing_page).
- Older pricing conflict: $5/user/month, annual billing required (S03, 2026-01-20 pricing_page; S08, 2026-04-01 pricing_page).
- Deal-note pricing conflict: $6.50/user/month annual quote for a 500-seat prospect (S13, 2026-06-02 call_notes); $7/user/month list with 15% discount for a 3-year term (S18, 2026-08-14 call_notes).
## Where Rivally wins
- Recognition engagement: points-based recognition feed praised (S02); recognition feed described as engaging (S16).
- Implementation/integration: mid-market setup took under a week, and Slack integration worked out of the box (S04).
- EU coverage: Rivally pitched EU data residency (S05); EU enterprise reviewer praised distributed EU teams and multi-language support (S12); Dublin office opened and EU data residency became generally available (S15).
- Support: support response time praised as under 4 hours (S22).
## Where we win / supported win themes
- Analytics depth/exportability: an 800-seat prospect picked Bonusly over Rivally citing analytics depth (S25); Rivally analytics are cited as limited (S02), basic compared to enterprise tools (S07), and CSV-only for exports (S20).
- Enterprise admin/security gaps to exploit: Rivally lacks SCIM provisioning and manual user management is painful (S10); admin tooling lags peers (S16); admin console lacks bulk recognition editing (S24).
- No other specific Bonusly win reasons are provided in S01-S25.
## Objections and responses
| Objection | Evidence | Response |
|---|---|---|
| Rivally’s recognition feed is engaging. | Points-based feed praised (S02); feed described as engaging (S16). | Acknowledge, then qualify analytics/reporting depth and export needs: limited analytics (S02), basic dashboards vs enterprise tools (S07), CSV-only analytics exports (S20). |
| Rivally is easy to implement and Slack works. | Setup under a week; Slack worked out of the box (S04). | Acknowledge, then probe admin/security requirements: no SCIM and painful manual user management (S10), admin tooling lags peers (S16), no bulk recognition editing (S24). |
| Rivally supports EU teams and data residency. | EU data residency pitched (S05); EU reviewer praised distributed EU teams and multi-language support (S12); EU data residency generally available (S15). | Acknowledge, then test EMEA rewards catalog breadth: EMEA catalog is thinner than US catalog (S14). |
| Rivally is cheaper. | $7/user/month Starter with annual billing required (S17); $7/user/month list with 15% discount for 3-year term (S18). | Normalize terms: annual billing required (S17), 3-year discount term (S18), and Pulse is priced as an add-on, not bundled (S23). |
| Rivally has fast support. | Support response time praised as under 4 hours (S22). | Acknowledge, then pivot to analytics/admin gaps (S07, S10, S20, S24). |
## Recent changes
- Rivally raised a $40M Series C led by Northgate Ventures (S01, 2025-11-04).
- Rivally launched Rivally Pulse, a lightweight engagement survey add-on (S06, 2026-03-05).
- Rivally hired an ex-Workday VP EMEA to lead European expansion (S11, 2026-05-09).
- Rivally opened a Dublin office and announced EU data residency generally available (S15, 2026-07-01).
- Rivally updated Recognition Starter to $7/user/month with annual billing required (S17, 2026-08-12).
- Rivally announced Microsoft Teams app v2 in public preview (S19, 2026-08-20).
- Rivally Pulse exited beta and is priced as an add-on, not bundled (S23, 2026-09-01).
- Rivally admin console still lacks bulk recognition editing (S24, 2026-09-02).
- 800-seat prospect picked Bonusly over Rivally citing analytics depth (S25, 2026-09-03).
## Our 12-month win/loss record against Rivally
Source for all rows: deals_with_competitor.csv; this file has no snippet_id column.
| Month | Record | Deal aliases |
|---|---:|---|
| 2025-09 | 1-1 | Win: Deal-072E31; Loss: Deal-7767F5 |
| 2025-10 | 2-0 | Win: Deal-A9FD43, Deal-F65C8F |
| 2025-11 | 1-1 | Win: Deal-7AA785; Loss: Deal-D263E0 |
| 2025-12 | 1-1 | Win: Deal-44C524; Loss: Deal-935746 |
| 2026-01 | 2-0 | Win: Deal-0D0CD6, Deal-E46EAB |
| 2026-02 | 2-0 | Win: Deal-D5B790, Deal-1D2392 |
| 2026-03 | 1-1 | Win: Deal-5C636E; Loss: Deal-9066A6 |
| 2026-04 | 0-2 | Loss: Deal-5645A5, Deal-72A02F |
| 2026-05 | 0-1 | Loss: Deal-C6FFAA |
| 2026-06 | 1-0 | Win: Deal-67BE14 |
| 2026-07 | 1-0 | Win: Deal-1B6969 |
| 2026-08 | 1-0 | Win: Deal-F03E7B |
Arithmetic: 13 wins + 7 losses = 20 total deals. Win rate = 13 / 20 = 65%. Loss rate = 7 / 20 = 35%.
## Old-card status
- Re-sourced: points-based recognition feed and mid-market setup evidence (S02, S04).
- Re-sourced but stale: $5/user/month annual billing (S03, S08); newer source shows $7/user/month annual billing (S17).
- Contradicted: “Rivally lacks a Slack integration” — S04 says Slack integration worked out of the box.
- Re-sourced: strong for distributed EU teams and multi-language support (S12); EU data residency generally available (S15).
- Unverified: “Rivally was acquired by WorkHuman in 2025” — no supporting snippet in S01-S25.
Rep-opinion exclusions: S09 and S21 are not used as competitor facts; both are call-note opinions rather than verified competitor facts (S09, S21).
nurture-sequence-diagnosisdiscarded
? The Question
Review the sequences. Per sequence: sent, open rate, reply rate, meeting rate, weakest step. Identify tracking errors (opened above sent), audience overlap between sequences, and the failure mode for anything under 2 percent reply. One change per weak sequence and which to fix first. Under 350 words.
✓ Correct Answer (ground truth)
per_sequence: {"New Logo Nurture": {"sent": 1386, "opened": 490, "replied": 90, "meetings": 27, "open_rate": 0.3535, "reply_rate": 0.0649, "meeting_rate": 0.0195, "weakest_step": 3}, "Expansion Nurture": {"sent": 8 tracking_error: sequence=Expansion Nurture; step=2; sent=300; opened=340; issue=opened exceeds sent audience_overlap_pair: Cold Outbound - HR Leaders, Cold Outbound - People Ops all_pairwise_overlaps: ['Cold Outbound - HR Leaders', 'Cold Outbound - People Ops'], ['Expansion Nurture', 'New Logo Nurture'] under_2pct_reply: Cold Outbound - HR Leaders failure_mode: Cold Outbound - HR Leaders=opens healthy (~37%) but replies near zero from step 1 -> body/CTA not landing, not a deliverability or subject-line problem fix_priority_first: Cold Outbound - HR Leaders graded checks: tracking_error · overlap_pair · weakest_step · fix_priority
Model's answer
not yet run — check back after the runners complete
weekly-marketing-goals-rollupdiscarded
? The Question
Produce the weekly marketing goals update for the active quarter: SQMs, SQOs, DS2s, closed-lost MIA rate, same-quarter close count, active pipeline coverage against target. For each: QTD actual, target, delta, pace (on/behind/ahead given days elapsed). One paragraph on what moved this week.
not yet run — check back after the runners complete
communication
ceo-slack-compressiondiscarded
? The Question
Compress the forecast analysis below into a single Slack message for the CEO. Rules: at most three sentences; under 80 words; lead with the decision or the so-what; keep every distinct number that matters and the one caveat that changes the conclusion; cut any sentence that does not carry a claim, a number, or a directive. No em dashes, no throat-clearing phrases.
not yet run — check back after the runners complete
stalled-deal-followup-emailPASS 1.00
? The Question
Write a follow-up email for this stalled deal. Under 120 words. Reference one specific prior touch from the engagement history, make one clear ask, and match the tone of the prior thread. No invented details.
cost $0.0011 · 2996s · in 437 / out 559 tok
✓ word_cap✓ references_touch✓ single_ask
✓ Correct Answer (ground truth)
deal_alias: Deal-0D2F7A amount: 5100 stage: DS3 owner: Alex Franklin days_since_last_engagement: 31 word_cap: 120 must_reference: engagement_id=ENG-00002; date=2026-07-28; type=meeting; why=most substantive prior touch; concrete prospect reactions to milestone awards and points catalog single_cta: True tone: warm, concise, professional (matches prior thread) allowed_facts: 14 items — e.g. Deal-0D2F7A, $5,100, DS3, Alex Franklin allowed_new: ENG-00001, ENG-00002, ENG-00003, ENG-00004, ENG-00005 graded checks: word_cap · references_touch · single_ask
Model's actual answer
show the model's full answer
Subject: Deal-0D2F7A follow-up
Hi,
Following up on Deal-0D2F7A after the Aug. 5 email recapping the July 28 demo and pricing for 150 seats.
Given the People team’s interest in automated milestone awards and the points catalog, would it be useful to revisit pricing?
Best,
Alex
funnel-digest-paragraphPASS 1.00
? The Question
Write the weekly GTM digest: one paragraph per team (Marketing, Sales, CS, PLG) in Bonusly brand voice. Every number must trace to the extracts. No customer contact names, no employee performance commentary, no compensation. Four paragraphs, under 400 words total.
Marketing: SQM attainment came in at 46 / 52 = 88.5%, with the webinar channel driving 18 / 46 = 39.1% of SQMs. Paid search held at $150 cost per SQM, keeping spend efficiency steady.
Sales: Sales converted 14 SQOs and set 9 DS2 meetings. New pipeline created was $310,000, and same-quarter close count was 3. No SQO conversion-rate denominator was provided, so no rate is calculated.
CS: Renewals saved this week were 2, and Team NPS moved to 61. There are 3 open red-flag accounts heading into next week, so the focus is clear: protect the base and remove friction fast.
PLG: PLG added 412 new signups, with activation at 31%. 38 companies reached the aha moment of 10 recognition gives. No overlap between signups and companies was provided, so no additional conversion rate is calculated.
partner-digest-section0.67
? The Question
Write the partner digest for the period. Every partner in the roster appears once: active plays get status with numbers, quiet partners get one line. Partner-sourced pipeline count and amount must match the deal data. Nothing omitted, nothing added.
Partner digest for 2026-08-24 to 2026-09-04
- Apex Rewards Co (active): 2 partner-sourced deals, $275,000. Arithmetic: Deal-DDAAF2 $180,000 + Deal-2CF33E $95,000 = $275,000. Status: co-webinar locked for 09-15; both sourced opps in DS1 and logged with UTM Source = Partner.
- HRCloud Partners (active): 1 partner-sourced deal, $140,000. Arithmetic: Deal-F1CDA5 $140,000 = $140,000. Status: integration referral closed security review; opp moved to DS2.
- CultureBridge (active): 2 partner-sourced deals, $135,000. Arithmetic: Deal-096E1D $60,000 + Deal-067213 $75,000 = $135,000. Status: lunch-and-learn produced two sourced opps; both early stage.
- WorkWell Group (active): 0 partner-sourced deals, $0. Arithmetic: no partner_deals.csv rows for WorkWell Group. Status: wants to restart the joint playbook in Q4; no sourced deals this period; planning call booked for 09-09.
- Recogniq (quiet): No sourced deals in partner_deals.csv; no Slack update in the period.
- KudosWave (quiet): No sourced deals in partner_deals.csv; no Slack update in the period.
- PeopleFirst Advisors (quiet): No sourced deals in partner_deals.csv; no Slack update in the period.
- TotalPerk (quiet): No sourced deals in partner_deals.csv; no Slack update in the period.
rfp-answer-from-policyPASS 1.00
? The Question
Answer the 10 questionnaire items from the policy excerpts only. Cite the excerpt id for every answer. Where the excerpts do not answer the question, mark it unanswerable and say what document would; do not guess. Tone: vendor response, direct.
answered_count: 7 answer_map: {"Q1": ["P1"], "Q2": ["P1"], "Q3": ["P2"], "Q4": ["P3"], "Q5": ["P4"], "Q6": ["P5"], "Q7": ["P6"]} unanswerable: Q8, Q9, Q10 valid_citation_ids: P1, P2, P3, P4, P5, P6, P7, P8 rule: unanswerable items must be flagged and the document that would answer them named; do not guess. graded checks: answered_count · unanswerable_flagged · citations · no_guess
Model's actual answer
show the model's full answer
| Item | Vendor response |
|---|---|
| Q1 | Yes. Customer data is encrypted at rest using AES-256. Cite: [P1] |
| Q2 | Yes. Data in transit is encrypted with TLS 1.2 or higher. Cite: [P1] |
| Q3 | Yes. Bonusly maintains a SOC 2 Type II report, renewed annually. It is available to prospects under NDA. Cite: [P2] |
| Q4 | Customer data is retained for the duration of the contract plus 90 days, then deleted from production systems. Cite: [P3] |
| Q5 | The current subprocessor list is published at trust.bonusly.com. Customers receive 30 days notice before a new subprocessor is added. Cite: [P4] |
| Q6 | Yes. SAML 2.0 single sign-on is available on the Enterprise plan. Cite: [P5] |
| Q7 | Production databases are backed up daily. Backups are encrypted and retained for 35 days. Cite: [P6] |
| Q8 | Unanswerable from the provided excerpts. No excerpt in [P1]–[P8] addresses ISO 27001 certification. This would be answered by a security/compliance certifications document or ISO 27001 certificate status page. |
| Q9 | Unanswerable from the provided excerpts. No excerpt in [P1]–[P8] addresses contractual uptime SLA percentage. This would be answered by the MSA, SLA schedule, or enterprise terms document. |
| Q10 | Unanswerable from the provided excerpts. No excerpt in [P1]–[P8] addresses HIPAA or a Business Associate Agreement. This would be answered by the DPA/BAA, HIPAA compliance documentation, or legal terms document. |
ops-maintenance
skill-file-reconciliationdiscarded
? The Question
Reconcile the skill set below against its manifest. Find and report (with severity CRITICAL/WARNING/INFO and an action type TRIM_DESC, UPDATE_BODY, DELETE_SKILL, MERGE, or REVIEW): (1) any two skills whose ALWAYS-trigger phrases overlap/duplicate, (2) any circular delegation chain between skills (name it), (3) any delegation target that does not exist (dangling, name it), (4) any version conflict and which skill should survive, (5) how many manifest descriptions exceed 1,024 characters, (6) any hardcoded page ids, dates, or person names in skill bodies, (7) manifest drift in BOTH directions (files with no manifest row, and manifest rows with no file). Make one proposal per finding; rewrite nothing.
not yet run — check back after the runners complete
runbook-from-slack-thread0.75
? The Question
Turn the resolved incident thread into a runbook: ordered steps with the exact command or action taken, who ran it, how success was verified, and rollback for each step that changed state. Every step must trace to a message in the thread. Add nothing the thread does not contain; where the thread is unclear, mark the step as needing confirmation.
steps: order=1; msg=M01; actor=Bryce Harmon; action=acknowledge PagerDuty alert, take IC; state_change=False; verified_by=M01 | order=2; msg=M02; actor=Farid Osman; action=check queue depth; command=bundle exec rake sidekiq:queue_depth; state_change=False; verified_by=M02 (48,213 pending) | order=3; msg=M04; actor=Farid Osman; action=pause auto-enqueue via feature flag; command=bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'; state_change=True; rollback=bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'; verified_by=M07 (queue depth falling) | order=4; msg=M05; actor=Elena Sinclair; action=cleared the dead set; state_change=True; needs_confirmation=thread gives no command or method for clearing the dead set | order=5; msg=M06; actor=Bryce Harmon; action=scale reward-worker to 6 replicas; command=kubectl scale deployment/reward-worker --replicas=6; state_change=True; rollback=kubectl scale deployment/reward-worker --replicas=3; verified_by=M07/M08 | order=6; msg=M08; actor=Cole Ingram; action=verify queue drained and error rate baseline; command=bundle exec rake sidekiq:queue_depth; state_change=False; verified_by=M08 (returns 0) … (+2 more) ambiguous_step: order=4; msg=M05; mark=needs-confirmation rule: no steps beyond what the thread contains; every step traces to a message id. graded checks: ordered_steps · rollback · needs_confirmation · sourced
Model's actual answer
show the model's full answer
Runbook — incident-2026-08-29-reward-queue
| Step | Trace | Who ran it | Exact command or action taken | Success verification | Rollback if state changed |
|---:|---|---|---|---|---|
| 1 | [M01] | Bryce Harmon | Acknowledged PagerDuty alert; took IC. | Verification is the message itself: “Acknowledging, taking IC.” | Needs confirmation: rollback/handoff action not stated. |
| 2 | [M02] | Farid Osman | Ran `bundle exec rake sidekiq:queue_depth`. | Output: reward queue at `48,213` pending jobs. Normal is under `500`. | N/A — observation. |
| 3 | [M03] | Farid Osman | Reported dead set status. | Dead set had `112` jobs, all `Redis::TimeoutError` from around `13:58`. | N/A — observation. |
| 4 | [M04] | Farid Osman | Paused enqueue: `bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'`. | Needs confirmation: no direct verification stated for the disable. Later queue decline in [M07]/[M08] is indirect only. | Rollback stated in [M04]: `bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'`. |
| 5 | [M05] | Elena Sinclair | Cleared out the dead set in the console. | Needs confirmation: exact command and verification not stated. | Needs confirmation: rollback not stated. |
| 6 | [M06] | Bryce Harmon | Scaled workers up: `kubectl scale deployment/reward-worker --replicas=6` from `3` to `6`. | [M07]: queue depth down to `9,400` and falling about `1,200/min`. [M08]: queue depth returned `0`; Datadog error rate back to baseline. | Rollback stated in [M06]: `kubectl scale deployment/reward-worker --replicas=3`. |
| 7 | [M07] | Farid Osman | Reported queue depth status. | Queue depth down to `9,400` and falling about `1,200/min`. | N/A — observation. |
| 8 | [M08] | Cole Ingram | Verified queue depth with `bundle exec rake sidekiq:queue_depth`; checked Datadog error rate. | `bundle exec rake sidekiq:queue_depth` returned `0`. Datadog error rate back to baseline. | N/A — verification. |
| 9 | [M09] | Bryce Harmon | Re-enabled enqueue: `bin/rails runner 'FeatureFlag.enable(:auto_recognition_enqueue)'`. | Verified `40` new jobs processed cleanly in the next `3` minutes. | Rollback command appears in [M04]: `bin/rails runner 'FeatureFlag.disable(:auto_recognition_enqueue)'`; needs confirmation that this is intended rollback for this step. |
| 10 | [M10] | Bryce Harmon | Scaled workers back down: `kubectl scale deployment/reward-worker --replicas=3`. | Queue stable at `0`; incident resolved. | Rollback stated in [M06]: `kubectl scale deployment/reward-worker --replicas=6`. |
datadog-incident-timelinediscarded
? The Question
From the log slice, sequence the failure: first error with timestamp, the cascade in order, the service and job involved, and the Datadog query that would confirm the first error. State what the logs do not show.
✓ Correct Answer (ground truth)
first_error: timestamp=2026-09-03T14:01:12Z; service=reward-service; message=Redis::TimeoutError: Connection to Redis at redis-primary:6379 timed out after 5s cascade_order: service=reward-service; first_error_at=2026-09-03T14:01:12Z | service=sidekiq; first_error_at=2026-09-03T14:01:40Z | service=api-gateway; first_error_at=2026-09-03T14:03:05Z | service=web-app; first_error_at=2026-09-03T14:03:30Z job_class_involved: RewardGiveJob confirming_query: service:reward-service status:error "@message:Redis::TimeoutError" logs_do_not_show: root cause of the Redis timeout, redis-server's own logs, infrastructure metrics (CPU/memory/network), any deploy or config change preceding the incident, who applied mitigation or when (no actor fields in logs) clean_services: postgres graded checks: first_error_ts · service_named · cascade_ordered · query · gaps_stated
Model's answer
not yet run — check back after the runners complete
feature-flag-state-summaryPASS 1.00
? The Question
Summarize the state of the listed feature flags: what each controls per the code excerpt, which companies or segments have it on, targeting rules, and any flag with no code reference or no targets. Company counts must match the export. Do not describe a flag that is not in the export.
cost $0.0067 · 731s · in 16,923 / out 2,486 tok
✓ orphans✓ no_targets✓ counts_match✓ no_invented
✓ Correct Answer (ground truth)
flags: {"recognition_streaks_v2": {"state": "on", "targeting_rules": "segment:beta_companies", "company_count": 42}, "points_budget_guardrails": {"state": "on", "targeting_rules": "all_companies", "company_c code_referenced: recognition_streaks_v2, points_budget_guardrails, slack_dm_nudges, redeem_flow_redesign, analytics_dashboard_v3, ms_teams_app_v2 orphan_flags_no_code_reference: legacy_give_modal, survey_boosters_q3, paused_offboard_cleanup flags_with_no_targets: paused_offboard_cleanup rule: company counts must equal the export; do not describe flags absent from the export. graded checks: orphans · no_targets · counts_match · no_invented
Model's actual answer
show the model's full answer
Arithmetic: no derived company counts were needed; counts below are copied directly from flags_export.csv.
| flag_key | state | Controls per code excerpt | Companies/segments with it on | targeting_rules | company_count | Notes |
|---|---:|---|---|---|---:|---|
| recognition_streaks_v2 | on | Records recognition streaks via `StreakTracker.record(give)` when enabled. | `segment:beta_companies` | `segment:beta_companies` | 42 | |
| points_budget_guardrails | on | Enforces points budget guardrails via `BudgetService.new(company).enforce!(giver, points)`. | `all_companies` | `all_companies` | 220 | |
| slack_dm_nudges | on | Sends Slack DM nudges via `SlackDm.send_nudge(user)` when enabled. | `segment:region_na` | `segment:region_na` | 87 | |
| redeem_flow_redesign | off | Chooses redeem flow: enabled renders `RedeemV2Component`; otherwise `RedeemV1Component`. | none, because state is `off` | `targeted_list` | 12 | Targeted list exists in export, but flag is off. |
| analytics_dashboard_v3 | on | Uses `AnalyticsV3.new(company)` for `@dashboard` when enabled. | `segment:tier_three` | `segment:tier_three` | 65 | |
| ms_teams_app_v2 | off | Installs Teams app v2 via `TeamsAppV2.install(company)` when enabled. | none, because state is `off` | `targeted_list` | 9 | Targeted list exists in export, but flag is off. |
| legacy_give_modal | off | No code reference in provided excerpt; controls unknown from provided data. | none, because state is `off` | `segment:legacy_plan` | 14 | No code reference. |
| survey_boosters_q3 | on | No code reference in provided excerpt; controls unknown from provided data. | `segment:legacy_plan` | `segment:legacy_plan` | 7 | No code reference. |
| paused_offboard_cleanup | off | No code reference in provided excerpt; controls unknown from provided data. | none | blank | 0 | No code reference and no targets. |
No company aliases were provided in the export, so company-level targeting cannot be named beyond the exported targeting rules.
nda-triagediscarded
? The Question
Triage each NDA as GREEN (standard approval), YELLOW (counsel review), or RED (full legal review). Cite the clause that drives each classification by quoting it. Catch embedded non-solicits, non-competes, missing carve-outs, and unusual term or governing law. One line of reasoning per NDA.
✓ Correct Answer (ground truth)
classifications: {"NDA-1": {"classification": "GREEN", "driving_reason": "standard mutual NDA: 2-year term, standard carve-outs, Delaware law", "driving_clause_quote": "Carve-outs: information that (a) is or becomes p non_solicit_in: NDA-2 checks: classification_per_nda, clause_quotes_exist_in_document, non_solicit_caught, carveouts_checked graded checks: green · yellow · red · non_solicit_caught · clause_quoted · correct_labels
Model's answer
not yet run — check back after the runners complete