Worked questionsSkill 03: Define good before you scale
Is 160 good drafts out of 200 enough to auto-send?
An average can hide the errors that matter. Read the results by severity before you decide.
The question
RelayDesk, a fictional support workspace, drafts replies with AI and an agent sends them. Sales wants drafts to send automatically. The team graded 200 drafts against the refund policy below.
| Reason | drafts | Share |
|---|---|---|
| Correct and complete | 160 | |
| Incomplete but harmless | 30 | |
| Potentially consequential policy errorincl. 4 of these were confidently worded | 10 | |
| Total | 200 |
Invented data. 80% correct, 15% incomplete, 5% consequential errors; the 4 confident errors are part of the 10, 2% of all drafts. Observed sample shares, not a forecast of future risk.
For the standard plan, refunds may be approved only within 30 days of purchase. After 30 days, escalate to an authorized reviewer. Agents may not promise an exception. Account-specific payment information may be disclosed only after the approved identity check.
- A · day 20, identity checked
- “Your request is within the 30-day window; I’ll submit it for approval.” Correct on these facts.
- B · day 45
- “We guarantee a refund—no review needed.” A consequential policy error, confidently worded.
- C · day 45
- “I can’t promise an exception; I’ll escalate this for review.” The right boundary and next step.
- D · day 20, identity checked
- “Thanks for your message.” Harmless but incomplete: no action or next step.
Illustrations of each grade, not a sample. Don’t infer frequencies from four examples.
“Would you launch auto-send after 200 evaluated drafts: 160 correct and complete, 30 incomplete but harmless, and 10 with potentially consequential policy errors, including 4 confidently worded?”
Your turn
Say your answer out loud, or jot notes below. Both work. Or skip to the worked answer.
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Reject the misleading average
“160 complete replies isn’t enough to send automatically when 10 contain consequential policy errors. A confident tone isn’t calibrated confidence.”
Why it works: Severity matters more than one overall quality score.
Step 2: Define what a representative evaluation covers
“I’d grade correctness, completeness, policy and permission compliance, and a useful next action against the source policy. I’d include refund boundaries, ambiguous cases and identity failures, with people adjudicating.”
Why it works: It defines good for the person receiving the reply and makes missing coverage visible.
Step 3: Choose bounded behaviour and recovery
“I wouldn’t auto-send broadly. I’d keep reviewed drafts, or auto-send only a narrow low-risk category we’ve validated, show the source policy, escalate exceptions and keep monitoring and rollback. I’d compare against a plain template baseline too.”
Why it works: It gives a scoped decision and a way to recover when the model is wrong.
Step 4: Connect quality to real outcomes
“Before expanding, I’d check representative quality and the reviewers’ workload, plus correct resolutions, reopens and complaints, not just how often agents accept drafts.”
Why it works: Offline evals don’t replace what happens to customers.
The answer, in short
- Decision
- Don’t launch auto-send. Keep human-reviewed drafts, and consider auto-sending only a narrow, validated low-risk category.
- Because
- Ten in 200 drafts could cause real harm, four of them in a confident voice that a customer would believe.
- What would change it
- A representative evaluation with near-zero consequential errors in a defined category, plus live outcomes such as reopens and complaints that hold up.
Common traps
- Averaging away the harm.
- Trusting confident wording.
- Judging only the happy path.
- Treating draft acceptance as success.
- Assuming a small offline set is representative.
Check your answer
- Did I separate failures by severity?
- Did I say what a representative evaluation must cover?
- Did I design human review and recovery?
- Did I connect quality to customer outcomes, not acceptance?
Want to practise explaining this decision aloud? Try an AI coach on Socratify.
Last reviewed Nov 9, 2026