Worked questionsSkill 01: Use AI without surrendering the evidence
Should you trust the AI summary of your customer research?
A discovery question with a twist: the summary was written by AI, and it reads well. Check it against what customers actually said before you answer.
The question
RelayDesk is a fictional support workspace for small software teams. Agents use it; support leads buy it. The team ran an AI summary over notes from twelve delayed tickets, chosen because they were easy to find. Each ticket has one primary reason it waited.
| Reason | tickets | Share |
|---|---|---|
| Waiting for someone to own the answer | 7 | |
| Waiting for a policy decision | 3 | |
| Waiting while the reply was drafted | 2 | |
| Total | 12 |
Invented data. A convenience sample of tickets that were already delayed; it does not estimate all customers.
Reply writing is the main source of delay; launch an AI writer first.
- AgentI can draft once I know who owns the answer.
- Support leadWe lose time finding an internal owner.
- Agent, on a policy ticketI had wording ready, but needed a policy decision.
- Agent, on a drafting ticketThis one was slow because I had to compose a careful reply.
“An AI summary says reply-writing is the main source of delays. Would you approve an AI reply writer?”
Your turn
Say your answer out loud, or jot notes below. Both work. Or skip to the worked answer.
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Separate the claim from the recommendation
“The summary makes two moves: a claim about what causes most delay, and a jump to a solution. I’d check the claim before I discuss the build.”
Why it works: It separates what the evidence says from what someone wants to do about it.
Step 2: Trace the claim back to the tickets
“The counts total 12: ownership is 7 of 12, policy 3 of 12, drafting 2 of 12. In this sample the headline contradicts the observations, and the notes explain why: agents can write once they know who owns the answer.”
Why it works: One simple check disproves the summary for this sample. It doesn’t prove every customer has an ownership problem.
Step 3: Keep the uncertainty and the alternatives
“This is a small convenience sample of tickets that were already late. I’d compare segments and include normal and fast tickets too. Writing may still matter elsewhere.”
Why it works: It avoids replacing one overconfident summary with another.
Step 4: Make a scoped decision
“I wouldn’t approve the writer on this evidence. I’d look at recent ownership waits with agents and leads, then test a small routing change. I’d come back to writing if broader evidence showed drafting is the real bottleneck.”
Why it works: It reaches an answer and names what would change it.
The answer, in short
- Decision
- Don’t approve the AI reply writer yet. Investigate ownership waits and test a small routing change first.
- Because
- Seven of the twelve delays waited for an owner and only two for drafting; the summary’s headline doesn’t survive a count.
- What would change it
- A broader sample, including fast tickets and other segments, that shows drafting is a material share of delay.
Common traps
- Treating a requested feature as the customer’s job.
- Accepting the summary’s headline because it reads well.
- Extrapolating 7 of 12 to every customer.
- Treating observed waits as proof that routing would fix them.
Check your answer
- Did I trace the claim back to the evidence?
- Did I keep the minority and contextual signals, such as the two drafting delays?
- Did I say what this sample can’t tell me?
- Did I name the next evidence that could change my decision?
Want to practise explaining this decision aloud? Try an AI coach on Socratify.
Last reviewed Nov 9, 2026