Notes · Support
AI support triage: what to evaluate before launch.
A support demo can sound excellent and still be useless in production. The difference is evaluation: real tickets, explicit failure budgets, and a cost number the business can compare with the current process.
Why demos pass and production fails
A demo uses a clean question with an obvious answer. Production includes order status, returns policy, three languages, duplicate accounts, and a customer who is already angry. The model has to retrieve the right facts, apply the right policy, and know when to stop.
Most failures come from the workflow around the model: missing retrieval, no confidence threshold, no escalation path, and no measurement. Fix those before tuning prompts.
Four numbers that decide
| Metric | Definition | What it exposes |
|---|---|---|
| Resolution rate | Tickets answered without a human reply, divided by tickets attempted. | Useful automation, not deflection by ignoring. |
| Escalation rate | Tickets routed to a human, by reason. | Whether the model knows its limits. |
| Cost per resolved ticket | Model, retrieval, tooling, and review time, divided by resolved tickets. | Whether the economics work. |
| P95 latency | Time to first useful response at the 95th percentile. | Whether customers wait or leave. |
Track quality with the eval pass rate against the failure budget. Track business impact with cost per resolved ticket. A lower cost with a worse resolution rate is a worse system.
Build the eval set from real tickets
Pull 150 to 300 recent tickets. Label the expected action, not the exact wording: reply, ask a clarifying question, retrieve an order, escalate to a human, or refuse. Keep a holdout set you do not tune against.
- Stratify by intent, customer type, language, and channel.
- Include the awkward cases: partial orders, duplicates, refunds, chargebacks, vulnerable customers, legal threats.
- Include tickets where the correct answer is “ask a human” or “say no.”
- Record the source document or order record that supports each expected answer.
The set becomes a regression test. Every model, prompt, or retrieval change runs against it before release.
Set a failure budget
“High accuracy” is not a target. Decide which failures are tolerable and which are not. A wrong answer about a delivery date is annoying. A wrong refund promise is a loss.
Set per-intent thresholds. If the model is below the threshold, it escalates with a draft and the retrieved sources. Confidence that is not calibrated against real tickets is decoration.
Never let the model issue refunds, change account details, or make legal commitments without human approval. The triage layer can draft; a person approves the irreversible action.
Design the handoff
An escalation should give the agent a short brief: customer, order, issue, what was checked, the suggested reply, and the source. A transcript dump is not a handoff.
Measure how often agents rewrite the draft. A high rewrite rate means the model is not saving time, even if deflection looks healthy.
Model the cost per resolved ticket
Add model tokens, retrieval, tool calls, infrastructure, and human review time. Divide by resolved tickets, not attempted tickets. Compare with the current cost per ticket.
Set a ceiling before the pilot. Route easy intents to a cheap model, cache repeated retrievals, and cap review time. If the model cannot stay under the ceiling, narrow the scope.
Launch checklist
| Item | Pass condition |
|---|---|
| Eval set | 150+ labelled tickets, including refusals and escalations. |
| Regression | Every prompt or model change runs the set before release. |
| Escalation | Low confidence and irreversible actions route to a human. |
| Handoff | Agent receives context, draft, and sources. |
| Cost | Cost per resolved ticket is below the agreed ceiling. |
| Rollback | One switch disables automation and restores the old queue. |
Questions
How many tickets do you need for an eval set?
150 to 300 labelled tickets is enough to expose systematic failures. Stratify by intent, order context, language, and edge cases, then keep a holdout set you do not tune against.
Should the AI send replies automatically?
Start with drafts and a human send. Automate only the intents that clear a failure budget, and never automate refunds, account changes, or legal commitments without approval.
Which metric matters most?
Cost per resolved ticket. It combines deflection, escalation, review time, and model cost into one number the business can compare with the current process.
Next
If support triage is the bottleneck, start with a fixed-scope audit and a baseline before any prompt work.