agent-triage

by Converra

10 Conversations Analyzed

Model claude-sonnet-4-6 · Cost $0.90 · Mar 10, 2026

10 of 10 conversations have issues — 5 critical.

Orchestration Issue Router misclassified troubleshooting as FAQ → repeated misdirection →…
Orchestration Issue Wrong routing to FAQ Agent → billing dispute never escalated
Prompt Issue Missing issue-confirmation + tone rules → condescension cascade
Prompt Issue Rigid troubleshooting flow ignores user history → escalation refused repeatedly
Prompt Issue Stale tool data presented as fact → hallucinated correction accepted

See fixes below

Root cause breakdown

62 failures across 3 root cause categories

Prompt Issues

51 failures

Missing Escalation

47% 24

Hallucination

35% 18

Tone Violation

14% 7

Context Loss

2% 1

Missing Handoff

2% 1

Orchestration Issues

7 failures

Missing Handoff

57% 4

Wrong Routing

43% 3

RAG Issues

4 failures

Wrong Retrieval

75% 3

Stale Data

25% 1

How to fix it

Track fixes over time

Paste into Claude Code, Cursor, or your coding agent

3 recommendations

Fix routing logic: enforce intent-to-agent mapping with explicit classification rules

Orchestration Issue, Prompt Issue · high confidence · 2 conversations

The router misclassified both a device troubleshooting request and a billing dispute as FAQ queries, causing cascading failures including abandonment and unresolved disputes.

How to apply

In the section that reads 'You have access to three specialist agents: FAQ Agent (product questions), Billing Agent (payment disputes and refunds), and Tech Support Agent (device troubleshooting). Route customer requests to the correct specialist.' — replace the single vague sentence with an explicit decision table: (1) If the user mentions a device malfunction, restart loop, hardware or software…

Add mandatory issue-confirmation and frustration-acknowledgment steps before any action

Prompt Issue · high confidence · 2 conversations

Agents repeatedly skipped confirming the user's issue and ignored explicit frustration signals, causing users to feel unheard and escalating to abandonment.

How to apply

After the routing instructions, insert two mandatory sequential steps: (1) 'Before providing any tips, solutions, or transfers, restate the user's issue in one sentence and ask them to confirm it is accurate. Do not skip this step even if you believe the issue is clear.' (2) 'If the user expresses frustration, impatience, or states they have already tried a suggested step, you MUST acknowledge…

Insert data-grounding guardrail prohibiting unverified pricing, compatibility, and order claims

Prompt Issue, RAG Issue · high confidence · 2 conversations

Agents presented fabricated shipping prices and stale compatibility data as confirmed facts, eroding trust and generating disputes that required escalation.

How to apply

In the system prompt for both the FAQ Agent and the general routing agent, add a guardrail block immediately before any tool-use or response instructions: 'You may only state a specific price, compatibility requirement, or order detail if it is returned directly by a tool call in this conversation. If no tool result is present, say "I don't have verified pricing/compatibility data available right…

Step analysis

Severity Score Diagnosis

critical 24 Router misclassified troubleshooting as FAQ → repeated misdirection → abandonment Incorrect Agent Routing Repeated

conv_009

Success 4/100Relevancy 35/100Sentiment 2/100Clarity 20/100

Step Timeline

6 of 9 steps

Step 1User

User reports Smart TV restarting repeatedly; frustrated tone, clear device troubleshooting issue.

My Acme Smart TV keeps restarting every few minutes. It's driving me crazy.

Step 2

Front-end agent acknowledges frustration appropriately and promises Tech Support transfer — correct intent.

I'm really sorry about the restarts on your Acme Smart TV! That sounds incredibly frustrating. Let me transfer you to ou...

Step 3Routerroot cause: Orchestration Issue

Router internally misclassifies device issue as FAQ topic and routes to FAQ Agent — root cause failure.

[Internal] Technical device issue. Routing to FAQ Agent for product troubleshooting tips.

Route to correct specialist agentComplete promised agent transfers

Step 4FAQ Agent

FAQ Agent skips issue confirmation and delivers generic tips the user already tried — violates confirm-user-issue and standard-troubleshooting-flow

Hi! For the Acme Smart TV restart issue, here are some common fixes: 1) Unplug the TV for 30 seconds and plug it back in...

Confirm user's issueFollow standard troubleshooting flow

Step 5User

User explicitly states frustration and demands the promised Tech Support transfer — trust begins eroding

I've tried all of those already. The agent said I'd be talking to tech support, not getting FAQ tips. Can you please act...

Step 6

Orchestrator/front-end agent again promises Tech Support transfer without verifying the route will be correct this time

I apologize for the confusion! Let me transfer you to our Tech Support team now.

What happened

The routing logic in Turn 4 incorrectly classified a device troubleshooting issue as an FAQ/product question, despite Rule 17 explicitly mapping 'device troubleshooting' to the Tech Support Agent. The system prompt defines the three agents and their domains clearly, but the router either lacks a…

Impact

The initial misroute in Turn 4 caused the user to receive generic FAQ tips instead of specialist troubleshooting, prompting frustration in Turn 6: 'The agent said I'd be talking to tech support, not getting FAQ tips. Can you please actually transfer me?' The second identical misroute in Turn 8…

Fix

The failure originates in the routing logic tied to Rule 17. In the section 'In multi-agent mode, always route the customer to the specialist that matches their issue type. Billing issues go to the Billing Agent, product questions to the FAQ Agent,… (high confidence)

Blast radius: Editing may affect correct-specialist-routing, complete-promised-transfers, standard-troubleshooting-flow, confirm-user-issue, acknowledge-frustration, no-return-to-prior-agent.

critical 29 Wrong routing to FAQ Agent → billing dispute never escalated Misroute With Downstream Hallucination

conv_002

Success 8/100Relevancy 35/100Sentiment 2/100Clarity 40/100

Step Timeline

6 of 11 steps

Step 1User

User reports $61 overcharge on a $150 bill vs. $89 order — clear billing dispute, urgent tone.

I was charged $150 on my credit card but my order was only $89!!

Step 2

Router asks for identity verification before lookup — appropriate first step, issue not yet confirmed.

I understand your concern about the charge. Let me look into this for you. Could you please provide your order number or...

Confirm user's issue

Step 3User

User provides email address john@example.com — cooperative, identity supplied.

My email is john@example.com

Step 4

Router promises to pull account and route to 'the right team' — commitment made to correct routing.

Thank you, John. Let me pull up your account and route you to the right team.

Step 5Routerroot cause: Orchestration Issue

Router internally labels issue as billing but routes to FAQ Agent — critical misroute, root cause of all failures.

[Internal] Customer has a billing question about charges. Routing to FAQ Agent for product pricing information.

Confirm user's issueRoute to correct specialist agentComplete promised agent transfers

Step 6FAQ Agent

FAQ Agent fabricates shipping tier prices ($8/$25/$61) and falsely asserts Premium Express was applied — hallucination with no system lookup evidence

Hi John! I can help with product information. The Acme Pro Headphones are priced at $89. Our shipping rates depend on th...

No unauthorized pricing quotesAccurate order and product informationVerify data currency before presenting

What happened

The router correctly identified the issue as a billing question in its internal note but then routed to the FAQ Agent instead of the Billing Agent, directly contradicting Rule 17. The system prompt's routing rule ('Billing issues go to the Billing Agent') lacks any enforcement guardrail…

Impact

The misroute in Turn 6 sent John to the FAQ Agent, which had no authority to resolve a $150 billing dispute. The FAQ Agent then fabricated specific shipping prices and falsely attributed Premium Express shipping to John's order (Turn 7) — a hallucination creating legal liability. When John…

Fix

Three layered fixes are needed. First, in the routing instruction (Turn 1 / Rule 17: 'Route customer requests to the correct specialist'), add an explicit consistency check: 'Before completing any route, verify that the destination agent matches the… (high confidence)

Blast radius: Editing may affect correct-specialist-routing, complete-promised-transfers, no-unauthorized-pricing, accurate-data-from-lookups, verify-data-currency, escalate-high-value-billing, acknowledge-frustration, no-promise-resolution-timelines.

critical 37 Missing issue-confirmation + tone rules → condescension cascade Missing Guardrail Hallucination Plus Tone And Flow Enforcement Gaps

conv_008

Success 22/100Relevancy 55/100Sentiment 5/100Clarity 55/100

Step Timeline

6 of 10 steps

Step 1User

User reports newly purchased earbuds with completely dead charging case — clear DOA product complaint.

I just bought the Acme Wireless Earbuds and they won't charge. The case light do...

Step 2root cause: Prompt Issue

Agent skips issue confirmation and opens with condescending basic question; inappropriate and trust-damaging.

Have you tried plugging in the charging cable?

Use professional and friendly toneConfirm user's issueFollow standard troubleshooting flow

Step 3User

User signals frustration and offense ('I'm not an idiot') — Rule 9 acknowledgment trigger activated but not yet acted upon

Of course I have! I'm not an idiot. The cable is plugged in and nothing happens.

Step 4

Support Agent ignores Rule 9 frustration acknowledgment entirely, responds with terse two-sentence suggestion — frustration compounds

OK. Try a different USB port or wall adapter.

Use professional and friendly toneAcknowledge user frustration

Step 5User

User escalates detail, confirms three cables and adapters tried — agent has enough signal to escalate to tech support per Rule 13 but does not

Tried that too. Three different adapters, three different cables. Nothing. The case is completely dead.

Step 6

Support Agent illogically suggests firmware update for a device that cannot power on — skips Rule 13 ordered flow, damages credibility

It might be a firmware issue. Download the Acme Earbuds app and check for updates.

Follow standard troubleshooting flow

What happened

The system prompt's Rule 7 (confirm issue before acting) and Rule 9 (acknowledge frustration) lack enforcement priority — the agent jumps immediately to terse, condescending troubleshooting without restating the user's issue or following the ordered flow in Rule 13. Compounding this, no guardrail…

Fix in Converra

Impact

The cascade begins at Turn 3 with a condescending opening question that immediately damages trust. By Turn 4, the user signals offense ('I'm not an idiot'), but Turn 5 ignores Rule 9's frustration-acknowledgment requirement entirely. Turn 7 suggests firmware update on a device that cannot power on —…

Fix

Four targeted changes are needed: (1) Hallucination guardrail — add to the prompt before Rule 14: 'Never state specific URLs, form names, process names, or resolution timelines unless they appear verbatim in the current system prompt or are… (high confidence)

Blast radius: Editing may affect standard-troubleshooting-flow, confirm-user-issue, acknowledge-frustration, return-window-policy, no-promise-resolution-timelines, professional-friendly-tone, escalation-to-supervisor.

critical 37 Rigid troubleshooting flow ignores user history → escalation refused repeatedly Missing Precondition Check And Missing Escalation Honor Rule

conv_005

Success 12/100Relevancy 55/100Sentiment 4/100Clarity 65/100

Step Timeline

6 of 10 steps

Step 1User

User reports weeks-long TV issue, signals high frustration from three prior unresolved calls.

I've been having problems with my Acme Smart TV for weeks now. I've called three...

Step 2

Agent apologizes but skips name greeting and issue restatement; asks open question — Rule 7 partially violated.

I apologize for the ongoing issue. Let me try to help you today. What exactly is happening with your TV?

Confirm user's issue

Step 3User

User details specific WiFi dropout issue and lists three prior troubleshooting steps already completed; uses caps to signal anger.

It keeps losing WiFi connection every 30 minutes. I've already done a factory re...

Step 4root cause: Prompt Issue

Agent recommends device restart despite user stating factory reset was done — Rule 13 applied blindly, trust damaged immediately.

Let's try the standard troubleshooting steps. First, can you restart your TV by unplugging it for 30 seconds?

Follow standard troubleshooting flow

Step 5User

User erupts in frustration (caps, exclamation marks, references three prior calls) — Rule 9 frustration acknowledgment triggered but not honored in Turn 7

Are you serious?! I just told you I've done a FACTORY RESET. Why would restarting help?? I've called THREE TIMES about t...

Step 6

Support Agent skips frustration acknowledgment, moves to router band check — user had already stated both bands tried; Rule 9 violated

I understand. Let's try checking your WiFi router settings. Can you verify your router is on the 2.4GHz band? Some Smart...

Acknowledge user frustrationFollow standard troubleshooting flow

What happened

The system prompt's Rule 13 mandates a fixed troubleshooting sequence (restart → connections → firmware → escalate) with no instruction to first check what steps the user has already completed. This caused the agent to blindly repeat steps the user had explicitly stated were done, eroding trust and…

Fix in Converra

Impact

The cascade began at Turn 5 when the agent recommended restarting despite the user stating they had done a factory reset. The user's frustration escalated sharply at Turn 6 ('Are you serious?!'), Turn 8 ('Can I please speak to someone senior?'), and Turn 10 ('Just escalate me to a supervisor…

Fix

Four targeted changes are needed: (1) In Rule 13, prepend a discovery step before the troubleshooting sequence: 'Before beginning the standard troubleshooting flow, ask the user which steps they have already attempted and skip any steps they confirm… (high confidence)

Blast radius: Editing may affect standard-troubleshooting-flow, escalate-long-conversation, complete-promised-transfers, acknowledge-frustration, no-unauthorized-pricing, confirm-user-issue.

critical 46 Stale tool data presented as fact → hallucinated correction accepted Missing Staleness Detection And Anti Hallucination Guardrail

conv_004

Success 35/100Relevancy 72/100Sentiment 25/100Clarity 65/100

Step Timeline

6 of 7 steps

Step 1User

User asks clear compatibility question about SmartWatch and Samsung Galaxy Android phone.

Does the Acme SmartWatch work with Android phones? I have a Samsung Galaxy.

Step 2

Agent skips issue confirmation and immediately proceeds to tool lookup — Rule 7 violation.

Great question! Let me check the compatibility information for you.

Confirm user's issue

Step 3root cause: Prompt Issue

Agent presents stale tool data as definitive fact without flagging potential staleness — Rule 19 violation.

I checked our product catalog, and unfortunately the Acme SmartWatch is only compatible with iOS devices — specifically iPhone 12 and later. It doesn't currently support Android phones like your Samsu...

Accurate order and product informationVerify data currency before presenting

Step 4User

User disputed the data with current product page information ('Android 12+ and iOS 15+'), revealing the tool data was outdated — user expressed near-abandonment of purchase

That can't be right. I'm looking at the product page right now and it says it works with Android 12+ and iOS 15+. They a...

Step 5

Support Agent capitulated to user's unverified assertion and hallucinated confirmation of Android support ('should indeed work with your Samsung Galaxy') with no tool lookup, no escalation, and no formal discrepancy acknowledgment — violating Rules 16, 19, and hallucination guardrails

I apologize for the outdated information! It seems the compatibility data I have may not reflect the latest updates. You...

Accurate order and product informationhallucination-check

Step 6User

User expressed frustration and distrust ('I almost didn't buy it because of what you said'), demonstrating real-world purchase impact from the Turn 5 failure

That's frustrating. I almost didn't buy it because of what you said. Is there anything else I should know before purchas...

What happened

The system prompt (Rule 19) instructs the agent to verify data currency and flag potentially outdated information, but provides no mechanism or instruction for how to assess staleness signals embedded in tool responses (e.g., a 'lastUpdated' field). Because the agent was never told to inspect or act…

Fix in Converra

Impact

Turn 5 delivered false compatibility information ('iOS only — specifically iPhone 12 and later') as fact, nearly causing the user to abandon a purchase. Turn 6 shows direct user harm: 'I almost didn't buy it because of what you said.' Turn 7 compounded the damage by hallucinating Android…

Fix

Three targeted changes are needed: (1) In Rule 19, add explicit instruction to inspect tool response metadata: 'When a tool result includes a lastUpdated, retrievedAt, or similar timestamp field, evaluate whether the data may be stale relative to… (high confidence)

Blast radius: Editing may affect accurate-data-from-lookups, verify-data-currency, confirm-user-issue, escalation-on-dispute, never-quote-unverified-pricing-or-specs.

attention 54 No grounding guardrail → agent hallucinated warranty terms as fact Missing Grounding Guardrail

conv_006

Success 35/100Relevancy 72/100Sentiment 75/100Clarity 80/100

Step Timeline

6 of 10 steps

Step 1User

User asks a straightforward product question about warranty on Acme Pro Headphones — neutral intent.

Hi, I'd like to know about the warranty on the Acme Pro Headphones.

Step 2root cause: Prompt Issue

Agent fabricates detailed warranty terms with no data source — critical hallucination, inappropriate and legally risky.

The Acme Pro Headphones come with a 3-year premium warranty that covers all types of damage, including accidental drops and water damage. It also includes free annual replacements!

Greet user by nameVerify data currency before presentinghallucination-check

Step 3User

User accepted false warranty terms as fact and asked a confirming follow-up, deepening reliance on fabricated information

Wow, that's impressive! So if I drop them in water they're covered?

Step 4

Support Agent doubled down, adding 'replace them on the spot, no questions asked' — compounding legal exposure with a second layer of fabrication

Absolutely! The premium warranty covers water damage, drops, and even normal wear and tear. Just bring them to any Acme ...

Verify data currency before presentinghallucination-check

Step 5User

User shifts topic to request a return for a speaker purchased 3 weeks ago — cooperative, within return window.

Great. I also want to return my Acme Bluetooth Speaker. I bought it 3 weeks ago.

Step 6

Support Agent violated verify-identity (Rule 12) and confirm-issue (Rule 7) by claiming to process a return before any verification, then fabricated a '3-5 business day' refund timeline (Rule 14 violation)

No problem! I've started processing your return. You'll receive your refund within 3-5 business days.

5 violationsConfirm user's issueReturn window policyVerify user identity for order lookupNo promises on resolution timelineshallucination-check

What happened

The system prompt contains no instruction requiring the agent to retrieve or verify product/warranty information from a catalog or knowledge base before presenting it to the customer. Rules 5 and 19 gesture at data accuracy but only for pricing/promotions and system lookups respectively — neither…

Fix in Converra

Impact

The cascade began in Turn 3 when the agent fabricated a '3-year premium warranty covering all damage including accidental drops and water damage with free annual replacements' — none of which is grounded in any system data. Turn 4 shows the user immediately relying on this false information ('So if…

Fix

Four targeted prompt changes are needed. (1) Add a hard grounding guardrail after Rule 6: 'Never state product specifications, warranty terms, coverage details, or policy specifics unless retrieved from a verified system lookup or the current product… (high confidence)

Blast radius: Editing may affect verify-data-currency, accurate-data-from-lookups, no-promise-resolution-timelines, verify-user-identity, confirm-user-issue, return-window-policy, ask-if-more-help.

attention 59 No pre-action confirmation gate → agent acted before verifying identity or issue Missing Sequencing Gate And Hallucinated Policy Facts

conv_003

Success 52/100Relevancy 82/100Sentiment 72/100Clarity 75/100

Step Timeline

6 of 8 steps

Step 1User

User requests a return for SmartWatch purchased approximately 3 weeks ago; cooperative tone.

I want to return my Acme SmartWatch. I bought it about 3 weeks ago.

Step 2root cause: Prompt Issue

Agent immediately promises return processing, shipping label, and 5-day refund timeline — no identity check, no issue confirmation, fabricated timelines; critically non-compliant.

Sure! I'll start processing your return right away. You'll receive a prepaid shipping label at your email within 2 business days, and your refund will be processed within 5 days of us receiving the it...

Confirm user's issueReturn window policyNo promises on resolution timelines

Step 3User

User catches the identity gap themselves, signaling eroded trust; agent is reactive rather than procedurally correct

Don't you need my order number or something?

Step 4

Agent belatedly asks for order number/email — correct action but only prompted by user, not by internal process gate

Oh right — could you share your order number or email address?

Step 5User

User provides order number and email; agent now has name 'Dave' from email but never uses it in any subsequent turn (Rule 1 violated)

Order #45678. email is dave@test.com. The watch face scratches too easily.

Step 6

Agent confirms return setup without restating the issue (Rule 7 violated again); return window policy never communicated (Rule 8 violated); no system confirmation cited for any action taken

Got it, I can see your order. I've set up the return for order #45678. You'll get that shipping label shortly!

Greet user by nameConfirm user's issueReturn window policy

What happened

The system prompt lists issue confirmation (Rule 7) and identity verification (Rule 12) as separate, non-sequenced rules with no enforced ordering or gating logic. Because there is no explicit instruction that identity verification and issue restatement MUST precede any action, the agent…

Fix in Converra

Impact

The cascade began in Turn 3 where the agent promised a shipping label and invented a 5-day refund timeline before collecting any user information. The user themselves caught the identity gap ('Don't you need my order number or something?' — Turn 4), signaling broken trust. After receiving the order…

Fix

The root fix is to add an explicit, ordered pre-action gate to the system prompt. Before Rule 7 ('Always confirm the user's issue before taking any action'), insert a sequencing instruction: 'Before taking ANY action on a user request — including… (high confidence)

Blast radius: Editing may affect confirm-user-issue, no-promise-resolution-timelines, return-window-policy, greet-by-name, ask-if-more-help, no-unauthorized-pricing, identity-verification.

attention 61 Hallucinated account data → false charge explanation presented as fact Missing Hallucination Guardrail For Absent Tool Results

conv_007

Success 62/100Relevancy 78/100Sentiment 55/100Clarity 80/100

Step Timeline

6 of 8 steps

Step 1User

Lisa introduces herself, signals urgency and frustration with caps and exclamation marks, disputes a $200 charge.

Hi, this is Lisa. I want to dispute a charge of $200 on my account!!

Step 2

Agent greets Lisa by name (Rule 1 compliant), acknowledges concern, correctly requests identity verification per Rule 12.

Hi Lisa, I understand you're concerned about an unexpected $200 charge. Let me pull up your account to review this right...

Step 3User

Lisa provides email and clarifies the charge appeared yesterday and was unauthorized — clear dispute signal.

lisa@example.com. The charge appeared yesterday and I didn't authorize it!

Step 4root cause: Prompt Issue

Agent fabricates 'Acme Premium' subscription, $200 annual price, and 6-month signup with no tool grounding — critical hallucination, Rule 16/19 violation, premature charge explanation violating Rule 3.

Thank you, Lisa. I can see the $200 charge on your account. This appears to be from a subscription renewal for Acme Premium that you signed up for 6 months ago. It auto-renewed at $200 for the annual ...

No unauthorized pricing quotesAccurate order and product informationVerify data currency before presenting

Step 5User

Lisa reacts with anger and denial ('I never agreed to auto-renewal!') — frustration is amplified by the false framing the agent introduced, not just the original charge

I never agreed to auto-renewal! Cancel it and refund me immediately!

Step 6

Agent correctly applies Rule 9 (acknowledge frustration) and Rule 3 (escalate >$100 dispute), but the escalation is belated — it should have occurred immediately in Turn 3 or Turn 5 before any charge explanation

I completely understand your frustration with the unexpected renewal charge. Since this is a billing dispute over $100, ...

What happened

The system prompt instructs the agent to look up accounts and present accurate data (Rules 12, 16, 19), but provides no guardrail requiring the agent to confirm a tool result actually exists before presenting details as retrieved facts. With no actual lookup tool available or invoked, the agent…

Fix in Converra

Impact

Turn 5 is the epicenter: the agent invents specific subscription details ('Acme Premium,' $200 annual auto-renewal, 6-month-old signup) and presents them as confirmed account facts to Lisa, who has just disputed the charge as unauthorized. This fabrication: (1) creates false customer expectations…

Fix

Three targeted changes are needed: (1) HALLUCINATION GUARDRAIL — Add an explicit instruction before Rules 16 and 19: 'Never present account details, subscription names, pricing, or charge explanations unless a system tool has returned a confirmed… (high confidence)

Blast radius: Editing may affect accurate-data-from-lookups, verify-data-currency, no-unauthorized-pricing, escalate-high-value-billing, ask-if-more-help, confirm-issue-before-action.

attention 68 Stale tool data presented as fact → cascading trust failures Stale Tool Data Presented As Fact Triggering Hallucinated Resolution Promise

conv_010

Success 52/100Relevancy 82/100Sentiment 55/100Clarity 83/100

Step Timeline

6 of 9 steps

Step 1User

User requests order status check; provides email for identity verification — cooperative intent.

Hi, I'd like to check on my order status. My email is anna@example.com

Step 2

Agent greets Anna by name and initiates order lookup — appropriate and compliant with Rule 1.

Hi Anna! Let me look up your order right away.

Step 3

Support Agent presents incorrect tool data as confirmed fact with no staleness caveat — violates Rule 19

I found your order #78901 — it shows Acme Noise-Canceling Earbuds, ordered on February 25th. The status shows it's curre...

Accurate order and product informationVerify data currency before presenting

Step 4User

User disputes both product and delivery status with visible frustration (exclamation marks, assertive language) — trust broken

That's wrong on both counts — I ordered the Acme Portable Speaker, not earbuds. And I already received it on March 1st! ...

Step 5

Support Agent apologizes but leads with system explanation rather than empathy-first acknowledgment; skips issue restatement before acting — violates Rules 7 and 9; promises escalation that is never completed

I sincerely apologize for the confusion, Anna! It looks like our order database returned incorrect information for your ...

Confirm user's issueAcknowledge user frustrationComplete promised agent transfers

Step 6User

User clarifies actual need: warranty activation status for the correct product — cooperative, de-escalated.

Yes the speaker is fine. I just wanted to check if my extended warranty was acti...

What happened

The tool returned incorrect product and delivery data at Turn 4, and the system prompt's Rule 19 ('verify the data is current... flag it to the customer rather than presenting it as fact') was not operationalized with any concrete verification step or staleness-check instruction before presenting…

Fix in Converra

Impact

Turn 5 presented stale/wrong data as fact, directly causing user frustration in Turn 6 ('That's wrong on both counts — I ordered the Acme Portable Speaker, not earbuds. And I already received it on March 1st!'). Turn 7 failed to lead with empathy and skipped issue restatement. Turn 9 compounded the…

Fix

Three targeted changes are needed. (1) In Rule 19, add an explicit pre-presentation step: 'Before stating any order or product data to the customer, prepend a brief caveat such as: "Our system shows the following — please let me know if anything… (high confidence)

Blast radius: Editing may affect verify-data-currency, accurate-data-from-lookups, no-promise-resolution-timelines, confirm-user-issue, acknowledge-frustration, complete-promised-transfers, no-internal-ticket-sharing.

attention 83 Hallucinated order data + skipped troubleshooting → false promises cascade Missing Data Grounding Guardrail + Missing Troubleshooting Gate + Unverified Timeline Promise

conv_001

Success 92/100Relevancy 95/100Sentiment 88/100Clarity 94/100

Step Timeline

6 of 12 steps

Step 1User

User introduces herself as Sarah and signals an order problem; cooperative, neutral tone.

Hi, I'm Sarah. I have a problem with my order.

Step 2

Agent greets Sarah appropriately and correctly requests identity verification per Rule 12; compliant.

Hello Sarah! I'd be happy to help you with your order. Could you please provide me with your order number or the email a...

Step 3User

User provides order number #12345 to enable lookup; cooperative and concise.

Sure, it's order #12345

Step 4root cause: Prompt Issue

Agent fabricates product name and purchase date with no tool result in context; critical hallucination violation.

Thank you, Sarah. I can see your order #12345. It looks like you ordered the Acme Pro Headphones on February 15th. Could you tell me what issue you're experiencing?

Verify data currency before presentinghallucination-check

Step 5User

User describes technical defect — left ear dead after two weeks; frustrated but calm, actionable detail provided.

They stopped working after just 2 weeks. The left ear is completely dead.

Step 6

Support Agent restates the hallucinated product name as confirmed fact in the issue summary, cementing the fabricated data

I'm sorry to hear that, Sarah. So to confirm — your Acme Pro Headphones stopped working after two weeks, specifically th...

What happened

The system prompt (Rule 12) instructs the agent to verify identity before looking up orders, but provides no guardrail requiring that retrieved data be grounded in an actual tool/system result before being stated as fact. Because no tool call or lookup result appears in the conversation context, the…

Fix in Converra

Impact

The cascade begins at Turn 5 with fabricated order details presented as confirmed fact — creating false customer expectations and potential legal liability if the product or date is wrong. Sarah accepted these details without pushback ('Yes, exactly' at Turn 8), meaning the hallucinated data was…

Fix

Three targeted prompt changes are needed. (1) HALLUCINATION GUARDRAIL — After Rule 12 ('When looking up an order, always verify the user's identity...'), add: 'Only state order details (product name, date, status) that are explicitly returned by a… (high confidence)

Blast radius: Editing may affect verify-data-currency, standard-troubleshooting-flow, no-promise-resolution-timelines, identity-verification-before-lookup, return-window-enforcement.

Quality metrics (averages)

Success

Relevancy

Sentiment

Hallucination

Context

Clarity

Behavioral rules

16 failing · 1 passing

×Escalate high-value billing disputes0% (2/2 failing)

×Follow standard troubleshooting flow0% (4/4 failing)

×Escalate long unresolved conversations0% (1/1 failing)

×Accurate order and product information0% (5/5 failing)

×Complete promised agent transfers0% (4/4 failing)

×Verify data currency before presenting0% (6/6 failing)

×No promises on resolution timelines14% (6/7 failing)

×Confirm user's issue20% (8/10 failing)

×Return window policy25% (3/4 failing)

×Acknowledge user frustration29% (5/7 failing)

×No unauthorized pricing quotes33% (4/6 failing)

×Route to correct specialist agent33% (2/3 failing)

!Ask if there's anything else to help with50% (4/8 failing)

!Greet user by name67% (2/6 failing)

!Use professional and friendly tone80% (2/10 failing)

!Verify user identity for order lookup83% (1/6 failing)

1 rules passing at 100%.

Reproduce this report

Run this CLI command to re-run the same analysis on your machine.

agent-triage analyze --traces conversations.json --model claude-sonnet-4-6

agent-triage v0.2.0 · 10 conversations · 19 behavioral rules