Automating Insight: Saving Teams 6 Hours a Week with AI-Powered Bot Analysis
“We had dashboards, but we still didn’t know why the bot was failing.”
Brands invest heavily in automating support, but most teams are left with shallow metrics like handover rates or resolution times. That’s where Analyze comes in—an AI-powered analytics platform that surfaces topic-level insights, tracks sentiment, and learns over time to drive continuous improvement.
Note: I’ve used randomly generated data in the screenshots to protect client confidentiality. The visuals show real features, but the content is illustrative.
THE PROBLEM
Too much data, not enough direction — teams were stuck digging through chat logs without knowing what to fix.
My Role
8 people (PM, 2 designers, 4 devs, QA)
Team
Lead Product Designer
Timeline
4 months
Key CHALLENGES
Turning unstructured conversations
into clear insights
Challenge
Raw chatbot logs were all over the place—messy, inconsistent, and hard to use.
Solution
I added structure with Progressive Disclosure. It cut the noise, surfaced what mattered, and gave teams clarity without overload.
Designing on the go
Challenge
AI moves fast; models change, timelines shrink. The tricky part was keeping clarity while moving quickly.
Solution
I leaned on quick research, rough prototypes, and close teamwork with data and engineering. That way, we adapted fast without hurting usability.
MY DESIGN PROCESS
Build with users, not just for them
This wasn’t just a design sprint —
it was a continuous loop of insight → validation → iteration.
RESEARCH
From noise to clarity
Our goal was to understand not just what metrics users needed but how they made decisions about bot performance and improvements. Here's how we approached it:
User persona
We interviewed internal CS teams and reviewed user behaviors to build two detailed personas:
Customer success manager
Aisha Mehra
33 y/o, CSM
“I need to know where the bot is failing and how to fix it fast.”
🎯
Goals
Understand where the bot is failing and how to fix it fast.
😓
Frustrations
Relies on manual log reviews/spreadsheets, no clarity on whether issues stem from bot flow, content, or intent.
👇
Needs
Clear view of unresolved queries, sentiment trends, and actionable insights to improve containment.
🧠
How Might We
Summarize unresolved queries, link metrics to conversations, and surface automation opportunities to reduce prep time for QBRs.
⚙️
Behaviors & Tool Gaps
Tasks
Reviewing issues
Client reporting
Automation tracking
Current Behavior
Reads raw logs
Manual exports
Gut feeling
Limitations
Time-consuming
Not scalable
Risky
Bot Builder / Automation Specialist
“I need to know where the bot is failing and how to fix it fast.”
Aisha Mehra
33 y/o, CSM
🎯
Goals
Fix flows fast, cut unresolved queries, and boost CSAT with smoother self-service.
😓
Frustrations
Lacks a clear view of flow issues, switches tools often, and wastes time finding root causes.
👇
Needs
Quick filters, context-rich logs, and AI suggestions for KB and resolutions.
🧠
How Might We
Help Ravi find issues faster and fix them with less effort.
⚙️
Behaviors & Tool Gaps
Tasks
Fixing flows
Reviewing sentiment
Updating KB
Current Behavior
Manual debugging
Ignores it
Based on gut
Limitations
Slow & reactive
Not connected to flows
Not data-driven
Research Methods
Affinity mapping
Grouped feedback into six themes to uncover recurring pain points in analysis workflows.
Key inshights
Unclear resolution & sentiment
Disconnected dashboards
Manual reviews took 5–6 hours weekly
Agent replies weren’t reused
Data lacked “how to fix” context
Impact v/s effort analysis
We prioritized what to build based on the cost vs value of solving each pain point.
Competitive Audit
We evaluated how leading platforms like Ada and Intercom handled similar problems.
Targeted pain points
Reduce manual effort and increase decision clarity
After mapping feedback, identifying behavioural gaps, and evaluating competitor gaps, we narrowed down to the most critical problems our MVP needed to address. These were selected based on a combination of user pain, feasibility, and impact potential.
Manual effort to tag or categorize issues
CSMs and bot builders spent hours reviewing logs weekly. We saw teams averaging 5–6 hours/week just to prepare reports.
Auto tag conversations
Enable auto-tagging with LLMs, allowing users to filter by topic, failure, intent, or escalation
No clarity on resolution
Teams couldn’t tell if a user query was actually resolved, even if the bot completed a flow. This limited trust in the data.
Conversation classification
Give resolution status. Classify each conversation as resolved, unresolved, or handed over
No actionable suggestions
Even with charts and trends, teams didn’t know what to do next. Fixing issues relied heavily on intuition.
Actionable suggestions
Provide insight that highlight top failure reasons, drop-off patterns, and automation suggestions
No automation signals from agent replies
Valuable agent messages were not reused, even when they could have trained the bot. Automation potential was left untapped.
Auto learn from agent messages
Build an engine to identify repetitive agent replies that can be converted to KB articles or bot responses.
Disconnected dashboards and logs
Teams couldn't drill down from metrics to actual conversations. This broke the feedback loop and slowed issue resolution.
Add conversation to metrics
Link every metric to specific conversations, making insights traceable to the source
DISCOVERY
Validating hypothesized solution
Validation
Co-built with early customers by manually tagging chat logs to reveal why bots failed. Expanded across industries, refined tags, and introduced CRR & sentiment. Partnered with engineering to scale into an AI model that automated tagging and surfaced clear automation opportunities.
Client feedback
When we presented the insights, the reaction was immediate
“This is the first time we’ve seen why our bot is actually failing.”
“Can you make this process of analysis faster and smarter ”
The structured clarity clicked. For them and for us.
Key insights
Manual → AI tagging cut effort
Resolution tagging unlocked true success
Reports linked metrics to conversations
Agent replies revealed automation gaps
Need for filters & cleaner handover
Insights 3× faster on average
AI MODEL
Reduce manual effort and increase decision clarity
To move beyond dashboards, we started thinking differently, what if the bot could learn directly from real failures?
Our engineering team built an LLM-based system that analyzes conversations between agents and customers to surface where automation failed and how it could improve.
Conversations
AI model reviews conversations that were transferred to a human and got resolved
An AI model reviews conversations
Labels each conversation
Analyse how human solved query
Suggests articles to add to bot knowledge to improve automation
Topic
Model assigns a topic based on user intent
Containment
Checks if the query was resolved with/without agent handover
Resolution status
Checks if the query was resolved or not
User sentiment
Detects final user sentiment — positive, neutral, or negative
Design
Turning vision into reatilty
Analyze is built for fast-moving teams who can’t waste time in chat logs or static dashboards — it turns conversations into clear insights so they know exactly what to fix and why.
Topics list
We turned noisy chat data into a clear table that helps teams spot trends, gaps, and next steps.
Familiar table view, enriched with AI metrics like sentiment and resolution rate.
Every row highlights an action showing what’s working and what needs fixing.
Clear, high signal insights reduce overload and give teams confidence to act.
Topic details
Users needed more than high-level stats they wanted to see why a topic was struggling and what to do about it. This screen made that possible.
Key metrics up front show how often a topic appears, how bots handle it, and how users feel.
Clear next steps guide teams to act fast, like adding an article or resolving chats.
Smart suggestions explain the “why,” giving teams confidence to improve without analysts.
Conversation analysis
We connected insights to real conversations so teams could see exactly where the bot failed and why. By pairing full transcripts with AI analysis, users got both the story and the signal in one place.
Searchable queries, full transcripts, and AI tags are all shown side by side.
Unresolved chats are flagged instantly, making failures easy to spot.
Raw conversations build trust, while reducing screen-hopping speeds up diagnosis and decisions.
Outcome
Assessing the impact
We rolled out in phases, first to 15 enterprise customers for beta, then to our entire base. Every feedback loop helped us refine the product before the big launch.
30%
faster analysis saving teams 5–6 hours weekly
3x
quicker issue detection with AI driven tagging and insights
10%
increase in CRR for top accounts during beta
200+
users subscribed to analyse module
Improvements and future
While the early impact was promising, real user behavior uncovered areas for growth. Many teams weren’t acting on KB suggestions a sign we needed better integration with content workflows. We also heard a clear ask for a holistic view of topic performance, not just isolated metrics. Moving forward, we’re building features like multi-language support, deeper reporting, and easier adoption paths to turn insights into real action faster, and for more teams.
Thanks for reading! :)
Automating Insight: Saving Teams 6 Hours a Week with AI-Powered Bot Analysis
“We had dashboards, but we still didn’t know why the bot was failing.”
Brands invest heavily in automating support, but most teams are left with shallow metrics like handover rates or resolution times. That’s where Analyze comes in—an AI-powered analytics platform that surfaces topic-level insights, tracks sentiment, and learns over time to drive continuous improvement.
Note: I’ve used randomly generated data in the screenshots to protect client confidentiality. The visuals show real features, but the content is illustrative.
THE PROBLEM
Too much data, not enough direction — teams were stuck digging through chat logs without knowing what to fix.
My Role
8 people (PM, 2 designers, 4 devs, QA)
Team
Lead Product Designer
Timeline
4 months
Key CHALLENGES
Turning unstructured conversations
into clear insights
Challenge
Raw chatbot logs were all over the place—messy, inconsistent, and hard to use.
Solution
I added structure with Progressive Disclosure. It cut the noise, surfaced what mattered, and gave teams clarity without overload.
Designing on the go
Challenge
AI moves fast; models change, timelines shrink. The tricky part was keeping clarity while moving quickly.
Solution
I leaned on quick research, rough prototypes, and close teamwork with data and engineering. That way, we adapted fast without hurting usability.
MY DESIGN PROCESS
Build with users, not just for them
This wasn’t just a design sprint —
it was a continuous loop of insight → validation → iteration.
RESEARCH
From noise to clarity
Our goal was to understand not just what metrics users needed but how they made decisions about bot performance and improvements. Here's how we approached it:
User persona
We interviewed internal CS teams and reviewed user behaviors to build two detailed personas:
Customer success manager
Aisha Mehra
33 y/o, CSM
“I need to know where the bot is failing and how to fix it fast.”
🎯
Goals
Understand where the bot is failing and how to fix it fast.
😓
Frustrations
Relies on manual log reviews/spreadsheets, no clarity on whether issues stem from bot flow, content, or intent.
👇
Needs
Clear view of unresolved queries, sentiment trends, and actionable insights to improve containment.
🧠
How Might We
Summarize unresolved queries, link metrics to conversations, and surface automation opportunities to reduce prep time for QBRs.
⚙️
Behaviors & Tool Gaps
Tasks
Reviewing issues
Client reporting
Automation tracking
Current Behavior
Reads raw logs
Manual exports
Gut feeling
Limitations
Time-consuming
Not scalable
Risky
Bot Builder / Automation Specialist
“I need to know where the bot is failing and how to fix it fast.”
Aisha Mehra
33 y/o, CSM
🎯
Goals
Fix flows fast, cut unresolved queries, and boost CSAT with smoother self-service.
😓
Frustrations
Lacks a clear view of flow issues, switches tools often, and wastes time finding root causes.
👇
Needs
Quick filters, context-rich logs, and AI suggestions for KB and resolutions.
🧠
How Might We
Help Ravi find issues faster and fix them with less effort.
⚙️
Behaviors & Tool Gaps
Tasks
Fixing flows
Reviewing sentiment
Updating KB
Current Behavior
Manual debugging
Ignores it
Based on gut
Limitations
Slow & reactive
Not connected to flows
Not data-driven
Research Methods
Affinity mapping
Grouped feedback into six themes to uncover recurring pain points in analysis workflows.
Key inshights
Unclear resolution & sentiment
Disconnected dashboards
Manual reviews took 5–6 hours weekly
Agent replies weren’t reused
Data lacked “how to fix” context
Impact v/s effort analysis
We prioritized what to build based on the cost vs value of solving each pain point.
Competitive Audit
We evaluated how leading platforms like Ada and Intercom handled similar problems.
Targeted pain points
Reduce manual effort and increase decision clarity
After mapping feedback, identifying behavioural gaps, and evaluating competitor gaps, we narrowed down to the most critical problems our MVP needed to address. These were selected based on a combination of user pain, feasibility, and impact potential.
Manual effort to tag or categorize issues
CSMs and bot builders spent hours reviewing logs weekly. We saw teams averaging 5–6 hours/week just to prepare reports.
Auto tag conversations
Enable auto-tagging with LLMs, allowing users to filter by topic, failure, intent, or escalation
No clarity on resolution
Teams couldn’t tell if a user query was actually resolved, even if the bot completed a flow. This limited trust in the data.
Conversation classification
Give resolution status. Classify each conversation as resolved, unresolved, or handed over
No actionable suggestions
Even with charts and trends, teams didn’t know what to do next. Fixing issues relied heavily on intuition.
Actionable suggestions
Provide insight that highlight top failure reasons, drop-off patterns, and automation suggestions
No automation signals from agent replies
Valuable agent messages were not reused, even when they could have trained the bot. Automation potential was left untapped.
Auto learn from agent messages
Build an engine to identify repetitive agent replies that can be converted to KB articles or bot responses.
Disconnected dashboards and logs
Teams couldn't drill down from metrics to actual conversations. This broke the feedback loop and slowed issue resolution.
Add conversation to metrics
Link every metric to specific conversations, making insights traceable to the source
DISCOVERY
Validating hypothesized solution
Validation
Co-built with early customers by manually tagging chat logs to reveal why bots failed. Expanded across industries, refined tags, and introduced CRR & sentiment. Partnered with engineering to scale into an AI model that automated tagging and surfaced clear automation opportunities.
Client feedback
When we presented the insights, the reaction was immediate
“This is the first time we’ve seen why our bot is actually failing.”
“Can you make this process of analysis faster and smarter ”
The structured clarity clicked. For them and for us.
Key insights
Manual → AI tagging cut effort
Resolution tagging unlocked true success
Reports linked metrics to conversations
Agent replies revealed automation gaps
Need for filters & cleaner handover
Insights 3× faster on average
AI MODEL
Reduce manual effort and increase decision clarity
To move beyond dashboards, we started thinking differently, what if the bot could learn directly from real failures?
Our engineering team built an LLM-based system that analyzes conversations between agents and customers to surface where automation failed and how it could improve.
Conversations
AI model reviews conversations that were transferred to a human and got resolved
An AI model reviews conversations
Labels each conversation
Analyse how human solved query
Suggests articles to add to bot knowledge to improve automation
Topic
Model assigns a topic based on user intent
Containment
Checks if the query was resolved with/without agent handover
Resolution status
Checks if the query was resolved or not
User sentiment
Detects final user sentiment — positive, neutral, or negative
Design
Turning vision into reatilty
Analyze is built for fast-moving teams who can’t waste time in chat logs or static dashboards — it turns conversations into clear insights so they know exactly what to fix and why.
Topics list
We turned noisy chat data into a clear table that helps teams spot trends, gaps, and next steps.
Familiar table view, enriched with AI metrics like sentiment and resolution rate.
Every row highlights an action showing what’s working and what needs fixing.
Clear, high signal insights reduce overload and give teams confidence to act.
Topic details
Users needed more than high-level stats they wanted to see why a topic was struggling and what to do about it. This screen made that possible.
Key metrics up front show how often a topic appears, how bots handle it, and how users feel.
Clear next steps guide teams to act fast, like adding an article or resolving chats.
Smart suggestions explain the “why,” giving teams confidence to improve without analysts.
Conversation analysis
We connected insights to real conversations so teams could see exactly where the bot failed and why. By pairing full transcripts with AI analysis, users got both the story and the signal in one place.
Searchable queries, full transcripts, and AI tags are all shown side by side.
Unresolved chats are flagged instantly, making failures easy to spot.
Raw conversations build trust, while reducing screen-hopping speeds up diagnosis and decisions.
Outcome
Assessing the impact
We rolled out in phases, first to 15 enterprise customers for beta, then to our entire base. Every feedback loop helped us refine the product before the big launch.
30%
faster analysis saving teams 5–6 hours weekly
3x
quicker issue detection with AI driven tagging and insights
10%
increase in CRR for top accounts during beta
200+
users subscribed to analyse module
Improvements and future
While the early impact was promising, real user behavior uncovered areas for growth. Many teams weren’t acting on KB suggestions a sign we needed better integration with content workflows. We also heard a clear ask for a holistic view of topic performance, not just isolated metrics. Moving forward, we’re building features like multi-language support, deeper reporting, and easier adoption paths to turn insights into real action faster, and for more teams.
Thanks for reading! :)
Automating Insight: Saving Teams 6 Hours a Week with AI-Powered Bot Analysis
“We had dashboards, but we still didn’t know why the bot was failing.”
Brands invest heavily in automating support, but most teams are left with shallow metrics like handover rates or resolution times. That’s where Analyze comes in—an AI-powered analytics platform that surfaces topic-level insights, tracks sentiment, and learns over time to drive continuous improvement.
Note: I’ve used randomly generated data in the screenshots to protect client confidentiality. The visuals show real features, but the content is illustrative.
THE PROBLEM
Too much data, not enough direction — teams were stuck digging through chat logs without knowing what to fix.
My Role
8 people (PM, 2 designers, 4 devs, QA)
Team
Lead Product Designer
Timeline
4 months
Key CHALLENGES
Turning unstructured conversations
into clear insights
Challenge
Raw chatbot logs were all over the place—messy, inconsistent, and hard to use.
Solution
I added structure with Progressive Disclosure. It cut the noise, surfaced what mattered, and gave teams clarity without overload.
Designing on the go
Challenge
AI moves fast; models change, timelines shrink. The tricky part was keeping clarity while moving quickly.
Solution
I leaned on quick research, rough prototypes, and close teamwork with data and engineering. That way, we adapted fast without hurting usability.
MY DESIGN PROCESS
Build with users, not just for them
This wasn’t just a design sprint —
it was a continuous loop of insight → validation → iteration.
RESEARCH
From noise to clarity
Our goal was to understand not just what metrics users needed but how they made decisions about bot performance and improvements. Here's how we approached it:
User persona
We interviewed internal CS teams and reviewed user behaviors to build two detailed personas:
Customer success manager
Aisha Mehra
33 y/o, CSM
“I need to know where the bot is failing and how to fix it fast.”
🎯
Goals
Understand where the bot is failing and how to fix it fast.
😓
Frustrations
Relies on manual log reviews/spreadsheets, no clarity on whether issues stem from bot flow, content, or intent.
👇
Needs
Clear view of unresolved queries, sentiment trends, and actionable insights to improve containment.
🧠
How Might We
Summarize unresolved queries, link metrics to conversations, and surface automation opportunities to reduce prep time for QBRs.
⚙️
Behaviors & Tool Gaps
Tasks
Reviewing issues
Client reporting
Automation tracking
Current Behavior
Reads raw logs
Manual exports
Gut feeling
Limitations
Time-consuming
Not scalable
Risky
Bot Builder / Automation Specialist
“I need to know where the bot is failing and how to fix it fast.”
Aisha Mehra
33 y/o, CSM
🎯
Goals
Fix flows fast, cut unresolved queries, and boost CSAT with smoother self-service.
😓
Frustrations
Lacks a clear view of flow issues, switches tools often, and wastes time finding root causes.
👇
Needs
Quick filters, context-rich logs, and AI suggestions for KB and resolutions.
🧠
How Might We
Help Ravi find issues faster and fix them with less effort.
⚙️
Behaviors & Tool Gaps
Tasks
Fixing flows
Reviewing sentiment
Updating KB
Current Behavior
Manual debugging
Ignores it
Based on gut
Limitations
Slow & reactive
Not connected to flows
Not data-driven
Research Methods
Affinity mapping
Grouped feedback into six themes to uncover recurring pain points in analysis workflows.
Key inshights
Unclear resolution & sentiment
Disconnected dashboards
Manual reviews took 5–6 hours weekly
Agent replies weren’t reused
Data lacked “how to fix” context
Impact v/s effort analysis
We prioritized what to build based on the cost vs value of solving each pain point.
Competitive Audit
We evaluated how leading platforms like Ada and Intercom handled similar problems.
Targeted pain points
Reduce manual effort and increase decision clarity
After mapping feedback, identifying behavioural gaps, and evaluating competitor gaps, we narrowed down to the most critical problems our MVP needed to address. These were selected based on a combination of user pain, feasibility, and impact potential.
Manual effort to tag or categorize issues
CSMs and bot builders spent hours reviewing logs weekly. We saw teams averaging 5–6 hours/week just to prepare reports.
Auto tag conversations
Enable auto-tagging with LLMs, allowing users to filter by topic, failure, intent, or escalation
No clarity on resolution
Teams couldn’t tell if a user query was actually resolved, even if the bot completed a flow. This limited trust in the data.
Conversation classification
Give resolution status. Classify each conversation as resolved, unresolved, or handed over
No actionable suggestions
Even with charts and trends, teams didn’t know what to do next. Fixing issues relied heavily on intuition.
Actionable suggestions
Provide insight that highlight top failure reasons, drop-off patterns, and automation suggestions
No automation signals from agent replies
Valuable agent messages were not reused, even when they could have trained the bot. Automation potential was left untapped.
Auto learn from agent messages
Build an engine to identify repetitive agent replies that can be converted to KB articles or bot responses.
Disconnected dashboards and logs
Teams couldn't drill down from metrics to actual conversations. This broke the feedback loop and slowed issue resolution.
Add conversation to metrics
Link every metric to specific conversations, making insights traceable to the source
DISCOVERY
Validating hypothesized solution
Validation
Co-built with early customers by manually tagging chat logs to reveal why bots failed. Expanded across industries, refined tags, and introduced CRR & sentiment. Partnered with engineering to scale into an AI model that automated tagging and surfaced clear automation opportunities.
Client feedback
When we presented the insights, the reaction was immediate
“This is the first time we’ve seen why our bot is actually failing.”
“Can you make this process of analysis faster and smarter ”
The structured clarity clicked. For them and for us.
Key insights
Manual → AI tagging cut effort
Resolution tagging unlocked true success
Reports linked metrics to conversations
Agent replies revealed automation gaps
Need for filters & cleaner handover
Insights 3× faster on average
AI MODEL
Reduce manual effort and increase decision clarity
To move beyond dashboards, we started thinking differently, what if the bot could learn directly from real failures?
Our engineering team built an LLM-based system that analyzes conversations between agents and customers to surface where automation failed and how it could improve.
Conversations
AI model reviews conversations that were transferred to a human and got resolved
An AI model reviews conversations
Labels each conversation
Analyse how human solved query
Suggests articles to add to bot knowledge to improve automation
Topic
Model assigns a topic based on user intent
Containment
Checks if the query was resolved with/without agent handover
Resolution status
Checks if the query was resolved or not
User sentiment
Detects final user sentiment — positive, neutral, or negative
Design
Turning vision into reatilty
Analyze is built for fast-moving teams who can’t waste time in chat logs or static dashboards — it turns conversations into clear insights so they know exactly what to fix and why.
Topics list
We turned noisy chat data into a clear table that helps teams spot trends, gaps, and next steps.
Familiar table view, enriched with AI metrics like sentiment and resolution rate.
Every row highlights an action showing what’s working and what needs fixing.
Clear, high signal insights reduce overload and give teams confidence to act.
Topic details
Users needed more than high-level stats they wanted to see why a topic was struggling and what to do about it. This screen made that possible.
Key metrics up front show how often a topic appears, how bots handle it, and how users feel.
Clear next steps guide teams to act fast, like adding an article or resolving chats.
Smart suggestions explain the “why,” giving teams confidence to improve without analysts.
Conversation analysis
We connected insights to real conversations so teams could see exactly where the bot failed and why. By pairing full transcripts with AI analysis, users got both the story and the signal in one place.
Searchable queries, full transcripts, and AI tags are all shown side by side.
Unresolved chats are flagged instantly, making failures easy to spot.
Raw conversations build trust, while reducing screen-hopping speeds up diagnosis and decisions.
Outcome
Assessing the impact
We rolled out in phases, first to 15 enterprise customers for beta, then to our entire base. Every feedback loop helped us refine the product before the big launch.
30%
faster analysis saving teams 5–6 hours weekly
3x
quicker issue detection with AI driven tagging and insights
10%
increase in CRR for top accounts during beta
200+
users subscribed to analyse module
Improvements and future
While the early impact was promising, real user behavior uncovered areas for growth. Many teams weren’t acting on KB suggestions a sign we needed better integration with content workflows. We also heard a clear ask for a holistic view of topic performance, not just isolated metrics. Moving forward, we’re building features like multi-language support, deeper reporting, and easier adoption paths to turn insights into real action faster, and for more teams.
Thanks for reading! :)
Automating Insight: Saving Teams 6 Hours a Week with AI-Powered Bot Analysis
“We had dashboards, but we still didn’t know why the bot was failing.”
Brands invest heavily in automating support, but most teams are left with shallow metrics like handover rates or resolution times. That’s where Analyze comes in—an AI-powered analytics platform that surfaces topic-level insights, tracks sentiment, and learns over time to drive continuous improvement.
Note: I’ve used randomly generated data in the screenshots to protect client confidentiality. The visuals show real features, but the content is illustrative.
THE PROBLEM
Too much data, not enough direction — teams were stuck digging through chat logs without knowing what to fix.
My Role
8 people (PM, 2 designers, 4 devs, QA)
Team
Lead Product Designer
Timeline
4 months
Key CHALLENGES
Turning unstructured conversations
into clear insights
Challenge
Raw chatbot logs were all over the place—messy, inconsistent, and hard to use.
Solution
I added structure with Progressive Disclosure. It cut the noise, surfaced what mattered, and gave teams clarity without overload.
Designing on the go
Challenge
AI moves fast; models change, timelines shrink. The tricky part was keeping clarity while moving quickly.
Solution
I leaned on quick research, rough prototypes, and close teamwork with data and engineering. That way, we adapted fast without hurting usability.
MY DESIGN PROCESS
Build with users, not just for them
This wasn’t just a design sprint —
it was a continuous loop of insight → validation → iteration.
RESEARCH
From noise to clarity
Our goal was to understand not just what metrics users needed but how they made decisions about bot performance and improvements. Here's how we approached it:
User persona
We interviewed internal CS teams and reviewed user behaviors to build two detailed personas:
Customer success manager
Aisha Mehra
33 y/o, CSM
“I need to know where the bot is failing and how to fix it fast.”
🎯
Goals
Understand where the bot is failing and how to fix it fast.
😓
Frustrations
Relies on manual log reviews/spreadsheets, no clarity on whether issues stem from bot flow, content, or intent.
👇
Needs
Clear view of unresolved queries, sentiment trends, and actionable insights to improve containment.
🧠
How Might We
Summarize unresolved queries, link metrics to conversations, and surface automation opportunities to reduce prep time for QBRs.
⚙️
Behaviors & Tool Gaps
Tasks
Reviewing issues
Client reporting
Automation tracking
Current Behavior
Reads raw logs
Manual exports
Gut feeling
Limitations
Time-consuming
Not scalable
Risky
Bot Builder / Automation Specialist
“I need to know where the bot is failing and how to fix it fast.”
Aisha Mehra
33 y/o, CSM
🎯
Goals
Fix flows fast, cut unresolved queries, and boost CSAT with smoother self-service.
😓
Frustrations
Lacks a clear view of flow issues, switches tools often, and wastes time finding root causes.
👇
Needs
Quick filters, context-rich logs, and AI suggestions for KB and resolutions.
🧠
How Might We
Help Ravi find issues faster and fix them with less effort.
⚙️
Behaviors & Tool Gaps
Tasks
Fixing flows
Reviewing sentiment
Updating KB
Current Behavior
Manual debugging
Ignores it
Based on gut
Limitations
Slow & reactive
Not connected to flows
Not data-driven
Research Methods
Affinity mapping
Grouped feedback into six themes to uncover recurring pain points in analysis workflows.
Key inshights
Unclear resolution & sentiment
Disconnected dashboards
Manual reviews took 5–6 hours weekly
Agent replies weren’t reused
Data lacked “how to fix” context
Impact v/s effort analysis
We prioritized what to build based on the cost vs value of solving each pain point.
Competitive Audit
We evaluated how leading platforms like Ada and Intercom handled similar problems.
Targeted pain points
Reduce manual effort and increase decision clarity
After mapping feedback, identifying behavioural gaps, and evaluating competitor gaps, we narrowed down to the most critical problems our MVP needed to address. These were selected based on a combination of user pain, feasibility, and impact potential.
Manual effort to tag or categorize issues
CSMs and bot builders spent hours reviewing logs weekly. We saw teams averaging 5–6 hours/week just to prepare reports.
Auto tag conversations
Enable auto-tagging with LLMs, allowing users to filter by topic, failure, intent, or escalation
No clarity on resolution
Teams couldn’t tell if a user query was actually resolved, even if the bot completed a flow. This limited trust in the data.
Conversation classification
Give resolution status. Classify each conversation as resolved, unresolved, or handed over
No actionable suggestions
Even with charts and trends, teams didn’t know what to do next. Fixing issues relied heavily on intuition.
Actionable suggestions
Provide insight that highlight top failure reasons, drop-off patterns, and automation suggestions
No automation signals from agent replies
Valuable agent messages were not reused, even when they could have trained the bot. Automation potential was left untapped.
Auto learn from agent messages
Build an engine to identify repetitive agent replies that can be converted to KB articles or bot responses.
Disconnected dashboards and logs
Teams couldn't drill down from metrics to actual conversations. This broke the feedback loop and slowed issue resolution.
Add conversation to metrics
Link every metric to specific conversations, making insights traceable to the source
DISCOVERY
Validating hypothesized solution
Validation
Co-built with early customers by manually tagging chat logs to reveal why bots failed. Expanded across industries, refined tags, and introduced CRR & sentiment. Partnered with engineering to scale into an AI model that automated tagging and surfaced clear automation opportunities.
Client feedback
When we presented the insights, the reaction was immediate
“This is the first time we’ve seen why our bot is actually failing.”
“Can you make this process of analysis faster and smarter ”
The structured clarity clicked. For them and for us.
Key insights
Manual → AI tagging cut effort
Resolution tagging unlocked true success
Reports linked metrics to conversations
Agent replies revealed automation gaps
Need for filters & cleaner handover
Insights 3× faster on average
AI MODEL
Reduce manual effort and increase decision clarity
To move beyond dashboards, we started thinking differently, what if the bot could learn directly from real failures?
Our engineering team built an LLM-based system that analyzes conversations between agents and customers to surface where automation failed and how it could improve.
Conversations
AI model reviews conversations that were transferred to a human and got resolved
An AI model reviews conversations
Labels each conversation
Analyse how human solved query
Suggests articles to add to bot knowledge to improve automation
Topic
Model assigns a topic based on user intent
Containment
Checks if the query was resolved with/without agent handover
Resolution status
Checks if the query was resolved or not
User sentiment
Detects final user sentiment — positive, neutral, or negative
Design
Turning vision into reatilty
Analyze is built for fast-moving teams who can’t waste time in chat logs or static dashboards — it turns conversations into clear insights so they know exactly what to fix and why.
Topics list
We turned noisy chat data into a clear table that helps teams spot trends, gaps, and next steps.
Familiar table view, enriched with AI metrics like sentiment and resolution rate.
Every row highlights an action showing what’s working and what needs fixing.
Clear, high signal insights reduce overload and give teams confidence to act.
Topic details
Users needed more than high-level stats they wanted to see why a topic was struggling and what to do about it. This screen made that possible.
Key metrics up front show how often a topic appears, how bots handle it, and how users feel.
Clear next steps guide teams to act fast, like adding an article or resolving chats.
Smart suggestions explain the “why,” giving teams confidence to improve without analysts.
Conversation analysis
We connected insights to real conversations so teams could see exactly where the bot failed and why. By pairing full transcripts with AI analysis, users got both the story and the signal in one place.
Searchable queries, full transcripts, and AI tags are all shown side by side.
Unresolved chats are flagged instantly, making failures easy to spot.
Raw conversations build trust, while reducing screen-hopping speeds up diagnosis and decisions.
Outcome
Assessing the impact
We rolled out in phases, first to 15 enterprise customers for beta, then to our entire base. Every feedback loop helped us refine the product before the big launch.
30%
faster analysis saving teams 5–6 hours weekly
3x
quicker issue detection with AI driven tagging and insights
10%
increase in CRR for top accounts during beta
200+
users subscribed to analyse module
Improvements and future
While the early impact was promising, real user behavior uncovered areas for growth. Many teams weren’t acting on KB suggestions a sign we needed better integration with content workflows. We also heard a clear ask for a holistic view of topic performance, not just isolated metrics. Moving forward, we’re building features like multi-language support, deeper reporting, and easier adoption paths to turn insights into real action faster, and for more teams.