Intelligence Satisfaction Score (ISAT)

An industry-first framework to measure the effectiveness of an intelligent virtual assistant

Limitations in the Current Measurement Frameworks

CSAT & NPS

Only 2% of users share feedback about their experience with a virtual assistant

Automation

Useful but does not say anything about queries resolved, false positives, confidence threshold, etc.

Engagement Rate

Standard market metrics such as retention rate, engagement, avg time spent, etc. are not applicable for virtual assistants

Sentiment Analysis

Complex multi-turn conversations can have various sentiments that make it difficult to

The ISAT Score Framework

Steps to Arrive at the ISAT for a given virtual assistant or chatbot

1
Categorize
Conversations
2
Calculate ISAT
Score
3
Optimize
Conversation
Flows

Classifying User Conversations

Analyzing Positive, Negative and Neutral ISAT Conversations

Positive
  • User comes with a query
  • IVA responds appropriately and solves the query
  • Task completion
Negative
  • High user intent. User repeatedly asks the same query
  • Sends > 2 messages
  • IVA unable to comprehend repeats itself, leading to negative sentiments and unhappy users
Neutral
  • Low user intent
  • User interacts with the IVA with no clear goal or objective in mind
  • User drops off randomly

Analyzing Negative ISAT Conversations

Conversations with a negative ISAT score are poor experiences that users undergo when the IVA fails to understand and resolve the user’s query. This model is validated by adding all conversations that had a user repetition (UR), bot repetition (BR), and cuss words (CW).

Score
Clients
  • Insurance
  • Finance
  • Healthcare
  • Govt
  • FMCG
ISAT
Bad Chats
  • 12.2%
  • 1.1%
  • 10.4%
  • 18.0%
  • 3.2%
Machine Analysis
UR BR CW Total Negative Conversations
6.0% 3.0% 2.0% 11.0%
1.0% 0.0% 0.0% 1.0%
4.0% 2.0% 2.0% 8.0%
14.0% 3.0% 3.0% 17.0%
1.0% 1.0% 1.0% 2.0%

Decoding False Positives in ISAT

When the IVA fails to recognize the real intent of the user, it sends a wrong response causing False positives, thereby affecting the user experience negatively.

Combusting Bot Breaks in ISAT

An IVA ‘breaks’ when it is unable to recognize the intent of the user. An IVA designer configures a response to be sent to the user in such a scenario to convey the context and scope of the IVA.

ISAT Case Study

For one of the world’s largest insurance companies, we used the ISAT as a benchmark to improve the quality of the IVA, thereby improving the overall NPS

“Our customers often contact us with the same kind of requests. Using Haptik, we built Zuri and we have been able to achieve greater business efficiency and 70% end-to-end query resolution. In addition to the positive response from our customers, Zuri has helped us to offer 24x7 support to complement our HelpPoint timings and ensure we are there when our customers need us the most.”

Mark Cady, Head of Operations, Zurich International

This website stores cookies in order to collect information about how you interact with our website. By continuing to the website, you allow the use of these cookies. To find out more, see our Cookies Policy.

Upgrade your Customer Experience with Conversation AI