SofTeCode Blogs

One Place for all Tech News and Support

Case study of Chatbot

7 min read

Introduction of Chatbot
I have worked on building internal employee-facing chatbots for quite two years. Our current chatbot platform only allows 10 training examples for every intent. to several people, 10 training examples sound sort of a lot, cause it’s hard for an individual to return up with 10 alternative ways to ask an equivalent question. However, with many thousands of users, it’s quite common to possess quite 10 verbiages for an equivalent intent. Most chatbot tools nowadays have a point of tongue understanding (NLU) or tongue processing (NLP) capacity, which suggests we don’t get to provide all the possible variations as training examples, we only got to provide the foremost popular and representative ones. However, will 10 be enough?

In this article, I will be able to list the important user queries of a really common intent within the human resource domain, analyze where the variations come from, so that you’ll be ready to estimate the amount and sort of coaching examples you would like to coach an intent. In the end, I also listed some recommendations for evaluating chatbot tools.

Sample Intent

The sample intent is to urge my employer’s holiday schedule for the present year. Let’s assume my employer’s name is Microsoft.
The sample response to the above intent is

  • New Year’s Day — January 1
  • Martin Luther King Day — January 20
  • President’s Day — February 17
  • Memorial Day — May 25
  • Independence Day — July 3
  • Labor Day — September 7
  • Columbus Day — Columbus Day
  • Veterans’ Day — Veterans’ Day
  • Thanksgiving Day — November 26
  • Day After Thanksgiving — November 27
  • Christmas — December 25

Before I start

  • I want to pause right here and ask you to think about:
  • how will you express this intent to a chatbot?
  • how many alternative ways are you able to consider to precise this intent to a chatbot?
case study of chatbot
credit unsplash

Real User Queries

  • I fount 31 different user queries that would be a match of the sample intent.
  • What is the company’s holiday schedule for 2020?
  • What are the 2020 company holidays?
  • what are the additional holidays in 2020?
  • Where am I able to find the vacation calendar?
  • what is this year’s holiday schedule?
  • what are Microsoft holidays?
  • What holiday do I buy off?
  • what are our 2020 holidays
  • list of Microsoft holidays
  • list of holidays
  • 2020 holiday calendar
  • 2020 paid holidays
  • 2020 holiday schedule
  • 2020 vacation days
  • 2020 holidays
  • 2020 vacation
  • company holiday
  • corporate holidays
  • Microsoft holiday 2020
  • Microsoft holiday
  • ms holiday
  • ms holidays
  • holidays
  • holiday
  • is Christmas Eve 2020 a holiday?
  • is Columbus Day a Microsoft holiday?
  • is July 5 a holiday?
  • Which day can we get off for the 4th of July?
  • What’s the subsequent holiday?
  • When is my next holiday
  • Where is that the vacation calendar?


The above variations mainly come from different word/phrase choices, different syntactic structures, and therefore the need for all or partial responses.

  • Different word and phrase choices (without stemming)
  • Common synonyms — holiday / holidays / vacation / vacations, schedule / calendar, company / corporate
  • Synonyms only in specific context — MS / Microsoft / company / our, this year / 2020, holiday / get off
  • Acronyms — MS / Microsoft
  • Incorrect spacing — Microsoft / Micro Soft
  • Typos — vacation / vocation / vaacation

Most intent identifiers nowadays can handle some typos and therefore the commonest synonyms. However, for special acronyms like “MS” for “Microsoft”, and for words with an equivalent meaning only during a specific context like “our” and “company”, and intent identifier might not be ready to recognize the similarity of these words without explicit training examples.

Different syntactic structures

The different syntactic structure of full sentences — what’s the company’s holiday schedule? / Where am I able to find the vacation calendar?

  • what’s a subsequent holiday? / When is that the next holiday?
  • Full-sentences / keyphrases — what’s the company’s holiday schedule? / list of holidays
  • Informal language and incorrect grammar

One could argue that “What is that the schedule?” is posing for the schedule while “Where is that the schedule?” is posing for the situation of the schedule, therefore they’re two separate intents. However, during this case, most users who invite the situation want to ascertain the schedule. “Where is that the schedule?” may be a short version of “Where is that the schedule? I would like to ascertain it”. Also, it saves users longer when responding with the schedule directly than showing was to urge it. for instance, If the response provides the link to the schedule, it costs a user another click then putting the schedule within the response. Therefore I like better to combine those two intents.
Another issue is a few users prefer keywords then full sentences, which make the intent ambiguous. for instance, if a user asks “holiday”, does he want to urge the employer’s holiday schedule, the federal holidays, holiday event, holiday gift service, or anything associated with the holiday? The intent recognizer doesn’t have enough information to rank all the possible intents and won’t return the foremost popular intents for these keywords. Sometimes we put keywords and phrases as training examples within the hottest intents to form sure the favored intents rank higher for those keywords.
Informal language (e.g. acronyms and emojis) and incorrect grammar also are quite common in user queries. Assume the interior employees call the corporate holiday schedule “CHS”, the intent identifier won’t know it without a training example says “What is CHS?” is that the same with “What is corporate holiday schedule”.

Intents and sub-intents

Some users don’t want to ascertain the entire holiday schedule. They only got to know the date of a selected holiday. In some situations, it’d worth creating one intent for every holiday. for instance, “Which day can we get off for 4th of July?”, “What is that the holiday schedule for Christmas?” However, after creating 11 more intents, one for every holiday, there also are intents like “What is that the next holiday?”, “Is July 5th a holiday?”. it’s probably not worth adding another 365 intents, one for every day during a year. The intent is typically a gaggle of the many sub-intents. Although I would like my chatbot to know all the littlest intents and answer properly with only the specified information, less and not less, it’s not feasible to try to so with a little team. In practice, we usually group all the intents with an equivalent response together, so that if the response needs an update, we only got to update one intent. Therefore, to coach an intent, we’d like to supply training samples of different sub-intents.

Chatbot tool evaluation tips

I don’t really know exactly what percentage training examples are needed for an intent. It depends on how intelligent your chatbot is, how broad your intents are, and also how high the users’ expectation is for the chatbot.
When choosing chatbot tools, the power to handle typos and grammar mistakes may be a must. it’s hard to coach a chatbot to acknowledge typos and grammar mistakes using training examples because there are too many typos and you can’t enumerate all of them. However, typos are so common in user queries and if your chatbot cannot handle them, it’ll cause a very bad user experience. When a chatbot responds “Sorry I don’t understand.” Users won’t blame themselves for his or her typos and grammar mistakes, they’re going to just say the chatbot isn’t intelligent.
The ability to know common synonyms is additionally vital . for instance if a training example has three words and every word has three common synonyms. Without the power to know common synonyms, you would like to supply 3 * 3 * 3 = 27 training examples in total, just to hide the essential synonyms. Ideally, the chatbot tool should also allow you to define your own all-time synonyms and synonyms within a specific context.
The ability to know sentences with different syntactic structures is good to possess, but I wouldn’t rely on an excessive amount of thereon. Questions that look very different but mean similar things usually got to be added as training examples.
The ability to know sub-intent is never seen within the current chatbot tools, especially if the chatbot is trained only with questions. If a chatbot is trained on both questions and responses, and therefore the responses have traces of the sub-intents, there’s an opportunity that the chatbot can understand sub-intents without additional training examples.
The ability to handle ambiguous intents will improve the user experience also. When the chatbot cannot determine which intent it’s, why not provide multiple options for the user to settle on from? it’s not practical to ask users to always express their intent clearly with full sentences and excellent language.
Different intents require different numbers of coaching examples. it’s fine to possess a max number of coaching examples, but 10 is just too small in my opinion. Dialogflow allows 250 training examples for every intent. It doesn’t mean all intent will have 250 training examples, but a minimum of we are allowed to supply more when needed.
I have heard repeatedly from different chatbot vendors that their tool only needs one training example for every intent. At first, I got really excited when hearing it. Now it’s an enormous red flag on my behalf of me. I prefer vendors being honest about what their chatbot can handle and show us the ways to optimize than saying how easy it’s to create a chatbot with their tool which seems to be useless.

source: Feedspot

Why NLP Technology is a Promising Technology?

Top 10 Artificial Intelligence- AI Softwares

How useful was this post?

Click on a star to rate it!

Average rating 0 / 5. Vote count: 0

No votes so far! Be the first to rate this post.

Give your views

This site uses Akismet to reduce spam. Learn how your comment data is processed.