#2 Natural Language Processing and Machine Learning: The core of the modern, smart chatbot

This article is the second part in a series about Chatbots and Machine Learning. Other articles that were published in this series can be found here:

#1: Chatbots: A bright future in IoT?

#3: Machine learning, neural networks and algorithms

In the previous article about chatbots we discussed how chatbots are able to translate and interpret human natural language input. This is done through a combination of NLP (Natural Language Processing) and Machine Learning. The dialog system shortly explained in a previous article, illustrates the different steps it takes to process input data into meaningful information. The same system then gives feedback based on the interpretation, which relies on the ability of the NLP components to interpret the input. Today we will talk about NLP components and what they are able to do.

NLP, the dialog system and the most common tasks

A lot of companies are trying to develop the ideal chatbot, that can have a conversation that is as natural as possible and that it is indistinguishable from a normal one between humans.

The simpler older chatbots, are the chatbots that employ heuristics with pattern recognition, rule based expression matching or very simple machine learning. The important aspect is that these systems are good at comparing a fixed set of rules.

The newer smarter chatbots employ deep learning to not only analyze human input but also generate a response. The response analysis and generation is learned through the deep learning algorithm that is employed in decoding input and generating a response. NLP then also translates the input and output into a textual format that is both understood by the machine and the human.

If you look at the simpler chatbots, any response (provided it was correct grammar beforehand) is void of any grammatical error. This is of course due to the pre-written sentences in the repository. It might however be unable to handle any input it does not recognize because of human grammatical errors or not matching sentences. The newer smarter chatbots are the exact opposite, if they are well “trained” they can recognize the human natural language and can react accordingly to any situation. However, the big disadvantages is that these natural responses require a great amount of learning time and data to be able to learn the vast amount of possible inputs. The training will prove if the bots are able to handle the more challenging issues that are normally obstacles for simpler chatbots.

Depending on the question, these can be long or short conversations. Longer conversations tend to have deeper meanings and multiple questions that the chatbot would have to consider in its extrapolation of the total picture.

Ultimately the tasks that NLP should be able to handle are in the following summary. The following task can be text, speech and even image related:

• SummarizationSummarization deals with summarizing large amount of text in a short but precise explanation. A great example of this is the summary or tl;dr bot from reddit.

• Example of summary bot from Reddit, original source is here.
• Open and Closed QuestionsModern chatbots should be able to answer any question whether it is open or closed. There is for example a huge difference between, “Is London the capital of the UK ?” instead of “Why is London the capital of the UK?”
• ConferenceThis section has to do with relating objects with words. For example “The office is situated in Rotterdam.” the bot has then to be able to confer from other sentences which office is meant. It should be able to connect the previously mentioned office owner to this particular sentence.
• AmbiguityAmbiguity is related to the context and meaning of the sentence. Not only is the bot responsible for correctly associated the meaning with the word, but also some languages are more ambiguous than others. This is especially true when analyzing human speech.
• MorphologyEach language has a different morphology, the chatbot has to able to separate words into individual morphemes.
• SemanticsSemantics is the meaning of sentences or words in the associated human natural language. This section particularly deals with natural language understanding and natural language generation. The ability for the chatbot to translate any human natural language, whether its for creating a response or analyzing questions.
• Text structureRelated to the structure of texts, punctuation and use of spaces. This greatly differs between languages.
• SentimentThe chatbot should be able to detect the emotional polarity of the subject the human is talking about. It should be able to tell from the way the text or speech pattern is presented whether the human is angry, sad or happy.

Today we have discussed older chatbots, smart chatbots and various elements of NLP. In this series, the previous article was about the use of chatbots in various situation, the current article is about NLP and the future article will be about machine and deep learning. Another future item will include programming languages for developing a chatbot.

---