Teneo Sentiment and Intensity NL Analyzer
Introduction
The Teneo Sentiment and Intensity NL Analyzer is Artificial Solutions' proprietary resources containing off-the-shelf building blocks to be used for detecting positive and negative sentiment as well as intensity in user inputs.
Companies have a high interest in learning about the interactions their customers make via support channels, through tweets, emails or chat with conversational AI applications. To fulfil this need, companies turn to one of the many suppliers of sentiment analysis; the sentiment analysis solutions today take mainly a statistical approach as well as using syntax analyses to make their conclusion about the content. The statistical approach has its limitations and often does not consider other important aspects of the communication, such as intent, thematic errors or intensity of the communication. At Artificial Solutions this has been attempted to resolve by presenting clear components to answer the high linguistic demands placed on such analyses.
The objects in the Teneo Sentiment and Intensity NL Analyzer can be used in Teneo Studio solutions in Listeners, Triggers, or Transitions to keep track of various occurrences during the conversation and to support various actions:
- Flag specific dialogue events,
- influence both proactive and direct responses or actions, and
- compute or track user sentiment.
Available languages
The Teneo Sentiment and Intensity NL Analyzer is currently available in the below listed languages.
| Language | |
|---|---|
| Chinese (Mandarin) | German |
| Dutch | Italian |
| English | Swedish |
Teneo Sentiment
The Teneo Sentiment NL Analyzer categorizes sentiment into negative and positive categories; the category of neutral sentiment is given by an absence of negative and positive sentiment. Additionally, the determination of sentiment is scaled into two categories:
- high confidence (clear, specific statements)
- low confidence (partial statements or keywords that can vary according to context).
The language rules used to determine sentiment apply the complete range of Teneo Linguistic Modeling Language as well as Part-of-Speech (POS) tags, to support high-performance precisions and are varied and numerous enough to achieve industrial-level recall.
Language Objects
The main objects to use to detect sentiment, when not interested in confidence levels, are:
| Sentiment object | Example variants |
|---|---|
SENTIMENT_POSITIVE.INDICATOR | I like you / It’s a lovely day / You are beautiful / You make a good point |
SENTIMENT_NEGATIVE.INDICATOR | Go to hell / I don’t like you / Fuck you / This is the worst thing that ever happened |
SENTIMENT_NEUTRAL.INDICATOR | I am going home / It is raining / Winter is coming / You look busy |
Please note that a number of Language Objects are used internally by the resource to achieve a better structured approach to recognizing sentiment, but these objects are not relevant for external use and will therefore not be documented here.
Use of the objects is straightforward: if the object is true, the applicable sentiment was detected in the input. Note that it is also possible for an input to contain both negative and positive sentiment, so both objects might be true. Sentiment is defined as neutral if neither positive nor negative sentiment was detected.
For more complex sentiment detecting involving confidence levels, the following objects are available:
| Sentiment object | Type of input |
|---|---|
SENTIMENT.INDICATOR | Used to set NLU variables containing sentiments and confidence levels found |
SENTIMENT_NEGATIVE_HIGH_CONFIDENCE.INDICATOR SENTIMENT_POSITIVE_HIGH_CONFIDENCE.INDICATOR | Statements, phrases |
SENTIMENT_NEGATIVE_LOW_CONFIDENCE.INDICATOR SENTIMENT_POSITIVE_LOW_CONFIDENCE.INDICATOR | Emojis, hashtags, keywords |
The sentiment objects do not detect strength or intensity of the sentiment, e.g. I hate you versus I really really hate you!! although the latter statement is clearly more extreme than the first. The degree of sentiment can be determined by combining the detected sentiment with a detection of intensity described in the Teneo Intensity section.
Detecting sentiment
Within a Teneo solution, sentiment can be detected via Listeners and used for various purposes, for example:
- tagging specific transactions or sessions (for later analyses)
- defining specific actions during the session
- tracking changes in sentiment, and
- computation of sentiment score for the entire conversation.
Tracking sentiment score
One way to use sentiment objects is to create per sentiment (negative and positive) Listeners that keep track of the items during each transaction. The user of Teneo Studio may use these if not interested in confidence level or it is also possible to take an approach of allowing one sentiment to supersede the other.
By maintaining a running total of sentiment hits, it is possible to register and respond to trends, for example, if a user is especially negative, handing over to a live agent might be a good option. This can be done in a Flow using a Prompt Trigger.
Detecting sentiment and confidence level
If concerned about confidence levels for each sentiment, this can be detected using the Language Object SENTIMENT.INDICATOR, which checks the sentiment objects and returns the sentiments found along with a confidence score. If positive or negative sentiment were not found, it returns neutral.
The Studio user can define a single Global Pre-listener for sentiment and use the object as follows, taking care of retrieving the NLU variable that is set:
The variable sSentiment is a list which in some cases may contain two sentiments; the value of the entries can be as follows:
Positive High ConfidenceorPositive Low Confidence(only one of these)Negative High ConfidenceorNegative Low Confidence(only one of these)Neutral(only if positive nor negative were not found).
Customization
The sentiment analysis objects have been prepared to allow easy customization by users, for instance, if a specific product-name is erroneously classified as negative sentiment, four Language Objects can be created locally and filled with items to include or exclude from sentiment analysis. The Language Objects in question exist as placeholders in the NLU Ontology and Semantic Networks.
| Language objects |
|---|
SENTIMENT_NEGATIVE_EXCLUDE.PROJ |
SENTIMENT_NEGATIVE_INCLUDE.PROJ |
SENTIMENT_POSITIVE_EXCLUDE.PROJ |
SENTIMENT_POSITIVE_INCLUDE.PROJ |
Limitations
Absolute detection of sentiment is accepted as an unobtainable goal. Existing systems must deal with this limitation. Sentences containing two conflicting sentiments at the same time, e.g. Sometimes I feel so happy, sometimes I feel so sad, are especially difficult to detect.
Accuracies of 70-90% can be considered acceptable. 90% is often quoted as the level at which two independent human raters can agree on sentiment. Since sentiment is not absolute, an absolute detection of sentiment as a goal is not a wise choice. False positives and false negatives will always exist, as language is infinite and often ambiguous. In a rule-based approach each new rule that is added while decreasing the detection of false negatives will invariably increase the incidence of false positives.
Teneo Intensity
Users say things with different intensity; the statement I hate you in itself does not contain cues as to the vehemence of the sentiment, but conclusions can be drawn as to vehemence by looking for cues that enhance the statement beyond the pure informational content. If a user says I hate you!!, I despise you or I really, really hate you, it can be said that these sentiments are expressed with higher intensity. It might not be possible to know with certainty whether the expressed intensity corresponds with actual intensity of emotion, but it does correspond to the socially accepted way of representing an expression of an intense emotion.
Note that some interpretations of sentiment analysis do not separate intensity from sentiment; at Artificial Solutions this was not found a good approach because, firstly, it increases the difficulty in distinguishing categories and, secondly, it does not consider that neutral statements can also be expressed with different intensity.
Language Objects
In an examination of thousands of tweets and user inputs, six distinct categories of intensity are defined which are for the most part easy to detect with the Teneo technology.
| Intensity object | Description | Example variants |
|---|---|---|
INTENSITY.INDICATOR | Collects all intensity LOBs, returns a list with all types of detected intensities | |
INTENSITY_CAPITALIZATION.INDICATOR | Capitalizing single words, phrases or an entire statement to show emphasis | I AM NOT HAPPY ABOUT THIS / This is NOT good / You DO NOT want to shop there |
INTENSITY_DUPLICATION.INDICATOR | Using repetition of words or phrases to show emphasis | I know, I know / I always always liked you / Run run run |
INTENSITY_EMOJIS.INDICATOR | Using emojis, emoticons or emotion-abbreviations to express intensity | I like you / Rotfl / Lol / 😠 / 🙂 |
INTENSITY_PUNCTUATION.INDICATOR | Using punctuation to express intensity | I love your hat!! / You’re the greatest!!!!!!!! / Why are you doing this?? / You said what!?! |
INTENSITY_SENTENCES.INDICATOR | Using one-word sentences to emphasize a point | It just. won’t. work. / I am so glad. you. came. / Please. go. away. |
INTENSITY_VOCABULARY.INDICATOR | Using heavily weighted words, superlatives or intensity modifiers | I am very happy / This is awesome / You’re the greatest |
Detecting Intensity
In a solution, a listener can be created to fill a global variable with the types of intensity cues a user input contains. The INTENSITY.INDICATOR object returns a list containing the types of intensity via an NLU variable, and if no intensity is detected, an empty list is returned. The list's size may be queried in the log data to determine the degree of intensity, e.g. three types of intensity cues are present. This in turn can be combined with sentiment to determine especially happy or dissatisfied users.
The values in the intensity list can be as follows:
- capitalization
- duplication
- emoji
- punctuation
- sentences
- vocabulary
Since the definition of intensity includes six unique types, one can consider that each input will have an intensity score ranging from 0 to 6. As a rule, it is rare to find more than four types of intensity grouped into a single statement; even three different types are unusual to find. However, occurrences of two different intensities simultaneously are quite common.
When combining intensity with sentiment, it is possible to locate inputs that stand out in some way, for example if intense negative sentiments are expressed, it might be used as a trigger for human intervention via live chat, etc. or to activate feedback, allowing the user a chance to give constructive criticism. If the system takes emphatically upset users seriously, it may defuse the situation as well as deliver information about how to prevent such situations in the future.
A beneficial side-effect of combining intensity with sentiment is that the subset of inputs is likely to contain fewer false positives, as sentiment and intensity often occur together. This can be used as an additional factor in determining confidence level.
Limitations
Five of the six categories (capitalization, duplication, emoticons, punctuation and sentences) can be detected to near perfection, as they are implemented algorithmically using inline groovy scripting. Intense vocabulary is more reliably determined than sentiment, as the vocabulary is limited to a specific, well-defined mean of expressions, superlatives, etc. Capitalization is the trickiest, as acronyms are often capitalized, but are not cues to intensity, as are names and personal pronouns. But as the number of capitalized words in an input increases, so does the likelihood that it is about intensity.
Sentiment, Intensity and Teneo Query Language
In the Log Data tab (Teneo backstage > Optimization > Log Data) it is possible to use sentiment and intensity objects together with the Teneo Query Language (TQL) to perform direct queries on log data; it is also possible to adorn transactions to facilitate queries.
Adorners can be created, for example to apply to the user input the sentiment classification provided by the Sentiment and Intensity resource and could help to illuminate the causes of negative sentiments and to highlight when users aren't satisfied with a particular response given by a conversational AI application and they complain.
For information on how to create, apply and query adorners, please see the Adorners section.