Teneo Developers

Teneo Offensive Language Detector NL Analyzer


The Teneo Offensive Language Detector is an Artificial Solutions' proprietary resource containing off-the-shelf building blocks to be used to detect abusive and controversial language in user inputs.

Abusive and controversial inputs are a necessary evil that NLI systems need to deal with; by abusive language we mean severe insults, profanity, violent threats, etc., while other subject areas may be deemed as too controversial or sensitive to go into, e.g. subjects such as suicide, abortion or death in the family. It is a good strategy for a conversational AI system to show that such unprofessional language and subjects are not acceptable as a response that doesn't respond to the language or subject in an appropriate way can be a poor reflection of the system as well as alienating the user.

The Teneo Platform provides components for the detection of abusive inputs as well as controversial subject matter, for use in Listeners, Triggers or Transitions, enabling the system to respond as the situation requires.

Note that offensive language and negative sentiment should not be confused as identical concepts; abusive language or certain controversial inputs may be flagged as negative sentiment, but a one to one correspondence is not given. The Teneo Offensive Language Detector may be seen as supplementing the Teneo Sentiment and Intensity NL Analyzers.

The Teneo Offensive Language Detector is distributed as a set of Language Objects available in the English Teneo NLU Ontology and Semantic Network.

Available languages

The Offensive Language Detector is currently available in the below listed language:


Teneo Abusive Language Detector

Our vast experience in building conversational AI systems has unfortunately exposed us to a creative variety of abusive language. Many users attempt to test a system by seeing how it will respond to profanity, and some will even persist in misusing the system by bombarding it with abusive inputs only. On the other hand, if a system is not well-prepared in its topic area, users may become frustrated and switch to abuse. Abusive language occurs in many different situations and contexts. Our detection of abusive language used the following classification:

  • Hate speech: racist of homophobic inputs, sexist language
  • Profanity: general profanities, the "f" word, etc.
  • Sexual harassment: sexual, pornographic inputs
  • Threats of violence: murder, physical abuse, etc.

By offering a detailed diagnosis of the type of abuse given to a system, we enable the developer to take appropriate type-specific actions.

One approach to abusive language in CAI systems is to, for example, give the user two warnings; if the user does not drop the abuse they are given a "time out" in which no questions are answered until user apologizes. If, after the apology, the user reverts to abusive language, the conversation is terminated and the session slot made available for a serious user.

Language Objects

Abusive language can be detected using the single Language Object ABUSE.INDICATOR. The object returns two NLU Variables, list objects containing the type of abuse detected along with a confidence level for the detection. In general, a low confidence refers to a single questionable word, whereas high confidence items will tend to be phrases or specific unambiguous terms. The following table illustrates the values returned by the ABUSE.INDICATOR object.

NLU variableTypePossible values
lob.sAbuseString []{ “hatespeech” | "profanity" | "sexual" | "violence" }
lob.sConfidenceString []{ “high” | "low" }

An overview of the main Language Objects available, including the specific types is as follows; usually it is enough to use ABUSE.INDICATOR, but if interested in only a specific type of abuse, one can apply the individual object as well.

Abusive language objectSample inputs
ABUSE_HATESPEECH.INDICATORYou son of a bitch / I hate n******
ABUSE_PROFANITY.INDICATORf*** you / kiss my ass
ABUSE_SEXUAL.INDICATORSuck my d*** / I want to sleep with you
ABUSE_VIOLENCE.INDICATORI’ll smash your face / I could kill you

Please note that a number of Language Objects are used internally by the resource to achieve a better structured approach to recognizing abuse, but these objects are not relevant for external use and will therefore not be documented here.

Since language is sometimes ambiguous the areas of abuse are not always clearly delineated, a single abusive input may produce multiple determinations of mixed or equal confidence levels. This is not a bug but a feature.


If project-specific customizations are required, a local ABUSE.INDICATOR Language Object can be created, and the user can add the additional variations or categories as needed as well as follow the NLU Variable structure.

Teneo Controversial Language Detector

When building a conversational AI system, there is often concerns that the system may respond in inappropriate ways when users broach sensitive off-topic areas; it might be a test, but if a user says I want to kill myself or My mother just died, it can make the system look very bad if it responds in a flippant manner. Other inputs might imply criminal activities and should in no way be responded to in ways that encourage the activity, e.g. Should I kill my husband? With the Controversial Language Detector, the solution developer can redirect such inputs to Flows that deal with them, as well as suppressing certain Flows that would be inappropriate in sensitive contexts. A results is that the Controversial Language Detector shows end-users that their inputs are taken seriously.

Language Objects

The detection of controversial language is designed to focus on specific areas often seen in past systems; the developer may use the Language Object CONTROVERSY.INDICATOR to detect all of the areas in question. The Language Object returns a list via an NLU Variable of all types of controversy that were detected in the input. This Language Object does not make a distinction of high versus low confidence.

NLU variableTypePossible values
lob.sControversystring []{ “abortion” | "crime" | "death" | "fascism" | "sex" | "suicide" | "terrorism" }
Controversy language objectSample inputs
CONTROVERSY_ABORTION.INDICATORWhere can I get an abortion? / I want to end my pregnancy.
CONTROVERSY_CRIME.INDICATORWhere can I hide a body? / I want to buy some LSD.
CONTROVERSY_DEATH.INDICATORMy dog just died. / My grandmother just passed away.
CONTROVERSY_FASCISM.INDICATORHeil Hitler! / The Holocaust never happened.
CONTROVERSY_SEX.INDICATORI am looking for a prostitute. / I am a pedophile.
CONTROVERSY_SUICIDE.INDICATORI want to slit my wrists. / How can I kill myself?
CONTROVERSY_TERRORISM.INDICATORHow do I build a bomb? / What happened on nine eleven?


Since the definition of controversial subjects will vary from one project to the next, the CONTROVERSY_CUSTOM.PROJ Language Object has been developed which developers can adjust to include new categories or variants of the existing categories. The Language Object returns a default categorization in lob.sControversy of "customized", but this category can and should be expanded depending on the content added.

To add customizations, create a local Language Object with the name CONTROVERSY_CUSTOM.PROJ making sure to add the corresponding NLU Variable; the developer can also copy and paste the content from the Language Object in the Teneo NLU Ontology and Semantic Network.