Teneo Query Language Syntax Overview

When analyzing a conversation, we often look at whether certain occurrences took place, as well as their frequencies. However, it is equally important to be able to know when and in which circumstances those occurrences took place.

For example, knowing that someone has bought a product is important, but to get comprehensive conversational insights (and to optimize and analyze the interaction), it is important to assess what happened before and after the purchase, as well as the full sequence of events, and much more.

This is why we have built our own proprietary query language to cater to specific conversational analysis needs. With this language, data analysts and project teams are able to easily query for sequences of events during the optimization process. This is something that can be comparatively cumbersome in other existing query langauges (like SQL).

A TQL query consists of a command, a selection, and, optionally, a set of constraints and/or transformers and meta-parameters. The command is separated from the selection by a space, the selection is separated from the constraints by a colon, the constraints/transformers are separated from each other by commas, and finally, the meta-parameters are included after a space. This is reflected in the pseudo-code example below:

1<command> <selection> : [<constraint1>|<transformer1>],
2[<constraint2>|<transformer2>],... <meta-parameters>
3

In this query,

• The command is the operation that the user wants to carry out;
• The selection indicates what is retrieved and computed by the query. The selection can hold more than one variable;
• The constraints (optional) can be used to filter out undesired data from results, e.g. by specifying a date range or a required variable value, or it can be used to indicate particular relationships among the items, e.g. by indicating that an event should happen before another event;
• The transformers (optional) can be used to generate new data from the source data, e.g. by manipulating date strings, obtaining top N items from ranking lists, or matching strings against Teneo Solution language objects;
• The meta-parameters (optional) are additional operators used to indicate how to organize the query results.

The following are general tips to keep in mind when working with TQL:

• TQL syntax is case sensitive, hence TQL queries must use the correct capitalization.
• TQL syntax supports commenting out parts of the query using // and /* */.

The next sections in this page provide a complete description of TQL commands, constraints, transformers, and meta-parameters, as well as the support commands for sub queries and saving query results.

Query constructs

The structure of Inquire data (sessions containing transactions, transactions containing events, and the three of them containing properties) is reflected in the TQL syntax, which requires an explicit construct marker indicating the path to reach any property in the data. The following notations are used:

• s refers to the session.
• s.<attribute_name> refers to an attribute of the session, e.g.: s.beginTime
  .
• t or s.t to refer to a transaction. A session can consist of multiple transactions;
• t1, t2, t3, etc. refer to different transactions. The digit does NOT indicate order of precedence.
• t.<attribute_name> refers to a transaction attribute, e.g. t.time
  .
• e, t.e or s.t.e to refer to an event. A transaction can consist of multiple events.
• e1, e2, e3, etc. refer to different events. The digit does NOT indicate order of precedence.
• e.<attribute_name> refers to an event attribute, e.g. e.type
  .

Querying events

Events’ properties are identified using the e. construct.

For example, the fname property of an event is referred to like this:

1e.fname
2

A notation digit can be added to the construct to differentiate different events. In the example below, fname and userInput are properties of the same events:

1e.fname, e.userInput
2

In the next example, on the other hand, fname and userInput are properties of two events that may not be the same one (although they can be):

1e1.fname, e2.userInput
2

Note that the digits here do not imply order or sequence, merely difference. To specify the precedence order of events, use Skip-constraints (described in another section of this document) or the index property.

Querying transactions

Transactions are queried using the t. construct, which has two uses: to access a property at the transaction level, and to constrain a set of events to occur on the same transaction.

The example below specifies that duration is a property of a transaction:

1t.duration
2

The following example refers to two event properties that co-occur in different events within the same transaction:

1t.e1.fname, t.e2.userInput
2

Notation digits can also be added to the t. construct – e.g. t1, t2 – to refer to distinct transactions. The following example specifies that duration and time are properties of two distinct transactions:

1t1.duration, t2.time
2

Similarly, it is possible to combine independent transactions and events in a single construct. The example below refers to two independent transactions (t1 and t2). Properties fname and userInput are obtained from two independent events (e1 and e2) that co-occur in the same transaction (t1), whereas the answerText property is obtained from event e3 in transaction t2.

1t1.e1.fname, t2.e2.userInput, t2.e3.answerText
2

Note however that the above TQL selection is not allowed in Inquire because it requires additional constraints. See the section Computational Costs of TQL below for more information about this constraint.

When the t. construct is not present, a query will search for all events matching a constraint, regardless of the transaction they belong to. For example:

1e1.fname, t1.e2.userInput, t2.e3.answerText
2

This example does not constrain the event e1 to belong to the same transaction as e2, and so all the events in the data will be considered.

Querying sessions

Session level properties are accessed by means of the s. construct. The TQL query is automatically scoped to a single session at a time, so this is the only construct that can be used to refer to session properties. For example:

1s.beginTime
2

Note that this means that it is not possible to use digits – such as s1, s2 – to refer to two different sessions.

Querying backstage

Query scope

Teneo Inquire considers all elements in a single LDS. When a query is executed, Teneo Inquire looks into all the sessions, transactions, and events, as well as user-generated data, such as augmenters, saved results, and solutions that are available in the LDS.

As mentioned previously, a TQL query is automatically scoped to work within individual sessions. This means that an individual query can operate across transactions and events within a single session at a time (querying elements intra-session) but does not perform searches in elements that belong to two different sessions (querying inter-session).

Computational costs of TQL and good practices

As mentioned above, it is possible to combine properties from independent events and transactions by adding digits to their construct – e1, e2, t1, t2, etc. This TQL ability is very powerful but it also has computational implications given that all events and transactions will be taken in consideration for every distinct construct. For example, consider the following construct:

1t1.e1.fname, t1.e2.userInput
2

Here, e1 and e2 are explicitly constrained to belong to the same transaction. Such a construct takes into consideration all events having the property fname in a transaction and computes the cross product with all events having the property userInput within the same transaction.

In contrast, consider the following example:

1e1.fname, e2.userInput
2

This example computes the cross product between all events having the property fname and those having the property userInput, regardless of the transaction they belong to.

Therefore, the more distinct constructs that are used, the more computationally demanding the query is. Hence, indicating which transaction an event should belong to is highly recommended.

In fact, Teneo Inquire includes a query validation mechanism that checks for potentially dangerous queries that may be highly demanding. If such a query is identified, Teneo Inquire rejects it and does not execute it, and Studio displays a warning message informing the user the query has not been carried out. The user will need to review the constructs in both the query selections and the constraints and adjust them where possible.

A good practice is to include the event type in the queries; this reduces the number of comparisons required to compute the results. For instance, when selecting the userInput variable in the query, the type==”request” constraint could be added:

1la e.userInput: e.type == "request"
2

When selecting the answerText variable, the type==”response” constraint could be added:

1la e.answerText: e.type == "response"
2

Similarly, when selecting properties from a path event, it can be constrained by means of the pathType:

1lu t1.e.fname : t1.e.pathType=="raise-flow"
2

Working with variables

There are several type of variables logged in Inquire data: flow variables (fv), session variables (sv), log variables (lv), and annotations variables (annotation.variables). This section shows the difference among them and how to query them.

Session and flow variables

Teneo Inquire logs the values of these variables according to their type: string, integer, etc. However, a variable can be assigned different types in different events. In order to enable Teneo Inquire to deal with varying types, variable names are prefixed according to the table below. Also, flow and session variables are prefixed in Inquire with fv: and sv:, respectively. Hence, the full name of a variable in Inquire becomes fv:TYPE:VARIABLE_NAME for flow variables and sv:TYPE:VARIABLE_NAME for session variables.

As seen in the table, arrays are prefixed with two values: a: which indicates the variable contains an array, and the TYPE: which indicates the type of the values in the array, e.g.: a:f:myArrayofDouble

TQL queries can be constrained using array variables by using the set-constraint (== in) operator. For example:

1la e.fv:a:f:myArrayofDobule : e.fv:a:f:myArrayofDobule == in {0.0,1.0,2.0}
2

Log variables

These variables are logged at transaction level and they can only contain a String, though their contents represent an Object. Hence, they do not need to be prefixed and they can only be constrained by means of string operators. Example:

1la t.lv:intVar : t.lv:intVar ~= ".*\d.*"
2

Annotation variables

These variables can only happen within an annotation. They are logged as a map where each property is a key-value pair of string-object, i.e., the key is a string and the value is an arbitrary object.

Hence, annotation variable names are also prefixed according to their type, the same way flow and session variables are. Note that strings in JSON are stored with quotes. Examples:

1la e.annotation : e.annotation.variables.f:confidence > 0.5
2

1la e.annotation : e.annotation.variables.n:order == 0
2

1la e.annotation : e.annotation.variables.s:name ~= "\"NN.POS\""
2

Working with annotations

Annotations are a special type of data in Teneo Inquire organized in a nested structure.

1Map annotation {
2    Long sentence-index,
3    Map<String,String> variables,
4    Long[] word-indices,
5    String name,
6    String action
7}
8

The String value in the attributes of the variables property represents a JSON object with nested structures such as numbers, strings, maps, or arrays. They can be used in the same way as session or flow variables.

It is possible to use Inquire to query some of the Annotation data, for instance:

Annotation properties:

1lu e.annotation.sentence-index, e.annotation.variables,  
2e.annotation.word-indices, e.annotation.name,  
3e.annotation.action
4

A specific property in the variables map

1lu e.annotation.variables.f:confidence
2

Numeric conditions

• Over annotation.sentence-index:

1la e.annotation : e.annotation.sentence-index == 0
2

1la e.annotation : e.annotation.sentence-index == in {0,1}
2

String comparisons

• Over annotation.name and annotation.action:

1lu e.annotation : e.annotation.action == "added",  
2e.annotation.name == "VB.POS"
3

1lu e.annotation : e.annotation.action == "added",  
2e.annotation.name ~= ".*SENTENCE_WORDS.*"
3

Numeric comparisons on annotation variables properties with numbers:

1lu e.annotation.variables.f:confidence :  
2e.annotation.variables.f:confidence > 0.8
3

1lu e.annotation :  
2e.annotation.variables.n:sentenceLength == 1
3

String comparisons on annotation variables properties with strings:

1lu e.annotation :  
2e.annotation.variables.s:words ~= ".*[hH]ello.*"
3

1lu e.annotation :  
2e.annotation.variables.s:words == "hello"
3

Array comparisons over annotation word-indices properties (note that values are numeric):

1lu e.annotation : e.annotation.word-indices ~~ 1,  
2e.annotation.word-indices ~~ 2  
3

Note the use of the contains operator; this query returns annotations that have the values 1 and 2 in the word-indices array.

1lu e.annotation : e.annotation.word-indices == in {0,2}
2

Note the use of the range operator; this query return annotations that have exactly the array [0,1,2] in the word-indices array.

Search for empty values:

1lu e.annotation :  
2notexists e.annotation.variables.s:words
3

→ returns annotations that have no words property in the variables.

1lu e.annotation : notexists e.annotation.word-indices
2

→ returns annotations that have empty arrays in word-indices.

1lu e1.id, !e1.annotation.word-indices :  
2notexists e1.annotation.word-indices  
3

→ returns empty arrays AND events without annotations.

However, Teneo Inquire has some limitations when dealing with annotations. It does NOT allow certain constraints such as:

Array comparisons over properties of the annotation variables

1lu e.annotation : e.annotation.variables.a:s:words ~~ "hello",
2e.annotation.variables.a:s:words ~~ "Hello"
3

Regular expressions over arrays or maps in the annotation variables:

1lu e.annotation : e.annotation.variables.a:s:words ~= ".*[hH]ello.*"
2

Use of ranges with arrays:

1lu e.annotation : e.annotation.word-indices == in {0..2}
2

Check for empty values:

1lu e.annotation : e.annotation.word-indices == {0..2}
2

Working with dates

Teneo Inquire data includes some date properties, such as the session begin time, which are all logged in Z zone using the format: yyyy-MM-ddTHH:mm:ssZ (e.g.: 2018-12-12T09:07:26Z). However, in the front end, dates are displayed using the locale of the host.

On the other hand, users can assign Date values to variables in Teneo using the scripting functionalities, e.g. assign a Date value to a flow variable using the on-drop script. These dates can be stored using the user's preferred date/time format. However, Teneo Inquire will identify and store as Date those that comply with ISO-8601, otherwise they will be stored as a String. The following table summarizes the date/time formats that Teneo Inquire can handle as dates: