Teneo Studio

Teneo Studio Desktop

Conversational Language Understanding

Teneo 7.3 brings Microsoft's Conversational Language Understanding (CLU) closer to the Studio developer with the possibility to create and maintain CLU intent models directly in Teneo Studio.

Conversational Language Understanding (CLU) is a service provided by Microsoft in the Cognitive Services layer in the Azure ecosystem. The service allows developers to train and query Natural Language Understanding (NLU) models using state-of-the-art, transformer-based, pre-trained language models and, furthermore, CLU provides multilingual capabilities.

Teneo allows to set up the account to work with CLU directly in Studio, where classes, training data examples and test data examples are added in the Class Manager and the CLU model is trained in the CLU Manager.

CLU Settings

Before opening a solution, Teneo Studio Desktop incorporates a new Account tab which displays the account information, i.e., name and Id of the Teneo account, and allows to introduce the CLU Settings required to work with Conversational Language Understanding models in Teneo Studio; note that the Studio properties file also include new properties to allow Studio to connect to CLU.

To add a Resource Name and the API Key, simply click Edit; once the values are added, click Save to save the credentials to Teneo.

Notes on CLU Credentials

The CLU Settings - or credentials - consist of the Azure Resource Name and the API Key.
In Teneo, it is the users themselves who mange the CLU credentials although it is limited to users with either the "Teneo admin" or the new "Modify account settings"permission. After saving the CLU credentials to Teneo, these are stored encrypted and the API key is no longer visible in the Account page nor is it possible to read them (unencrypted) via the API. With one of the beforementioned user permissions it is though possible to overwrite and update the credentials as needed.
When a Resource name and API Key is provided in the Account's page, all solutions within the Teneo account may make use of CLU.

In summary:

• The Teneo user must provide the CLU credential themselves
• Only Teneo users with correct permissions can manage (including update/overwrite) the CLU Settings
• CLU credentials are uploaded to Teneo where they are stored encrypted
• No matter which permissions the user might have, it is not possible to read the credentials (unencrypted) neither via the API nor the Studio frontend
• All solutions in the Studio account can use CLU once defined on the Account page.

CLU Manager

When the CLU Settings are defined and a solution opened, the CLU Manager is accessed via the button in the top ribbon of Teneo Studio Desktop; the CLU Manager allows to train CLU models directly in Studio based on the training data examples added in the Class Manager without the need for any integrations nor work in other systems.

Note that the CLU Manager is only available when the CLU Settings are defined on the Account page as described in the previous section.

The fist time the CLU Manager is opened, an informative message encourages the developer to train a CLU model and training a new model is as simple as clicking the Train button in the top ribbon.
Once one or more models are trained, the CLU Models list is populated with information related to each training and the far left-hand side provide information related to the training date, the user who initiated the training as well as the model Id and the expiration date of the model (as defined by Azure). In the middle, information is displayed when the mode is used in a publish environment, while the far right-hand side displays information as to which solution version the model is assigned (i.e., latest and / or stable).
Double-clicking the information related to the publish environment(s) allows to navigate directly to the Publish Environment in the backstage of Teneo Studio.

CLU Model Training

As mentioned above, training of a CLU model is done based on the classes and training data examples provided in the Class Manager and is as simple as opening the CLU Manager and clicking Train in the top ribbon. When a model training starts, the status can be seen at the top of the CLU Models list and different status messages are available depending on the progress:

• Importing in progress
• Training is queued: happens because models usually are queued on the CLU server before the training actually starts
• Training in progress: is displayed while the training is running with a progress bar, the duration (until now) and the expected remaining time
• Cancelling: is shown if the user clicks the Cancel button
• Training cancelled: is displayed when a training is successfully cancelled
• Trained CLU model available: successful training of CLU model, the model can now be assigned and used in the solution.

Only one CLU model can be trained at a time and if the developer performs changes in the Class Manager to either Class names and / or training examples during a running training, the training will automatically be cancelled.

Model Training Info

Model trainings in CLU can be rather time-consuming and therefore, to let developers keep an eye on the status while working on other solution areas, the status bar in the lower, left corner of Studio Desktop windows now displays information related to the training status; this includes a coral colored spinning wheel when a training is running, a green checkmark when a training has finished, a warning triangle in case a training failed, etc.

Duplicated Training Examples

CLU doesn't allow for duplicated training data examples so, in Teneo, duplicated examples are raised as Suggestions (see Optimization Suggestions further below) both when duplicated examples are detected in different classes and in the same class.
On top of this, duplicated examples within the same class are automatically removed by Teneo before sending the data to CLU, but even so, the developer must manually correct duplicated examples detected in various classes as the training of a new CLU model is prevented until no such duplicated examples exist in the solution.
In case of duplicated examples in different classes, the Train button is disabled in the CLU Manager and a message encouraging the user to review the Suggestions displayed.

The detection of duplicated examples is - as of Teneo 7.3 - case insensitive, meaning that, for example, "LOL" and "lol" are detected as duplicates of one another.

Also see the section Optimization Suggestions further below.

Assign CLU Model to Solution

When a model training has finished, the new model becomes available at the top of the CLU Models list and, to be able to use the model in the solution, the developer must assign it to the solution by selecting the model and clicking Assign in the top ribbon.

When the model is assigned, Used By information is displayed in the right side of the CLU Models list and also in the status bar of Teneo Studio windows; also see Assigned Model Info below.

Once a CLU Model is assigned to the solution, Teneo features such as the Auto-test, Tryout, Classifier and so on, automatically make use of the CLU model. Of course, the developer can at any time switch back to Learn if so desired by simply clicking Unassign in the top ribbon of the CLU Manager.

Assigned Model Info

The status bar in the lower, left corner of the various Studio Desktop windows (such as for example, the main window, the Tryout, Flow window, Language Object window, Class Manager, Solution Log, etc.) now provide information related to the assigned intent classifier for both the stable and the latest solution version. The possible values are CLU, Learn, and - in case of not making use of a stable model - None. For example, in the below image, Learn is used in the latest solution version while CLU is used in the stable solution version.

Delete CLU Models

CLU models can be removed from Teneo Studio Desktop by selecting the model to remove and clicking Delete in the top ribbon of the CLU Manager. CLU models in use, i.e., used in the latest solution version, in a stable solution version or in a published solution, cannot be removed.
In case a CLU model was used in older publications, the user should consider carefully if the model can be deleted: if the model is deleted it is no longer available if the user re-publish the old solution publication again in the future.

Note that when a CLU model is deleted from the CLU Manager, the model is removed both from Teneo and from the CLU server.

Valid CLU Models

A CLU model is always trained on the latest solution revision and it is only valid and in an assignable state as long as the training examples in the Class Manager does not change.
When - or if - updates are performed to the training data examples, then any unassigned models or a model used by the latest solution revision are automatically invalidated and a new CLU model must be trained again based on the latest changes and the, now, current training data. When this happens, Teneo automatically defaults to the Learn Classifier for the latest solution version. If a model is used in the stable solution or has been published in one of the publish environments of the solution, then that particular model is still available for the stable solution/in any published solution.

In case of performing changes which will invalidate the currently assigned CLU model, Teneo Studio will display warning messages informing the developer about the CLU model becoming unavailable; again, if the changes are saved, Teneo Learn will be used as the intent classifier until a CLU model is trained and assigned to the solution anew.

More details about working in the CLU Manager in the reference documentation

Stable Intent Model Assignment

The Intent Model Assignment is a versionable solution document which is created when the user assign a CLU model to the solution; currently only CLU models can be assigned as intent model and if no CLU model is assigned, Learn is used. History - and thereby versions - are created when the assignment of the intent model changes, either by assigning a CLU model or by unassigning a CLU model (and reverting to Teneo Learn).

A version of the Intent Model Assignment can be set as stable.
In a stable solution version, the model assigned as the stable Intent Model Assignment is used, but unlike other solution documents that can be set as stable, setting an Intent Model Assignment as stable is not mandatory for a stable solution: if no stable version of the Intent Model Assignment exists, Learn is used!
This is underlined by the fact that it is not possible to set an Intent Model Assignment which uses Learn as stable; only Intent Model Assignments of CLU models can be set as stable. And, to use Learn, the Intent Model Assignment simply needs to be unset from stable.

When trying to set an Intent Model Assignment version as stable, if the CLU model referenced by the Intent Model Assignment no longer exists, then Studio doesn't allow the user to set the particular version as stable.

Similarly to the latest solution version where consistency between the training data and the assigned model is guaranteed by resetting the Intent Model Assignment from CLU to Learn if the training data changes, consistency must also be guaranteed in the stable solution version and therefore, when assigning an Intent Model Assignment version that uses a certain CLU model, all versions of the class documents that were used to train the model are automatically set as stable and class documents that were not used in the model are unset from stable.
Correspondingly, if CLU is used in the stable Intent Model Assignment, setting/unsetting class documents as stable will unset the stable Intent Model Assignment version.

The Intent Model Assignment can be set/unset as stable in the CLU Manager by clicking the History button, selecting the wanted CLU Intet Model Assignment and lastly clicking Set Stable or Unset Stable depending on the desired behavior. If CLU is not available in the Studio account, there is no need to work directly with the Intent Model Assignment as the solution always uses the Learn Classifier.

Multilingual CLU Models for Localization Setup

CLU models are multilingual and can be shared between Master and Local solutions.
CLU models are always trained per solution with a limit of maximum 10 models per solution; however, solutions in a Localization setup can use a CLU model from any of the solutions involved as long as the model is valid for that particular solution, i.e. if all solutions define exactly the same classes (same class Ids).

A model can only be used in a solution if it contains the latest training data of that solution, the model, however, does not have to contain the latest training data from other solutions. Developers can try to train a new model if they think other solutions contain updated training data but no model was yet trained.

Models can only be deleted from the solution that trained it. However, it is not possible to remove models if other solutions still use them. Similarly, it is not possible to remove solutions if the solution contains one or more models that are still in use in other solutions.

If a Master solution has a CLU model assigned and all class are included, during branching the same model will be automatically assigned in the new Local solution.

Class Manager

The Class Manager, in Teneo 7.3, brings support to test data examples and introduces linked test examples! The test data examples are added either manually or by import to classes while the linked test examples are the positive User Intent examples of flow triggers which make use of a Class Match.
Both the test data examples and linked test examples are used when performing Test Data Evaluation of the intent classifier.

Test Data Examples

Test data examples can now be added to a class directly in the Class Manager allowing developers to maintain not only the intent model with its classes and training data examples but also the test data needed to evaluate the intent model. Of course, it is also possible to add test data examples in Studio Web.

To add test examples, simply select the Test Examples tab - next to the Training Examples tab - with the class in edit mode and write the example text in the Enter Test Example field. Press Enter or click Add to add it to the list of examples and continue writing a new example.
Examples can also be moved between the two tabs by simply dragging/dropping the example(s); multi-select by holding Ctrl down while selecting the examples to drag.

The test data examples are searchable in the advanced Search in Teneo Studio Desktop, just as the training data examples and class names.

Linked Test Examples

Linked test examples are the positive examples of User Intent added in a trigger or transition which uses a class as a Class Match. The linked test examples are viewable in the right-side panel of the Class Manager and the developer can by double-clicking navigate directly to the flow where the User Intent example are included.

The linked test examples are used together with any test examples added directly in the Class Manager when running Test Data Evaluations.

If for any reason, the developer doesn't want to use the User Intent examples as Linked Test Examples, in the Flow, untick the Include in tests option below the User Intent examples. Note that this also excludes the User Intent examples when running Auto-test.

Import of Classes

The Import Classes functionality is extended in Teneo 7.3 to support import of both training examples and test examples.
The old format, specifying class name and training example only, is still supported while a new format is available supporting data type, class name as well as test and training data examples.

With the new format, developers can:

• create new class(es) with training data examples
• create new class(es) with training data examples and test data examples
• update existing class(es) with training data examples and/or test data examples.

The new format consists of three columns separated by tab where each line must contain the data type (either TEST or TRAINING), a valid class name, and depending on the specified data type either a training or test data example. Note that the file must still be saved in the tab-separated values format (.tsv).

More details in the Class Manager page

Test Data Evaluation

The Class Performance section in the backstage of Teneo Studio now allows to perform Test Data Evaluations of the model in a solution by testing the provided test examples against the model. Test Data Evaluations can be performed for both Learn and CLU models (depending on which is currently used by the solution) and make use of all test examples added directly in the Class Manager as well as any Linked Test Examples.

To evaluate the classifier by using test data, each class should have at least one test data example (either directly added to the class or as a linked test example) and the functionality is limited to running one test evaluation at a time. Note that duplicated test examples are removed from the test data when running Test Data Evaluation.

Now, to start a new Test Data Evaluation, simply click Run in the right side of the view. Clicking the down arrow on the button allows to select between Test Data Evaluation and Cross Validation when Learn is used as the solution's classifier, i.e., Cross Validation is not available for CLU models. As always, the results can be compared and viewed either in the Class Performance Table or in the Confidence Threshold Graph and it is also possible to compare the results of Test Data Evaluations of both Learn and CLU models.

To differentiate Test Data Evaluations performed on Learn and CLU models, a flamingo is used to visualize a Learn model evaluation while a blue brain icon visualizes the use of a CLU model; note that when hovering over the results more info is displayed in an informative tooltip.

Read more about Test Data Evaluation in the Class Performance page.

Please note that on upgrade to Teneo 7.3 results of Cross Validations from previous Teneo versions are removed; also see Upgrade to 7.3.

Classifier

Teneo 7.3 brings two filters to the Classifier (Optimization area of Studio's backstage) which allow developers to restrict the scope of the displayed data by selecting a Revision Range and/or a Date Range.
Hereby the developer can now load a subset of the available data in the Log Data Source based on the selected revision(s) and/or date(s). Furthermore, the loading of data has been delayed and will only first happen when the developer actively clicks the Run button to load the data into the Classifier.

To load the data in the Classifier view, a query is run in the selected Log Data Source and the views are then populated. In previous versions this would happen automatically when opening the Classifier and selecting a view, but as of Teneo 7.3, to enable the developer to apply filters (i.e., setting revision and date ranges), the Run button must be clicked before results are loaded. Setting the Revision Range and/or Date Range is completely optional, but highly recommended when working with big Log Data Source (i.e., that contains lots of data).

The Reload button allows to reload the LDS and the classes data (but doesn't execute any query) and while running a query, a Cancel button becomes available for the developer to cancel the process if needed.

Suggestions

In Teneo 7.3, two suggestions are now available related to duplicated training data, one for duplicated examples found within the same class and one for duplicated examples found across different classes:

• Error suggestion: Class has training examples duplicated in multiple classes
• Warning suggestion: Class has duplicated training data

Note that as of Teneo 7.3, the detection of duplicated examples is case in-sensitive, meaning that, for example, "LOL" and "lol" are detected as duplicated examples.

When CLU is used as the solution's intent classifier, training a new model is prevented when the error suggestion occurs, i.e., training examples are duplicated across classes, and the developer must review and correct these before being able to train a new CLU model. In case of duplicated examples in the same class, Teneo removes the duplicated examples before sending the data to CLU.

Read more on the Suggestions page.

Tryout

The Learn model updates on Engine start have been optimized, meaning that until now, the model in Engine would always be updated in a second step after Tryout had loaded the solution which would be visible in the model update messages displayed in Tryout, but as of Teneo 7.3 the model update messages will only be visible when the model is actually updated because it has to be trained again. The model update messages, when Tryout is running, will still appear when changes are made to training data.

Standard or Forced Classification

Due to the Engine changes related to the deferred intent classification, Tryout comes with two new buttons allowing the developer to test inputs in two different classification modes: Standard classification mode or Forced classification mode.

The Standard classification mode makes use of the deferred intent classification where Predict annotations are only create if - during the matching process - references to class annotations are detected.

The Forced classification mode allows to re-create the behavior pre-Teneo 7.3, where class annotations are created for all tested inputs independently of whether these annotations were later used during the matching process or not.

Note that it is only in Tryout users can enforce creation of class annotations; all other functionalities automatically use the deferred intent classification.

Messages

The Messages section of the advanced Tryout now includes error messages coming from the Predict Input Processor and covering failed requests to a CLU model at runtime, for example, if the Azure/CLU service is down; also see the Teneo Engine section.

Publish

History

As of Teneo 7.3, Publish History is always saved and the option to select if the publish history should be saved or not is removed from the settings of publish environments.

CLU Model

Information is also displayed in the publish environments when a CLU model was used in the published solution; clicking the Id of the CLU model takes the developer to the CLU Manager.

On publish, a Learn model is always send along with the published solution and in case of issues with the CLU model (e.g., network problems, Azure downtime, or similar), the Learn model is used as fallback instead.

Augmenters Date Ranges

An issue with Augmenters and date ranges was detected in Teneo 7.2 and 7.2 SP1 which implies that:

• Augmenters with a date range created or changed in 7.2 looks correct but doesn't work
• Augmenters with a date range created in 7.1 or prior and upgraded/migrated to 7.2 looks wrong but works
• Augmenters without date range are unaffected.

This has been fixed on Teneo 7.3 and augmenters making use of date ranges now works as expected.

Also see KI Resolutions

Further Improvements and Fixes

Further improvements are also implemented in Teneo Studio Desktop 7.3:

• Improvement to and visualization of Folders after adding content to a solution either by Add Content or Bulk Import as well as when restoring from Recycle Bin or performing copy/paste of documents/full folder structures have been improved.

• Setting a Folder (and its content) to Enabled / Disabled or Set / Unset stable is reviewed to ensure correct functionality and display.

• A bug detected in Teneo 7.1 and 7.2 in the Class Performance where running a test and next deleting the user would make the Class Performance area always display an Unable to find User by Key error has been reviewed and fixed.

• Minor text updates in the About box and on different labels in Teneo Studio Desktop to ensure coherent messaging.

• A Solution Log error occurring after unsetting a solution from stable has been amended.

• Some selection and filter issues with Global documents have been reviewed to ensure correct behavior.

• A crash of Teneo Studio occurring due to error loading and flow loading timing issues has been fixed. And, in cases where the Studio application crashes, the user can again write a message to the send with the crash log in the pop up window; minor improvements are also performed to the log format.

Teneo Generative QnA

Teneo Generative QnA is a starter kit which enables projects to quickly set up a Conversational AI application using OpenAI services on the Teneo Platform to use it to answer questions based on a custom controlled source of information.
Using the template solution as a starting point makes it possible to benefit from the many values of Generative AI and Large Language Models while still getting access to the extensive conversational AI features and functionalities of the Teneo Platform, providing projects with full control over the overall behavior of the conversational AI application.

Generative AI is a type of artificial intelligence technology that produces content using machine learning algorithms; it "learns" from the existing data and produces new content based on it. The foundation models of Generative AI in natural language processing are Large Language Models (LLMs) which are "computer programs" that use artificial neural networks to generate text based on a prompt, typically a question or a command, and are trained with self-supervised learning on large amounts of unlabeled text, an example of these types of computer programs is ChatGPT.

Teneo Generative QnA is a Teneo template solution, providing features, functionalities, and structural ideas; settings in the template solution are fully exposed and configurable, giving developers full control to adapt, add or remove features. The solution can be fine-tuned, enhanced further or just serve as inspiration when developing a new Conversational AI project.
The Teneo Generative QnA template comes out-of-the-box in Studio and is available upon creation of new solutions; after solution creation the developer will find a User Guide together with the solution's other Resource Files which helps with step-by-step guides to set up everything needed.

Read more in the reference documentation

Flow Variable Description

Teneo 7.3 also brings Flow Variable descriptions to both Teneo Studio Desktop and Web! In both frontends, clicking Add description allows to unfold the description field and the developer can now add a text describing, for example, the purpose of the Flow Variable.

Note that in Localization setup, the description field is editable in the Local solution.

Visual Warning for Non-Input Consuming Transitions

Teneo 7.3 brings a visual warning for non-input consuming transitions leaving the Give Response label to help clear any confusion: with the Give Response label in the Flow editor, the Flow graph kind of indicates that all transitions leaving the Give Response label are also "Get input before continuing" or input consuming.
But not all the transitions have to be input consuming when leaving the Give Response label - and when they are not, the often expected behavior is quite different from the one the Studio developer sees when testing, for example, in Tryout, and this is why Teneo Studio Desktop and Studio Web introduce a new visual warning icon in the Flow editor for developers to easily spot the difference in the transitions leaving the Give Response label.

For example, in the below flow where the "No" User Intent at the top is not marked as "Get input before continuing", if the user says "no" at the first Give Response label (at the top of the image), the first User Intent matching on "No" is triggered giving the Bot Output "Say No again" and the same "no" is used again to continue through the flow (without the need for the user to provide a new input) and triggers the latter "No" User Intent and thereby the Bot Output "No Thanks!". These two answers are now concatenated and given together as displayed in the right side of below image.

In most cases - the above described behavior is not the desired behavior - and the developer would expect the bot to stop and wait for a new user input after providing the first Bot Output (Say No again). This is why in Studio, non-input consuming transitions leaving a Give Response label where the other transitions are input-consuming, are now visualized with warning triangle.

As mentioned above, the visual marker is also available in Studio Web, and both in Web and Desktop an informative tooltip is displayed on hover-over.

Learn more in the reference documentation

Teneo Studio Web

Add Flow Scripts

Developers of Teneo solutions can now add On-top and On-drop Flow scripts directly in Flows in Teneo Studio Web.
The Scripts button is available in the higher left-side of the Flow window just below the Flow overview, Variables and Listeners and once clicked, the editor to add On top and On drop flow scripts becomes available.

Read more in the reference documentation about Flow scripts

Enable and Disable Documents

As of Teneo 7.3, developers can enable and disable documents in Teneo Studio Web.
In the documents' lists, a new icon is available in the far right hand side on hover-over to respectively disable / enable the selected document; disabled documents are written in italic and marked as such in the new Disabled column.

An open document can be enabled / disabled by clicking Info in the top ribbon and (de)selecting the Enable checkbox at the bottom of the Edit document info box. Remember to Save the document to preserve the modification.

Note that for Integrations, the Availability setting which controls if an integration can be added in new places in the solution, is - in Studio Web - controlled by the Enable / Disable functionality.

Class Test Examples

Just as for Teneo Studio Desktop, as of Teneo 7.3, developers can add both training data examples and test data examples to classes also in Teneo Studio Web. The new Test Examples can later be used, in Studio Desktop, to test the solution's intent model. Similarly to the training examples, the test examples should be unique, but unlike the training examples it is not obligatory to add test data.

Read more in the Class Management section of the reference documentation.

Assigned Intent Model in Tryout

Due to the implementation of native CLU in Teneo and the option for projects and developers to use CLU models, the Tryout in Studio Web now provide information about the currently assigned intent model in the upper, right corner.

Note that the actual assignment of an intent model is managed in Teneo Studio Desktop as described here.

Upgrade to 7.3

Please notice that old results of Cross Validations are removed upon upgrade to Teneo 7.3 and therefore the results from previous versions are not available in Teneo Studio Desktop 7.3.