CLU Manager

The options and functionalities described in this page currently only apply toTeneo Studio Desktop

Introduction

Conversational Language Understanding (CLU) is a service provided by Microsoft in the Cognitive Services layer in the Azure ecosystem. The service allows developers to train and query Natural Language Understanding (NLU) models using state-of-the-art, transformer-based, pre-trained language models and, furthermore, CLU provides multilingual capabilities.

Teneo allows to set up the account to work with CLU directly in Studio, where classes, training data examples and test data examples are added in the Class Manager and the CLU model then is trained in the CLU Manager.

CLU Settings must be defined in the Account tab of Teneo Studio to work with CLU

CLU Manager Window

CLU Manager is accessed by clicking the CLU Manager button in the top ribbon of Teneo Studio and consists of two tabs: Home and CLU.

The CLU Manager button is only visible when CLU Settings are defined in the Account tab

The Home tab contains the following:

• Top ribbon providing buttons to
  • Close the window
  • Train a CLU model
  • Delete an existing CLU model
  • Assign a CLU model to the solution
  • Unassign a CLU model
• CLU Models list displays the available CLU models and previously trained models, providing information related to:
  • the training date (Started) and the duration of the training
  • the user who started the training
  • expiration date of the CLU model; read more in the Azure documentation
  • stable/latest version of a CLU model (when applicable)
  • publish events related to a specific CLU model (when applicable)

The CLU tab consists of a left-side menu with the following:

• Close allows to close the CLU Manager.

Basics

Train

To train a new CLU model based on changes performed in the Class Manager, follow the below steps:

• Open CLU Manager
• Click Train in the top ribbon
• Teneo will now start the CLU model training

A new CLU model can only be trained if no previous model exists, if changes were performed to the training data since the last training and if no duplicated examples are detected.

Cancel Training

To cancel a training, simply click the Cancel button which appears in the CLU Models list for the running training.

Assign

When a new CLU model has been trained, to assign the model to the latest solution revision, follow the below steps:

• Open CLU Manager
• Select the new model in the CLU Models list
• Click Assign in the top ribbon
• Teneo confirms the assignment with a "Assignment process has finished" message, click Close and proceed to also close the CLU Manager.

A CLU model can only be assigned to the solution if the training data used to train the model is the same as the training data currently in the solution.

Unassign

To unassign a CLU model, follow the below steps:

• Open CLU Manager
• Select the currently assigned model in the CLU Models list
• Click Unassign in the top ribbon
• Teneo Studio prompts the user to confirm the action, optionally enter a comment (for the version history)
• Click Yes
• Teneo confirms the unassignment with a "Unassignment process has finished" message, click Close and proceed to also close the CLU Manager.

When unassigning a CLU model, Teneo will automatically use the Learn Intent Classifier instead; a previously assigned model might still be used for the stable solution revision or in a published solution.

Delete

To delete a CLU model, follow the below steps:

• Open CLU Manager
• Select the model to remove
• Click Delete
• Teneo Studio prompts to confirm the deletion, click Yes
• The model is now deleted and will disappear from the CLU Models list.

Also see Delete CLU Models

Close

To close the CLU Manager, either:

• Click the X in the top, right corner of the CLU Manager window
• Click the Close button available in the top ribbon
• In the CLU Manager backstage, click Close in the left-side menu.

Stable Version

The Intent model assignment can be set to stable as follows:

• First at least one CLU model must be trained and assigned to the latest solution
• In the top ribbon of the CLU Manager, click Intent model assignment
• In the History, select the wanted model assignment version; note that only versions with "Intent model assignment type CLU" can be set as stable
• Click Set Stable in the upper, right corner of the History view.

Also see History and Version Flag further below.

Assigned Model Information

Details

General

Conversational Language Understanding (CLU) is a service provided by Microsoft in the Cognitive Services layer in the Azure ecosystem. The service allows developers to train and query Natural Language Understanding (NLU) models using state-of-the-art transformer-based pre-trained language models. CLU, furthermore, provides multilingual capabilities.

CLU models can be build, trained and maintained directly in Teneo Studio by adding class names and training examples in the Class Manager and training the model in the CLU Manager where it is also possible to assign the model to the solution and manage other models.
A CLU model is always trained for the latest solution revision (see below section) and, of course, the model can be set to stable in projects which make use of the Version flag feature. Note that there is a limit of maximum 10 CLU models per solution.

CLU is a cloud-based service and therefore there may be limitations regarding the number of deployed (i.e., runtime) models and number of API calls per minute; for more information please see the Azure documentation. Also, as the deployed CLU model sits in the cloud, it might be impossible to query it in case of a problem in Azure or the network and in this case Teneo will default to Learn as a Learn model is always sent along when publishing a solution.

Valid CLU Models

A CLU model is always trained on the latest solution revision and it is only valid and in an assignable state as long as the training examples in the Class Manager does not change, or - said in other words - for a CLU model to be in a valid and assignable state, the training examples must be exactly the same as when the CLU model was trained.
If updates are performed to the training examples, then:

• an unassigned model becomes un-assignable
• a model assigned to the latest solution revision is automatically unassigned (and Studio defaults to the Learn classifier).

In both of the above cases, a new CLU model must be trained based on the updated training examples in order to assign the new model to the solution.

If a model is used in the stable solution revision or has been published in one of the solution's publish environments, then that particular model is still available (after modifying the training examples) for the stable solution/in any published solution. So, the stable solution can still be using a previous model if the model was set to stable while the model was in a valid state just as a previous model can be available in a published solution if the model was valid and assigned to the solution at the moment of publish.

Note that Teneo Studio displays warning messages when performing changes (updates to training examples) which will invalidate the currently assigned CLU model, and that Teneo Learn will be used by the solution as the intent classifier until a new CLU model is trained and assigned to the solution.

Duplicated Examples

CLU doesn't allow for duplicated training data examples. If duplicated examples are detected by Teneo, these are raised as Suggestions in the backstage of Studio as either Class has training examples duplicated in multiple classes or Class has duplicated training data. In case of duplicated examples across classes, the Train button is deactivated in the CLU Manager and an informative message displayed at the top of the CLU Models list to draw the developers attention to these as they need to be removed manually. Duplicated examples within the same class are automatically removed by Teneo before sending the data to CLU. Note that the detection of duplicated training data examples is case insensitive.

Training Status

Information related to the CLU model training is displayed in the CLU Manager where a new training is listed at the top of the CLU Models list and the status available in the right-hand side.

The status messages include:

• Importing in progress
• Training is queued: happens because models usually are queued on the CLU server before the training actually starts
• Training in progress: is displayed while the training is running with a progress bar, the duration (until now) and the expected remaining time
• Cancelling: is shown if the user clicks the Cancel button
• Training cancelled: is displayed when a training is successfully cancelled
• Trained CLU model available: successful training of CLU model, the model can now be assigned and used in the solution.

Information related to the CLU model trainings is also displayed in the Status bar of Teneo Studio windows to allow developers to work on other solution documents while the CLU model is being trained.

History and Version Flag

The Intent Model Assignment is a versionable solution document which keep track of the assigned intent model. Currently only CLU models can be assigned as intent models; if no CLU model is assigned, Learn is used.
History - and thereby versions - are created when the assignment of the intent model changes, either by being assigned to a CLU model or by being reverted to Learn by unassigning a CLU model.

The Intent Model Assignment document's version can be set as stable. In a stable solution version, the model assigned as the stable Intent Model Assignment is used, but unlike other solution documents that can be set as stable, setting an Intent Model Assignment as stable is not mandatory for a stable solution: if no stable version of the Intent Model Assignment exists, Learn is used!
This is underlined by the fact that it is not possible to set an Intent Model Assignment which uses Learn as stable. Only Intent Model Assignments that use CLU can be set as stable. To use Learn, the Intent Model Assignment simply needs to unset stable.

When setting an Intent Model Assignment version as stable, it is possible that the CLU model referenced by that version no longer exists. In that case, Studio doesn't allow setting that particular version as stable.

Similarly to the latest solution version, where consistency between the training data and the assigned model is guaranteed by resetting the Intent Model Assignment from CLU to Learn if the training data changes, consistency must also be guaranteed in the stable solution version and, therefore, when assigning an Intent Model Assignment version that uses a certain CLU model, all versions of the class documents that were used to train the model are automatically set as stable and class documents that were not used in the model are unset from stable. Correspondingly, if CLU is used in the stable Intent Model Assignment, setting / unsetting class documents as stable will unset the stable Intent Model Assignment version.

The Intent Model Assignment can be set / unset as stable in the CLU Manager as described here. If CLU is not available in the Studio account, there is no need to work directly with the Intent Model Assignment as the solution will always use the Learn Classifier.

Read more about Version History

Localization Setup

CLU models can be trained in one language (or even in various languages) and used to predict intents in another language making CLU an interesting option for projects which make use of Localization setup.

In Localization setup, CLU models are still trained per solution and there is - as with other solutions - a limit of 10 CLU models per solution. However, solutions in Localization setups can use trained CLU models from any of the other solutions of the setup if the model is valid for that particular solution, i.e., if all solutions define exactly the same classes (based on the class Id).
Changes in training data still unassign the CLU model automatically in the same solution, but doesn't affect other solutions directly! When training data changes happen outside a solution, the user will be ablet o train a new CLU model.
A model can only be used in a solution if it contains the latest training data of that solution. The model, however, does not have to contain the latest training data from other solutions. Users can try to train a new model if they believe the other solutions contain updated training data but no model has been trained yet.

This also implies that models from other solutions may be displayed in the Models list in the following three cases:

• if the model (either trained or training) can be assigned to the solution
• if the model has usages in the solution
• if the model was used in a historical publication of the solution.

Models can only be deleted from the solution which trained it. However, it is not possible to remove models if other solutions still use them. Similarly, it is not possible to remove solutions if the solution contains models that are still in use by other solutions.

Further, if a master solution has a CLU model assigned and all classes are included, during branching the same model will already be assigned to the local solution.

Note that the Test Data Evaluation runs on the solution's test data examples, i.e., test data examples are not shared across Master and Local solutions.

Read more about supported languages in CLU or see the list of supported Teneo languages

Deleting CLU Models

A solution can have a maximum of 10 CLU models, therefore inevitable at some point one or more models must be removed from the solution.
When deleting a CLU model in Teneo, please take into account:

• CLU models used by latest or stable solution versions cannot be deleted
• a CLU model used by an active publish target cannot be deleted
• please consider carefully if a CLU model previously used in a publication can be deleted; when the model is deleted it will no longer be available if old solution versions are re-published in the future
• Deleting a CLU model implies removing the CLU deployment and the CLU model from the CLU servers.

Predict Annotations

The Predict Input Processor makes use of the intent model to annotate user inputs with the defined model as described in more details in other sections (for example, in the conceptual overview of Intent Classification or in this section describing the input processor itself) and it also creates a Classifier variable which in its value provide information about the uses classifier, i.e., either CLU or Learn. In cases where the CLU model is unavailable, the Learn model is used as the fallback model and the value of the Classifier variable will be LearnFallback (note that a Learn model is always sent along with the published solution).

Troubleshooting

Warnings and errors

Error type	Error message	Implication	Resolution
Error	There are classes with duplicate examples. Go to the Suggestions panel for details.	CLU model cannot be trained	Remove one of the duplicated examples from either of the classes; more information here
Warning	At least two intents with training data are required to train a model	CLU model cannot be trained	Open the Class Manager to add more classes and training data
Warning	Changing the training data will make the assigned CLU model unavailable; Learn will be used instead	CLU model becomes invalid	The CLU model becomes invalid when changes are performed to the solution's training data examples, Studio defaults back to Teneo Learn until a new CLU model is trained and assigned; learn more
Warning	Restoring a previous version of the class will make the assigned CLU model unavailable; Learn will be used instead	CLU model becomes invalid	The CLU model becomes invalid when changes are performed to the solution's training data examples, Studio defaults back to Teneo Learn until a new CLU model is trained and assigned; learn more
Error	Error response from CLU server (429): Requests to the Get training job status and result details. Operation under Language - Conversational Language Understanding Authoring APIs (2022-05-01) have exceeded call rate limit of your current TextAnalytics S pricing tier. Please retry after 1 second. Please contact Azure support service if you would like to further increase the default rate limit.	CLU training is not possible	Wait a moment and try to train the CLU model again. The error is returned by Azure / CLU, please visit Azure Language Service for more information
Error	Error response from CLU server (429): Requests to the Delete a trained model. Operation under Language - Conversational Language Understanding Authoring APIs (2022-05-01) have exceeded call rate limit of your current TextAnalytics S pricing tier. Please retry after 5 seconds. Please contact Azure support service if you would like to further increase the default rate limit.		The error is returned by Azure / CLU, please visit Azure Language Service for more information
Error	Error response from CLU server (429): Requests to the Trigger a new deployment or replace an existing one. Operation under Language - Conversational Language Understanding Authoring APIs (2022-05-01) have exceeded call rate limit of your current TextAnalytics S pricing tier. Please retry after 47 seconds. Please contact Azure support service if you would like to further increase the default rate limit.		The error is returned by Azure / CLU, please visit Azure Language Service for more information