Automated testing

User Intents

Teneo Studio Desktop allows performing automatic tests to ensure that the right Flow triggers and transitions are triggered by a given input; the Auto-test uses the positive and negative examples of User Intent to test the Flows, so - before running an Auto-test - examples of User Intent should be added to the Flows in the solution.

Run an Auto-test

To see and run an Auto-test, follow the below steps:

Go to the backstage of Teneo Studio Desktop (Solution tab)
Select Auto-test in the left-side menu
In the top of the Test page, click Run Test to run an Auto-test with the default settings.

Defining Auto-test settings

In the top of the Test page, it is possible to define the settings of the Auto-test to run.

The first three icons allows the user to specify which elements to include in the test: Triggers, Transitions, and Links. By default, all three elements are pre-selected. To deselect any, simply click the icon.

Auto-test levels

Auto-tests can be done at three levels:

Flow (note that this test level is performed within the Flow)
Folder
Solution

The Solution button at the top of the Test page allows the user to browse for a specific folder and define the level of the test.

The Run Test button allows the user to choose between two different test options:

Run Test and
Run Test Using Flow scope

When selecting Run Test, the triggers and transitions in the scope (Flow, Folder or Solution) are tested in the following ways:

If the positive examples of User Intent match the Matches of the trigger/transition they belong to,
If the positive examples of User Intent do not match the Matches of the trigger/transition they belong to, and
if the trigger/transition is actually triggered for the examples that match the Matches, or
if the example(s) "is stolen" by another trigger/transition with a higher ranking in Ordering.

When selecting Run Test Using Flow Scope, which is particularly useful when working with Matches to ensure that these cover the examples of User Intent, the performed test checks:

How the triggers perform as a stand-alone, and
all other triggers in the solution are ignored.

Auto-test at Flow level

Follow these steps to set up and run an Auto-test at Flow level:

Open the Flow to be tested
Select the items to be included in the test: Triggers - Transitions - URLs
Click Run Test or Run Test Using Flow Scope at the top of the view.

When the Auto-test has finished, the results can be filtered by Succeeded, Failed, Succeeded with warnings or by Non-testable items.

Auto-test at Folder level

Follow one of the below options to set up and run an Auto-test at Folder level:

Either, in the Solution Explorer, right-click the folder to test and select Auto-test in the context menu, or
go to the Auto-test page in the backstage of Teneo Studio (Solution tab > Auto-test)
- Select the items to be included in the test: Triggers - Transitions - URLs
- Click the Solution button to open the drop-down menu and select Browse for folder
In the Folder Browser window, select the folder and click OK
Lastly, click Run Test or Run Test Using Flow Scope in the top of the Auto-test page.

Auto-test at Solution level

To test all Triggers, Transitions and URLs set up in the solution, follow these steps:

Go to the Auto-test page in the backstage of Teneo Studio (Solution tab > Auto-test)
Select the items to be included in the test: Intent Triggers - Transitions - URLs
Lastly, click Run Test.

Use Stable Documents

The Use Stable Documents/Use Latest Documents refers to Version Flags and allows to choose between running Auto-test on all documents in the solution marked as stable or on the latest version of all documents within the solution.

Auto-tests run on the stable documents are marked with a Tested using stable versions flag in the top of the test results.

Cancel an Auto-test

As soon as an Auto-test has started, the Cancel test button appears in the page. The user can, by clicking it, cancel the run of the Auto-test. Note that the cancellation can take a little as it needs to be communicated to the server for the Auto-test to be cancelled.

Intent Model

In case the Learn model isn't ready when starting an Auto-test, for example, because the model cache has expired or because a model training is in progress, the execution of the Auto-test will launch a new model training (or wait for the running training to finish) before starting the actual test of the User Intents in the solution. While the model isn't ready, a Training model... message is displayed below the Cancel test button.

When a CLU intent model is assigned to the solution, Auto-test uses the CLU model for the generation of the test results.

Learn more about Intent Classification in Teneo.

Review former Auto-test results

The History panel, at the bottom of the Auto-test page, displays a summary of former Auto-tests. It is possible to Open any of the previous Auto-tests and Reload the list by using the available buttons.

Export test report

The user can export the Auto-test results in Excel format. To do so, simply click the Get Report button available in the top of the Auto-test page.
For further information on how to review the test results, please see the Auto-test results panel section.