Blog

Automated GUI Testing: Some Feedback On Squish

Homepage Automated GUI Testing: Some Feedback On Squish

When designing a graphics application, making sure it works all the way through development can become complicated. Unit tests cover possible regressions of the functional logic, but how do you make sure that the graphical part is always hunky-dory? You need an automated GUI testing tool. Some do exist. They provide a test framework to ensure that the graphical part does not regress during changes. Of all the tools, there are few that are designed to be used with Qt. Squish, published by FrogLogic, is one of these.

How does automated GUI testing work with Squish?

Squish allows us to create a test suite, which will then be played automatically, to be sure of the quality of the GUI of the software developed. Squish can be used to test desktop, web or mobile GUIs, whether they are developed in Java, with Qt or through the native tools of the many supported OS (Windows, Linux, MacOs, Android, iOs …).

The operation of Squish is quite simple: the software will convert the actions we perform on the GUI into scripts in order to be able to replay these actions sequentially thereafter. Squish does not repeat the user actions. It takes into account the user actions and replays them as quickly as possible for the interface. In order for the test to continue to completion, Squish waits until the object with which the user interacted is available before replaying the action.
In order to make sure that the interface functions properly, the user can add conformance tests that Squish will then run to test whether the result corresponds to what is expected.

Once the test suite has been run, Squish produces an xml file containing the test results. This file can then be used as a result in a continuous integration tool like Jenkins for example.

Tutorial: Using Squish to test a graphical interface developed for Linux using Qt

For our automated GUI testing purposes, we will use the GUI of an industrial coffee machine developed at Witekio using Qt.

Note that the test creation process seems to be similar to other platforms.

Step 1, create your test suite

Figure 1 test suite creation page for automated GUI testing

Figure 1. Test suite creation page

First, you have to create your automated testing suite, which will contain all the test cases you want to run. Logic dictates that all cases that test the same set of features are grouped in the same test suite. In our example, I created a test suite to check the operation of adding a new drink to the list of those available. When creating this suite, we can choose the language in which the scripts of the test cases will be written. Javascript is selected by default, but it is possible to choose creation of these scripts in Python, Perl, Ruby or even Tcl. This choice only matters if you have a preferred scripting language.

Step 2, associate your test suite with the target application

Once the test suite has been created, it must be associated with the application you wish to test. In my case, I selected the executable generated by QtCreator. If the application needs specific environment variables in order to run, remember to enter these in the field on the same page.

Figure 2 Test Suite Settings

Figure 2. Test suite settings

Step 3, make sure that it works properly

Figure 3 automated testing Launch button
Figure 3. Button to launch the application

Once the application is correctly configured, it can be launched via the interface to make sure everything works as it should.

Step 4, create your test scenarios

it is now time to create the test cases. Once the script is created and named, the empty file is added to the list of test cases for the created test suite. To complete this script, the application will launch and then simply perform the actions that you want to be automatically repeated thereafter. In the case of tests for a Qt application, Squish will identify the QML elements that we have clicked, double clicked or even dragged and dropped. There is no need to rush, the waiting time is not taken into account in the script.

Figure 4 Record Button

Figure 4. Button to start recording the script

Step 5, Add checks on the status of your GUI

It is possible to add GUI status checks at any time throughout this process. Once the verification type has been chosen, the user is asked to select the QML object to be used for validation. It is possible to explore the hierarchy of objects in Squish directly, but it is more convenient to use the pipette tool and select the object that you want to use in the GUI.

Figure 5 possible automated testing for your Qt GUI
Figure 5. Possible tests

There are several types of verification.

  • « Properties » is used to check the value of a QML object property.
  • « Screenshot » captures the graphical state of the application and saves this capture so that it can be compared with the state of the application when the check has to be done.
  • « Table » compares the contents of a table, and finally
  • « Visual » compares the graphical state of an interface element.

The tests concerning the graphical state of the interface (« Visual » and « Properties ») must be adjusted according to test conformity expectations

Once the check is inserted, you can continue to record the test.

If you have saved everything you want, you can stop the recording, which will close the application and complete the script, or add elements to the test script for your application.

If necessary, you can add new events to the script easily without having to re-record everything. Just start the application via Squish, then, by clicking on the « Record snippet » button, the actions will be added to the active script. For ease of readability, it is possible to directly add comments to the script flow when saving the actions.

Figure 6 automated GUI testing Insert Menu
Figure 6. Eventual insertion

Once you think your test is complete, you can run it through Squish to make sure everything works the way you want.

Challenges of Squish as an automated GUI testing tool

Tests can be tricky to create

When designing automated GUI testing for my Qt application, I encountered cases where tests did not finish correctly. Some actions of the script played did not have the expected effect. It turned out that the problem came from the speed of execution of the script, which sometimes simulated the pressing of a button before it was actually accessible, although it was in a correct state. The solution: I added a very small delay prior to the problematic action.

If Squish cannot run the script to the end, either because the tested application crashes, or because the application is not in the expected state, the script stops and an error is added to the test case results.

The main concern with Squish is probably the lack of robustness of checks that can be added to the test cases.

Visual tests are of relative reliability

During my tests, I failed to pass a single visual check, even though the captures that were compared seemed to be identical in every respect.

Similarly, it is difficult to visually test a GUI that has animations. There also seems to be some problems with GUI objects that are generated dynamically, such as beverage lists. Squish uses a property as a marker for the QML element to be used, and if the object is not identical, Squish may not recognize the element it needs to compare.

An interesting automated GUI testing tool that has not yet reached maturity

While Squish seems to be the only tool in its category, I still believe it is not a complete success.

It is difficult to have complete confidence in the results given by Squish. The difficulty of creating the tests on the properties and the relative reliability of the visual tests leaves the complete running of the test script as the only robust test option.

And the only certainty that can be deduced if the test script has run until the end is that Squish has managed to perform all the actions, and therefore that the application has responded correctly. This does not indicate that the application works as expected.

In addition, without any special attention or effort to integrate graphical tests during the development of the application, it is likely that each graphical modification of the interface requires a redesign of tests in Squish.

And when we see that the test creation time is about 40 minutes per test case for a beginner, it certainly gives pause for thought on the use of Squish while the development of the GUI is still incomplete.

From my experience, finishing the graphical interface only comes late in the development cycle of a product. Therefore, Squish rather becomes an appropriate tool to ensure the functioning of an application whose development is finished, and for which any modifications will be related to. However, its use must be anticipated during the development phase in order to allow the designers of the tests to work efficiently once a state of stability is reached on the various sections of the developed application.

 

If you wonder about the coffee machine we are working on, you can read the summary of what we are doing for Evoca coffee vending machine software.

To learn more about Squish, click here

 

And if you need to know more about Squish,  want to set automated testing for your UI development, search for an IoT and embedded software expert to assist you, contact us.

- Website Administrator
15 juillet 2020