When designing a graphics application, making sure it works all the way through development can become complicated. Unit tests cover possible regressions of the functional logic, but how do you make sure that the graphical part is always hunky-dory? You need an automated GUI testing tool. Some do exist. They provide a test framework to ensure that the graphical part does not regress during changes. Of all the tools, there are few that are designed to be used with Qt. Squish, published by FrogLogic, is one of these.
How does automated GUI testing work with Squish?
Squish allows us to create a test suite, which will then be played automatically, to be sure of the quality of the GUI of the software developed. Squish can be used to test desktop, web or mobile GUIs, whether they are developed in Java, with Qt or through the native tools of the many supported OS (Windows, Linux, MacOs, Android, iOs …).
The operation of Squish is quite simple: the software will convert the actions we perform on the GUI into scripts in order to be able to replay these actions sequentially thereafter. Squish does not repeat the user actions. It takes into account the user actions and replays them as quickly as possible for the interface. In order for the test to continue to completion, Squish waits until the object with which the user interacted is available before replaying the action.
In order to make sure that the interface functions properly, the user can add conformance tests that Squish will then run to test whether the result corresponds to what is expected.
Once the test suite has been run, Squish produces an xml file containing the test results. This file can then be used as a result in a continuous integration tool like Jenkins for example.
Tutorial: Using Squish to test a graphical interface developed for Linux using Qt
For our automated GUI testing purposes, we will use the GUI of an industrial coffee machine developed at Witekio using Qt.
Note that the test creation process seems to be similar to other platforms.
Step 1, create your test suite
Figure 1. Test suite creation page
Step 2, associate your test suite with the target application
Once the test suite has been created, it must be associated with the application you wish to test. In my case, I selected the executable generated by QtCreator. If the application needs specific environment variables in order to run, remember to enter these in the field on the same page.
Figure 2. Test suite settings
Step 3, make sure that it works properly
Figure 3. Button to launch the application
Once the application is correctly configured, it can be launched via the interface to make sure everything works as it should.
Step 4, create your test scenarios
it is now time to create the test cases. Once the script is created and named, the empty file is added to the list of test cases for the created test suite. To complete this script, the application will launch and then simply perform the actions that you want to be automatically repeated thereafter. In the case of tests for a Qt application, Squish will identify the QML elements that we have clicked, double clicked or even dragged and dropped. There is no need to rush, the waiting time is not taken into account in the script.
Figure 4. Button to start recording the script
Step 5, Add checks on the status of your GUI
It is possible to add GUI status checks at any time throughout this process. Once the verification type has been chosen, the user is asked to select the QML object to be used for validation. It is possible to explore the hierarchy of objects in Squish directly, but it is more convenient to use the pipette tool and select the object that you want to use in the GUI.
Figure 5. Possible tests
There are several types of verification.
- « Properties » is used to check the value of a QML object property.
- « Screenshot » captures the graphical state of the application and saves this capture so that it can be compared with the state of the application when the check has to be done.
- « Table » compares the contents of a table, and finally
- « Visual » compares the graphical state of an interface element.
The tests concerning the graphical state of the interface (« Visual » and « Properties ») must be adjusted according to test conformity expectations
Once the check is inserted, you can continue to record the test.
If you have saved everything you want, you can stop the recording, which will close the application and complete the script, or add elements to the test script for your application.
If necessary, you can add new events to the script easily without having to re-record everything. Just start the application via Squish, then, by clicking on the « Record snippet » button, the actions will be added to the active script. For ease of readability, it is possible to directly add comments to the script flow when saving the actions.
Figure 6. Eventual insertion
Once you think your test is complete, you can run it through Squish to make sure everything works the way you want.
Challenges of Squish as an automated GUI testing tool
Tests can be tricky to create
When designing automated GUI testing for my Qt application, I encountered cases where tests did not finish correctly. Some actions of the script played did not have the expected effect. It turned out that the problem came from the speed of execution of the script, which sometimes simulated the pressing of a button before it was actually accessible, although it was in a correct state. The solution: I added a very small delay prior to the problematic action.
If Squish cannot run the script to the end, either because the tested application crashes, or because the application is not in the expected state, the script stops and an error is added to the test case results.
The main concern with Squish is probably the lack of robustness of checks that can be added to the test cases.
Visual tests are of relative reliability
During my tests, I failed to pass a single visual check, even though the captures that were compared seemed to be identical in every respect.
Similarly, it is difficult to visually test a GUI that has animations. There also seems to be some problems with GUI objects that are generated dynamically, such as beverage lists. Squish uses a property as a marker for the QML element to be used, and if the object is not identical, Squish may not recognize the element it needs to compare.
An interesting automated GUI testing tool that has not yet reached maturity
While Squish seems to be the only tool in its category, I still believe it is not a complete success.
It is difficult to have complete confidence in the results given by Squish. The difficulty of creating the tests on the properties and the relative reliability of the visual tests leaves the complete running of the test script as the only robust test option.
And the only certainty that can be deduced if the test script has run until the end is that Squish has managed to perform all the actions, and therefore that the application has responded correctly. This does not indicate that the application works as expected.
In addition, without any special attention or effort to integrate graphical tests during the development of the application, it is likely that each graphical modification of the interface requires a redesign of tests in Squish.
And when we see that the test creation time is about 40 minutes per test case for a beginner, it certainly gives pause for thought on the use of Squish while the development of the GUI is still incomplete.
From my experience, finishing the graphical interface only comes late in the development cycle of a product. Therefore, Squish rather becomes an appropriate tool to ensure the functioning of an application whose development is finished, and for which any modifications will be related to. However, its use must be anticipated during the development phase in order to allow the designers of the tests to work efficiently once a state of stability is reached on the various sections of the developed application.
If you wonder about the coffee machine we are working on, you can read the summary of what we are doing for Evoca coffee vending machine software.
To learn more about Squish, click here
And if you need to know more about Squish, want to set automated testing for your UI development, search for an IoT and embedded software expert to assist you, contact us.