When designing a graphic application, making sure it works all the way through development can become complicated. Unit tests cover possible regressions of the functional logic, but how do you make sure that the graphical part is always hunky-dory?
Tools do exist, automated testing frameworks, to ensure that the graphical part of your UI does not regress during changes. Of all the tools, there are few that are designed to be used with Qt. Squish, published by FrogLogic is one of these tools.
Squish Automation : principles
Squish proposes an IDE and a framework allowing to implement acceptance testing for GUI, to automate execution and check in real-time if the application is true to its specifications.
Squish is compatible with desktop, web and mobile graphic interfaces, whether they are developed in Java, Qt or with OS native tools among the wide choice of supported OS (Windows, Linux, MacOS, Android, iOS, …).
The operation of Squish is quite simple: the software will convert the actions we perform on the GUI into scripts in order to be able to automatically replay these actions sequentially thereafter. Squish does not repeat the user’s actions. It takes into account the user actions and replays them as quickly as possible for the interface. In order for the test to continue to completion, Squish waits until the object with which the user interacted is available before replaying the action.
In order to make sure that the interface functions properly, you can add conformity tests that Squish will then run to test whether the result corresponds to what is expected.
Once the test suite has been run, Squish produces an xml file, in Junit format, containing the test results. This file can then be used as a result in a continuous integration tool like Jenkins or Bitbucket Pipelines.
5 steps to use Squish for GUI automated testing
For testing purposes, we will use the GUI of an industrial coffee machine developed at Witekio using Qt. The graphical interface has been developed for Linux using Qt.
Note that the test creation process will be quite similar with other development platforms.
Figure 1. Test suite creation page.
Step 1: create your test suite
First, you have to create a test suite, which will contain all the test cases you want to run. Logic dictates that all cases that test the same set of features are grouped in the same test suite. In our example, I created a test suite to check the operation of adding a new beverage to the list of those available.
When creating this suite, the IDE will let you choose the language in which the scripts of the test cases will be written. Javascript is selected by default, but you can change it to Python, Perl, Ruby or even Tcl.
Step 2: associate your test suite with the target application
Once the test suite has been created, it must be associated with the application you wish to test. In my case, I selected the executable generated by QtCreator. If the application needs specific environment variables in order to run, remember to enter these in the field on the same page.
Figure 2. Test suite parameters.
Step 3: make sure your automated testing framework works properly
Once the application is correctly configured, it can be launched via the interface to make sure everything works as it should.
Figure 3. Button to launch the application.
Step 4: create automated test cases
It is now time to create the test cases. Once the script is created and named, the empty file is added to the list of test cases for the created test suite.
To complete this script, the IDE proposes a tool to record actions. You just need to click on the record button, then the application launches and can proceed to the actions allowing to run the test.
In the case of tests for a Qt application, Squish will identify the QML elements that we have clicked, double clicked or even dragged and dropped. There is no need to rush, the waiting time is not taken into account in the script.
Figure 4. Button to start recording the script.
Step 5: Verify the status of your GUI
It is possible to add GUI status verifications at any time throughout this process. Once the verification type has been chosen, the user is asked to select the QML object to be used for validation. It is possible to explore the hierarchy of objects in Squish directly, but it is more convenient to use the pipette tool and select the object that you want to use in the GUI.
Figure 5. Possible tests.
There are several types of verifications:
- “Properties” is used to check the value of a QML object property.
- “Screenshot” captures the graphical state of the application and saves this capture so that it can be compared with the state of the application when the check has to be done.
- “Table” compares the contents of a table.
- “Visual” compares the graphical state of an interface element.
- “OCR” uses Tesseract, the open-source engine, to proceed to character recognition on the application interface.
The tests concerning the graphical state of the interface (“Visual”, “Properties ” and “OCR”) must be adjusted according to test conformity expectations, in particular the validity threshold (the pixel by pixel correspondence between the reference image and the image captured during the test).
Once the check is inserted, you can continue to record the test.
If you have saved everything you want, you can stop the recording, which will close the application and complete the script, or add elements to the automated test script for your application.
If necessary, you can add new events to the script easily without having to re-record everything. Just start the application via Squish, then, by clicking on the “Record snippet” button, the actions will be added to the active script. For ease of readability, it is possible to directly add comments to the script flow when saving the actions.
Figure 6. Insert specific interactions.
Once you think the implementation covers your acceptance testing steps and verifications, you can run it through Squish to make sure everything works the way you want.
Figure 7. Test in Gherkin (left) and script steps (right).