Automated Testing Framework: Try Squish With A Qt GUI
When designing a graphics application, making sure it works all the way through development can become complicated. Unit tests cover possible regressions of the functional logic, but how do you make sure that the graphical part is always hunky-dory?
Tools do exist, automated testing frameworks, to ensure that the graphical part does not regress during changes. Of all the tools, there are few that are designed to be used with Qt. Squish, published by FrogLogic, is one of these tools.
Principles Squish as an automated testing framework.
Squish proposes an IDE and a framework allowing to implement acceptance testing for GUI, to automate execution and check in real time if the application is true to its specifications.
Squish is compatible with desktop, web and mobile graphic interfaces, whether they are developed in Java, Qt or with OS native tools among the wide choice of supported OS (Windows, Linux, MacOS, Android, iOS, …).
The operation of Squish is quite simple: the software will convert the actions we perform on the GUI into scripts in order to be able to automatically replay these actions sequentially thereafter. Squish does not repeat the user actions. It takes into account the user actions and replays them as quickly as possible for the interface. In order for the test to continue to completion, Squish waits until the object with which the user interacted is available before replaying the action.
In order to make sure that the interface functions properly, you can add conformity tests that Squish will then run to test whether the result corresponds to what is expected.
Once the test suite has been run, Squish produces an xml file, in Junit format, containing the test results. This file can then be used as a result in a continuous integration tool like Jenkins or Bitbucket Pipelines.
4 steps to use Squish to automate testing of a graphical interface developed for Linux using Qt:
For testing purposes, we will use the GUI of an industrial coffee machine developed at Witekio using Qt.
Note that the test creation process will be quite similar with other development platforms.
Figure 1. Test suite creation page
Step 1, create your test suite
First, you have to create a test suite, which will contain all the test cases you want to run. Logic dictates that all cases that test the same set of features are grouped in the same test suite. In our example, I created a test suite to check the operation of adding a new beverage to the list of those available.
Step 2, associate your test suite with the target application
Once the test suite has been created, it must be associated with the application you wish to test. In my case, I selected the executable generated by QtCreator. If the application needs specific environment variables in order to run, remember to enter these in the field on the same page.
Figure 2. Test suite parameters
Step 3, make sure that it works properly
Once the application is correctly configured, it can be launched via the interface to make sure everything works as it should.
Figure 3. Button to launch the application
Step 4, create test cases
It is now time to create the test cases. Once the script is created and named, the empty file is added to the list of test cases for the created test suite.
To complete this script, the IDE proposes a tool to record actions. You just need to click on the record button, then the application launches and can proceed to the actions allowing to run the test.
In the case of tests for a Qt application, Squish will identify the QML elements that we have clicked, double clicked or even dragged and dropped. There is no need to rush, the waiting time is not taken into account in the script.
Figure 4. Button to start recording the script
Verify the status of your GUI
It is possible to add GUI status verifications at any time throughout this process. Once the verification type has been chosen, the user is asked to select the QML object to be used for validation. It is possible to explore the hierarchy of objects in Squish directly, but it is more convenient to use the pipette tool and select the object that you want to use in the GUI.
Figure 5. Possible tests
There are several types of verifications:
- "Properties" is used to check the value of a QML object property.
- "Screenshot" captures the graphical state of the application and saves this capture so that it can be compared with the state of the application when the check has to be done.
- "Table" compares the contents of a table
- "Visual" compares the graphical state of an interface element, and finally
- “OCR” uses Tesseract, the open-source engine, to proceed to character recognition on the application interface.
The tests concerning the graphical state of the interface ("Visual", "Properties" and “OCR”) must be adjusted according to test conformity expectations, in particular the validity threshold (the pixel by pixel correspondence between the reference image and the image captured during the test).
Once the check is inserted, you can continue to record the test.
If you have saved everything you want, you can stop the recording, which will close the application and complete the script, or add elements to the test script for your application.
If necessary, you can add new events to the script easily without having to re-record everything. Just start the application via Squish, then, by clicking on the "Record snippet" button, the actions will be added to the active script. For ease of readability, it is possible to directly add comments to the script flow when saving the actions.
Figure 6. Insert specific interactions
Once you think the implementation covers your acceptance testing steps and verifications, you can run it through Squish to make sure everything works the way you want.
A key advantage: Squish allows to integrate Behavior-Driven Development formalism (BDD)
BDD is an agile programming process. It makes collaboration easy between the different stakeholders of a project by encouraging conversation and practical examples to formalize the common understanding of the application’s functioning. BDD borrows its principles and methods to Test-Driven Development and Domain-Driven Design to elaborate common processes and tools for the development team (technical approach), quality engineers and non-technical people (business approach).
One of these tools is a formal dedicated language based on natural language: Gherkin.
It proposes a syntax allowing to define acceptance tests: perimeter of a functionality, initial application stage, steps that have to be validated to use the functionality, final expected state of the application after the functionality has been used.
Squish offers an interpreter to define every step, interaction and verification of the script in order to automate them.
Figure 7. Test in Gherkin (left) and script steps (right)
A few alternatives to Squish
You can come across Ranorex. This automated testing framework is similar to Squish but uses a Visual Studio base for the IDE and C#VB.NET for the scripting. Due to these restrictions, it can only be used for testing implementation and automation for Windows environments.
Sikuli and RobotFramework offer some alternative. Sikuli is a FOSS tool similar to Squish. It uses image comparison and OCR (respectively via OpenCV and Tesserac) to proceed to acceptance testing verifications. Sikuli integrates with RobotFramework to offer a bridge to BDD formalism and scripted test automation. But this framework does not allow introspection into the interface code and therefore depends of the graphical context to set testing parameters (referral screenshots, windows manager, screen size, etc.).
Disadvantages of Squish as an automated testing solution for a graphical user interface
Tests not always easy to create
Implementing testing with Squish as a framework is based on the introspection mechanism in order to interact with the interface without having to rely on the graphical context of the testing environment (screen size – 1080p., 1440p., ect.- windows managers – Gnome for Ubuntu, Desktop Windows Manager for Windows, or the different versions of Android). This introspection requires that the application interface has been designed taking into account automated testing (limited dynamic components creation, limited business intelligence in the interface)
When implementing my tests for the coffee machine Qt application, I encountered cases where tests did not finish correctly. Some actions of the script played did not have the expected effect. It turned out that the problem came from the speed of execution of the script, which sometimes simulated the pressing of a button before it was actually accessible, although it was in a correct state. To solve this issue, I added a very small delay prior to the problematic action.
Visual tests of relative reliability
All automation tools are based on comparing and recognizing images to establish testing verification points. But this method highly depends on the visual aspect of your application (if you have to make a change in font size or color, you will have to make all the screenshots again to adapt to the new style).
It is also highly dependent on the comparison algorithm’s fidelity and reliability.
For example, during my tests, I failed to pass a single visual check, even though the captures that were compared seemed to be identical in every respect.
Similarly, you can imagine that it is difficult to visually test a GUI that has animations.
To compensate this drawback, Squish offers an introspection mechanism that allows to access directly to the properties of the interface elements. Squish creates a marker for the QML element using a collection of properties. But if the object is not identical, Squish may not automatically recognize the element it needs to compare, from a test to another. You will then need to manually adjust the markers. This kind of problems usually happens when your graphical interface objects are created dynamically, like dynamic elements lists (list of beverages, presentation grids, etc.).
An automated test tool category that has not yet reached maturity
While Squish seems to be one of the rare tools in its category, I still believe it is not a complete success yet.
Test results are reliable only if verification methods are reliable. While unit tests frameworks can directly test code functions and do verifications right in the application development language, verifications for interface testing need bridges to validate results: a human eye to check the presence of a configuration menu, a OCR engine to validate the text of an address.
The need to use these bridges requires high precision and rigor in the implementation to make sure you get reliable results.
Finally, from experience, finalizing the graphical interface only comes late in the development cycle of a product. Therefore, Squish rather becomes an appropriate tool to ensure the functioning of an application whose development is finished, and for which any modifications will be related to correction. But if you plan to use Squish it will have to be anticipated during the development phase in order to allow the designers of the tests to work efficiently once a state of stability is reached on the various sections of the developed application.