Linux Debugging and Maintenance

Debugging is an integral part of every embedded and IoT software engineer’s role. Pushing working, secure, and robust firmware or embedded software to a device and then never returning to that software when new kernel versions or security patches are released is incredibly shortsighted. Instead, the best engineers understand that returning to the code to maintain and debug it is as much a part of their job as writing the code in the first place.

Linux debugging is necessary every time a new version of a Linux kernel or a new Yocto layer is released. More than just a replacement of the existing kernel or layer, though, debugging means updating and testing the new elements as well as all the dependencies throughout the software stack. Thus, even if updating the layer is relatively straightforward, there is often extensive work to be done to test all the other software that is interconnected with or dependent on that new layer.

BSP Maintenance: The Key to Product Security

At the heart of Witekio’s Linux debugging and maintenance, offer is its broad commitment to the maintenance of many different board support packages or BSPs.

Every day new vulnerabilities are discovered in the Linux kernel or in the drivers associated with the kernel. These vulnerabilities leave impacted devices open to being hacked or otherwise impacted by bad actors with ill intentions. The result of any such hack could be significant and carry heavy financial, legal, and reputational risk. Suffice to say, any IoT device vendor has a distinct interest in avoiding a breach of its systems and therefore a clear interest in maintaining the security of its Linux systems.

Linux vulnerabilities are reported and stored in a range of different databases known as CVEs, for Common Vulnerabilities and Exposures. Monitoring all these databases for new vulnerabilities or potential attack vectors is normally a tedious job for an engineer but not so at Witekio. Our team has fully automated the monitoring of the various CVE databases, and thanks to our automated tools we can regularly scan a client’s Yocto BSP and generate a list of all new CVEs relevant to that BSP.

The client’s focus, like ours, is on maintaining a working product that is fully secure. Towards this end, when providing the list of CVEs Witekio also advises the client as the best next steps to take. This advice usually includes a recommendation of which updates are best suited to your Yocto BSP, the automated and non-regression tests required to ensure the update or new layer do not impact the product’s useability, and any additional software services recommended to ensure that the client’s product is free from the risks of buggy code.

Witekio’s approach of automated monitoring for CVEs along with automated testing, extensive BSP support, and software service expertise ensures that a client’s software is always up to date, always secure, and free from the bugs that can impact the user experience and provoke customer discord.

Debugging Diagnostic Tools in the Witekio Arsenal

Witekio’s engineers have broad experience in how to debug the Linux kernel, Yocto layers, and the hundreds of software dependencies that connect with these. This experience and the combined expertise of engineers in the EU, the UK, and North America means that there is often no need to resort to formal debugging tools to maintain and update code. Only when an issue falls outside of this vast experience do engineers turn to Linux debugging tools understanding, as they do, the benefits and drawbacks of using each.

Debugging tools such as JTAG (Joint Test Action Group) and GDB (GNU Debugger) are considered akin to ‘the biggest hammer’ in the Witekio toolbox. Both are high-level debugging tools and JTAG is often deemed essential when it comes to debugging embedded systems that cannot be debugged by any other channel. Yet being high-level debugging tools, they are often akin to using a hammer to swat a fly. GDB, too, while effective, lacks the detail that JTAG offers. For Witekio engineers, then, both tools are essential but to be used sparingly when other approaches prove unsuccessful or inapplicable.

KDbg is an alternative GUI debugger for the KDE Desktop Environment. It can be considered something like a more user-friendly interface to the GDB tool, making it more intuitive and potentially faster to resolve issues. Valgrind is a memory debugger that can be run on all major desktop platforms (Linux, macOS, Windows) and offers a suite of tools for analyzing embedded software. What’s more, Valgrind can also be used as a framework for creating custom dynamic analysis tools, including profilers.

Occasionally and when tracing a particularly elusive bug, Witekio engineers will also utilize a ‘print’ statement or prtmsg approach. This somewhat primitive debugging approach involves adding print statements into the software to understand a control flow, gather key variables, and identify points of failure. While effective, there are disadvantages to this approach, not least that the additions to the code will have to be removed, the code recompiled, and the software re-run to get the output. It’s time-consuming but sometimes necessary.