Tools & methods

Linux Debugging and Maintenance

Homepage Linux Debugging and Maintenance

Debugging is an integral part of every embedded and IoT software engineer’s role. Pushing working, secure, and robust firmware or embedded software to a device and then never returning to that software when new kernel versions or security patches are released is incredibly shortsighted. Instead, the best engineers understand that returning to the code to maintain and debug it is as much a part of their job as writing the code in the first place.

Linux debugging is necessary every time a new version of a Linux kernel or a new Yocto layer is released. More than just a replacement of the existing kernel or layer, though, debugging means updating and testing the new elements as well as all the dependencies throughout the software stack. Thus, even if updating the layer is relatively straightforward, there is often extensive work to be done to test all the other software that is interconnected with or dependent on that new layer.

At Witekio we have the long experience in embedded Linux development and deep expertise to maintain Linux systems, in Linux debugging, and offer support for a host of different boards.

We have the skills to identify and deploy kernel patches but also understand where and how those patches might have unintended consequences and undesirable effects on a client’s board and create new bugs of their own.

Embedded Linux Development services

BSP Maintenance: The Key to Product Security

At the heart of Witekio’s Linux debugging and maintenance offer is its broad commitment to the maintenance of many different board support packages or BSPs.

Every day new vulnerabilities are discovered in the Linux kernel or in the drivers associated with the kernel. These vulnerabilities leave impacted devices open to being hacked or otherwise impacted by bad actors with ill intentions. The result of any such hack could be significant and carry heavy financial, legal, and reputational risk. Suffice to say, any IoT device vendor has a distinct interest in avoiding a breach of its systems and therefore a clear interest in maintaining the security of its Linux systems.

Linux vulnerabilities are reported and stored in a range of different databases known as CVEs, for Common Vulnerabilities and Exposures. Monitoring all these databases for new vulnerabilities or potential attack vectors is normally a tedious job for an engineer but not so at Witekio. Our team has fully automated the monitoring of the various CVE databases, and thanks to our automated tools we can regularly scan a client’s Yocto BSP and generate a list of all new CVEs relevant to that BSP.

The client’s focus, like ours, is on maintaining a working product that is fully secure. Towards this end, when providing the list of CVEs Witekio also advises the client as the best next steps to take. This advice usually includes a recommendation of which updates are best suited to your Yocto BSP, the automated and non-regression tests required to ensure the update or new layer do not impact the product’s useability, and any additional software services recommended to ensure that the client’s product is free from the risks of buggy code.

Witekio’s approach of automated monitoring for CVEs along with automated testing, extensive BSP support, and software service expertise ensures that a client’s software is always up to date, always secure, and free from the bugs that can impact the user experience and provoke customer discord.

Debugging Diagnostic Tools in the Witekio Arsenal

Witekio’s engineers have broad experience in how to debug the Linux kernel, Yocto layers, and the hundreds of software dependencies that connect with these. This experience and the combined expertise of engineers in the EU, the UK, and North America means that there is often no need to resort to formal debugging tools to maintain and update code. Only when an issue falls outside of this vast experience do engineers turn to Linux debugging tools understanding, as they do, the benefits and drawbacks of using each.

Debugging tools such as JTAG (Joint Test Action Group) and GDB (GNU Debugger) are considered akin to ‘the biggest hammer’ in the Witekio toolbox. Both are high-level debugging tools and JTAG is often deemed essential when it comes to debugging embedded systems that cannot be debugged by any other channel. Yet being high-level debugging tools, they are often akin to using a hammer to swat a fly. GDB, too, while effective, lacks the detail that JTAG offers. For Witekio engineers, then, both tools are essential but to be used sparingly when other approaches prove unsuccessful or inapplicable.

KDbg is an alternative GUI debugger for the KDE Desktop Environment. It can be considered something like a more user-friendly interface to the GDB tool, making it more intuitive and potentially faster to resolve issues. Valgrind is a memory debugger that can be run on all major desktop platforms (Linux, MacOS, Windows) and offers a suite of tools for analyzing embedded software. What’s more, Valgrind can also be used as a framework for creating custom dynamic analysis tools, including profilers.

Occasionally and when tracing a particularly elusive bug, Witekio engineers will also utilize a ‘print’ statement or prtmsg approach. This somewhat primitive debugging approach involves adding print statements into the software to understand a control flow, gather key variables, and identify points of failure. While effective, there are disadvantages to this approach, not least that the additions to the code will have to be removed, the code recompiled, and the software re-run to get the output. It’s time consuming, but sometimes necessary.

Witekio engineers are never stumped by bugs in the code and pride themselves on applying the right tool in the right way to deliver software that is secure, reliable, robust, and working as expected.

No matter the device and no matter the BSP in a client’s product, Witekio engineers can maintain and debug the Linux kernel or Yocto layer involved. With decades of experience and a long history of client projects, Witekio’s record of effectively updating and debugging devices in all domains stands as evidence of high skill, deep expertise, and a capacity to deliver the Linux software updates devices need but without the bugs that emerge when less careful engineers manage that update project. 

In need of Linux debugging or maintenance?
You might also like...
Embedded Linux Systems & the Yocto Project

Embedded Linux Systems & the Yocto Project Creating a new product today often requires the choice of an embedded operating system - so why choose Linux? In this article we will also present the steps to build a Yocto-based BSP for your embedded Linux system.

Read more

Witekio’s software development service is core to the company and key to delivering value for our customers. Our teams are skilled in all types of software development for smart, connected, and handheld devices, industrial IoT devices, embedded security devices, and all cloud-connected products. Dozens of customers that return time and again to Witekio for their software development needs are a testament to the quality of the work we do and the cutting-edge software that we write. From start to finish, we help you make the right choices to support your vision, and we write clean and efficient code to extract the most value from your device.

Geoffrey Wright - Senior Embedded Software Engineer
13 October 2021