Linux Debugging and Maintenance

Debugging is an integral part of every embedded and IoT software engineer’s role. Pushing working, secure, and robust firmware or embedded software to a device and then never returning to that software when new kernel versions or security patches are released is incredibly shortsighted. Instead, the best engineers understand that returning to the code to maintain and debug it is as much a part of their job as writing the code in the first place.

Linux debugging is necessary every time a new version of a Linux kernel or a new Yocto layer is released. More than just a replacement of the existing kernel or layer, though, debugging means updating and testing the new elements as well as all the dependencies throughout the software stack. Thus, even if updating the layer is relatively straightforward, there is often extensive work to be done to test all the other software that is interconnected with or dependent on that new layer.

At Witekio we have long experience in embedded Linux development and deep expertise in maintaining Linux systems, in Linux debugging, and we offer support for a host of different boards.

We have the skills to identify and deploy kernel patches but also understand where and how those patches might have unintended consequences and undesirable effects on a client’s board and create new bugs of their own.

At the heart of Witekio’s Linux debugging and maintenance, offer is its broad commitment to the maintenance of many different Linux board support packages (or Linux BSPs). Linux BSP development is crucial to getting the most from your device. From hardware integration to software optimization, your Linux BSP can make the difference between a good product and a great one.

Every day new vulnerabilities are discovered in the Linux kernel or in the drivers associated with the kernel. These vulnerabilities leave impacted devices open to being hacked or otherwise impacted by bad actors with ill intentions. The result of any such hack could be significant and carry heavy financial, legal, and reputational risk. Suffice to say, any IoT device vendor has a distinct interest in avoiding a breach of its systems and therefore a clear interest in maintaining the security of its Linux systems.

Linux vulnerabilities are reported and stored in a range of different databases known as CVEs, for Common Vulnerabilities and Exposures. Monitoring all these databases for new vulnerabilities or potential attack vectors is normally a tedious job for an engineer but not so at Witekio. Our team has fully automated the monitoring of the various CVE databases, and thanks to our automated tools we can regularly scan a client’s Linux BSP and generate a list of all new CVEs relevant to that BSP.

The client’s focus, like ours, is on maintaining a working product that is fully secure. Towards this end, when providing the list of CVEs Witekio also advises the client as the best next steps to take. This advice usually includes a recommendation of which updates are best suited to your Linux BSP, the automated and non-regression tests required to ensure the update or new layer does not impact the product’s useability, and any additional software services recommended to ensure that the client’s product is free from the risks of buggy code.

Witekio’s Long-term Maintenance service includes automated monitoring for CVEs along with automated testing, extensive Linux BSP support, as well as regular kernel updates. This ensures that your software is always up to date, always secure, and free from the bugs that can impact the user experience and provoke customer discord. Our proactive approach to IoT device security means your product can stay secure and evolving for longer.

Witekio’s engineers have broad experience in how to debug the Linux kernel, Yocto layers, and the hundreds of software dependencies that connect with these. This experience and the combined expertise of engineers in the EU, the UK, and North America means that there is often no need to resort to formal debugging tools to maintain and update code. Only when an issue falls outside of this vast experience do engineers turn to Linux debugging tools understanding, as they do, the benefits and drawbacks of using each.

Debugging tools such as JTAG (Joint Test Action Group) and GDB (GNU Debugger) are considered akin to ‘the biggest hammer’ in the Witekio toolbox. Both are high-level debugging tools and JTAG is often deemed essential when it comes to debugging embedded systems that cannot be debugged by any other channel. Yet being high-level debugging tools, they are often akin to using a hammer to swat a fly. GDB, too, while effective, lacks the detail that JTAG offers. For Witekio engineers, then, both tools are essential but to be used sparingly when other approaches prove unsuccessful or inapplicable.

KDbg is an alternative GUI debugger for the KDE Desktop Environment. It can be considered something like a more user-friendly interface to the GDB tool, making it more intuitive and potentially faster to resolve issues. Valgrind is a memory debugger that can be run on all major desktop platforms (Linux, macOS, Windows) and offers a suite of tools for analyzing embedded software. What’s more, Valgrind can also be used as a framework for creating custom dynamic analysis tools, including profilers.

Occasionally and when tracing a particularly elusive bug, Witekio engineers will also utilize a ‘print’ statement or prtmsg approach. This somewhat primitive debugging approach involves adding print statements into the software to understand a control flow, gather key variables, and identify points of failure. While effective, there are disadvantages to this approach, not least that the additions to the code will have to be removed, the code recompiled, and the software re-run to get the output. It’s time-consuming but sometimes necessary.

Witekio’s embedded Linux development service is based on a foundation of over 20 years of experience helping out customers get the most from their devices. Our teams are skilled in all types of embedded software development for smart, connected, and handheld devices, industrial IoT devices, embedded security devices, and all cloud-connected products. From start to finish, we help you make the right choices to support your vision.

We pride ourselves on applying the right tool in the right way to deliver software that is secure, reliable, and robust. No matter the device and no matter your needs, Witekio engineers can maintain and debug the Linux kernel or Yocto layer involved. If you want to find out more about how Witekio can help maintain your device, get in touch today for a free consultation.