Operating Systems

Why You Need Yocto for NVIDIA Jetson Edge AI

Homepage Why You Need Yocto for NVIDIA Jetson Edge AI

At Witekio, we work with companies all over the world who need help connecting  everything from interactive screens to industrial equipment. Being a Nvidia preferred partner made sense for software lovers as we are.

As IoT technologies become more mature and widely adopted, the amount of data that devices are typically shipping to the cloud is increasing and becoming more expensive. In order to combat this shift, many are deploying edge computing architectures in order to perform some of the computational work locally at the “edge” of the network. This allows for reduced cloud traffic as well as reduced latency on the edge since pre-analyzed data can be sent to the cloud and feedback sent back to edge devices more quickly.

NVIDIA Jetson BSP challenges when coming to edge AI and machine learning

However, an edge computing device often requires special hardware in order to efficiently and effectively analyze the data coming in, namely a sizeable GPU, TPU (Tensor Processing Unit), or NPU (Neural Processing Unit). As well as having the appropriate capabilities to process data at the edge, the edge computing device must be robust and reliable as it acts as the gateway to the core network.  Therefore, it is important that the edge computing device must not only have the appropriate software to properly drive its special hardware (GPU, TPU, or NPU), but the entire BSP must be sufficiently stable in order to continue to perform its integral role in the network.

If you are considering integrating an edge-computing device into your network, chances are you’ve considered using an NVIDIA Jetson platform. The devices in the Jetson family provide powerful hardware for computation, particularly for AI and machine learning tasks, as well as extensive software support that allows for integrating with many existing platforms and libraries, such as TensorFlow and PyTorch. However, you also understand the importance of stability regarding your edge device, so you are likely asking yourself how you can create a stable BSP sufficient for productization for an NVIDIA Jetson platform. The answer to that question is a simple one: Yocto!

Along with the Jetson platforms, NVIDIA provides Jetpack L4T which is an Ubuntu-based BSP with the Jetson kernel and bootloader, Ubuntu filesystem, and all of the CUDA packages which make the Jetson special. Jetpack L4T is a great way to get started developing applications on a Jetson platform. Not only can it make full use of the GPU for machine learning and AI applications off the bat, it also has access to the Canonical aarch64 apt repositories so most tools and libraries can be easily loaded onto the platform.

Where Jetpack L4T Falls Short as a Production BSP

While Jetpack L4T is an efficient way to begin developing applications, it is not a great selection as a BSP for a product. When it comes to developing a stable BSP for a product, there are two key components when considering the build system:

  • Repeatability
  • Modifiability

The build system used should be able to reliably and repeatably generate an identical image for the device. Each time the build system generates an image, the same software components must be compiled and appropriately installed. Similarly, Developer A must be able to easily generate the exact same image as Developer B at any point in time. If this is not the case, there is no guarantee that the image generated will work properly. Perhaps an improper version of the kernel was built which is not fully compatible with the userspace tools built, or improper versions of libraries were installed which are not compatible with the application dependencies.

Unfortunately, Jetpack L4T suffers from poor repeatability. It is not designed to build an image which always contains the same software components, it is designed to build an image which always contains the most up-to-date versions of software components currently supplied by the repository. Without a good way to fully control the versions of the tools and dependencies installed on the system, BSP stability cannot be guaranteed and ongoing maintenance will be a struggle throughout the product’s lifetime.

The build system should also provide a robust method for modification of the BSP image. A developer must be able to modify the BSP to select specific versions for software components, apply patches, and incorporate predefined configuration files onto the final image. Just as importantly, these modifications must be able to be easily tracked in version control software (i.e. git) to ensure that developers can safely collaborate.

Again, Jetpack L4T falls short in its ability to provide modifiability since the BSP itself is not designed to be customized at build time.  A basic image, which is created using a generic tarball containing an Ubuntu root filesystem, is installed onto the target. NVIDIA-specific packages, such as the CUDA libraries, are then installed. Any further modification is only possible manually either on the booted target itself or by manually modifying the root filesystem tarball before installation. Neither of these options offers a robust method of generating a stable and version control trackable custom BSP.

Furthermore, Jetpack L4T does not provide a robust method for modifying the bootloader and kernel.  This is typically not an issue when just doing application development on an EVK, but custom hardware is very likely to require bootloader and kernel customizations.

Yocto Saves the Day!

This is where Yocto steps in.

Yocto is a widely used and mature build system which can be employed to generate a stable Embedded Linux BSP. Yocto has quite a lot to offer, including flexibility, robustness, and a wide variety of existing support for many hardware platforms and software packages. Most importantly, however, Yocto will reliably and repeatably generate a final image. It is also configured through a set of configuration files which makes it easy to track changes using version control software.

Support for specific software components and hardware platforms is introduced into a Yocto project by integrating layers. Layers contain recipes which are configuration files describing to the build system how to fetch, build, and install software components into the final image. When integrating software into your platform image, you can create your own layers and recipes for custom source code (such as your product’s application), include one from a 3rd-party repository, or select them from the wide variety of layers directly supported by the Yocto Project.

https://layers.openembedded.org/layerindex/branch/master/layers/

In order to integrate support for NVIDIA Jetson platforms to a Yocto project, you need to include a layer which contains recipes for your hardware platform’s kernel, bootloader, userspace dependencies, and CUDA libraries (assuming you wish to use these). Luckily, you won’t need to do this yourself as most of the legwork has been done by the open source community to create meta-tegra, a Yocto layer containing support for the following Jetson platforms:

  • Jetson-TX1 development kit
  • Jetson-TX2 development kit
  • Jetson AGX Xavier development kit
  • Jetson Nano development kit
  • Jetson Nano eMMC module with rev B01 carrier board
  • Jetson Nanon 2GB development kit
  • Jetson Xavier NX Development Kit
  • Jetson Xavier NX eMMC module in dev kit or Nano carrier board
  • Jetson-TX2i module
  • Jetson-TX2 4GB module
  • Jetson AGX Xavier 8GB module

https://github.com/OE4T/meta-tegra

Challenges of Yocto on NVIDIA Jetson

It is important to be aware, however, that while meta-tegra is officially endorsed by NVIDIA, it is not maintained by NVIDIA. It started as a private project by madisongh (https://github.com/madisongh) and then later gained traction within the community. This is an important caveat to be aware of when developing a BSP for a Jetson platform using Yocto. NVIDIA does not provide source code for their CUDA libraries, but instead only provide them as Ubuntu packages for their Jetpack L4T BSP.  meta-tegra has managed to integrate these into a Yocto layer by deconstructing the packages, but it is key to note that these libraries have been built in a specific, Ubuntu-based environment. Since the toolchain, build-time dependency tree, compilation flags and discovered host OS features used when building the CUDA libraries are completely different than the rest of the software components generated by your Yocto project, care must be taken when integrating these libraries and other packages into your BSP.

As Jetson platforms are commonly used as edge-computing devices, a stable BSP is crucial to the integrity of the entire network. Using Yocto, a repeatable, modifiable, and robust custom BSP can be generated, which is unfortunately just not yet the case when using Jetpack L4T. While Yocto on the Jetson platform poses some challenges of its own, the quality of production-ready BSP which Yocto can provide is unmatched. Though NVIDIA themselves don’t directly maintain the meta-tegra project, it has gained significant traction within the community. Additionally, a number of contributors to the project are part of the official NVIDIA Partner Network, a network of skilled third-parties with close ties to NVIDIA. Due to this, we can expect to see the maturity of this project continue to grow as well as expect sustained maintenance which is key to the viability of the Jetson family as a production-ready platform.

You might also like...
Why Edge AI matters on your technology roadmap?

Why Edge AI matters on your technology roadmap?

Read more
Nick Elsmore - Embedded Software Engineer
24 December 2020