How did we do this and what where our objectives?
Discover the whole story.
Remember how Docker containers has completely changed the way applications are deployed and orchestrated in the Cloud?
A year ago, with a bunch of colleagues, we started imagining that we could apply the same kind of solution, with the same technologies to IoT devices instead of the cloud.
This approach would allow to organize IoT devices OTA update in a simple and robust way. So we decided to investigate if anybody had had the same idea. Through our research we came across resin.io (since then changed their name to balena) and it looked as if they offered exactly what we had in mind.
Resin.io, our reference for software update for IoT devices showed some weaknesses
After digging into the solution, we spotted out two major weaknesses:
First weakness is that this Balena solution uses the full-blown Docker stack and registries to deploy updates. The question then is how you generate containers compatible with the processor architecture you use on your IoT device on your computer (usually your computer is an Intel processor when most IoT devices use ARM processors)? The issue has been partly sorted out by providing the most usual containers (e.g., to run an application coded in Javascript) for the most usual architectures. But then, what happens if you are on something a bit unusual? You might not be able to address it with their technology and you may well get stuck with your IoT device. Which can be off course slightly annoying…
The second drawback we identified is that you, IoT maker, will have to disclose your code and give it to Balena. Their solution implies indeed, that you first need to push your source code to their git server, then they proceed to the compilation and finally push code back to your devices with the result. If you work on an open source project, this doesn’t really matter. But for quite a few innovative companies, giving away their source code is simply unthinkable.
So we came to the conclusion that IoT makers need a more flexible device software update solution for the cross compilation of applications and higher privacy for source code. And we started thinking about developing a new tool based on the same technologies.This idea eventually grew into FullMetalUpdate!
FullMetalUpdate is a fully integrated solution for simple and robust OTA updates. And it is open source! It includes everything you will need for device software update, from the embedded software to the web server. And it is easy to deploy.
Our device software update solution: key aspects
First of all, we decided our device software update solution would be open source. Just because OTA update is too crucial for innovators to be left in the hands of others.
It had to be open-source!
Then, we set up our expectations:
- An update should never brick or put a device in an unknown state.
- It should be secure, efficient and thrifty and therefore container based
- It should work with any cloud provider.
An OTA software update compatible with any IoT platform
Yocto and containers, the perfect start to make sure we are compatible with all IoT software platforms
As you probably already know, Yocto is a good solution to build custom Linux distributions for embedded systems. A majority of processors that can run Linux are supported. Yocto tool-set is highly flexible. With some imagination and work, you can use it to generate the main image for your operating system (OS). At the same time it can also generate an independent filesystem for each application with just the right dependencies to run them. This is exactly what we call a container. But why would you put your application in container? In fact, it brings multiple benefits:
- Security: Your application is isolated from the rest of the system. The configuration file of the container allows to finely tune what resources from the host system are accessible. In the unlikely event that an application running in a container would be hacked. It will be stuck in the container with a very limited access to the rest of the system.
- Resource usage limitation: You can configure a container to only have a restricted access to the hardware by limiting its maximum bandwidth usage, its processor usage, its memory usage, …
Dependency isolation: Because each of your applications will be running in its own container, all issues with library incompatibilities are solved. You can have an application running with Qt4 in one container and another application using Qt5 in a different container. You will never run anymore into incompatibilities between different version of the same library.
Now we had every application in a separate container, the next question was how to run these applications on an IoT device.
To make the challenge even greater we wanted our solution to be compliant with the Open Container Initiative standard.
So only 3 options were left:
- RunC. It is the runtime used by Docker to run their container. This option is proven effective as it is used by basically all servers/computers running Docker.
- RailCar. This runtime has been developed by Oracle. But we found out that maintenance could be an issue as the solution had not been maintained for about 7 months.
- Clearcontainers. This runtime has been developed by Intel and is leveraging the VT-x technology. But it is not compatible with ARM processors which was a clear elimination factor.
In the end, for our device software update solution, the obvious choice was RunC. It is not tied to any specific processor technology and we can be quite sure it will be maintained as the community using it is quite huge and growing.
Now that we had chosen how to build containers and decided what technology would help to run these on an embedded system, it was high time to think about how to deploy software updates smartly. There, our main goal was to limit the bandwidth usage to realize an update and benefit from it’s highly appreciated side effect:limit the overall power consumption of the device. Basically, that meant we needed to realize delta updates. But what could we use to do that?
Deploy containers from a remote host to multiple IoT devices
FullMetalUpdate uses OSTree to Deploy containers to multiple IoT devices
One feature coming from the full-blown Docker implementation is the ability to create delta software updates to limit the bandwidth usage. Docker uses specific filesystems like AUFS or Overlayfs to deal with the deltas. These solutions are great on servers but not usable on embedded systems because they are not reliable in case of power failure.
That is where OSTree came in!
OSTree generates delta updates and handles the deployment in a GIT fashion but for binaries. OSTree manages updates like git commits and generates hard links of the used commit to the filesystem. This last operation is realized atomically which guarantees data consistency. As a result, it is completely independent of the filesystem type used on the device and extremely efficient on the storage usage.
The most popular implementation of OSTree for Yocto has been designed specifically to update Linux with no separation between the operating system (OS) and the applications:
This implementation is used by projects like ATS-Garage or the Automotive Grade Linux project.
But, one drawback is that the system needs to restart after each update. This can make sense if you update the Linux kernel, but if you update an application, it would be great to restart this specific application only after the update.
To solve this issue, we created a new OSTree repository specific to applications that will be updated independently of the rest of the system:
With this implementation, we had the best of two worlds.
On the one hand, we had OSTree, a solution used by multiple industrial-grade applications, to update the OS. On the other hand, we had an OSTree used to update your applications which are boxed within OCI containers (an architecture very close to the one used by Flatpak but adapted to the embedded world).
However, we were still missing one piece: The next question was how to manage the life cycle of the applications running within the different containers.
Manage the life cycle of the applications running in the different containers
Development of FullMetalUpdate Client
We needed a tool able to manage dependencies between the different containers. To reach this goal, we decided to develop our own software (FullMetalUpdate Client). And to avoid starting from scratch when solutions already existed, we decided to use SystemD as a back-end to manage the dependencies between the containers and to start and stop them.
Manage and deploy our device software update
Hawkbit and its challenges to build a web app in the Cloud
Next, we needed to provide a web application that could be used to manage and deploy the updates to multiple IoT devices. The obvious solution was to go for Hawkbit. Why was it so obvious? because it has the advantage of being proven effective. It is an open-source project developed by Bosh and used on numerous industrial projects.
But because our approach was based on OSTree for the deployment of software updates, we faced the challenge of integrating OSTree with Hawkbit. Our solution was to push the information for the update and the reference to the OSTree commit instead of uploading the actual update package to Hawkbit. In other words, with FullMetalUpdate, you use Yocto to build the OS and your containers. Then, Yocto automatically deploys the new software version to a remote OSTree, gets the commit reference, and pushes this information to Hawkbit. Once this is done, it is up to you to use the Hawkbit UI to start updating your different IoT devices.
Of course, for all this to be possible, we still needed to create a communication layer between the IoT devices and Hawkbit. Because the FullMetalUpdate client had been developed in Python, a natural choice was to reuse parts of the RAUC project to handle the communication protocol with the Hawkbit server.
Connect the different parts of our OTA update solution together
Docker Compose to bring everything together in your device software update
As a last step to complete the FullMetalUpdate project, we used Docker-Compose to connect the different parts composing the server on a virtual network. To make sure that our solution will build on any computer running Docker containers, we embedded the Yocto build system within a Docker container.
We finally ended up with the architecture below:
We now held our Grail! A fully integrated solution for OTA software updates that is simple, robust and open to all.
FullMetalUpdate is now available on Github and currently already includes containers to run Qt applications in full screen and neural networks using Tensor Flow Lite. This is only the beginning because the Yocto ecosystem is huge and can easily be extended to re-compile literally every project for your specific hardware!
We are actually working at some new commits that will include:
- Full support of the Raspberry Pi 3
- Skopeo to use containers from Docker and OCI registries
- CNI to easily create virtual networks between your containers
And as this device software update solution is open-source, we leave it up to the community for use, optimization, growth, for the benefit of all IoT makers.
