OSTREE Tutorial for your system updates

Whatever type of embedded system or IoT solutions you create today, you will need to update it tomorrow. Either because there are new security patches or simply for an improved version of your application. But a new system update could just mean one line of code was changed and ideally should not lead to a complete reinstallation of all software present on your device. Therefore, it would be better to only update the differences between two versions of software to install. Hence, this type of technology could limit the bandwidth usage for each update, limit the installation time and consequently the overall offline time of your system during an update. A great solution that addresses all these points is called OSTree.

OSTree is a system for versioning updates of Linux-based operating systems. It’s both a shared library and a suite of command line tools. It features a “git-like” model for committing and downloading bootable filesystem trees, and a layer for deploying these trees and managing the bootloader configuration.

How OSTree works for differentials atomic system updates

From the documentation, “OSTree is deeply inspired by git; the core layer is a userspace content-addressed versioning filesystem”. 

Now let’s have a look at the Git Internals (just to be sure) :

Git tracks content – files and directories. It is at its heart a collection of simple tools that implement a tree history storage and directory content management system. It is simply used as a Software Configuration Management tool, not really designed as one. When most Software Configuration Management tools store a new version of a project, they store the code delta or diff. When Git stores a new version of a project, it stores a new tree – a bunch of blobs of content and a collection of pointers that can be expanded back out into a full directory of files and subdirectories.

If you want a diff between two versions, it doesn’t add up all the deltas, it simply looks at the two trees and runs a new diff on them. This is what fundamentally allows the system to be easily distributed, it doesn’t have issues figuring out how to apply a complex series of deltas, it simply transfers all the directories and content that one user has and another does not have but is requesting. It is efficient about it – it only stores identical files and directories once and it can compress and transfer its content using delta-compressed packfiles – but in concept, it is a very simple beast. Git is at its heart very stupid-simple.

To rephrase, Git thinks of its data more like a series of snapshots of a miniature filesystem. With Git, every time you commit, or save the state of your project, Git basically takes a picture of what all your files look like at that moment and stores a reference to that snapshot. To be efficient, if files have not changed, Git doesn’t store the file again, just a link to the previous identical file it has already stored. Git thinks about its data more like a stream of snapshots. Now that you have that in mind, let’s see how OSTree works for system update:

For instance, on Figure 1 between version 1 and 2, File A and C have been modified. When pulling the version 2 you will receive a compressed patch including just the differences between A and A1 and just the differences between C and C1. This patch includes a signature for A1 and C1 and it will be used to validate the success of the system update. Then OSTree will generate a new directory with a copy of File A and will apply the patch to update it to A1. The same will be done for C with the patch to update it to C1. B has not been changed and will be directly reused. As a last step, OSTree will check whether the signature computed after the update matches the one received within the compressed patch.

If not OSTree will discard the update and roll back to File A and File C. In case both signatures match, the update will be flagged as successful and used when your device will be restarted. By using 2 steps with first a copy and then an update, it makes sure in case of power failure that your system will boot using either A or A1 and C or C1. One downside of differential system update is the RAM usage during the update, hence the file to be updated and the patch are both loaded in RAM before they are merged together. Which means that you need enough RAM to store File A and the update A1. Another downside of OSTree is in the case the filesystem is corrupted it will be impossible to update the device. Therefore, you should always provision an alternative way to restore a corrupted device.  For instance, by using a dual partition scheme but that’s a different story and will be explained in a different blog article.

A summary of pros and cons of OSTree for system updates:

Knowing more about how OSTree manages new updates, let’s dig on how it internally stores them and links them to the “final” filesystem. OSTree only supports recording and deploying complete (bootable) filesystem trees.

OSTree supports two persistent writable directories that are preserved across upgrades: /etc and /var. OSTree relies on a new toplevel ostree directory;

On each client machine, there is an OSTree repository stored in /ostree/repo, and a set of “deployments” stored in /ostree/deploy/$STATEROOT/$CHECKSUM. Each deployment is primarily composed of a set of hardlinks into the repository.

To illustrate the Ostree structure, let’s search for the Cinematic experience application:

# find / -name "*Cinematic*"
 
/sysroot/ostree/deploy/poky/deploy/c6708386f3752a06074769aeaed2bd801b00a1f68370de9908968de7f25327d1.0/usr/bin/Qt5_CinematicExperience
==> That's where the "real" file is located
 
/sysroot/usr/share/cinematicexperience-1.0/Qt5_CinematicExperience
/usr/bin/Qt5_CinematicExperience

Therefore, you have a directory with a commit number (c6708…), this is the real file which is linked to /usr/bin/

If you modify the cinematic experience and commit this updated version to your OSTree server. Then, when you will pull this version on your embedded device, the deploy path directory will be changed:

/sysroot/ostree/deploy/poky/deploy/NEW_COMMIT_NUMBER/usr/bin/Qt5_CinematicExperience

When the device will be restarted the hard link to the “real” filesystem will be modified to point on this new path. Because an embedded device has a limited mass storage, for each new commit on the server side, only the two last commits are stored on the embedded device (the previous one is kept to easily roll back to the previous version even if no connectivity is available). In this way, the size used by OSTree to store your different files is always kept to the minimum.

How to set up OSTree with Yocto

You probably wonder but how can I install OSTree on my embedded device? One easy solution consists in using Yocto. You are not familiar with Yocto? Follow the link below and you will get the perfect introduction to get started:

• Embedded Linux Demystified 

Then to install OSTree is pretty straight forward, there is one meta available just for that:

• meta-updater

It provides everything you need to generate OSTree tools directly into a build using Yocto. But of course, you will need to tweak a bit the configuration of this meta to adapt OSTree to your needs.

Therefore, let’s get started:

To enable all features, you need to add to your local.conf

require conf/distro/sota.conf.inc
INHERIT += " sota"

Within meta-updater, there are several classes ready to do the job:

image_types_ostree

Contains everything to prepare and maintain the OSTree repository.
It depends on several environment variables (defined by default in sota.bbclass ):

• OSTREE_REPO` – path to your OSTree repository. Defaults to `${DEPLOY_DIR_IMAGE}/ostree_repo
• OSTREE_OSNAME` – OS deployment name on your target device.
• OSTREE_INITRAMFS_IMAGE` – initramfs/initrd image that is used as a proxy while booting into OSTree deployment. Do not change this setting unless you are sure that your initramfs can serve as such a proxy.
• OSTREE_BRANCHNAME ?= “${MACHINE}”
• GARAGE_SIGN_REPO ?= “${DEPLOY_DIR_IMAGE}/garage_sign_repo”
• GARAGE_SIGN_KEYNAME ?= “garage-key”
• GARAGE_TARGET_NAME ?= “${OSTREE_BRANCHNAME}”
• SOTA_MACHINE ??=”none”
• SOTA_PACKED_CREDENTIALS (path to your credentials.zip from ATS Garage). If this variable is not defined, all remote operations are skipped. But all modifications are still committed to your local OSTree repository.

image_types_otaimg

Creates an `otaimg` bootstrap image, which is an OSTree physical sysroot as a burnable filesystem image.

At the point you should have set up OSTree in your Yocto build environment and generate a new set of images ready to be programmed on your target.

The next step is to start playing with it!

Use OSTree to run differential system updates on your embedded device

OSTree used to include a simple HTTP server as part of the ostree binary, but this has been removed in more recent versions. However, OSTree repositories are self-contained directories, and can be trivially served over the network using any HTTP server. For example, you could use Python’s SimpleHTTPServer:

cd tmp/deploy/images/imx6qdlsabresd/ostree_repo
python -m SimpleHTTPServer <port> # port defaults to 8000

You can then run OSTree from your device by adding an OSTree repository:

mkdir ~/ostree_repo
ostree init --repo=~/ostree_repo

This behaves like adding a Git remote; you can name it anything

ostree remote add --no-gpg-verify my-remote http://<your-ip>:<port> --repo=~/ostree_repo

If OSTREE_BRANCHNAME is set in local.conf, that will be the name of the branch. If not set, it defaults to the value of MACHINE (e.g. qemux86-64).

ostree pull my-remote <branch>

In our case “poky” is the OS name as set in OSTREE_OSNAME

ostree admin deploy --os=poky my-remote:<branch>

Let’s check the status of this deploy

==> checking the new release. 
==> (*) indicate the current release used. 
==> (pending) means mean that this release will be used after rebooting
# ostree admin status
 
poky 6c9973369710028eff6d94e44d3f20e25f93e89219940ca146a3193287209eee.2 (pending)
origin refspec: my-remote:imx6qdlsabresd
 
* poky 6c9973369710028eff6d94e44d3f20e25f93e89219940ca146a3193287209eee.1
origin refspec: my-remote:imx6qdlsabresd

Once the system is correctly rebooted on the new release, rollback is defined

# ostree admin status

* poky 6c9973369710028eff6d94e44d3f20e25f93e89219940ca146a3193287209eee.2
origin refspec: my-remote:imx6qdlsabresd
 
poky 6c9973369710028eff6d94e44d3f20e25f93e89219940ca146a3193287209eee.1 (rollback)
origin refspec: my-remote:imx6qdlsabresd

How does OSTree swap between boot configurations?

OSTree allows swapping between boot configurations by implementing the “swapped directory pattern” in /boot. This means it is a symbolic link to one of two directories

# ls -l /ostree/
lrwxrwxrwx 1 root root 8 May 15 09:51 boot.0 -> boot.0.1
drwxr-xr-x 3 root root 1024 May 15 09:51 boot.0.1
drwxr-xr-x 3 root root 1024 May 15 09:51 boot.0.0
drwxr-xr-x 3 root root 1024 May 15 09:51 deploy
drwxr-xr-x 7 root root 1024 May 15 09:51 repo

# ls -l /boot/
lrwxrwxrwx 1 root root 8 May 15 09:51 loader -> loader.0
drwxr-xr-x 3 root root 1024 May 15 09:51 loader.0
drwxr-xr-x 3 root root 1024 May 15 09:51 loader.1
drwxrwxr-x 4 root root 1024 May 15 09:51 ostree

To swap the contents atomically, if the current version is 0, OSTree creates /ostree/boot.1, populate it with the new contents, then atomically swap the symbolic link. Finally, the old contents can be garbage collected at any point.

OSTree implements a special optimization where you want to avoid touching the bootloader configuration if the kernel layout hasn’t changed. This is handled by the ostree= kernel argument referring to a “bootlink”.

But you do need to update the bootloader configuration if the kernel arguments change.

# cat /boot/loader/uEnv.txt
 
 kernel_image=/ostree/poky-5376a2f53f7be8328d4f0109c9a398b0bc0b4b9b17e73160bc5f0de180b4c672/vmlinuz
 ramdisk_image=/ostree/poky-5376a2f53f7be8328d4f0109c9a398b0bc0b4b9b17e73160bc5f0de180b4c672/initramfs
 bootargs=ostree=/ostree/boot.1/poky/5376a2f53f7be8328d4f0109c9a398b0bc0b4b9b17e73160bc5f0de180b4c672/0 root=foo
 
 kernel_image2=/ostree/poky-5376a2f53f7be8328d4f0109c9a398b0bc0b4b9b17e73160bc5f0de180b4c672/vmlinuz
 ramdisk_image2=/ostree/poky-5376a2f53f7be8328d4f0109c9a398b0bc0b4b9b17e73160bc5f0de180b4c672/initramfs
 bootargs2=ostree=/ostree/boot.1/poky/5376a2f53f7be8328d4f0109c9a398b0bc0b4b9b17e73160bc5f0de180b4c672/1

In this case, adding bootargs when deploying the last release forced ostree to switch the /boot/loader symlink.

How does your system boot when using OSTree?

As an initial step, the meta-updater will change the location of your Linux kernel (zImage) from wherever it is located by default to the RootFS in a directory called boot:

At last, you will need to change some settings in U-Boot. OSTree already provides some elements in the file /boot/loader/uEnv.txt. 
Therefore, you need to modify U-boot configuration to load  this script and the kernel/initramfs provided. Here below is an example for imx6 platform:

#recipe-bsp/bootfiles/files/uEnv.txt
bootiaddr=0x40000000
bootmmc=1:2
bootargs_root=ostree_root=/dev/mmcblk2p2 root=/dev/ram0 ramdisk_size=8192
bootargs_extra=rw rootfstype=ext4 rootwait rootdelay=2 console=ttymxc0,115200
bootcmd_otenv=ext2load mmc ${bootmmc} $loadaddr /boot/loader/uEnv.txt; env import -t $loadaddr $filesize
bootcmd_load_kernel=ext4load mmc ${bootmmc} $loadaddr "/boot"$kernel_image
bootcmd_load_ramdisk=ext4load mmc ${bootmmc} $bootiaddr "/boot"$ramdisk_image
bootcmd_run=bootz ${loadaddr} ${bootiaddr} ${fdt_addr}
bootcmd=run findfdt; run loadfdt; run bootcmd_otenv; setenv bootargs ${bootargs} ${bootargs_root} ${bootargs_extra}; run bootcmd_load_kernel; 
run bootcmd_load_ramdisk; run bootcmd_ru

During the boot phase, Systemd will be used to switch from the original rootfs located in the ramfs to the new rootfs generated by OSTree:

...
 Waiting 2 sec before mounting root device...
 RAMDISK: gzip image found at block 0
 EXT4-fs (ram0): mounted filesystem with ordered data mode. Opts: (null)
 VFS: Mounted root (ext4 filesystem) on device 1:0.
 devtmpfs: mounted
 Freeing unused kernel memory: 752K
 /sbin/init[1]: Starting OSTree initrd script
 /sbin/init[1]: mounting FS: proc /proc
 /sbin/init[1]: mounting FS: sysfs /sys
 /sbin/init[1]: mounting FS: devtmpfs /dev
 /sbin/init[1]: /dev (devtmpfs) already mounted
 /sbin/init[1]: mounting FS: devpts /dev/pts
 /sbin/init[1]: mounting FS: tmpfs /dev/shm
 /sbin/init[1]: mounting FS: tmpfs /tmp
 /sbin/init[1]: mounting FS: tmpfs /run
 EXT3-fs (mmcblk2p2): error: couldn't mount because of unsupported optional features (240)
 EXT2-fs (mmcblk2p2): error: couldn't mount because of unsupported optional features (240)
 EXT4-fs (mmcblk2p2): mounted filesystem with ordered data mode. Opts: (null)
 Examining /sysroot//ostree/boot.0/poky/5376a2f53f7be8328d4f0109c9a398b0bc0b4b9b17e73160bc5f0de180b4c672/0
 Resolved OSTree target to: /sysroot/ostree/deploy/poky/deploy/6c9973369710028eff6d94e44d3f20e25f93e89219940ca146a3193287209eee.2
 /sbin/init[1]: Moving /dev to new rootfs
 /sbin/init[1]: Moving /proc to new rootfs
 /sbin/init[1]: Moving /run to new rootfs
 /sbin/init[1]: Switching to new rootfs
 /sbin/init[1]: Launching target init
 umount: /run/initramfs: target is busy.
 systemd[1]: systemd 234 running in system mode. (-PAM -AUDIT -SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP -LIBCRYPTSETUP 
-GCRYPT -GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID -ELFUTILS +KMOD -IDN2 -IDN default-hierar)
 systemd[1]: Detected architecture arm.
 ...

When using OSTree for your system updates, please note some interesting remarks:

• If you remove packets, disk space won’t be immediately released after the new commit. Your packets will still be referenced by the rollback commit.
• Adding new packages and some big files on the disk won’t impact the boot time as long as these files exist on the filesystem.
• Filesystem limitation: There can be no hardlink between 2 partitions. This limits locations where files can be deployed.

In order to use OSTree, you will need to change the architecture of your boot process and filesystem:

• Changing your kernel location
• Changing the overall way on how your system boot
• Dynamically generating a new filesystem after each update

These are not trivial changes; However, they are completely automated when using the meta-updater and they bring new interesting features:

• Bandwidth usage: When upgrading a system, it downloads only the differences from the last commit
• Update time: as all files are centralized within the ostree repository. Upgrade process are relatively fast as it involved only updating the changed files and switch hardlinks to the new deployment

Therefore, this kind of system update is extremely interesting when the amount of data transferred to update an embedded system should be limited at its minimum (for instance if you use a satellite connection and pay per Megabyte transferred).

Moreover. when the online time of a system is critical, the fact to only update the modified part of your software speed up drastically the whole update and reboot process.

Of course, this technology is not perfect and comes with some downsides:

• The size of the RAM available in your system needs to be at least the size of the biggest file you will update plus the size of the biggest patch you will apply.
• It is impossible to restore a device with a filesystem corrupted.

Hence, if the bandwidth usage or the speed to update your system are critical then OSTree is your best option. However, if a 100% reliability is as well one of your hard requirement, you should consider tweaking OSTree to use a dual partition update system. By using such a system, you will have the guaranty that even if one filesystem is corrupted your system can always boot on the second one and restore the corrupted one. That’s the beauty of it.

If you found this article interesting you can share it or contact me for further questions.

Useful Links

• https://rpm-ostree.readthedocs.io/en/latest/#filesystem-layout
• https://reposcope.com/package/ostree
• https://news.ycombinator.com/item?id=13744559
• Scripts to handle OSTREE repo :  https://github.com/ostreedev/ostree-releng-scripts