Knowing more about how OSTree manages new updates, let’s dig on how it internally stores them and links them to the “final” filesystem. OSTree only supports recording and deploying complete (bootable) filesystem trees.
OSTree supports two persistent writable directories that are preserved across upgrades: /etc and /var. OSTree relies on a new toplevel ostree directory;
On each client machine, there is an OSTree repository stored in /ostree/repo, and a set of “deployments” stored in /ostree/deploy/$STATEROOT/$CHECKSUM. Each deployment is primarily composed of a set of hardlinks into the repository.
To illustrate the Ostree structure, let’s search for the Cinematic experience application:
# find / -name "*Cinematic*"
/sysroot/ostree/deploy/poky/deploy/c6708386f3752a06074769aeaed2bd801b00a1f68370de9908968de7f25327d1.0/usr/bin/Qt5_CinematicExperience
==> That's where the "real" file is located
/sysroot/usr/share/cinematicexperience-1.0/Qt5_CinematicExperience
/usr/bin/Qt5_CinematicExperience
|
Therefore, you have a directory with a commit number (c6708…), this is the real file which is linked to /usr/bin/
If you modify the cinematic experience and commit this updated version to your OSTree server. Then, when you will pull this version on your embedded device, the deploy path directory will be changed:
/sysroot/ostree/deploy/poky/deploy/NEW_COMMIT_NUMBER/usr/bin/Qt5_CinematicExperience
|
When the device will be restarted the hard link to the “real” filesystem will be modified to point on this new path. Because an embedded device has a limited mass storage, for each new commit on the server side, only the two last commits are stored on the embedded device (the previous one is kept to easily roll back to the previous version even if no connectivity is available). In this way, the size used by OSTree to store your different files is always kept to the minimum.
How to set up OSTree with Yocto
You probably wonder but how can I install OSTree on my embedded device? One easy solution consists in using Yocto. You are not familiar with Yocto? Follow the link below and you will get the perfect introduction to get started:
Then to install OSTree is pretty straight forward, there is one meta available just for that:
It provides everything you need to generate OSTree tools directly into a build using Yocto. But of course, you will need to tweak a bit the configuration of this meta to adapt OSTree to your needs.
Therefore, let’s get started:
To enable all features, you need to add to your local.conf
require conf/distro/sota.conf.inc
INHERIT += " sota"
|
Within meta-updater, there are several classes ready to do the job:
image_types_ostree
Contains everything to prepare and maintain the OSTree repository.
It depends on several environment variables (defined by default in sota.bbclass ):
- OSTREE_REPO` – path to your OSTree repository. Defaults to `${DEPLOY_DIR_IMAGE}/ostree_repo
- OSTREE_OSNAME` – OS deployment name on your target device.
- OSTREE_INITRAMFS_IMAGE` – initramfs/initrd image that is used as a proxy while booting into OSTree deployment. Do not change this setting unless you are sure that your initramfs can serve as such a proxy.
- OSTREE_BRANCHNAME ?= “${MACHINE}”
- GARAGE_SIGN_REPO ?= “${DEPLOY_DIR_IMAGE}/garage_sign_repo”
- GARAGE_SIGN_KEYNAME ?= “garage-key”
- GARAGE_TARGET_NAME ?= “${OSTREE_BRANCHNAME}”
- SOTA_MACHINE ??=”none”
- SOTA_PACKED_CREDENTIALS (path to your credentials.zip from ATS Garage). If this variable is not defined, all remote operations are skipped. But all modifications are still committed to your local OSTree repository.
image_types_otaimg
Creates an `otaimg` bootstrap image, which is an OSTree physical sysroot as a burnable filesystem image.
At the point you should have set up OSTree in your Yocto build environment and generate a new set of images ready to be programmed on your target.
The next step is to start playing with it!
Use OSTree to run differential system updates on your embedded device
OSTree used to include a simple HTTP server as part of the ostree binary, but this has been removed in more recent versions. However, OSTree repositories are self-contained directories, and can be trivially served over the network using any HTTP server. For example, you could use Python’s SimpleHTTPServer:
cd tmp/deploy/images/imx6qdlsabresd/ostree_repo
python -m SimpleHTTPServer <port> # port defaults to 8000
|
You can then run OSTree from your device by adding an OSTree repository:
mkdir ~/ostree_repo
ostree init --repo=~/ostree_repo
|
This behaves like adding a Git remote; you can name it anything
ostree remote add --no-gpg-verify my-remote http://<your-ip>:<port> --repo=~/ostree_repo
|
If OSTREE_BRANCHNAME is set in local.conf, that will be the name of the branch. If not set, it defaults to the value of MACHINE (e.g. qemux86-64).
ostree pull my-remote <branch>
|
In our case “poky” is the OS name as set in OSTREE_OSNAME
ostree admin deploy --os=poky my-remote:<branch>
|
Let’s check the status of this deploy
==> checking the new release.
==> (*) indicate the current release used.
==> (pending) means mean that this release will be used after rebooting
# ostree admin status
poky 6c9973369710028eff6d94e44d3f20e25f93e89219940ca146a3193287209eee.2 (pending)
origin refspec: my-remote:imx6qdlsabresd
* poky 6c9973369710028eff6d94e44d3f20e25f93e89219940ca146a3193287209eee.1
origin refspec: my-remote:imx6qdlsabresd
|
Once the system is correctly rebooted on the new release, rollback is defined
# ostree admin status
* poky 6c9973369710028eff6d94e44d3f20e25f93e89219940ca146a3193287209eee.2
origin refspec: my-remote:imx6qdlsabresd
poky 6c9973369710028eff6d94e44d3f20e25f93e89219940ca146a3193287209eee.1 (rollback)
origin refspec: my-remote:imx6qdlsabresd
|
How does OSTree swap between boot configurations?
OSTree allows swapping between boot configurations by implementing the “swapped directory pattern” in /boot. This means it is a symbolic link to one of two directories
# ls -l /ostree/
lrwxrwxrwx 1 root root 8 May 15 09:51 boot.0 -> boot.0.1
drwxr-xr-x 3 root root 1024 May 15 09:51 boot.0.1
drwxr-xr-x 3 root root 1024 May 15 09:51 boot.0.0
drwxr-xr-x 3 root root 1024 May 15 09:51 deploy
drwxr-xr-x 7 root root 1024 May 15 09:51 repo
|
# ls -l /boot/
lrwxrwxrwx 1 root root 8 May 15 09:51 loader -> loader.0
drwxr-xr-x 3 root root 1024 May 15 09:51 loader.0
drwxr-xr-x 3 root root 1024 May 15 09:51 loader.1
drwxrwxr-x 4 root root 1024 May 15 09:51 ostree
|
To swap the contents atomically, if the current version is 0, OSTree creates /ostree/boot.1, populate it with the new contents, then atomically swap the symbolic link. Finally, the old contents can be garbage collected at any point.
OSTree implements a special optimization where you want to avoid touching the bootloader configuration if the kernel layout hasn’t changed. This is handled by the ostree= kernel argument referring to a “bootlink”.
But you do need to update the bootloader configuration if the kernel arguments change.
# cat /boot/loader/uEnv.txt
kernel_image=/ostree/poky-5376a2f53f7be8328d4f0109c9a398b0bc0b4b9b17e73160bc5f0de180b4c672/vmlinuz
ramdisk_image=/ostree/poky-5376a2f53f7be8328d4f0109c9a398b0bc0b4b9b17e73160bc5f0de180b4c672/initramfs
bootargs=ostree=/ostree/boot.1/poky/5376a2f53f7be8328d4f0109c9a398b0bc0b4b9b17e73160bc5f0de180b4c672/0 root=foo
kernel_image2=/ostree/poky-5376a2f53f7be8328d4f0109c9a398b0bc0b4b9b17e73160bc5f0de180b4c672/vmlinuz
ramdisk_image2=/ostree/poky-5376a2f53f7be8328d4f0109c9a398b0bc0b4b9b17e73160bc5f0de180b4c672/initramfs
bootargs2=ostree=/ostree/boot.1/poky/5376a2f53f7be8328d4f0109c9a398b0bc0b4b9b17e73160bc5f0de180b4c672/1
|
In this case, adding bootargs when deploying the last release forced ostree to switch the /boot/loader symlink.
How does your system boot when using OSTree?
As an initial step, the meta-updater will change the location of your Linux kernel (zImage) from wherever it is located by default to the RootFS in a directory called boot:
At last, you will need to change some settings in U-Boot. OSTree already provides some elements in the file /boot/loader/uEnv.txt.
Therefore, you need to modify U-boot configuration to load this script and the kernel/initramfs provided. Here below is an example for imx6 platform:
#recipe-bsp/bootfiles/files/uEnv.txt
bootiaddr=0x40000000
bootmmc=1:2
bootargs_root=ostree_root=/dev/mmcblk2p2 root=/dev/ram0 ramdisk_size=8192
bootargs_extra=rw rootfstype=ext4 rootwait rootdelay=2 console=ttymxc0,115200
bootcmd_otenv=ext2load mmc ${bootmmc} $loadaddr /boot/loader/uEnv.txt; env import -t $loadaddr $filesize
bootcmd_load_kernel=ext4load mmc ${bootmmc} $loadaddr "/boot"$kernel_image
bootcmd_load_ramdisk=ext4load mmc ${bootmmc} $bootiaddr "/boot"$ramdisk_image
bootcmd_run=bootz ${loadaddr} ${bootiaddr} ${fdt_addr}
bootcmd=run findfdt; run loadfdt; run bootcmd_otenv; setenv bootargs ${bootargs} ${bootargs_root} ${bootargs_extra}; run bootcmd_load_kernel;
run bootcmd_load_ramdisk; run bootcmd_ru
|
During the boot phase, Systemd will be used to switch from the original rootfs located in the ramfs to the new rootfs generated by OSTree:
...
Waiting 2 sec before mounting root device...
RAMDISK: gzip image found at block 0
EXT4-fs (ram0): mounted filesystem with ordered data mode. Opts: (null)
VFS: Mounted root (ext4 filesystem) on device 1:0.
devtmpfs: mounted
Freeing unused kernel memory: 752K
/sbin/init[1]: Starting OSTree initrd script
/sbin/init[1]: mounting FS: proc /proc
/sbin/init[1]: mounting FS: sysfs /sys
/sbin/init[1]: mounting FS: devtmpfs /dev
/sbin/init[1]: /dev (devtmpfs) already mounted
/sbin/init[1]: mounting FS: devpts /dev/pts
/sbin/init[1]: mounting FS: tmpfs /dev/shm
/sbin/init[1]: mounting FS: tmpfs /tmp
/sbin/init[1]: mounting FS: tmpfs /run
EXT3-fs (mmcblk2p2): error: couldn't mount because of unsupported optional features (240)
EXT2-fs (mmcblk2p2): error: couldn't mount because of unsupported optional features (240)
EXT4-fs (mmcblk2p2): mounted filesystem with ordered data mode. Opts: (null)
Examining /sysroot//ostree/boot.0/poky/5376a2f53f7be8328d4f0109c9a398b0bc0b4b9b17e73160bc5f0de180b4c672/0
Resolved OSTree target to: /sysroot/ostree/deploy/poky/deploy/6c9973369710028eff6d94e44d3f20e25f93e89219940ca146a3193287209eee.2
/sbin/init[1]: Moving /dev to new rootfs
/sbin/init[1]: Moving /proc to new rootfs
/sbin/init[1]: Moving /run to new rootfs
/sbin/init[1]: Switching to new rootfs
/sbin/init[1]: Launching target init
umount: /run/initramfs: target is busy.
systemd[1]: systemd 234 running in system mode. (-PAM -AUDIT -SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP -LIBCRYPTSETUP
-GCRYPT -GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID -ELFUTILS +KMOD -IDN2 -IDN default-hierar)
systemd[1]: Detected architecture arm.
...
|
When using OSTree for your system updates, please note some interesting remarks:
- If you remove packets, disk space won’t be immediately released after the new commit. Your packets will still be referenced by the rollback commit.
- Adding new packages and some big files on the disk won’t impact the boot time as long as these files exist on the filesystem.
- Filesystem limitation: There can be no hardlink between 2 partitions. This limits locations where files can be deployed.
In order to use OSTree, you will need to change the architecture of your boot process and filesystem:
- Changing your kernel location
- Changing the overall way on how your system boot
- Dynamically generating a new filesystem after each update
These are not trivial changes; However, they are completely automated when using the meta-updater and they bring new interesting features:
- Bandwidth usage: When upgrading a system, it downloads only the differences from the last commit
- Update time: as all files are centralized within the ostree repository. Upgrade process are relatively fast as it involved only updating the changed files and switch hardlinks to the new deployment
Therefore, this kind of system update is extremely interesting when the amount of data transferred to update an embedded system should be limited at its minimum (for instance if you use a satellite connection and pay per Megabyte transferred).
Moreover. when the online time of a system is critical, the fact to only update the modified part of your software speed up drastically the whole update and reboot process.
Of course, this technology is not perfect and comes with some downsides:
- The size of the RAM available in your system needs to be at least the size of the biggest file you will update plus the size of the biggest patch you will apply.
- It is impossible to restore a device with a filesystem corrupted.
Hence, if the bandwidth usage or the speed to update your system are critical then OSTree is your best option. However, if a 100% reliability is as well one of your hard requirement, you should consider tweaking OSTree to use a dual partition update system. By using such a system, you will have the guaranty that even if one filesystem is corrupted your system can always boot on the second one and restore the corrupted one. That’s the beauty of it.
If you found this article interesting you can share it or contact me for further questions.
Useful Links