Bringing Improvements to your Yocto Compilation
It won’t take long for you to notice the limitations of the above version.
Installing missing packages
If you look for vim or nano when editing a file, you’re not going to find them. You’ll therefore have to install one of these packages. What’s more, the free version does not come with the auto-completion feature. Instead, you’ll have to install the “bash-completion” package and configure your .bashrc. Finally, you may have tried to configure your Linux kernel with the following command:
bitbake -c menuconfig virtual/kernel
It will probably fail and generate the following message:
ERROR: linux-yocto-5.0.19+gitAUTOINC+31de88e51d_00638cdd8f-r0 do_menuconfig: No valid terminal found, unable to open devshell.
Tried the following commands:
tmux split-window -c “{cwd}” “do_terminal”
tmux new-window -c “{cwd}” -n “linux-yocto Configuration” “do_terminal”
xfce4-terminal -T “linux-yocto Configuration” -e “do_terminal”
terminology -T=”linux-yocto Configuration” -e do_terminal
mate-terminal –disable-factory -t “linux-yocto Configuration” -x do_terminal
konsole –separate –workdir . -p tabtitle=”linux-yocto Configuration” -e do_terminal
gnome-terminal -t “linux-yocto Configuration” -x do_terminal
xterm -T “linux-yocto Configuration” -e do_terminal
rxvt -T “linux-yocto Configuration” -e do_terminal
tmux new -c “{cwd}” -d -s devshell -n devshell “do_terminal”
screen -D -m -t “linux-yocto Configuration” -S devshell do_terminal
ERROR: linux-yocto-5.0.19+gitAUTOINC+31de88e51d_00638cdd8f-r0 do_menuconfig:
ERROR: linux-yocto-5.0.19+gitAUTOINC+31de88e51d_00638cdd8f-r0 do_menuconfig: Function failed: do_menuconfig
ERROR: Logfile of failure stored in: /home/dev/poky/build/tmp/work/qemux86-poky-linux/linux-yocto/5.0.19+gitAUTOINC+31de88e51d_00638cdd8f-r0/temp/log.do_menuconfig.1785
ERROR: Task (/home/dev/poky/meta/recipes-kernel/linux/linux-yocto_5.0.bb:do_menuconfig) failed with exit code ‘1’
You can install “screen”, “vim”, and “bash-completion” to fix these problems. The line in our Dockerfile for installing the packages thus becomes:
RUN apt-get update && apt-get install -y gawk wget git-core diffstat unzip \
texinfo gcc-multilib build-essential chrpath socat cpio python \
python3 python3-pip python3-pexpect xz-utils debianutils iputils-ping \
python3-git python3-jinja2 libegl1-mesa libsdl1.2-dev xterm locales \
vim bash-completion screen
Next, go ahead and copy the .bashrc in the container:
COPY ./bashrc /home/dev/.bashrc
Then, create a “bashrc” file alongside your Dockerfile that contains:
# enable programmable completion features (you don’t need to enable
# this, if it’s already enabled in /etc/bash.bashrc and /etc/profile
# sources /etc/bash.bashrc).
if ! shopt -oq posix; then
if [ -f /usr/share/bash-completion/bash_completion ]; then
. /usr/share/bash-completion/bash_completion
elif [ -f /etc/bash_completion ]; then
. /etc/bash_completion
fi
fi
Securing your work by sharing it between the host machine and container
The result of the compilation only appears in the tree directory of the container. If you delete the container, you’ll lose all of the work done. You may therefore wish to share this data with the host machine:
docker run -i -t –name first-test -v <host directory>:/home/dev yocto-compile
As a result, any changes you make to your container’s /home/dev directory will also take place in the designated host directory. This may cause issues regarding access rights. We only set the PUID in the Dockerfile and the PGID was already set to 1000.
If the user of the host machine uses different IDs, you’re likely to run into problems when trying to write in your host machine’s tree directory from the container. We’ll therefore pass the PUID and PGID as parameters to the Dockerfile.
By the way, if you create the image for yourself, you can use the same username and the same path between the host machine and the container for the project. This will ensure there is a valid absolute path both on your host machine and in your container, thereby making it easier for you.
If a compilation error occurs, Yocto usually provides a path to a log file where the error is stored. As this path is an absolute path, you can open it on your host machine without having to change it. The Dockerfile will therefore look like this:
FROM ubuntu:18.04
RUN apt-get update && apt-get install -y gawk wget git-core diffstat unzip \
texinfo gcc-multilib build-essential chrpath socat cpio python \
python3 python3-pip python3-pexpect xz-utils debianutils iputils-ping \
python3-git python3-jinja2 libegl1-mesa libsdl1.2-dev xterm locales \
vim bash-completion screen
ARG USERNAME=dev
ARG PUID=1000
ARG PGID=1000
RUN groupadd -g ${PGID} ${USERNAME} \
&& useradd -u ${PUID} -g ${USERNAME} -d /home/${USERNAME} ${USERNAME} \
&& mkdir /home/${USERNAME} \
&& chown -R ${USERNAME}:${USERNAME} /home/${USERNAME}
RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
COPY ./bashrc /home/${USERNAME}/.bashrc
USER ${USERNAME}
WORKDIR /home/${USERNAME}
The command for generating the image is as follows:
docker build -t <image name>:<tag> –build-arg USERNAME=<user name> –build-arg PUID=<puid> –build-arg PGID=<pgid> .
Since the data has been shared with the host machine, you can destroy the container once you have exited it. The command for creating the container therefore becomes:
docker run -it –rm -v <working directory>:<working directory> <image name>:<tag>
You can subsequently compile Yocto by running:
cd <working directory>
git clone -b warrior git://git.yoctoproject.org/poky
cd poky
source oe-init-build-env
bitbake core-image-minimal
Creating scripts to automate and gain time
In this last section, we’ll create a few shell scripts to make our lives easier.
Here they are:
- StartBuild.sh, which will allow us to start the compilation by creating a container with the right parameters
- yocto-entrypoint.sh, or the script that will let us fetch sources and start the compilation of Yocto
- docker-entrypoint.sh, a simple script that serves as the entry point to the container
yocto-entrypoint.sh may need to be updated over time. We’ll use docker-entrypoint.sh (which will serve as the entry point to the container) to avoid having to generate a new image for each modification.
The main role of this script will be to call yocto-entrypoint.sh, which will be passed to the container by using the -v option of the docker run command. StartBuild.sh will execute the docker run with the right arguments.
We’ll start by creating a docker-entrypoint.sh script alongside our Dockerfile. It will contain:
#!/bin/bash
set -e
[ $# -eq 0 ] && set — yocto
if [ “$1” = “yocto” ]; then
shift; set — ${YOCTO_ENTRYPOINT} “$@”
fi
exec “$@”
As you can see, if we do not pass parameters or the first parameter of the script is Yocto, then we’ll call the Yocto compilation script whose path is made available by the environment variable YOCTO_ENTRYPOINT.
We’ll create the yocto-entrypoint.sh script in our work directory (a separate directory that does not include the Dockerfile):
#!/bin/bash
SUPPORTED_MACHINES=”
qemuarm
qemuarm64
qemumips
qemumips64
qemuppc
qemux86
qemux86-64
beaglebone-yocto
genericx86
genericx86-64
mpc8315e-rdb
edgerouter
”
SUPPORTED_YOCTO=”
thud
warrior
”
WORKDIR=”/”
yocto_sync()
{
if [ -d ${WORKDIR}/yocto/sources/poky ]; then
cd ${WORKDIR}/yocto/sources/poky
git fetch origin
git checkout origin/$1
else
mkdir -p “${WORKDIR}/yocto/sources”
cd “${WORKDIR}/yocto/sources”
git clone -b $1 git://git.yoctoproject.org/poky.git
fi
}
is_in_list()
{
local key=”$1″
local list=”$2″
for item in $list; do
if [ “$item” = “$key” ]; then
return 0
fi
done
return 1
}
is_machine_supported()
{
local MACHINE=”$1″
is_in_list “$MACHINE” “$SUPPORTED_MACHINES”
}
is_yocto_supported()
{
local YOCTO=”$1″
is_in_list “$YOCTO” “$SUPPORTED_YOCTO”
}
show_usage()
{
echo “Usage: StartBuild.sh command [args]”
echo “Commands:”
echo ” sync ”
echo ” Sync Yocto version”
echo ” E.g. sync quemux86 warrior”
echo
echo ” all”
echo ” Build Yocto”
echo
echo ” bash”
echo ” Start an interactive bash shell”
echo
echo ” help”
echo ” Show this text”
echo
exit 1
}
main()
{
if [ $# -lt 1 ]; then
show_usage
fi
if [ ! -d “${WORKDIR}/yocto/sources” ] && [ “$1” != “sync” ]; then
echo “The directory ‘yocto/sources’ does not yet exist. Use the ‘sync’ command”
show_usage
fi
case “$1” in
all)
cd ${WORKDIR}/yocto/
source sources/poky/oe-init-build-env build
bitbake core-image-minimal
;;
sync)
shift; set — “$@”
if [ $# -ne 2 ]; then
echo “sync command accepts only 2 arguments”
show_usage
fi
if ! is_machine_supported “$1”; then
echo “$1 is not a supported machine: ${SUPPORTED_MACHINES}”
show_usage
fi
if ! is_yocto_supported “$2”; then
echo “$2 is not a supported yocto version: ${SUPPORTED_YOCTO}”
show_usage
fi
yocto_sync $2
cd “${WORKDIR}/yocto”
source sources/poky/oe-init-build-env build
sed -i “s/^MACHINE ??= .*$/MACHINE ??= \”$1\”/” conf/local.conf
;;
bash)
cd “${WORKDIR}/yocto”
source sources/poky/oe-init-build-env build
bash
;;
help)
show_usage
;;
*)
echo “Command not supported: $1”
show_usage
esac
}
main $@
In the main function, there are three primary events:
- all, which lets you start the compilation of a core-image-minimal image.
- sync, which parameterizes the machine and the desired Yocto version. This lets you fetch or update the sources and modify the configuration file so the right machine is used.
- Bash, which lets you source the environment and access a bash. We can then immediately run the bitbake commands manually.
You may have noticed that the WORKDIR variable is currently defined as “improperly configured”. It will be configured by the StartBuild.sh script, which we will create in our work directory alongside the yocto-entrypoint.sh script:
#!/bin/bash
WORKDIR=”$(pwd)”
sed -i “s|^WORKDIR=.*$|WORKDIR=\”${WORKDIR}\”|” ./yocto-entrypoint.sh
docker run -i -t –rm \
-v ${WORKDIR}:${WORKDIR} \
-v $(pwd)/yocto-entrypoint.sh:/yocto-entrypoint.sh \
yocto-compile:latest \
yocto $@
You can see that the name of the image must be “yocto-compile” and the tag must be “latest”. Otherwise, you must modify the script. This script modifies the WORKDIR variable from the yocto-entrypoint.sh script. WORKDIR is the directory that will be shared between the host machine and the container.
Naturally, you’ll have to modify the Dockerfile to:
- define the environment variable YOCTO_ENTRYPOINT
- add the docker-entrypoint.sh script to the image and define it as the entry point.
The Dockerfile will then look like this:
FROM ubuntu:18.04
RUN apt-get update && apt-get install -y gawk wget git-core diffstat unzip \
texinfo gcc-multilib build-essential chrpath socat cpio python \
python3 python3-pip python3-pexpect xz-utils debianutils iputils-ping \
python3-git python3-jinja2 libegl1-mesa libsdl1.2-dev xterm locales \
vim bash-completion screen
ARG USERNAME=dev
ARG PUID=1000
ARG PGID=1000
ENV YOCTO_ENTRYPOINT ${YOCTO_ENTRYPOINT:-/yocto-entrypoint.sh}
RUN groupadd -g ${PGID} ${USERNAME} \
&& useradd -u ${PUID} -g ${USERNAME} -d /home/${USERNAME} ${USERNAME} \
&& mkdir /home/${USERNAME} \
&& chown -R ${USERNAME}:${USERNAME} /home/${USERNAME}
RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
COPY ./bashrc /home/${USERNAME}/.bashrc
USER ${USERNAME}
WORKDIR /home/${USERNAME}
COPY ./docker-entrypoint.sh /
ENTRYPOINT [“/docker-entrypoint.sh”]
Next, you must grant the three new bash scripts permission to run the command:
chmod +x <script name>
In summary, if:
- the username is dev
- the user has a PUID of 1000
- the user has a PGID of 1000
- the desired name for the Docker image is “yocto-compile”
- the desired tag for the Docker image is “latest”
- the target machine is qemuarm
- the branch of poky is warrior
You can generate the image by entering the directory containing the Dockerfile and running the following command:
docker build -t yocto-compile:latest –build-arg USERNAME=dev –build-arg PUID=1000 –build-arg PGID=1000 .
Then, in the work directory containing the StartBuild.sh and yocto-entrypoint.sh scripts, you can fetch the sources and configure the machine with:
./StartBuild.sh sync qemuarm warrior
Next, start the compilation with the command:
./StartBuild.sh all
