# Compiling Raspbian (Linux) Kernel for Running in QEMU

In this post, we go through the steps for cross-compiling the Linux kernel from source code for Raspberry Pi and running it with QEMU.

For those who work on the kernel stuff for Raspberry Pi boards, it’s not uncommon to play the SD card plug-and-unplug game every time you cross-compile your modified kernel and want to test it. A more elegant alternative is to run and debug your kernel in a virtual machine on your computer. Here, we are talking about running your custom kernel with QEMU.

Browsing around on the Internet, I’ve found some useful resources and pages that teach us how a given Raspberry Pi kernel image can be run with QEMU. Unfortunately, only a few of them include clues for making a custom kernel work with QEMU from scratch. Yet, they are not well documented and that’s why I’m summarizing the steps in this post.

### Assumptions and Limitations

In this post I mainly target running my custom Linux kernel in QEMU for Raspberry Pi 3B (RPi-3B). However, QEMU has not yet supported all the hardware on RPi-3B. For this reason, we’ll use the versatilepb machine type in QEMU to run our kernel. Obviously some drivers/modules will not work with this setup and this won’t be the right post for you if you need them working.

### [Step 0] Setup

Before we begin, please make sure you have the right environment being set up correctly:

• ARM Cross-Compiler Tool Chain and Build Dependencies

You can get the compiler as instructed here or using the commands below:

git clone https://github.com/raspberrypi/tools ~/tools
sudo apt update
sudo apt install git bison flex libssl-dev

• Linux Kernel Source Code from Raspberry Pi’s Git Repo

git clone https://github.com/raspberrypi/linux ~/
cd ~/linux
git fetch
git checkout -b rpi-4.19.y origin/rpi-4.19.y

• QEMU for ARM

sudo apt install qemu-system-arm

• (Optional) GDB for ARM

sudo apt install gdb-multiarch


After this step, I assume that you have your tools and linux source code in the following paths:

• ~/tools/: Raspberry Pi’s compiler tool set
• ~/linux/: your Linux kernel source code folder

### [Step 1] Patch Your Kernel

As mentioned earlier, we’ll use the machine type versatilepb to run our kernel in QEMU. For this, we need to patch our kernel for supporting versatilepb.

cd ~/linux
wget https://gist.githubusercontent.com/cchen140/0f06c4dd9c6c2cb0a219a4c523964699/raw/linux-arm-versatile.patch
patch -p1 < linux-arm-versatile.patch


### [Step 2] Revise defconfig

Next, some defconfig configurations must also be applied to make the kernel work in a versatile-pb machine in QEMU. Append all the configurations here (originally from here) or copy from below to your defconfig file located at ~/linux/arch/arm/configs/versatile_defconfig.

CONFIG_CPU_V6=y
CONFIG_ARM_ERRATA_411920=y
CONFIG_ARM_ERRATA_364296=y
CONFIG_AEABI=y
CONFIG_OABI_COMPAT=y
CONFIG_PCI=y
CONFIG_PCI_VERSATILE=y
CONFIG_SCSI=y
CONFIG_SCSI_SYM53C8XX_2=y
CONFIG_BLK_DEV_SD=y
CONFIG_BLK_DEV_SR=y
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_TMPFS=y
CONFIG_INPUT_EVDEV=y
CONFIG_EXT3_FS=y
CONFIG_EXT4_FS=y
CONFIG_VFAT_FS=y
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_ISO8859_1=y
CONFIG_FONT_8x16=y
CONFIG_LOGO=y
CONFIG_VFP=y
CONFIG_CGROUPS=y

CONFIG_MMC_BCM2835=y
CONFIG_MMC_BCM2835_DMA=y
CONFIG_DMA_BCM2708=y

CONFIG_FHANDLE=y

CONFIG_OVERLAY_FS=y

CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
CONFIG_FS_POSIX_ACL=y

CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y

CONFIG_MODVERSIONS=y

CONFIG_NET_9P=y
CONFIG_NET_9P_VIRTIO=y
CONFIG_9P_FS=y
CONFIG_9P_FS_POSIX_ACL=y

CONFIG_VIRTIO=y
CONFIG_VIRTIO_BLK=y
CONFIG_SCSI_VIRTIO=y
CONFIG_VIRTIO_NET=y
CONFIG_VIRTIO_CONSOLE=y
CONFIG_VIRTIO_BALLOON=y
CONFIG_VIRTIO_INPUT=y
CONFIG_VIRTIO_PCI=y

# for VIRTIO graphics
CONFIG_DRM=y
CONFIG_DRM_VIRTIO_GPU=y

CONFIG_HW_RANDOM=y
CONFIG_HW_RANDOM_VIRTIO=y


If you want to use GDB, then you may also want to add the following configurations to allow GDB to obtain sufficient information for debugging:

CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_KERNEL=y
CONFIG_GDB_SCRIPTS=y


What’s CONFIG_GDB_SCRIPTS for?

“Gdb comes with a powerful scripting interface for python. The kernel provides a collection of helper scripts that can simplify typical kernel debugging steps.” — source

### [Step 3] Compile the Kernel

Follow the instructions given in the Raspberry Pi’s page except that we’ll use versatile_defconfig revised in last step. The complete commands are given as follows.

1. Set up the environmental variables (please remember to change the path ~/tools/ to yours below)

echo PATH=\$PATH:~/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin >> ~/.bashrc source ~/.bashrc KERNEL=kernel7 2. Compile the kernel cd ~/linux make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- versatile_defconfig make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- zImage modules dtbs -j 6 ### [Step 4] Prepare a Disk Image After last step, we have our kernel image and device driver/module files ready. To run it in QEMU, we need a compatible disk image and it can be downloaded from Raspberry Pi’s website. As an example, we download a specific image version (the lite buster version released on 2020, 2/13) as follows: cd ~/linux wget https://downloads.raspberrypi.org/raspbian_lite/images/raspbian_lite-2020-02-14/2020-02-13-raspbian-buster-lite.zip unzip 2020-02-13-raspbian-buster-lite.zip  #### Optional Image Modification Some articles on the Internet have mentioned that the image may have to be revised to allow a successful boot. While I have no problem with booting up my kernel via QEMU without such a modification, I’m organizing the information here in case you (in fact, it’s me myself) bump into any relevant issue. What the following steps does is to disable by commenting. To do so, we need to mount the root partition encapsulated in the disk image to allow access to the files. 1. Mount the root partition First, let’s create a directory at /mnt/rpi_root (or any directory name you prefer) that we will use to mount our root partition: sudo mkdir /mnt/rpi_root  Then, there are several ways to find the right parameter to mount the root partition and you can choose either from below to proceed: • Use kpartx This is the easiest way among three presented here. However, you may have to install sudo apt install kpartx before you run the following commands to mount the root partition: sudo kpartx -av 2020-02-13-raspbian-buster-lite.img # then you should see the following messages: # add map loop0p1 (253:0): 0 524288 linear 7:0 8192 # add map loop0p2 (253:1): 0 3080192 linear 7:0 532480 sudo mount /dev/mapper/loop0p2 /mnt/rpi_root  • Use parted Determine the root partition offset in your image by using a cmd-styled tool: parted 2020-02-13-raspbian-buster-lite.img # Welcome to GNU Parted! Type 'help' to view a list of commands. # (parted) u # Unit? [compact]? B # (parted) print # Disk 2020-02-13-raspbian-buster-lite.img: 1849688064B # Sector size (logical/physical): 512B/512B # Partition Table: msdos # Disk Flags: # Number Start End Size Type File system Flags # 1 4194304B 272629759B 268435456B primary fat32 lba # 2 272629760B 1849688063B 1577058304B primary ext4 # (parted) q  What we are looking for is the offset of the root partition (the one with ext4 file system). In this case, it is 272629760. Then we can use the following command to mount such a partition: sudo mount -v -o offset=272629760 -t ext4 2020-02-13-raspbian-buster-lite.img /mnt/rpi_root  • Use fdisk This method requires some calculation, but I put it here since it is what a lot of articles proposed. We check the root partition offset by: fdisk -l 2020-02-13-raspbian-buster-lite.img # Disk 2020-02-13-raspbian-buster-lite.img: 1.7 GiB, 1849688064 bytes, 3612672 sectors # Units: sectors of 1 * 512 = 512 bytes # Sector size (logical/physical): 512 bytes / 512 bytes # I/O size (minimum/optimal): 512 bytes / 512 bytes # Disklabel type: dos # Disk identifier: 0x738a4d67 # Device Boot Start End Sectors Size Id Type # 2020-02-13-raspbian-buster-lite.img1 8192 532479 524288 256M c W95 FAT32 (LB # 2020-02-13-raspbian-buster-lite.img2 532480 3612671 3080192 1.5G 83 Linux  As you can see, the offset for the second partition is 532480 which is in sectors. To get the offset in bytes, we need to convert it by $$532480 \times 512 = 272629760$$. Then we can use the same command as above to mount such a partition: sudo mount -v -o offset=272629760 -t ext4 2020-02-13-raspbian-buster-lite.img /mnt/rpi_root  2. Disable preloaded shared libraries At this point, the root partition should already be mounted at /mnt/rpi_root where we can easily access. Open the file /etc/ld.so.preload with your favorite editor, for example: sudo vim /etc/ld.so.preload  then comment out every line with # in the file (in most cases, there should be only one line to be commented out in the file). Save and quit the file. You may also want to check the file /etc/fstab. What is /etc/ld.so.preload for? This file specifies a list of libraries to be loaded before any other libraries. See more details from some good references: ref1, ref2 3. Unmount the partition Use the following command to unmount the root partition: sudo umount /mnt/rpi_root  If you used kpartx earlier, you should also unmount from the tool: sudo kpartx -dv 2020-02-13-raspbian-buster-lite.img  ### [Step 5] Run QEMU With all the previous steps, we should get the kernel image, the device tree blob file and the disk image in the paths shown below: • ~/linux/arch/arm/boot/zImage • ~/linux/vmlinux • ~/linux/arch/arm/boot/dts/versatile-pb.dtb • ~/linux/2020-02-13-raspbian-buster-lite.img Then run the following QEMU command to start a VM running the kernel we just compiled: qemu-system-arm -kernel arch/arm/boot/zImage -cpu arm1176 -m 256 -M versatilepb -serial mon:stdio -append "root=/dev/sda2 panic=1 rw" -hda 2020-02-13-raspbian-buster-lite.img -no-reboot -dtb arch/arm/boot/dts/versatile-pb.dtb -redir tcp:5022::22 -device virtio-gpu-pci -device virtio-rng-pci  or qemu-system-arm -kernel arch/arm/boot/zImage -cpu arm1176 -m 256 -M versatilepb -serial mon:stdio -append "console=ttyAMA0 root=/dev/sda2 rootfstype=ext4 loglevel=8 rootwait fsck.repair=yes memtest=1 panic=1 rw" -drive file=2020-02-13-raspbian-buster-lite.img,format=raw -no-reboot -dtb arch/arm/boot/dts/versatile-pb.dtb -redir tcp:5022::22 -device virtio-gpu-pci -device virtio-rng-pci  What’s -device virtio-gpu-pci and -device virtio-rng-pci? The part -device virtio-gpu-pci (ref) avoids a DRM timeout on waiting for some non-existed driver signals. However, I am unable to see/use an emulated screen (i.e., what I see is all black) with this option. Remove this option if you really need GUI. The part -device virtio-rng-pci (ref) is to support a (virtual) hardware random number generator (RNG) device in QEMU. You can use the following command to verify if RNG is supported (returning none on unsupported): cat /sys/devices/virtual/misc/hw_random/rng_current  The part -redir tcp:5022::22 allows us to externally SSH to our QEMU VM. To do so, first enable the SSH service in your VM: # enable SSH on bootup sudo systemctl enable ssh # start SSH server now sudo systemctl start ssh  Then you can SSH to your VM from your computer by: ssh pi@127.0.0.1 -p 5022  Networking? With the above QEMU launching command your network should work by default — you should be able to link to your host’s network. Note that you will not get response from ping, so don’t use ping to verify your network (use something like sudo apt update instead. If your network doesn’t work. Then try to append the following two parameters in your QEMU launch command (source): -net user,hostfwd=tcp::5022-:22,vlan=0 -net nic,vlan=0  ### [Step 6] Run QEMU with GDB With necessary configurations set in defconfig in Step 2, we can connect a GDB to our running QEMU kernel. To allow a GDB to connect to the QEMU kernel, we need to add -s -S as the QEMU arguments. Such arguments make the QEMU kernel stop and wait for GDB to connect. So, the full command to start a QEMU kernel looks like this: qemu-system-arm -kernel arch/arm/boot/zImage -cpu arm1176 -m 256 -M versatilepb -serial mon:stdio -append "console=ttyAMA0 root=/dev/sda2 rootfstype=ext4 loglevel=8 rootwait fsck.repair=yes memtest=1 panic=1 rw" -drive file=2020-02-13-raspbian-buster-lite.img,format=raw -no-reboot -dtb arch/arm/boot/dts/versatile-pb.dtb -redir tcp:5022::22 -device virtio-gpu-pci -device virtio-rng-pci -s -S  Then, in another terminal, you can start gdb-multiarch and connect to your QEMU’ed kernel by: gdb-multiarch # enable Python scripts for kernel debugging with lx- commands python gdb.COMPLETE_EXPRESSION = gdb.COMPLETE_SYMBOL add-auto-load-safe-path scripts/gdb/vmlinux-gdb.py # read symbols file vmlinux # connect to the GDB server at the default address target remote :1234 # check the Linux version lx-version # run continue  or with using a one-liner to get everything settled while starting the GDB: gdb-multiarch -ex "python gdb.COMPLETE_EXPRESSION = gdb.COMPLETE_SYMBOL" -ex "add-auto-load-safe-path scripts/gdb/vmlinux-gdb.py" -ex "file vmlinux" -ex "target remote :1234"  GDB Doesn’t Work? In case you missed it, you should be using gdb-multiarch rather than the usual gdb since we are debugging a QEMU kernel running ARM instructions. If you haven’t already, install gdb-multiarch by using: sudo apt install gdb-multiarch  See value optimized out When Printing Variables in GDB? Linux kernel must be compiled with -O2 optimization level. As a result, some variables are being optimized and they are not traceable in GDB (since they are not there literally). Many people have been asking around if the optimization level can be decreased to avoid such an issue for debugging. Unfortunately, there is no one easy solution that fits all. There may be two workarounds as I searched on the Internet: • Some people point the direction of using the newly introduced optimization level -Og but I personally have not tested yet. • There is also a way to exclude those source files that you don’t want them to be optimized. To do so, go to the directory where your chosen source files are and edit the Makefile. As an example, let’s say I want to exclude kernel/sched/deadline.c. Then I should edit kernel/sched/Makefile to add the following lines: kernel/sched/Makefile CFLAGS_REMOVE_deadline.o := -O2 CFLAGS_deadline.o := -O0  Then save/recompile the kernel and then you should be able to inspect those variables (and even single-stepping through the code!) See an Empty history after Rebooting? If your history works well within your current session but shows nothing after rebooting, then you can add history -a in the file ~/.bash_logout. This way, the history in current session will automatically be appended to the history file ~/.bash_history when the session is terminating. The file should look like this with such an additional command: ~/.bash_logout # ~/.bash_logout: executed by bash(1) when login shell exits. # append history to .bash_history upon leaving hisotry -a # when leaving the console clear the screen to increase privacy if [ "$SHLVL" = 1 ]; then
[ -x /usr/bin/clear_console ] && /usr/bin/clear_console -q
fi


### References

• [ref] This Github repository contains a few pre-compiled kernel images and a useful automation script for patching and compiling the Raspberry Pi Linux kernel for running with QEMU.
• [ref] Instructions for setting up QEMU for Raspberry Pi kernel.
• [ref] Another good introduction for running a QEMU Raspberry Pi kernel.