Tags:

Compiling Raspbian (Linux) Kernel for Running in QEMU

In this post, we go through the steps for cross-compiling the Linux kernel from source code for Raspberry Pi and running it with QEMU.

For those who work on the kernel stuff for Raspberry Pi boards, it’s not uncommon to play the SD card plug-and-unplug game every time you cross-compile your modified kernel and want to test it. A more elegant alternative is to run and debug your kernel in a virtual machine on your computer. Here, we are talking about running your custom kernel with QEMU.

Browsing around on the Internet, I’ve found some useful resources and pages that teach us how a given Raspberry Pi kernel image can be run with QEMU. Unfortunately, only a few of them include clues for making a custom kernel work with QEMU from scratch. Yet, they are not well documented and that’s why I’m summarizing the steps in this post.

Go to top

Assumptions and Limitations

In this post I mainly target running my custom Linux kernel in QEMU for Raspberry Pi 3B (RPi-3B). However, QEMU has not yet supported all the hardware on RPi-3B. For this reason, we’ll use the versatilepb machine type in QEMU to run our kernel. Obviously some drivers/modules will not work with this setup and this won’t be the right post for you if you need them working.
Go to top

[Step 0] Setup

Before we begin, please make sure you have the right environment being set up correctly:

  • ARM Cross-Compiler Tool Chain and Build Dependencies

    You can get the compiler as instructed here or using the commands below:

    git clone https://github.com/raspberrypi/tools ~/tools
sudo apt update
sudo apt install git bison flex libssl-dev

  • Linux Kernel Source Code from Raspberry Pi’s Git Repo

    git clone https://github.com/raspberrypi/linux ~/
cd ~/linux
git fetch
git checkout -b rpi-4.19.y origin/rpi-4.19.y

  • QEMU for ARM

    sudo apt install qemu-system-arm

  • (Optional) GDB for ARM

    sudo apt install gdb-multiarch

After this step, I assume that you have your tools and linux source code in the following paths:

  • ~/tools/: Raspberry Pi’s compiler tool set
  • ~/linux/: your Linux kernel source code folder
Go to top

[Step 1] Patch Your Kernel

As mentioned earlier, we’ll use the machine type versatilepb to run our kernel in QEMU. For this, we need to patch our kernel for supporting versatilepb.

To do so, run the following commands to download the patch file and patch your kernel.

cd ~/linux
wget https://gist.githubusercontent.com/cchen140/0f06c4dd9c6c2cb0a219a4c523964699/raw/linux-arm-versatile.patch
patch -p1 < linux-arm-versatile.patch
Go to top

[Step 2] Revise defconfig

Next, some defconfig configurations must also be applied to make the kernel work in a versatile-pb machine in QEMU. Append all the configurations here (originally from here) or copy from below to your defconfig file located at ~/linux/arch/arm/configs/versatile_defconfig.

CONFIG_CPU_V6=y
CONFIG_ARM_ERRATA_411920=y
CONFIG_ARM_ERRATA_364296=y
CONFIG_AEABI=y
CONFIG_OABI_COMPAT=y
CONFIG_PCI=y
CONFIG_PCI_VERSATILE=y
CONFIG_SCSI=y
CONFIG_SCSI_SYM53C8XX_2=y
CONFIG_BLK_DEV_SD=y
CONFIG_BLK_DEV_SR=y
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_TMPFS=y
CONFIG_INPUT_EVDEV=y
CONFIG_EXT3_FS=y
CONFIG_EXT4_FS=y
CONFIG_VFAT_FS=y
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_ISO8859_1=y
CONFIG_FONT_8x16=y
CONFIG_LOGO=y
CONFIG_VFP=y
CONFIG_CGROUPS=y

CONFIG_MMC_BCM2835=y
CONFIG_MMC_BCM2835_DMA=y
CONFIG_DMADEVICES=y
CONFIG_DMA_BCM2708=y

CONFIG_FHANDLE=y

CONFIG_OVERLAY_FS=y

CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
CONFIG_FS_POSIX_ACL=y

CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y

CONFIG_MODVERSIONS=y

CONFIG_NET_9P=y
CONFIG_NET_9P_VIRTIO=y
CONFIG_9P_FS=y
CONFIG_9P_FS_POSIX_ACL=y

CONFIG_VIRTIO=y
CONFIG_VIRTIO_BLK=y
CONFIG_SCSI_VIRTIO=y
CONFIG_VIRTIO_NET=y
CONFIG_VIRTIO_CONSOLE=y
CONFIG_VIRTIO_BALLOON=y
CONFIG_VIRTIO_INPUT=y
CONFIG_VIRTIO_PCI=y

# for VIRTIO graphics
CONFIG_DRM=y
CONFIG_DRM_VIRTIO_GPU=y

CONFIG_HW_RANDOM=y
CONFIG_HW_RANDOM_VIRTIO=y

If you want to use GDB, then you may also want to add the following configurations to allow GDB to obtain sufficient information for debugging:

CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_KERNEL=y
CONFIG_GDB_SCRIPTS=y

What’s CONFIG_GDB_SCRIPTS for?

“Gdb comes with a powerful scripting interface for python. The kernel provides a collection of helper scripts that can simplify typical kernel debugging steps.” — source

Go to top

[Step 3] Compile the Kernel

Follow the instructions given in the Raspberry Pi’s page except that we’ll use versatile_defconfig revised in last step. The complete commands are given as follows.

1. Set up the environmental variables (please remember to change the path ~/tools/ to yours below)

echo PATH=\$PATH:~/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin >> ~/.bashrc
source ~/.bashrc
KERNEL=kernel7

2. Compile the kernel

cd ~/linux
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- versatile_defconfig
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- zImage modules dtbs -j 6
Go to top

[Step 4] Prepare a Disk Image

After last step, we have our kernel image and device driver/module files ready. To run it in QEMU, we need a compatible disk image and it can be downloaded from Raspberry Pi’s website. As an example, we download a specific image version (the lite buster version released on 2020, 2/13) as follows:

cd ~/linux
wget https://downloads.raspberrypi.org/raspbian_lite/images/raspbian_lite-2020-02-14/2020-02-13-raspbian-buster-lite.zip
unzip 2020-02-13-raspbian-buster-lite.zip

Optional Image Modification

Some articles on the Internet have mentioned that the image may have to be revised to allow a successful boot. While I have no problem with booting up my kernel via QEMU without such a modification, I’m organizing the information here in case you (in fact, it’s me myself) bump into any relevant issue. What the following steps does is to disable by commenting. To do so, we need to mount the root partition encapsulated in the disk image to allow access to the files.

  1. Mount the root partition

    First, let’s create a directory at /mnt/rpi_root (or any directory name you prefer) that we will use to mount our root partition:

    sudo mkdir /mnt/rpi_root

    Then, there are several ways to find the right parameter to mount the root partition and you can choose either from below to proceed:

    • Use kpartx

      This is the easiest way among three presented here. However, you may have to install sudo apt install kpartx before you run the following commands to mount the root partition:

      sudo kpartx -av 2020-02-13-raspbian-buster-lite.img
# then you should see the following messages:
# add map loop0p1 (253:0): 0 524288 linear 7:0 8192
# add map loop0p2 (253:1): 0 3080192 linear 7:0 532480

sudo mount /dev/mapper/loop0p2 /mnt/rpi_root

    • Use parted

      Determine the root partition offset in your image by using a cmd-styled tool: 

      parted 2020-02-13-raspbian-buster-lite.img

# Welcome to GNU Parted! Type 'help' to view a list of commands.
# (parted)
u

# Unit? [compact]?
B

# (parted)
print

# Disk 2020-02-13-raspbian-buster-lite.img: 1849688064B
# Sector size (logical/physical): 512B/512B
# Partition Table: msdos
# Disk Flags:
# Number Start       End          Size         Type     File system  Flags
# 1      4194304B    272629759B   268435456B   primary  fat32        lba
# 2      272629760B  1849688063B  1577058304B  primary  ext4

# (parted)
q

      What we are looking for is the offset of the root partition (the one with ext4 file system). In this case, it is 272629760. Then we can use the following command to mount such a partition:

      sudo mount -v -o offset=272629760 -t ext4 2020-02-13-raspbian-buster-lite.img /mnt/rpi_root

    • Use fdisk

      This method requires some calculation, but I put it here since it is what a lot of articles proposed. We check the root partition offset by:

      fdisk -l 2020-02-13-raspbian-buster-lite.img

# Disk 2020-02-13-raspbian-buster-lite.img: 1.7 GiB, 1849688064 bytes, 3612672 sectors
# Units: sectors of 1 * 512 = 512 bytes
# Sector size (logical/physical): 512 bytes / 512 bytes
# I/O size (minimum/optimal): 512 bytes / 512 bytes
# Disklabel type: dos
# Disk identifier: 0x738a4d67
# Device                                Boot  Start     End  Sectors  Size  Id  Type
# 2020-02-13-raspbian-buster-lite.img1         8192  532479   524288  256M   c  W95 FAT32 (LB
# 2020-02-13-raspbian-buster-lite.img2       532480 3612671  3080192  1.5G  83  Linux

      As you can see, the offset for the second partition is 532480 which is in sectors. To get the offset in bytes, we need to convert it by \(532480 \times 512 = 272629760\). Then we can use the same command as above to mount such a partition:

      sudo mount -v -o offset=272629760 -t ext4 2020-02-13-raspbian-buster-lite.img /mnt/rpi_root

  2. Disable preloaded shared libraries

    At this point, the root partition should already be mounted at /mnt/rpi_root where we can easily access. Open the file /etc/ld.so.preload with your favorite editor, for example:

    sudo vim /etc/ld.so.preload

    then comment out every line with # in the file (in most cases, there should be only one line to be commented out in the file). Save and quit the file.

    You may also want to check the file /etc/fstab.

    What is /etc/ld.so.preload for?

    This file specifies a list of libraries to be loaded before any other libraries. See more details from some good references: ref1, ref2

  3. Unmount the partition

    Use the following command to unmount the root partition:

    sudo umount /mnt/rpi_root

    If you used kpartx earlier, you should also unmount from the tool:

    sudo kpartx -dv 2020-02-13-raspbian-buster-lite.img
Go to top

[Step 5] Run QEMU

With all the previous steps, we should get the kernel image, the device tree blob file and the disk image in the paths shown below:

  • ~/linux/arch/arm/boot/zImage
  • ~/linux/vmlinux
  • ~/linux/arch/arm/boot/dts/versatile-pb.dtb
  • ~/linux/2020-02-13-raspbian-buster-lite.img

Then run the following QEMU command to start a VM running the kernel we just compiled:

qemu-system-arm -kernel arch/arm/boot/zImage -cpu arm1176 -m 256 -M versatilepb -serial mon:stdio -append "root=/dev/sda2 panic=1 rw" -hda 2020-02-13-raspbian-buster-lite.img -no-reboot -dtb arch/arm/boot/dts/versatile-pb.dtb -redir tcp:5022::22 -device virtio-gpu-pci -device virtio-rng-pci

or

qemu-system-arm -kernel arch/arm/boot/zImage -cpu arm1176 -m 256 -M versatilepb -serial mon:stdio -append "console=ttyAMA0 root=/dev/sda2 rootfstype=ext4  loglevel=8 rootwait fsck.repair=yes memtest=1 panic=1 rw" -drive file=2020-02-13-raspbian-buster-lite.img,format=raw -no-reboot -dtb arch/arm/boot/dts/versatile-pb.dtb -redir tcp:5022::22 -device virtio-gpu-pci -device virtio-rng-pci

What’s -device virtio-gpu-pci and -device virtio-rng-pci?

The part -device virtio-gpu-pci (ref) avoids a DRM timeout on waiting for some non-existed driver signals. However, I am unable to see/use an emulated screen (i.e., what I see is all black) with this option. Remove this option if you really need GUI.

The part -device virtio-rng-pci (ref) is to support a (virtual) hardware random number generator (RNG) device in QEMU. You can use the following command to verify if RNG is supported (returning none on unsupported):

cat /sys/devices/virtual/misc/hw_random/rng_current

The part -redir tcp:5022::22 allows us to externally SSH to our QEMU VM. To do so, first enable the SSH service in your VM:

# enable SSH on bootup 
sudo systemctl enable ssh

# start SSH server now
sudo systemctl start ssh

Then you can SSH to your VM from your computer by:

ssh pi@127.0.0.1 -p 5022

Networking?

With the above QEMU launching command your network should work by default — you should be able to link to your host’s network. Note that you will not get response from ping, so don’t use ping to verify your network (use something like sudo apt update instead. If your network doesn’t work. Then try to append the following two parameters in your QEMU launch command (source):

-net user,hostfwd=tcp::5022-:22,vlan=0 -net nic,vlan=0
Go to top

[Step 6] Run QEMU with GDB

With necessary configurations set in defconfig in Step 2, we can connect a GDB to our running QEMU kernel. To allow a GDB to connect to the QEMU kernel, we need to add -s -S as the QEMU arguments. Such arguments make the QEMU kernel stop and wait for GDB to connect. So, the full command to start a QEMU kernel looks like this:

qemu-system-arm -kernel arch/arm/boot/zImage -cpu arm1176 -m 256 -M versatilepb -serial mon:stdio -append "console=ttyAMA0 root=/dev/sda2 rootfstype=ext4  loglevel=8 rootwait fsck.repair=yes memtest=1 panic=1 rw" -drive file=2020-02-13-raspbian-buster-lite.img,format=raw -no-reboot -dtb arch/arm/boot/dts/versatile-pb.dtb -redir tcp:5022::22 -device virtio-gpu-pci -device virtio-rng-pci -s -S

Then, in another terminal, you can start gdb-multiarch and connect to your QEMU’ed kernel by:

gdb-multiarch 
# enable Python scripts for kernel debugging with lx- commands
python gdb.COMPLETE_EXPRESSION = gdb.COMPLETE_SYMBOL
add-auto-load-safe-path scripts/gdb/vmlinux-gdb.py

# read symbols
file vmlinux

# connect to the GDB server at the default address
target remote :1234

# check the Linux version
lx-version

# run
continue

or with using a one-liner to get everything settled while starting the GDB:

gdb-multiarch  -ex "python gdb.COMPLETE_EXPRESSION = gdb.COMPLETE_SYMBOL" -ex "add-auto-load-safe-path scripts/gdb/vmlinux-gdb.py" -ex "file vmlinux" -ex "target remote :1234"

GDB Doesn’t Work?

In case you missed it, you should be using gdb-multiarch rather than the usual gdb since we are debugging a QEMU kernel running ARM instructions. If you haven’t already, install gdb-multiarch by using:

sudo apt install gdb-multiarch

See value optimized out When Printing Variables in GDB?

Linux kernel must be compiled with -O2 optimization level. As a result, some variables are being optimized and they are not traceable in GDB (since they are not there literally).

Many people have been asking around if the optimization level can be decreased to avoid such an issue for debugging. Unfortunately, there is no one easy solution that fits all. There may be two workarounds as I searched on the Internet:

  • Some people point the direction of using the newly introduced optimization level -Og but I personally have not tested yet.

  • There is also a way to exclude those source files that you don’t want them to be optimized. To do so, go to the directory where your chosen source files are and edit the Makefile. As an example, let’s say I want to exclude kernel/sched/deadline.c. Then I should edit kernel/sched/Makefile to add the following lines:

    kernel/sched/Makefile 
    CFLAGS_REMOVE_deadline.o := -O2
CFLAGS_deadline.o := -O0

    Then save/recompile the kernel and then you should be able to inspect those variables (and even single-stepping through the code!)

See an Empty history after Rebooting?

If your history works well within your current session but shows nothing after rebooting, then you can add history -a in the file ~/.bash_logout. This way, the history in current session will automatically be appended to the history file ~/.bash_history when the session is terminating.

The file should look like this with such an additional command:

~/.bash_logout 
# ~/.bash_logout: executed by bash(1) when login shell exits.

# append history to .bash_history upon leaving
hisotry -a

# when leaving the console clear the screen to increase privacy

if [ "$SHLVL" = 1 ]; then
    [ -x /usr/bin/clear_console ] && /usr/bin/clear_console -q
fi
Go to top

References

  • [ref] This Github repository contains a few pre-compiled kernel images and a useful automation script for patching and compiling the Raspberry Pi Linux kernel for running with QEMU.
  • [ref] Instructions for setting up QEMU for Raspberry Pi kernel.
  • [ref] Another good introduction for running a QEMU Raspberry Pi kernel.

Was this post helpful?

1 thought on “Compiling Raspbian (Linux) Kernel for Running in QEMU”

Leave a Reply

Your email address will not be published.