For those who work on the kernel stuff for Raspberry Pi boards, it’s not uncommon to play the SD card plug-and-unplug game every time you cross-compile your modified kernel and want to test it. A more elegant alternative is to run and debug your kernel in a virtual machine on your computer. Here, we are talking about running your custom kernel with QEMU.
Browsing around on the Internet, I’ve found some useful resources and pages that teach us how a given Raspberry Pi kernel image can be run with QEMU. Unfortunately, only a few of them include clues for making a custom kernel work with QEMU from scratch. Yet, they are not well documented and that’s why I’m summarizing the steps in this post.
Assumptions and LimitationsIn this post I mainly target running my custom Linux kernel in QEMU for Raspberry Pi 3B (RPi-3B). However, QEMU has not yet supported all the hardware on RPi-3B. For this reason, we’ll use the versatilepb machine type in QEMU to run our kernel. Obviously some drivers/modules will not work with this setup and this won’t be the right post for you if you need them working.
[Step 0] Setup
Before we begin, please make sure you have the right environment being set up correctly:
ARM Cross-Compiler Tool Chain and Build Dependencies
You can get the compiler as instructed here or using the commands below:
git clone https://github.com/raspberrypi/tools ~/tools sudo apt update sudo apt install git bison flex libssl-dev
Linux Kernel Source Code from Raspberry Pi’s Git Repo
git clone https://github.com/raspberrypi/linux ~/ cd ~/linux git fetch git checkout -b rpi-4.19.y origin/rpi-4.19.y
QEMU for ARM
sudo apt install qemu-system-arm
(Optional) GDB for ARM
sudo apt install gdb-multiarch
After this step, I assume that you have your tools and linux source code in the following paths:
~/tools/: Raspberry Pi’s compiler tool set ~/linux/: your Linux kernel source code folder
[Step 1] Patch Your Kernel
As mentioned earlier, we’ll use the machine type versatilepb to run our kernel in QEMU. For this, we need to patch our kernel for supporting versatilepb.
To do so, run the following commands to download the patch file and patch your kernel.
cd ~/linux wget https://gist.githubusercontent.com/cchen140/0f06c4dd9c6c2cb0a219a4c523964699/raw/linux-arm-versatile.patch patch -p1 < linux-arm-versatile.patch
[Step 2] Revise defconfig
Next, some defconfig configurations must also be applied to make the kernel work in a versatile-pb machine in QEMU. Append all the configurations here (originally from here) or copy from below to your defconfig file located at
CONFIG_CPU_V6=y CONFIG_ARM_ERRATA_411920=y CONFIG_ARM_ERRATA_364296=y CONFIG_AEABI=y CONFIG_OABI_COMPAT=y CONFIG_PCI=y CONFIG_PCI_VERSATILE=y CONFIG_SCSI=y CONFIG_SCSI_SYM53C8XX_2=y CONFIG_BLK_DEV_SD=y CONFIG_BLK_DEV_SR=y CONFIG_DEVTMPFS=y CONFIG_DEVTMPFS_MOUNT=y CONFIG_TMPFS=y CONFIG_INPUT_EVDEV=y CONFIG_EXT3_FS=y CONFIG_EXT4_FS=y CONFIG_VFAT_FS=y CONFIG_NLS_CODEPAGE_437=y CONFIG_NLS_ISO8859_1=y CONFIG_FONT_8x16=y CONFIG_LOGO=y CONFIG_VFP=y CONFIG_CGROUPS=y CONFIG_MMC_BCM2835=y CONFIG_MMC_BCM2835_DMA=y CONFIG_DMADEVICES=y CONFIG_DMA_BCM2708=y CONFIG_FHANDLE=y CONFIG_OVERLAY_FS=y CONFIG_EXT4_FS_POSIX_ACL=y CONFIG_EXT4_FS_SECURITY=y CONFIG_FS_POSIX_ACL=y CONFIG_IKCONFIG=y CONFIG_IKCONFIG_PROC=y CONFIG_MODVERSIONS=y CONFIG_NET_9P=y CONFIG_NET_9P_VIRTIO=y CONFIG_9P_FS=y CONFIG_9P_FS_POSIX_ACL=y CONFIG_VIRTIO=y CONFIG_VIRTIO_BLK=y CONFIG_SCSI_VIRTIO=y CONFIG_VIRTIO_NET=y CONFIG_VIRTIO_CONSOLE=y CONFIG_VIRTIO_BALLOON=y CONFIG_VIRTIO_INPUT=y CONFIG_VIRTIO_PCI=y # for VIRTIO graphics CONFIG_DRM=y CONFIG_DRM_VIRTIO_GPU=y CONFIG_HW_RANDOM=y CONFIG_HW_RANDOM_VIRTIO=y
If you want to use GDB, then you may also want to add the following configurations to allow GDB to obtain sufficient information for debugging:
CONFIG_DEBUG_INFO=y CONFIG_DEBUG_KERNEL=y CONFIG_GDB_SCRIPTS=y
“Gdb comes with a powerful scripting interface for python. The kernel provides a collection of helper scripts that can simplify typical kernel debugging steps.” — source
[Step 3] Compile the Kernel
Follow the instructions given in the Raspberry Pi’s page except that we’ll use
1. Set up the environmental variables (please remember to change the path
echo PATH=\$PATH:~/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin >> ~/.bashrc source ~/.bashrc KERNEL=kernel7
2. Compile the kernel
cd ~/linux make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- versatile_defconfig make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- zImage modules dtbs -j 6
[Step 4] Prepare a Disk Image
After last step, we have our kernel image and device driver/module files ready. To run it in QEMU, we need a compatible disk image and it can be downloaded from Raspberry Pi’s website. As an example, we download a specific image version (the lite buster version released on 2020, 2/13) as follows:
cd ~/linux wget https://downloads.raspberrypi.org/raspbian_lite/images/raspbian_lite-2020-02-14/2020-02-13-raspbian-buster-lite.zip unzip 2020-02-13-raspbian-buster-lite.zip
Optional Image ModificationSome articles on the Internet have mentioned that the image may have to be revised to allow a successful boot. While I have no problem with booting up my kernel via QEMU without such a modification, I’m organizing the information here in case you (in fact, it’s me myself) bump into any relevant issue. What the following steps does is to disable by commenting. To do so, we need to mount the root partition encapsulated in the disk image to allow access to the files.
Mount the root partition
First, let’s create a directory at
/mnt/rpi_root(or any directory name you prefer) that we will use to mount our root partition:
sudo mkdir /mnt/rpi_root
Then, there are several ways to find the right parameter to mount the root partition and you can choose either from below to proceed:
This is the easiest way among three presented here. However, you may have to install
sudo apt install kpartxbefore you run the following commands to mount the root partition:
sudo kpartx -av 2020-02-13-raspbian-buster-lite.img # then you should see the following messages: # add map loop0p1 (253:0): 0 524288 linear 7:0 8192 # add map loop0p2 (253:1): 0 3080192 linear 7:0 532480 sudo mount /dev/mapper/loop0p2 /mnt/rpi_root
Determine the root partition offset in your image by using a cmd-styled tool:
parted 2020-02-13-raspbian-buster-lite.img # Welcome to GNU Parted! Type 'help' to view a list of commands. # (parted) u # Unit? [compact]? B # (parted) print # Disk 2020-02-13-raspbian-buster-lite.img: 1849688064B # Sector size (logical/physical): 512B/512B # Partition Table: msdos # Disk Flags: # Number Start End Size Type File system Flags # 1 4194304B 272629759B 268435456B primary fat32 lba # 2 272629760B 1849688063B 1577058304B primary ext4 # (parted) q
What we are looking for is the offset of the root partition (the one with ext4 file system). In this case, it is 272629760. Then we can use the following command to mount such a partition:
sudo mount -v -o offset=272629760 -t ext4 2020-02-13-raspbian-buster-lite.img /mnt/rpi_root
This method requires some calculation, but I put it here since it is what a lot of articles proposed. We check the root partition offset by:
fdisk -l 2020-02-13-raspbian-buster-lite.img # Disk 2020-02-13-raspbian-buster-lite.img: 1.7 GiB, 1849688064 bytes, 3612672 sectors # Units: sectors of 1 * 512 = 512 bytes # Sector size (logical/physical): 512 bytes / 512 bytes # I/O size (minimum/optimal): 512 bytes / 512 bytes # Disklabel type: dos # Disk identifier: 0x738a4d67 # Device Boot Start End Sectors Size Id Type # 2020-02-13-raspbian-buster-lite.img1 8192 532479 524288 256M c W95 FAT32 (LB # 2020-02-13-raspbian-buster-lite.img2 532480 3612671 3080192 1.5G 83 Linux
As you can see, the offset for the second partition is 532480 which is in sectors. To get the offset in bytes, we need to convert it by \(532480 \times 512 = 272629760\). Then we can use the same command as above to mount such a partition:
sudo mount -v -o offset=272629760 -t ext4 2020-02-13-raspbian-buster-lite.img /mnt/rpi_root
Disable preloaded shared libraries
At this point, the root partition should already be mounted at
/mnt/rpi_rootwhere we can easily access. Open the file /etc/ld.so.preloadwith your favorite editor, for example:
sudo vim /etc/ld.so.preload
then comment out every line with
#in the file (in most cases, there should be only one line to be commented out in the file). Save and quit the file.
You may also want to check the file
Unmount the partition
Use the following command to unmount the root partition:
sudo umount /mnt/rpi_root
If you used
kpartxearlier, you should also unmount from the tool:
sudo kpartx -dv 2020-02-13-raspbian-buster-lite.img
[Step 5] Run QEMU
With all the previous steps, we should get the kernel image, the device tree blob file and the disk image in the paths shown below:
~/linux/arch/arm/boot/zImage ~/linux/vmlinux ~/linux/arch/arm/boot/dts/versatile-pb.dtb ~/linux/2020-02-13-raspbian-buster-lite.img
Then run the following QEMU command to start a VM running the kernel we just compiled:
qemu-system-arm -kernel arch/arm/boot/zImage -cpu arm1176 -m 256 -M versatilepb -serial mon:stdio -append "root=/dev/sda2 panic=1 rw" -hda 2020-02-13-raspbian-buster-lite.img -no-reboot -dtb arch/arm/boot/dts/versatile-pb.dtb -redir tcp:5022::22 -device virtio-gpu-pci -device virtio-rng-pci
qemu-system-arm -kernel arch/arm/boot/zImage -cpu arm1176 -m 256 -M versatilepb -serial mon:stdio -append "console=ttyAMA0 root=/dev/sda2 rootfstype=ext4 loglevel=8 rootwait fsck.repair=yes memtest=1 panic=1 rw" -drive file=2020-02-13-raspbian-buster-lite.img,format=raw -no-reboot -dtb arch/arm/boot/dts/versatile-pb.dtb -redir tcp:5022::22 -device virtio-gpu-pci -device virtio-rng-pci
-device virtio-gpu-pci and
-device virtio-gpu-pci (ref) avoids a DRM timeout on waiting for some non-existed driver signals.
However, I am unable to see/use an emulated screen (i.e., what I see is all black) with this option. Remove this option if you really need GUI.
-device virtio-rng-pci (ref) is to support a (virtual) hardware random number generator (RNG) device in QEMU. You can use the following command to verify if RNG is supported (returning
none on unsupported):
-redir tcp:5022::22 allows us to externally SSH to our QEMU VM. To do so, first enable the SSH service in your VM:
# enable SSH on bootup sudo systemctl enable ssh # start SSH server now sudo systemctl start ssh
Then you can SSH to your VM from your computer by:
ssh email@example.com -p 5022
With the above QEMU launching command your network should work by default — you should be able to link to your host’s network. Note that you will not get response from
ping, so don’t use
ping to verify your network (use something like
sudo apt update instead.
If your network doesn’t work. Then try to append the following two parameters in your QEMU launch command (source):
-net user,hostfwd=tcp::5022-:22,vlan=0 -net nic,vlan=0
[Step 6] Run QEMU with GDB
With necessary configurations set in defconfig in Step 2, we can connect a GDB to our running QEMU kernel.
To allow a GDB to connect to the QEMU kernel, we need to add
-s -S as the QEMU arguments.
Such arguments make the QEMU kernel stop and wait for GDB to connect.
So, the full command to start a QEMU kernel looks like this:
qemu-system-arm -kernel arch/arm/boot/zImage -cpu arm1176 -m 256 -M versatilepb -serial mon:stdio -append "console=ttyAMA0 root=/dev/sda2 rootfstype=ext4 loglevel=8 rootwait fsck.repair=yes memtest=1 panic=1 rw" -drive file=2020-02-13-raspbian-buster-lite.img,format=raw -no-reboot -dtb arch/arm/boot/dts/versatile-pb.dtb -redir tcp:5022::22 -device virtio-gpu-pci -device virtio-rng-pci -s -S
Then, in another terminal, you can start
gdb-multiarch and connect to your QEMU’ed kernel by:
# enable Python scripts for kernel debugging with lx- commands python gdb.COMPLETE_EXPRESSION = gdb.COMPLETE_SYMBOL add-auto-load-safe-path scripts/gdb/vmlinux-gdb.py # read symbols file vmlinux # connect to the GDB server at the default address target remote :1234 # check the Linux version lx-version # run continue
or with using a one-liner to get everything settled while starting the GDB:
gdb-multiarch -ex "python gdb.COMPLETE_EXPRESSION = gdb.COMPLETE_SYMBOL" -ex "add-auto-load-safe-path scripts/gdb/vmlinux-gdb.py" -ex "file vmlinux" -ex "target remote :1234"
GDB Doesn’t Work?
In case you missed it, you should be using
gdb-multiarch rather than the usual
gdb since we are debugging a QEMU kernel running ARM instructions.
If you haven’t already, install
gdb-multiarch by using:
sudo apt install gdb-multiarch
value optimized out When Printing Variables in GDB?
Linux kernel must be compiled with
-O2 optimization level. As a result, some variables are being optimized and they are not traceable in GDB (since they are not there literally).
Many people have been asking around if the optimization level can be decreased to avoid such an issue for debugging. Unfortunately, there is no one easy solution that fits all. There may be two workarounds as I searched on the Internet:
Some people point the direction of using the newly introduced optimization level
-Ogbut I personally have not tested yet.
There is also a way to exclude those source files that you don’t want them to be optimized. To do so, go to the directory where your chosen source files are and edit the
Makefile. As an example, let’s say I want to exclude kernel/sched/deadline.c. Then I should edit kernel/sched/Makefileto add the following lines: kernel/sched/Makefile
CFLAGS_REMOVE_deadline.o := -O2 CFLAGS_deadline.o := -O0
Then save/recompile the kernel and then you should be able to inspect those variables (and even single-stepping through the code!)
See an Empty
history after Rebooting?
history works well within your current session but shows nothing after rebooting, then you can add
history -a in the file
The file should look like this with such an additional command:
# ~/.bash_logout: executed by bash(1) when login shell exits. # append history to .bash_history upon leaving hisotry -a # when leaving the console clear the screen to increase privacy if [ "$SHLVL" = 1 ]; then [ -x /usr/bin/clear_console ] && /usr/bin/clear_console -q fi
- [ref] This Github repository contains a few pre-compiled kernel images and a useful automation script for patching and compiling the Raspberry Pi Linux kernel for running with QEMU.
- [ref] Instructions for setting up QEMU for Raspberry Pi kernel.
- [ref] Another good introduction for running a QEMU Raspberry Pi kernel.