Skip to content

How to build and setup for Raspberry PI QEMU user.md

Jump to bottom
Alexander Pavlov edited this page Feb 21, 2024 · 3 revisions

How to build and configure for Raspberry PI 3B/B+ using QEMU user mode.

First we need to install the QEMU user static and binfmt packages on HOST system.

sudo apt-get update && sudo apt-get install -y qemu qemu-user-static binfmt-support parted wget dosfstools zip
sudo adduser $USER kvm

Checking the status of the binfmt daemon.

sudo systemctl status systemd-binfmt.service
● systemd-binfmt.service - Set Up Additional Binary Formats
   Loaded: loaded (/lib/systemd/system/systemd-binfmt.service; static; vendor preset: enabled)
   Active: inactive (dead)
Condition: start condition failed at Mon 2022-06-06 17:19:44 MSK; 10 months 12 days ago
     Docs: man:systemd-binfmt.service(8)
           man:binfmt.d(5)
           https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html
           https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems

Prepare binfmt config for Rasberry PI.

sudo nano -w /etc/binfmt.d/qemu-arm-rpi.conf
:arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm-rpi:
sudo nano -w /etc/binfmt.d/qemu-aarch64-rpi.conf
:aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\xfc\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64-rpi:

Restart the binfmt service.

sudo systemctl restart systemd-binfmt.service

Checking the status of the binfmt daemon again.

sudo systemctl status systemd-binfmt.service
● systemd-binfmt.service - Set Up Additional Binary Formats
   Loaded: loaded (/lib/systemd/system/systemd-binfmt.service; static; vendor preset: enabled)
   Active: active (exited) since Wed 2023-04-19 19:28:36 MSK; 9s ago
     Docs: man:systemd-binfmt.service(8)
           man:binfmt.d(5)
           https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html
           https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
  Process: 6580 ExecStart=/lib/systemd/systemd-binfmt (code=exited, status=0/SUCCESS)
 Main PID: 6580 (code=exited, status=0/SUCCESS)

апр 19 19:28:36 debian systemd[1]: Starting Set Up Additional Binary Formats...
апр 19 19:28:36 debian systemd[1]: Started Set Up Additional Binary Formats.

Checking the presence of the binfmt magic:

sudo cat /proc/sys/fs/binfmt_misc/arm
enabled
interpreter /usr/bin/qemu-arm-rpi
flags: 
offset 0
magic 7f454c4601010100000000000000000002002800
mask ffffffffffffff00fffffffffffffffffeffffff

sudo cat /proc/sys/fs/binfmt_misc/aarch64
enabled
interpreter /usr/bin/qemu-aarch64-rpi
flags: 
offset 0
magic 7f454c460201010000000000000000000200b700
mask fffffffffffffffcfffffffffffffffffeffffff

Let's create a wrapper to switch between configuration for Raspberry PI 3B and Raspberry PI 4

sudo mkdir /home/qemu 
sudo mkdir /home/qemu/qemu-user
cd /home/qemu/qemu-user
nano -w qemu-wrapper.c
// From https://wiki.gentoo.org/wiki/Embedded_Handbook/General/Compiling_with_qemu_user_chroot
/*
 * pass arguments to qemu binary
 * QEMU_CPU environment variable can be unset
 */

#include <string.h>
#include <unistd.h>

#define XSTR(cpu) STR(cpu)
#define STR(cpu) #cpu

int main(int argc, char **argv, char **envp) {
	int this_len = strlen(argv[0]);
	char staticpath[this_len];
	char *newargv[argc + 3];

	newargv[0] = argv[0];
	newargv[1] = "-cpu";
	newargv[2] = XSTR(QEMU_CPU);

	memcpy(&newargv[3], &argv[1], sizeof(*argv) * (argc -1));
	newargv[argc + 2] = NULL;
	memcpy(staticpath, argv[0], this_len - 1);
	staticpath[this_len - 1] = 0;
	return execve(staticpath, newargv, envp);
}

Compiling wrappers.

sudo gcc -static qemu-wrapper.c -DQEMU_CPU=cortex-a7  -O3 -s -o /usr/bin/qemu-arm-rpi3
sudo gcc -static qemu-wrapper.c -DQEMU_CPU=cortex-a7  -O3 -s -o /usr/bin/qemu-arm-rpi4
sudo gcc -static qemu-wrapper.c -DQEMU_CPU=cortex-a53 -O3 -s -o /usr/bin/qemu-aarch64-rpi3
sudo gcc -static qemu-wrapper.c -DQEMU_CPU=cortex-a72 -O3 -s -o /usr/bin/qemu-aarch64-rpi4

Switching wrappers between configurations for Raspberry PI 3B and Raspberry PI 4 in the future will be using the creation of symbolic links.

Create directories for future guests.

sudo mkdir /home/qemu/qemu-user/raspios-armhf-lite
sudo mkdir /home/qemu/qemu-user/raspios-arm64-lite

Clone the repository containing the CPU information needed to emulate the Raspberry PI 3B and 4B

sudo git clone https://github.com/pguyot/arm-runner-action.git

Let's create scripts for logging into the guest system using chroot.

sudo nano -w /home/qemu/qemu-user/run_armhf_rpi3.sh
#!/bin/bash

cd /usr/bin
rm -f qemu-arm-rpi
rm -f qemu-aarch64-rpi
ln -sf qemu-arm-rpi3 qemu-arm-rpi
ln -sf qemu-aarch64-rpi3 qemu-aarch64-rpi

cd /home/qemu/qemu-user

cd raspios-armhf-lite
mount --bind /proc proc
mount --bind /sys sys
mount --bind /dev dev
mount --bind /dev/pts dev/pts
mount --bind /home/qemu/qemu-user/arm-runner-actio/cpuinfo/raspberrypi_3b proc/cpuinfo
chroot . /bin/bash --login
sudo nano -w /home/qemu/qemu-user/run_armhf_rpi4.sh
#!/bin/bash

cd /usr/bin
rm -f qemu-arm-rpi
rm -f qemu-aarch64-rpi
ln -sf qemu-arm-rpi4 qemu-arm-rpi
ln -sf qemu-aarch64-rpi4 qemu-aarch64-rpi

cd /home/qemu/qemu-user

cd raspios-armhf-lite
mount --bind /proc proc
mount --bind /sys sys
mount --bind /dev dev
mount --bind /dev/pts dev/pts
mount --bind /home/qemu/qemu-user/arm-runner-actio/cpuinfo/raspberrypi_4b proc/cpuinfo
chroot . /bin/bash --login
sudo nano -w /home/qemu/qemu-user/run_arm64_rpi3.sh
#!/bin/bash

cd /usr/bin
rm -f qemu-arm-rpi
rm -f qemu-aarch64-rpi
ln -sf qemu-arm-rpi3 qemu-arm-rpi
ln -sf qemu-aarch64-rpi3 qemu-aarch64-rpi

cd /home/qemu/qemu-user

cd raspios-arm64-lite
mount --bind /proc proc
mount --bind /sys sys
mount --bind /dev dev
mount --bind /dev/pts dev/pts
mount --bind /home/qemu/qemu-user/arm-runner-actio/cpuinfo/raspberrypi_3b proc/cpuinfo
chroot . /bin/bash --login
sudo nano -w /home/qemu/qemu-user/run_arm64_rpi4.sh
#!/bin/bash

cd /usr/bin
rm -f qemu-arm-rpi
rm -f qemu-aarch64-rpi
ln -sf qemu-arm-rpi4 qemu-arm-rpi
ln -sf qemu-aarch64-rpi4 qemu-aarch64-rpi

cd /home/qemu/qemu-user

cd raspios-arm64-lite
mount --bind /proc proc
mount --bind /sys sys
mount --bind /dev dev
mount --bind /dev/pts dev/pts
mount --bind /home/qemu/qemu-user/arm-runner-actio/cpuinfo/raspberrypi_4b proc/cpuinfo
chroot . /bin/bash --login

Now our HOST is completely ready.

Next, you need to download and unpack the images of the Raspberry PI operating system into the directories "/home/qemu/qemu-user/raspios-armhf-lite" and "/home/qemu/qemu-user/raspios-arm64-lite"

We go to the page https://www.raspberrypi.com/software/operating-systems/ and select the desired system. In Let's take Raspbian bullseye as an example, the current version is 2023-02-22-raspios-bullseye-armhf.img and 2023-02-22-raspios-bullseye-arm64.img.

Download and unpack the compressed image:

sudo wget http://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2023-02-22/2023-02-21-raspios-bullseye-armhf-lite.img.xz
sudo xz -d 2023-02-21-raspios-bullseye-armhf-lite.img.xz
sudo wget http://downloads.raspberrypi.org/raspios_lite_arm64/images/raspios_lite_arm64-2023-02-22/2023-02-21-raspios-bullseye-arm64-lite.img.xz
sudo xz -d 2023-02-21-raspios-bullseye-armhf-lite.img.xz

First, get information about the image:

LC_ALL=en_US.utf8 fdisk -lu 2023-02-21-raspios-bullseye-armhf-lite.img
Disk 2023-02-21-raspios-bullseye-armhf-lite.img: 1.83 GiB, 1962934272 bytes, 3833856 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x34f4435e

Device                                      Boot  Start     End Sectors  Size Id Type
2023-02-21-raspios-bullseye-armhf-lite.img1        8192  532479  524288  256M  c W95 FAT32 (LBA)
2023-02-21-raspios-bullseye-armhf-lite.img2      532480 3833855 3301376  1.6G 83 Linux

LC_ALL=en_US.utf8 fdisk -lu 2023-02-21-raspios-bullseye-arm64-lite.img
Disk 2023-02-21-raspios-bullseye-arm64-lite.img: 1.95 GiB, 2097152000 bytes, 4096000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xe088fd39

Device                                      Boot  Start     End Sectors  Size Id Type
2023-02-21-raspios-bullseye-arm64-lite.img1        8192  532479  524288  256M  c W95 FAT32 (LBA)
2023-02-21-raspios-bullseye-arm64-lite.img2      532480 4095999 3563520  1.7G 83 Linux

We hook the entire first section (starts from sector 8192,each sector of 512 bytes) to the loop0 device, and the second section (starts from sector 532480,each sector of 512 bytes) to loop1.

sudo losetup -f --show -o $((8192*512)) 2023-02-21-raspios-bullseye-armhf-lite.img
sudo losetup -f --show -o $((532480*512)) 2023-02-21-raspios-bullseye-armhf-lite.img
sudo mkdir /mnt/rpi_boot
sudo mkdir /mnt/rpi_system
sudo mount /dev/loop0 /mnt/rpi_boot
sudo mount /dev/loop1 /mnt/rpi_system

Check the content of mounted directories.

ls  /mnt/rpi_boot
bcm2708-rpi-b.dtb       bcm2710-rpi-3-b.dtb       bootcode.bin   fixup.dat         overlays
bcm2708-rpi-b-plus.dtb  bcm2710-rpi-3-b-plus.dtb  cmdline.txt    fixup_db.dat      start4cd.elf
bcm2708-rpi-b-rev1.dtb  bcm2710-rpi-cm3.dtb       config.txt     fixup_x.dat       start4db.elf
bcm2708-rpi-cm.dtb      bcm2710-rpi-zero-2.dtb    COPYING.linux  issue.txt         start4.elf
bcm2708-rpi-zero.dtb    bcm2710-rpi-zero-2-w.dtb  fixup4cd.dat   kernel7.img       start4x.elf
bcm2708-rpi-zero-w.dtb  bcm2711-rpi-400.dtb       fixup4.dat     kernel7l.img      start_cd.elf
bcm2709-rpi-2-b.dtb     bcm2711-rpi-4-b.dtb       fixup4db.dat   kernel8.img       start_db.elf
bcm2709-rpi-cm2.dtb     bcm2711-rpi-cm4.dtb       fixup4x.dat    kernel.img        start.elf
bcm2710-rpi-2-b.dtb     bcm2711-rpi-cm4s.dtb      fixup_cd.dat   LICENCE.broadcom  start_x.el

ls  /mnt/rpi_system
bin  boot  dev  etc  home  lib  lost+found  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

Copy the contents of the mounted directories to our guest directory.

cp -ax /mnt/rpi_system/* /home/qemu/qemu-user/raspios-armhf-lite
cp -ax /mnt/rpi_boot/* /home/qemu/qemu-user/raspios-armhf-lite/boot

Unmount the directories and free the loop devices. Let's do the same operation with the second image of the system.

sudo umount /mnt/rpi_boot
sudo umount /mnt/rpi_system
sudo losetup -d /dev/loop0 /dev/loop1
sudo losetup -f --show -o $((8192*512)) 2023-02-21-raspios-bullseye-arm64-lite.img
sudo losetup -f --show -o $((532480*512)) 2023-02-21-raspios-bullseye-arm64-lite.img
sudo mount /dev/loop0 /mnt/rpi_boot
sudo mount /dev/loop1 /mnt/rpi_system
cp -ax /mnt/rpi_system/* /home/qemu/qemu-user/raspios-armhf-lite
cp -ax /mnt/rpi_boot/* /home/qemu/qemu-user/raspios-armhf-lite/boot
sudo umount /mnt/rpi_boot
sudo umount /mnt/rpi_system
sudo losetup -d /dev/loop0 /dev/loop1

Let's copy the qemu to the guest systems.

cp -vfr /usr/bin/qemu* /home/qemu/qemu-user/raspios-armhf-lite/usr/bin
cp -vfr /usr/bin/qemu* /home/qemu/qemu-user/raspios-arm64-lite/usr/bin

And finally, we will create scripts in the guest systems that should be run after the chroot.

sudo nano -w /home/qemu/qemu-user/raspios-armhf-lite/rpi3.sh
#!/bin/bash

cd /usr/bin
rm qemu-arm-rpi
rm qemu-aarch64-rpi
ln -sf qemu-arm-rpi3 qemu-arm-rpi
ln -sf qemu-aarch64-rpi3 qemu-aarch64-rpi
cd /
sudo nano -w /home/qemu/qemu-user/raspios-armhf-lite/rpi4.sh
#!/bin/bash

cd /usr/bin
rm qemu-arm-rpi
rm qemu-aarch64-rpi
ln -sf qemu-arm-rpi4 qemu-arm-rpi
ln -sf qemu-aarch64-rpi4 qemu-aarch64-rpi
cd /
sudo nano -w /home/qemu/qemu-user/raspios-arm64-lite/rpi3.sh
#!/bin/bash

cd /usr/bin
rm qemu-arm-rpi
rm qemu-aarch64-rpi
ln -sf qemu-arm-rpi3 qemu-arm-rpi
ln -sf qemu-aarch64-rpi3 qemu-aarch64-rpi
cd /
sudo nano -w /home/qemu/qemu-user/raspios-arm64-lite/rpi4.sh
#!/bin/bash

cd /usr/bin
rm qemu-arm-rpi
rm qemu-aarch64-rpi
ln -sf qemu-arm-rpi4 qemu-arm-rpi
ln -sf qemu-aarch64-rpi4 qemu-aarch64-rpi
cd /

Now our guest systems are fully prepared for chroot. Now it's time to check what we've got.

cd /home/qemu/qemu-user/
lscpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         48 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  12
  On-line CPU(s) list:   0-11
Vendor ID:               AuthenticAMD
  BIOS Vendor ID:        Advanced Micro Devices, Inc.
  Model name:            AMD Ryzen 5 5600G with Radeon Graphics
    BIOS Model name:     AMD Ryzen 5 5600G with Radeon Graphics         
    CPU family:          25
    Model:               80
    Thread(s) per core:  2
    Core(s) per socket:  6
    Socket(s):           1
    Stepping:            0
    Frequency boost:     enabled
    CPU max MHz:         4464.0000
    CPU min MHz:         400.0000
    BogoMIPS:            7803.38
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf
                          rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr
                         _core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xge
                         tbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold av
                         ic v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca fsrm
Virtualization features: 
  Virtualization:        AMD-V
Caches (sum of all):     
  L1d:                   192 KiB (6 instances)
  L1i:                   192 KiB (6 instances)
  L2:                    3 MiB (6 instances)
  L3:                    16 MiB (1 instance)
NUMA:                    
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-11
Vulnerabilities:         
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Not affected
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
  Srbds:                 Not affected
  Tsx async abort:       Not affected

gcc -v -E -x c /dev/null -o /dev/null -march=native 2>&1 | grep /cc1 
 /usr/libexec/gcc/x86_64-pc-linux-gnu/11.3.0/cc1 -E -quiet -v /dev/null -o /dev/null -march=znver3 -mmmx -mpopcnt -msse -msse2 -msse3 -mssse3 -msse4.1 -msse4.2 -mavx -mavx2 -msse4a -mno-fma4 -mno-xop -mfma -mno-avx512f -mbmi -mbmi2 -maes -mpclmul -mno-avx512vl -mno-avx512bw -mno-avx512dq -mno-avx512cd -mno-avx512er -mno-avx512pf -mno-avx512vbmi -mno-avx512ifma -mno-avx5124vnniw -mno-avx5124fmaps -mno-avx512vpopcntdq -mno-avx512vbmi2 -mno-gfni -mvpclmulqdq -mno-avx512vnni -mno-avx512bitalg -mno-avx512bf16 -mno-avx512vp2intersect -mno-3dnow -madx -mabm -mno-cldemote -mclflushopt -mclwb -mclzero -mcx16 -mno-enqcmd -mf16c -mfsgsbase -mfxsr -mno-hle -msahf -mno-lwp -mlzcnt -mmovbe -mno-movdir64b -mno-movdiri -mmwaitx -mno-pconfig -mpku -mno-prefetchwt1 -mprfchw -mno-ptwrite -mrdpid -mrdrnd -mrdseed -mno-rtm -mno-serialize -mno-sgx -msha -mshstk -mno-tbm -mno-tsxldtrk -mvaes -mno-waitpkg -mwbnoinvd -mxsave -mxsavec -mxsaveopt -mxsaves -mno-amx-tile -mno-amx-int8 -mno-amx-bf16 -mno-uintr -mno-hreset -mno-kl -mno-widekl -mno-avxvnni --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=512 -mtune=znver3 -dumpbase null

Run script "run_arm64_rpi4.sh" for example

sh run_arm64_rpi4.sh

In the guest system, run the script "rpi4"

sh rpi4.sh

Now, being in the guest system, we give the same commands.

Belka /home/qemu/qemu-user # ls
arm-runner-action  comands  download_image.sh  qemu-wrapper.c  raspios-arm64-lite  raspios-armhf-lite  run_arm64_rpi3.sh  run_arm64_rpi4.sh  run_armhf_rpi3.sh  run_armhf_rpi4.sh  umount
Belka /home/qemu/qemu-user # sh run_arm64_rpi4.sh
root@Belka:/# ls
bin  boot  dev	etc  home  lib	lost+found  media  mnt	opt  proc  root  rpi3.sh  rpi4.sh  run	sbin  srv  sys	tmp  usr  var
root@Belka:/# sh rpi4.sh
root@Belka:/# lscpu
Architecture:                    aarch64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
CPU(s):                          12
On-line CPU(s) list:             0-11
Thread(s) per core:              2
Core(s) per socket:              6
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       ARM
Model:                           3
Model name:                      Cortex-A72
Stepping:                        r0p3
Frequency boost:                 enabled
CPU max MHz:                     4464.0000
CPU min MHz:                     400.0000
BogoMIPS:                        108.00
L1d cache:                       192 KiB
L1i cache:                       192 KiB
L2 cache:                        3 MiB
L3 cache:                        16 MiB
NUMA node0 CPU(s):               0-11
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fp asimd evtstrm crc32 cpuid
root@Belka:/# gcc -v -E -x c /dev/null -o /dev/null -march=native 2>&1 | grep /cc1
 /usr/lib/gcc/aarch64-linux-gnu/10/cc1 -E -quiet -v -imultiarch aarch64-linux-gnu /dev/null -o /dev/null -mlittle-endian -mabi=lp64 -march=armv8-a+crc -fasynchronous-unwind-tables
root@Belka:/# su - pi
pi@Belka:~ $

Further steps you can see in the building and installation instructions for Raspberry PI OS.
How to build and setup on Raspberry PI
How to build and setup for Raspberry PI QEMU native

Unmount the various file systems when they are no longer in use:

root #umount  sys proc proc/cpuinfo dev/pts dev
Clone this wiki locally