Emulating aarch64 on x86_64 with qemu

This is what I had to do to get a aarch64 emulation on my x86 laptop.

The simplest way if you're running NixOS is setting

binfmt.emulatedSystems = ["aarch64-linux"];

pkgs = import nixpkgs { system = "aarch64-linux"; }

## Spawning a container with systemd-nspawn

### Getting debian linux distro

  $ qemu-img create aarch64-debian.img 128G
  $ mkfs.ext4 aarch64-debian.img
  $ mkdir mount-point.dir
  $ sudo mount -o loop aarch64-debian.img mount-point.dir
  $ sudo debootstrap --include=systemd-container --arch arm64 bookworm mount-point.dir

          

### Spawning the container

On host, set binfmt entry for aarch64-linux. On NixOS, the interpreter path is always in /run/binfmt/{aarch64-linux}. Download qemu-aarch64-static from here. And copy it to the path that is reported on the host in the container.

  $ cat /proc/sys/fs/binfmt_misc/aarch64-linux
  enabled
  interpreter /run/binfmt/aarch64-linux
  flags: P
  offset 0
  magic 7f454c460201010000000000000000000200b700
  mask ffffffffffffff00ffffffffffff00fffeffffff

          

systemd-nspawn mounts filesystems /dev/, /run/ and similar when container starts and are destroyed when container exits. So I can't just copy the downloaded qemu interpreter to /run. Instead I copy it to /bin in the distro and copy it /run when container is started.

  $ cp ~/Downloads/qemu-aarch64-static ~/mount-point.dir/bin/
  $ chmod 755 ~/mount-point.dir/bin/qemu-aarch64-static # can't login as any other user in the container without this

          

Update: You can instead create a symlink in a directory on host and mount it on to the container with --bind=binfmt:/run/binfmt/

  $ mkdir binfmt; cd binfmt
  $ ln -s /bin/qemu-aarch64-static aarch64-linux

          

Here, I am not sure how binfmt registries work across host and container, I don't see a binfmt entry in container's /proc/sys/fs/binfmt_misc though the documentation for nspawn says it limits access to /proc/sys as read only. However only when the qemu interpreter is present at the path set in the binfmt entry inside the container does it work. So this is what I am doing right now.

  $ sudo systemd-nspawn -D mount-point.dir -U --bind=binfmt:/run/binfmt/ --machine debaarch64 /bin/qemu-aarch64-static \
      -E QEMU_LD_PREFIX=/lib/ld-linux-aarch64.so.1 -E PATH=:/bin:/usr/bin:/usr/sbin:/usr/local/bin \
      -E TERM=xterm -E NIX_INSTALLER_EXTRA_CONF='filter-syscalls = false'
      /bin/bash -i

    Spawning container debaarch64 on /home/pranaysashank/mount-point.dir.
    Press ^] three times within 1s to kill container.
    Selected user namespace base 178716672 and range 65536.
    Failed to correct timezone of container, ignoring: Value too large for defined data type

  bash-5.1#

          

In the container I now copy qemu interpreter to the correct path. Update: This is now obselete with bind mount of binmount mentioned above

  bash-5.1# ls
  bash: /bin/ls: No such file or directory
  bash-5.1# qemu-aarch64-static bin/bash setup.sh
  bash-5.1# ls
  bin boot debootstrap dev  etc home lib lost+found proc root run sbin setup.sh sys tmp usr var
  bash-5.1# cat setup.sh
  qemu-aarch64-static /bin/mkdir /run/binfmt
  qemu-aarch64-static /bin/ln -s /bin/qemu-aarch64-static /run/binfmt/aarch64-linux

          

The debian distro downloaded might be extremely limited and you may need to have other packages like base-passwd and base-files. I couldn't get apt to work as it had some certificate verification errors. Following this work around fixed it. If you get an error about a missing _apt user, run

  adduser --force-badname --system --no-create-home _apt

          

If passwd, adduser, and similar commands are failing with a pam_entry not found or similar errors, or if you're unable to resolve localhost or container hostname check your /etc/hosts file and that /etc/nsswitch.conf exists and is properly configured. Update: if you install systemd-sysv you can pass /sbin/init and it starts systemd inside the container.

If you want to use sudo command inside the container, you need to set the C flag in binfmt entry file.

On Kernel version 5.15+ and systemd 249+ you can use idmapped bind mounts by passing the option

--bind=host_path:container_path:idmap

### Installing nix in the container

• You need to set filter-syscalls = false in /etc/nix/nix.conf before installing `nix` as the required seccomp filter is not available in the container