systemd-boot-252 loading initrd renders some systems unbootable with "Error preparing initrd: Bad Buffer Size" (suspected broken EFI implementation)

[ethorsoe]

Describe the bug

When selecting any boot entry that has separate initrd entry (not linux parameter initrd) boot fails with
Error preparing initrd: Bad Buffer Size

systemd/systemd#25911

Steps To Reproduce

Steps to reproduce the behavior:

1. upgrade to nixos-23.05 with boot.loader.systemd-boot.enable on a system with certain kind of EFI.
2. system is unbootable for all generations, for our case EFI edit command does not work either, maybe related

Expected behavior

System boots

Notify maintainers

@NixOS/systemd

Metadata

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.24, NixOS, 23.05 (Stoat), 23.05.20230419.645bc49`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.13.3`
 - nixpkgs: `/etc/tuxnix/channels/nixpkgs`

Workaround

boot.loader.systemd-boot.extraInstallCommands = ''
      extraInstallCommands = ''
        for i in /boot/loader/entries/nixos-generation-*.conf; do
          initrd="$(sed -n '/^initrd / {s/.* //;p}' "$i")"
          sed -i '/^initrd / d; /^options / s|$| initrd='"$initrd|" "$i"
        done
      '';

[ethorsoe]

Broken system details

[    0.000000] efi: EFI v2.80 by American Megatrends
[    0.000000] efi: ACPI=0xa8d99000 ACPI 2.0=0xa8d99014 SMBIOS=0xa9b23000 SMBIOS 3.0=0xa9b22000 MEMATTR=0x9ff25018 ESRT=0xa25d2998 RNG=0xa8913018 INITRD=0x9ff3ec18 
[    0.000000] SMBIOS 3.3.0 present.
[    0.000000] DMI: Supermicro Super Server/H12DSi-N6, BIOS 2.3 10/21/2021

[RaitoBezarius]

@ElvishJerricco Did we get somewhere regarding that?

[ElvishJerricco]

This is a bug in some firmwares. When an initrd is too big, the file reading APIs provided to the boot loader by the firmware fail in weird ways. The kernel knows to work around this when loading initrds passed on the cmdline by reading the initrd in chunks, but systemd-boot does not implement this workaround when loading large files currently. Until recently, systemd-boot has always passed the initrd to the kernel via the cmdline, but in systemd/systemd#22550 (first released in v253 I believe), they changed it to use LINUX_INITRD_MEDIA, which requires systemd-boot to load the file itself.

The fix is systemd/systemd#25948, which has not been merged upstream, although it appears to be approved. If we want to fix this in NixOS, we can include that patch, but again it's not even merged upstream yet, so I don't know if that's considered acceptable.

Another workaround is to revert the behavior indirectly by putting initrd in the options line of the entry file instead of in an initrd line.

[dawidd6]

The upstream PR is now merged!

[RaitoBezarius]

If we can cleanup things before branch-off regarding this, that'd be awesome.

[ElvishJerricco]

Should we wait for a stable backport from upstream, or include the patch in nixpkgs for the time being?

[RaitoBezarius]

Should we wait for a stable backport from upstream, or include the patch in nixpkgs for the time being?

In all the cases, a systemd rebuild will have to go through staging, so I'd say: we can afford one or two days and do the patch thing for the next staging-23.05.

[arianvp]

This fix has landed in staging-23.05. Should be available after the staging cycle

[arianvp]

https://nixpk.gs/pr-tracker.html?pr=235770 tracks when it lands. If you still have issues please let me know!