min
/
lanzaboote
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
lanzaboote/docs/TROUBLESHOOTING.md

69 lines
4.8 KiB
Markdown
Raw Normal View History

docs: Add troubleshooting documentation Running out of space on the ESP is an error that commonly happens. Document how to proceed in this case. Furthermore, in case the ESP gets corrupted, the system needs to be recovered. Document how to do this.
2023-09-24 06:30:36 -05:00
# Troubleshooting
## Bootloader installation fails with "Failed to install files. … No space left on device (os error 28)"
During the bootloader installation process, Lanzaboote must copy the kernel and initrd to the EFI system partition (ESP).
It is quite possible that the ESP is not large enough to hold these files for all installed generations, in which case this error occurs.
In this case, you must first delete some generations (e.g. run `nixos-collect-garbage --delete-older-than=7d` to delete all generations more than one week old).
After that, some space on the ESP must be freed manually.
To achieve this, delete some kernels and initrds in `/boot/EFI/nixos` (they will be recreated in the next step if they are in fact still required).
Finally, run `nixos-rebuild boot` again to finish the installation process that was interrupted by the error.
It is recommended run a garbage collection regularly, and monitor the ESP usage (particularly if it is quite small), to prevent this issue from happening again in the future.
**Warning:** It is recommended to not delete the currently booted kernel and initrd, and to not reboot the system before running `nixos-rebuild boot` again, to minimize the risk of accidentally rendering the system unbootable.
docs: Improve troubleshooting documentation Due to the temporarily doubled ESP space usage, it is now easier to run into the out of space issue (once). Document how to proceed in this case without having to delete any generations. Furthermore, recovery in case of ESP corruption is now slightly more involved, because not all files are rewritten all the time. Adjust the documentation accordingly.
2023-09-24 06:30:36 -05:00
**Note:** When upgrading Lanzaboote from version 0.3.0, or from git master prior to the merge of PR #204, ESP space usage is temporarily doubled.
Hence it is possible for this error to occur even if there was plenty (but less than half) free space available prior to the installation.
In this case, it is not necessary to delete any generations, and you can proceed directly to deleting some kernels and initrds before running the installation again.
docs: Add troubleshooting documentation Running out of space on the ESP is an error that commonly happens. Document how to proceed in this case. Furthermore, in case the ESP gets corrupted, the system needs to be recovered. Document how to do this.
2023-09-24 06:30:36 -05:00
## Power failed during bootloader installation, and now the system does not boot any more
Due to the shortcomings of the FAT32 filesystem, in rare cases, it is possible for the ESP to become corrupted after power loss.
With Lanzaboote enabled, this will lead to "secure boot errors" or "hash verification failures" (the exact wording depends on the firmware).
In these cases, recovery is usually still possible with the steps below.
**Note:** If the system fails to boot after the Linux kernel has already been started, then the problem is not caused by a corrupted ESP.
In this case, the steps below will not help, and standard rollback procedures should be followed instead.
### The system can still boot an older generation
In case an older generation still works, the recovery can be carried out from within the booted system.
docs: Improve troubleshooting documentation Due to the temporarily doubled ESP space usage, it is now easier to run into the out of space issue (once). Document how to proceed in this case without having to delete any generations. Furthermore, recovery in case of ESP corruption is now slightly more involved, because not all files are rewritten all the time. Adjust the documentation accordingly.
2023-09-24 06:30:36 -05:00
Run `nix-shell -p openssl sbctl` to ensure the tools required for recovery are available.
docs: Add troubleshooting documentation Running out of space on the ESP is an error that commonly happens. Document how to proceed in this case. Furthermore, in case the ESP gets corrupted, the system needs to be recovered. Document how to do this.
2023-09-24 06:30:36 -05:00
docs: Improve troubleshooting documentation Due to the temporarily doubled ESP space usage, it is now easier to run into the out of space issue (once). Document how to proceed in this case without having to delete any generations. Furthermore, recovery in case of ESP corruption is now slightly more involved, because not all files are rewritten all the time. Adjust the documentation accordingly.
2023-09-24 06:30:36 -05:00
1. Run `sudo sbctl verify /boot/EFI/Linux/nixos-generation-*.efi` to check the Lanzaboote stubs.
Files that have a crossmark on their left are corrupted and must be deleted.
2. Run `for file in /boot/EFI/nixos/*.efi; do hash=$(openssl dgst -sha256 -binary "$file" | base32 | tr -d = | LC_ALL=C tr [:upper:] [:lower:]); if [[ $file != *$hash.efi ]]; then echo $file; fi; done` to check the kernels and initrds.
Any files that are printed are corrupted and must be deleted.
3. Run `nixos-rebuild boot`.
This should reinstate all files that are required for the newer generations to boot.
4. Reboot the system, it should now work again.
docs: Add troubleshooting documentation Running out of space on the ESP is an error that commonly happens. Document how to proceed in this case. Furthermore, in case the ESP gets corrupted, the system needs to be recovered. Document how to do this.
2023-09-24 06:30:36 -05:00
### The system cannot boot any generation anymore
If no available generation can boot any more, the system must be recovered from a rescue system.
First make sure that you have a recent NixOS install medium available.
**Note:** Nix versions from before August 2023 contain a bug that can prevent `nixos-enter` from working.
A more recent medium must be used for the recovery procedure to work reliably.
1. Disable Secure Boot in the firmware settings.
The NixOS install medium is not signed and thus cannot be booted when Secure Boot is active.
2. Boot the NixOS install medium.
3. Mount all partitions belonging to the system to be recovered under `/mnt`, just like you would for installation.
1. In case the ESP does not mount, or only mounts in read-only mode, due to corruption, try `fsck.fat` first.
If that fails as well or the ESP still does not mount, it needs to be reformatted using `mkfs.fat`.
docs: Improve troubleshooting documentation Due to the temporarily doubled ESP space usage, it is now easier to run into the out of space issue (once). Document how to proceed in this case without having to delete any generations. Furthermore, recovery in case of ESP corruption is now slightly more involved, because not all files are rewritten all the time. Adjust the documentation accordingly.
2023-09-24 06:30:36 -05:00
4. Delete the corrupted files on the ESP, using `rm -fr /mnt/boot/EFI/Linux/nixos-generation-*.efi /mnt/boot/efi/nixos`.
5. Enter the recovery shell by running `nixos-enter`.
docs: Add troubleshooting documentation Running out of space on the ESP is an error that commonly happens. Document how to proceed in this case. Furthermore, in case the ESP gets corrupted, the system needs to be recovered. Document how to do this.
2023-09-24 06:30:36 -05:00
Then, run `nixos-rebuild boot` to install the bootloader again.
docs: Improve troubleshooting documentation Due to the temporarily doubled ESP space usage, it is now easier to run into the out of space issue (once). Document how to proceed in this case without having to delete any generations. Furthermore, recovery in case of ESP corruption is now slightly more involved, because not all files are rewritten all the time. Adjust the documentation accordingly.
2023-09-24 06:30:36 -05:00
6. Exit the recovery shell and unmount all filesystems.
7. Reboot the system to verify that everything works again.
8. Enable Secure Boot again in the firmware settings.
docs: Add troubleshooting documentation Running out of space on the ESP is an error that commonly happens. Document how to proceed in this case. Furthermore, in case the ESP gets corrupted, the system needs to be recovered. Document how to do this.
2023-09-24 06:30:36 -05:00
## The system doesn't boot with Secure Boot enabled
It is the most likely issue that Lanzaboote could not verify a cryptographic hash.
To recover from this, disable Secure Boot in your firmware settings.
Please file a bug, if you hit this issue.