How to use kexec for fast reboot/crashdump.

Hongyi Lu (jwnhy) Aug 21, 2023

What is `kexec`?

kexec is a system call that allows user to load a kernel into the memory, and boot directly from it, without time-consuming bootloading.

This is useful for kernel developers or other people who need to reboot very quickly without waiting for the whole BIOS boot process to finish. Moreover, it is also used to boot up an “emergency kernel” upon crash (e.g., panic()). This material explains this.

Install and Config `kexec`

Kernel Configuration

To use kexec, the following kernel config must be enabled.

Processor type and features --->
    [*] kexec system call
    [*] kexec file based system call

If you need crashdump, then the following config is needed as well.

Processor type and features --->
    [*] kexec system call
    [*] kernel crash dumps
    [*] Build a relocatable kernel
Kernel hacking  --->
    [*] Kernel debugging
    Compile-time checks and compiler options --->
        [*] Compile the kernel with debug info
File systems  --->
    Pseudo filesystems  --->
        -*- /proc file system support
        [*]   /proc/kcore support
        [*]   /proc/vmcore support

GRUB Configuration

One need to prepare sufficient space for kexec to put the new kernel. Therefore, we need to add/modify the following line in /etc/default/grub.

GRUB_CMDLINE_LINUX="crashkernel=1024M,high nokaslr"

The crashkernel might be a lower value if you have less memory available. My PC has 32 GiB of memory, just for reference.

The nokaslr disables address space randomization so that, GDB can correctly recognize debug symbols from /proc/vmcore.

Installation

Use the following to install userland utilities (on Gentoo).

sudo emerge --ask --verbose sys-apps/kexec-tools

Usage I: Fast Reboot

For fast reboot, the command is simple. Note that the --initrd is not mandatory if you don’t have one. Moreover, --append might be replaced with --reuse-cmdline if you want to use the same set of cmdline arguments.

kexec -l path_to_kernel_image --initrd=path_to_initrd_image \
--append=command-line-options

A concrete example would the following.

sudo kexec -l /boot/vmlinuz-6.1.38 --reuse-cmdline

With new kernel loaded, you can use the following command to fast reboot if you’re using systemd.

sudo systemctl kexec

Usage II: Crash Dump

For crashdump, things are a far more complicated. Here is a list of points to pay attention.

Use -p instead of -l to load crashdump kernel.
Remove unnecessary modules via modprobe.blacklist=<comma-separated-list>.
Limit the number of CPUs by maxcpus=1.
Use irqpoll for stable interrupt handling.

With all that, we can come up with the following command.

sudo kexec -p /boot/vmlinuz-6.1.31-gentoo --reset-vga --console-vga \ 
--command-line="root=/dev/nvme0n1p5 maxcpus=1 irqpoll quite splash loglevel=3 \
systemd.show_status=false modprobe.blacklist=iptable_nat,nvidia_drm \
,nvidia_modeset,nvidia,iwlmvm,kvm_intel,iwlwifi,fuse,efivarfs \
systemd.journald.forward_to_console=no"

If you’re using a X-based GUI environment, I would suggest that you use VTx via Ctrl+Alt+F{1-6} before you do something to crash your kernel otherwise kernel will just be stuck.

Sidenote: If you are using a display manager (e.g., lxdm) repeatedly respawn itself on failure. This repeated behavior should be disabled, since it will force the screen to constantly switch back to VT1. For systemd, just comment out Restart=Always to prevent this.