Knowing what happens between “power on” and the login prompt makes boot problems much less mysterious. When a server hangs at boot, the stage where it stops tells you what’s broken. Here’s the sequence on a modern Linux server, VPS or dedicated, and the tools to inspect each step.

The stages at a glance

Firmware (BIOS/UEFI)  →  Bootloader (GRUB)  →  Kernel  →  initramfs  →  systemd (PID 1)  →  targets & services  →  login

1. Firmware: BIOS or UEFI

The firmware initialises the hardware (on a VPS, the virtual hardware provided by the hypervisor) and finds something to boot:

  • BIOS (legacy): reads the first sector (MBR) of the boot disk and runs the bootloader code there.
  • UEFI: reads the EFI System Partition (a small FAT partition, usually mounted at /boot/efi) and runs a .efi bootloader file, often shimx64.efi then grubx64.efi with Secure Boot.

Check which one your system uses:

[ -d /sys/firmware/efi ] && echo UEFI || echo BIOS

2. Bootloader: GRUB

GRUB shows the menu (often hidden on cloud images), loads the selected kernel and initramfs from /boot into memory, and passes the kernel its command line (root device, options).

Problems here look like a grub> or grub rescue> prompt, or “no such device”. See GRUB bootloader.

ls /boot                 # vmlinuz-* (kernels), initrd.img-* / initramfs-* (initramfs)
cat /proc/cmdline        # the parameters GRUB passed to the running kernel

3. Kernel

The kernel decompresses itself, initialises CPUs, memory and core drivers, and mounts the initramfs as a temporary root filesystem. Its messages are what you see scrolling past at boot. Afterwards, read them with:

sudo dmesg -T | less
journalctl -k -b

A kernel panic at this stage (“VFS: Unable to mount root fs”, “not syncing”) usually means a missing driver in the initramfs, a wrong root= parameter, or a corrupted kernel or initramfs. Boot an older kernel from GRUB. See dmesg.

4. initramfs

The initramfs is a small filesystem image with the tools and drivers needed to find and mount the real root filesystem: storage drivers, LVM, RAID (mdadm), encryption (LUKS) and filesystem checks. It then switches to the real root and starts its init.

If it can’t find the root, you land in an initramfs / dracut emergency shell ((initramfs) prompt). Typical causes are a changed disk UUID, broken LVM or RAID, or a missing module. Regenerate after fixing:

sudo update-initramfs -u -k all     # Debian/Ubuntu
sudo dracut -f --regenerate-all     # RHEL family

5. systemd (PID 1)

Once the real root is mounted, the kernel starts /sbin/init, which is systemd on all mainstream distributions. systemd:

  1. reads unit files (services, mounts, sockets, timers);
  2. mounts filesystems from /etc/fstab (generated as .mount units);
  3. starts services in parallel, respecting dependencies;
  4. works towards the default target, multi-user.target on servers and graphical.target on desktops.

Failures here: “You are in emergency mode” (often fstab), or a hang at “A start job is running for…” (waiting for a device or network). See Linux rescue mode and systemd targets.

6. Login

getty provides the console login (what you see in the VNC console). sshd accepts network logins once networking is up.

Analyse boot time

systemd-analyze                     # firmware + loader + kernel + userspace times
systemd-analyze blame | head -15    # slowest units
systemd-analyze critical-chain      # the chain of units that determined the total time

Common slow items on servers:

  • systemd-networkd-wait-online or NetworkManager-wait-online, waiting for every interface. Configure only the needed interfaces, or set a timeout.
  • cloud-init stages on cloud images (usually a few seconds).
  • apt-daily and snapd seeding on first boot.
  • Mounts for missing devices (fstab without nofail), which wait 90 seconds.

Logs from a previous boot

journalctl --list-boots
journalctl -b -1 -p err           # errors from the last boot
systemctl --failed                # failed units in this boot

The journal must be persistent for previous boots: sudo mkdir -p /var/log/journal, then sudo systemctl restart systemd-journald. See journalctl.

Where to intervene

Stage What you can change Tool
GRUB Kernel choice, kernel parameters /etc/default/grub, grubby
Kernel Modules, parameters /etc/modprobe.d/, /etc/sysctl.d/
initramfs Included drivers and hooks update-initramfs, dracut
systemd Services, targets, mounts systemctl, unit files, fstab

Quick reference

Task Command
BIOS or UEFI ls /sys/firmware/efi
Kernel parameters cat /proc/cmdline
Kernel messages journalctl -k -b
Boot time systemd-analyze
Slowest units systemd-analyze blame
Failed units systemctl --failed
Last boot’s errors journalctl -b -1 -p err

Related: systemd unit files and upgrade the kernel.

  • Linux
  • Boot
  • systemd