The server doesn’t come back after a reboot. SSH times out and the website is down. Before reinstalling, open the console and look. Most boot failures come from a handful of causes that you can fix in minutes from emergency or rescue mode.

Step 1: open the console

Use your provider’s VNC console in the control panel. VPS.DO includes one for every VPS. It shows the server’s screen as if a monitor were attached, so you see boot messages and can type even when networking or SSH is broken.

Look at what’s on screen:

You see Likely cause Go to
“You are in emergency mode…” Failed mount (fstab), filesystem errors Fix fstab and filesystems
Login prompt, but no SSH Network config, firewall, sshd not starting Network and SSH
grub> or grub rescue> prompt Bootloader or /boot problem GRUB
Kernel panic Bad kernel, initramfs or root device Boot an older kernel
Stuck at “A start job is running…” Waiting for a missing disk or network mount Fix fstab and filesystems

Emergency and rescue targets

systemd has two minimal modes:

  • rescue.target — single-user mode. Local filesystems are mounted, there’s no network and only basic services run.
  • emergency.target — even less. Root is mounted read-only, nothing else starts.

Boot into one from GRUB. Press e on the entry, add this to the end of the linux line, then press Ctrl+X:

systemd.unit=rescue.target

Both modes ask for the root password. Ubuntu cloud images often have no root password set. Use the init=/bin/bash method below in that case.

Reset a lost root or user password

At the GRUB menu, press e and append to the linux line:

init=/bin/bash

Boot with Ctrl+X. You get a root shell without a password:

mount -o remount,rw /
passwd root            # or: passwd youruser
touch /.autorelabel    # RHEL family with SELinux only
sync
exec /sbin/init        # continue booting (or: reboot -f)

On RHEL-family systems, rd.break instead of init=/bin/bash stops in the initramfs. Then run mount -o remount,rw /sysroot; chroot /sysroot; passwd root; touch /.autorelabel; exit; exit.

Fix fstab and filesystems

The most common boot failure is an /etc/fstab entry for a disk that’s missing or misspelled.

mount -o remount,rw /
nano /etc/fstab               # comment out or fix the bad line; add nofail to data disks
systemctl daemon-reload
mount -a                      # must succeed without errors
reboot

See fstab.

Filesystem errors (“UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY”):

fsck -y /dev/vda1             # the filesystem must not be mounted read-write

For the root filesystem, run fsck from emergency mode (root mounted read-only) or with fsck.mode=force added to the kernel line.

Network and SSH

If you can log in at the console but not over SSH:

ip a                           # does the interface have its IP?
ip route                       # is there a default route?
ping -c2 1.1.1.1
systemctl status ssh           # 'sshd' on RHEL
sshd -t                        # config syntax check
ufw status / nft list ruleset  # did a firewall rule lock you out?

Typical culprits:

  • A netplan or ifupdown typo. Fix it, then run netplan apply. See netplan.
  • UFW enabled without allowing SSH, or the SSH port changed without opening it. ufw allow 22/tcp (or your port). See change SSH port.
  • An sshd_config error. sshd -t shows the line.

Boot an older kernel

If a kernel update caused the problem, pick the previous kernel under Advanced options in the GRUB menu, then fix or remove the bad kernel. See upgrade the kernel.

Disk full

A completely full root filesystem can stop services, and even logins, from working. From the console:

df -h /
journalctl --vacuum-size=200M
apt clean
du -xh / --max-depth=2 | sort -h | tail

See disk full troubleshooting.

Read the logs from the failed boot

Once the system is up, find out what went wrong:

journalctl -b -1 -p err        # errors from the previous boot (needs a persistent journal)
systemctl --failed
journalctl -xe

When to reinstall instead

If the system is badly damaged (core packages removed, a compromised server, unknown changes), restoring from backups onto a fresh OS is faster and safer. VPS.DO’s control panel offers one-click OS reinstall. It wipes the disk, so copy any data you can reach first (from the console, rsync it to another server once networking works). For hacked servers, always reinstall rather than repair.

Prevention

  • Test fstab changes with findmnt --verify and mount -a before rebooting.
  • Use nofail on non-root mounts.
  • Keep a root password set, or know the init=/bin/bash method.
  • Before risky firewall or SSH changes, keep a second session open, or schedule an automatic rollback (sleep 300 && ufw disable &).
  • Have off-server backups. See backup strategy.

Related: GRUB bootloader and SSH connection refused.

  • Linux
  • Boot
  • Troubleshooting