Couple days ago I reinstalled my laptop, moved my backed up data back into their rightful place, updated the system, shut it down and next morning came to a system that refused to boot. sigh

The reason seemed weird. The systemd-rfkill service and many others were not starting. And the root account is locked, so I can drop completely simply into rescue root shell. What to do next?

Systemd to the rescue

A golden trove has proven to be the documentation of systemd, the default init system of Fedora Linux. It talks about the different ways one can debug boot issues depending on whether you have access to a shell or not.

My case was that I did not have access to shell and thus I had to rely on adding kernel options in GRUB. My first choice was to boot into rescue mode or emergency mode by adding either systemd.unit=rescue.target or systemd.unit=emergency.target to the kernel command line but that did not work because my root account was locked after a fresh install where I did not set a password for it.

My next option was to enable debug shell through systemd. I added systemd.debug_shell=1 to the kernel command line and that did the trick. The systems launched a root shell on TTY9 with the filesystem mounted.

Diagnosis

With a root shell access and the filesystem mounted I could continue to diagnose the problem. I looked into the journal and saw that systemd-rfkill (and many other services) is complaining about permission issues when accessing directories under /var. And truly, the permissions were incorrectly set to my user instead of the root account. Why? I forgot to fix the ownership after restoring the state of /var from a backup. sigh

One chown later… Actually, two because I rewrote the ownership of my actual home directory 🙃… And the computer boots successfully again!