Have you ever set a GRUB boot password, unwittingly buggering it up with a typo, and then panicked when your rebooting host claimed you were giving it the wrong password so it wouldn’t boot? On an LVM disk system? I have. Just last week, in fact. Wanna know how I fixed it?
Hit the jump, then.
lolwut systemd
Our journey begins with the Center for Internet Security – or rather, my discovery of their lovely benchmarks for OS hardening. See, I wanted to deploy a fairly non-crucial VM so I figured I’d play with best-practice system hardening methods for, well, practice. So I threw some terms at a duck, and settled on a nice Server Fault question with which to guide that idea (Scott Pack’s lovely answer particular – gosh, I do like those Stack Exchange communities).
If you’re looking to spruce up the security of your Linux system(s), I heartily recommend you follow them links. For now, I’ll cut a very long story shorter (shock and horror!), and focus on the GRUB thing for this post.
Also, if you need to fix something LIKE NAOW, scroll all the way down for the dirty, unwaffled gotta-get-things-fixed-or-I-lose-my-job version.
So, the CIS Debian 8(.2) benchmark I was following had me set a boot loader password. First I generated an encrypted password using the grub-mkpasswd-pbkdf2 utility, and added it, along with the appropriate lines, to the /etc/grub.d/00_header file.
And that’s where it all went pear-shaped. See, I didn’t sanity check my changes to 00_header properly. I swopped a “d” with a “k,” copied my encrypted password in, exited edit mode and writ swiftly the :wq of my doom. Eager to test this new thing, I updated GRUB and rebooted straight-away – skipping the addition of the --unrestricted option that would’ve allowed GRUB to boot the system without a password (but with which I would not have known of my mistake until, probably, a long time to come twenty minutes later as I tested the next recommendations).
So how did I fix it, and manage to get this thing booting again? A live CD, of course! Normally, that would be the Ultimate Boot CD and its Parted Magic goodness, which has saved me from more jams than I have kept track of, but this time around I gave Knoppix a go. On booting the live CD/image, exchanging pleasant greetings with the Knoppix desktop and terminal, I set about mounting the volumes of my misbehaving system. Except, well, they’re LVM volumes and I hadn’t actually used LVM before, let alone with a live CD…
Knopp, knopp. Who’s there? GNU. … Ugh…
Thankfully, the helpful Knoppix wiki filtered to the top of my search for “how do I do the things,” and with their friendly guidance I found my way.
First I needed to scan for and activate my volume groups, by running # vgscan followed by # vgchange -a y. Then I wanted to refresh my memory of just what the hell I’d named my volumes, so I ran # lvs. Now it was time to mount said volumes. They’d need a mount point, so I threw out a # mkdir /mnt/tmp, followed by, initially anyway, the mounting of the root volume with # mount /dev/VG1/root /mnt/tmp (your volume and group names will probably vary).
With mounts out of the way (for now), I could scrutinise my 00_header file – after a little head-scratching before it occurred to me to look for it in my mount point, at /mnt/tmp/etc/grub.d/00_header – I spotted the devilish little typo and corrected it, and saved my correction.
(Skipping over the bit where I didn’t update GRUB with the new config and wondering why my system still wouldn’t boot…)
Next was to update GRUB with new config, which necessitated more mounting. Your mounting will vary, but because I’d seen fit to split my partitions (as per the hardening stuff), I had extra mounting to do. For this, another Stack Exchange question came to my rescue. Having already mounted my root partition, I now hammered out more mounts with # mount --bind /dev /mnt/tmp/dev, # mount --bind /sys /mnt/tmp/sys, and # mount --bind /proc /mnt/tmp/proc. Those are the three definitely-necessary ones, and for good measure I mounted the rest of my volumes as well (as you’ll see in the screenshots somewhere on this page).
With my mounts ready, I was all set to change root into the sick system (# chroot /mnt/tmp) and make the GRUB there aware that things had changed around here and it had best get with it (# update-grub).
All done, I exited from the changed root, shut the system down, unmounted the Knoppix image, and watched, expectantly, as I booted the server back up. Supplying the password, I went on watching, now triumphant, as first the familiar Linux boot messages appeared, followed systemd-fsck’s utterly hilarious time-travel complaints (no, really, I laughed. Out loud. People might have looked; I’m not sure.)
Then I did it all again to take screenshots and reaffirm all the processes for a blog post…
The TL;DR, no-waffling, less-explaining, bullet-point edition:
I’m assuming a /mnt/tmp as the mount destination within the live disc environment.
- Get a live CD/image, like Knoppix or Ultimate Boot CD.
- In the live disc environment, mount your host system’s root partition (and any other partitions that you’ve split off from root, like
/varand/tmp).- Use your preferred text editor to edit your GRUB config as needed – be it grub.cfg inside /boot or the config files within /etc/grub.d/
Remember to edit the GRUB files within of host system – editing the GRUB config of the live disc would be pointless.
- Run these necessary bind-mounts:
# mount --bind /dev /mnt/tmp/dev,# mount --bind /sys /mnt/tmp/sys, and# mount --bind /proc /mnt/tmp/proc. If/bootis a separate partition, mount that too.- Change root into your host system with
# chroot /mnt/tmp.- Make GRUB aware of the changes to your config files with
# update-grub.- gl;hf
