• PxeBooting

How the bits fit together for PXE boot to work

A PXE boot sequence looks something like this:

1. The box powers on and the BIOS or UEFI tells it to PXE boot
2. The NIC is initialised and starts its PXE mode, this is a tiny program stored an OPROM (Option ROM)
3. It does DHCP to get an IP address and whatever DHCP options the server sends to it, which generally consists of:

   • a path to an executable, like grub/grubx64.efi

   • a server IP address, like 192.168.1.12

   • a server name? like illustrious.thighhighs.top

4. The PXE client then downloads the specified executable from the specified server, via TFTP

5. Having been downloaded, GRUB now executes and attempts to download its config, which is normally a menu served up as grub/grub.cfg

6. GRUB is smart and tries to download a customised config made just for this machine though. It does this by requesting grub/grub.cfg-01-02-99-88-77-66-55

7. Once GRUB has found its config, it runs it. This usually means loading a couple of modules, then presenting a bunch of menu options. For our purposes, there's a single menu item with a timeout of 20sec. Without intervention, the timeout will expire then load that default menu item.

8. The menu item tells it to download a kernel and initrd, and supplies a bunch of kernel cmdline options. Here's an example entry that starts a kickstart install

   menuentry 'Install AlmaLinux 9.2' --class fedora --class gnu-linux --class gnu --class os {
       linuxefi /images/Alma-9.2/vmlinuz ip=dhcp inst.repo=https://repo.almalinux.org/almalinux/9.2/BaseOS/x86_64/os/ inst.ks=http://vector.thighhighs.top/ks/kalina.ks.cfg inst.nompath
       initrdefi /images/Alma-9.2/initrd.img
   }

9. GRUB downloads the kernel and initrd from the same server (ie. by TFTP), stages them in memory, then passes execution to the kernel

10. With the initrd unpacked, the kernel goes ahead and does its thing, starting up and probing all the hardware etc, then executing what it finds in the initrd filesystem. This is clasically a /init script (PID 1), which then spawns the rest of the entire system (sysvinit, systemd, etc)

Putting the right bits in the right places

This is where it is at the moment, it could change later.

• DHCP and PXE config is on helian

  • This provides the IP address and next-server and boot-filename

• The TFTP server is on illustrious or vector
  • This serves up GRUB, the GRUB menu, and the kernel and initrd

  • I'm running tftpd-hpa and it uses /srv/tftp/ as the docroot

• The web server is on illustrious or vector
  • This serves up the kickstart config for automated Almalinux installs

TFTP stuff

1. Upstream repo mirror is here, get the kernel and initrd: https://repo.almalinux.org/almalinux/9/BaseOS/x86_64/os/EFI/BOOT/

2. Drop that content in /srv/tftp/ like so:

   root@illustrious:/srv/tftp# tree
   .
   ├── BOOTX64.EFI
   ├── default.efi
   ├── grub
   │   ├── grub.cfg
   │   ├── grub.cfg-01-98-90-96-be-89-52
   │   └── grubx64.efi
   ├── images
   │   └── Alma-9.1
   │       ├── initrd.img
   │       └── vmlinuz
   ├── ipxe.efi
   └── shimx64.efi

3. Add a grub config fragment for the host's MAC address: grub.cfg-01-xx-xx-xx-xx-xx-xx

   • Make sure the grub config has the correct URL for its kickstart config

This was useful for figuring out the TFTP stuff for the first time: https://askubuntu.com/questions/1183487/grub2-efi-boot-via-pxe-load-config-file-automatically

Paths are hardcoded into the grubx64.efi binary, meaning HDD and PXE versions aren't the same. Make sure you put all the grub stuff in a grub/ directory. Check the $prefix to see where it's searching

HTTP stuff

On illustrious the kickstart files are being served from /data/www/illustrious/ks, and the final URL is something like https://illustrious.thighhighs.top/ks/persica1.ks.cfg

On vector I'm using micro-httpd, which serves up the classic /var/www/html, so put it in /var/www/html/ks/hostname.ks.cfg and the URL is http://vector.thighhighs.top/ks/kalina.ks.cfg

Make sure your per-host config file has the correct name!

Kickstart knowledge

Some good references:

• Reference manual: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/performing_an_advanced_rhel_9_installation/kickstart-commands-and-options-reference_installing-rhel-as-an-experienced-user#keyboard-required_kickstart-commands-for-system-configuration

• Generator tool: https://access.redhat.com/labs/kickstartconfig/

• k8s doesn't play well with swap so we need to disable it. You can provision a minimal swap volume of 1gb, then disable it later. Or just don't have one at all, that should work too.

Working with ISOs

Let's say you've got a CD ISO, but you want to boot that from the network, because using virtual media over an iDRAC sucks.

• https://askubuntu.com/questions/405712/how-to-convert-a-livecd-into-a-vmlinuz-and-initrd-to-pxe-boot-load-slowly

• https://oitibs.com/pxe-boot-almost-any-iso-image/

• And you wanna jam it into Foreman: https://projects.theforeman.org/projects/foreman/wiki/Fetch_boot_files_via_http_instead_of_TFTP

• Looks like memdisk is involved in "chaining" the ISO along: https://wiki.syslinux.org/wiki/index.php?title=MEMDISK

Trying to PXE boot a Pop OS install

Well I forgot I had this page, but I've been wanting it again because I tried to install Pop OS and had no luck with local USB.

It get somewhat more complex with UEFI, I think, but here are some notes. Pop OS has no published support for netinst, only USB and DVD installs. For some reason, illustrious simply will not read the squashfs from the USB without reporting errors, wtf O_o

• How it can work: https://unix.stackexchange.com/questions/218379/boot-iso-file-through-uefi-by-calling-efi-executable-inside-the-iso

• Live CD to netbootable: https://discourse.ubuntu.com/t/netbooting-the-live-server-installer/14510

• Ubuntu hardware enablement notes: https://wiki.ubuntu.com/Kernel/LTSEnablementStack

Maybe can PXE this with a pxelinux.cfg/default entry:

# taken from grub.conf on the iso
menuentry "Try or Install Pop_OS" --class pop-os {
        set gfxpayload=keep
        linux /casper_pop-os_20.10_amd64_nvidia_debug_22/vmlinuz.efi boot=casper live-media-path=/casper_pop-os_20.10_amd64_nvidia_debug_22 hostname=pop-os username=pop-os noprompt  modules_load=nvidia ---
        initrd /casper_pop-os_20.10_amd64_nvidia_debug_22/initrd.gz
}

How to run tftp server on suomi:

• /home/furinkan/Documents/tftp_root/uefi
• sudo py3tftp -p 69
• Unifi DHCP points to x.x.x.175 and looks for syslinux.efi

I found that Pop's initrd doesn't have support for a ramdisk, so copying Ubuntu's config didn't work with it. Here's some tricky stuff with initrds:

• Mangling your initrd: https://wiki.ubuntu.com/CustomizeLiveInitrd

• Modern initrds are whack: https://unix.stackexchange.com/questions/163346/why-is-it-that-my-initrd-only-has-one-directory-namely-kernel