persica cluster

This is a cluster of three identical nodes, named persica1/2/3

• Alma Linux 9.1 x64
• Dell Optiplex 9020 Micro
  • Intel Core i5-4590T @ 2.00 GHz
  • 16gb DDR3-1600
  • 128gb SSD

k8s notes

• Make a simple 3-node cluster
• Single-node control plane will run externally, on illustrious

• Use kubeadm to build the cluster: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

• Selected containerd as the container runtime
• Will use Flannel as the networking plugin
• Allocated IPs:
  • persica1 / 192.168.1.31
  • persica2 / 192.168.1.32
  • persica3 / 192.168.1.33
• Ingress: undecided so far
• Cgroup driver: let's use systemd
• k8s version: whatever is latest right now (2023-04-04)

Build notes

Per node

• Update the BIOS using this guide: https://www.dell.com/support/kbdoc/en-au/000131486/update-the-dell-bios-in-a-linux-or-ubuntu-environment#updatebios2015

  • Despite the usual Dell docs saying you need to make a DOS boot disk and run the flash updater app from there, it turns out that the BIOS Flash Update target (mash F12 to get the one-time boot menu) can read the 9020MA19.exe file from a FAT32 filesystem on a USB stick just fine

  • Not sure if this only works in UEFI mode or not, but I kinda don't care because we want to be in UEFI mode

  • This applies to systems made from 2015 or later
  • The latest BIOS update for the Optiplex 9020M is version A19, released
• Set BIOS to full UEFI mode, no legacy
• We'll be using DHCP, so find the MAC address so we can give it a consistent IP address when it boots
• Add the MAC address and IP assignment to dnsmasq on calico (a pihole box)

  • /etc/dnsmasq.d/02-pihole-dhcp-persica-cluster.conf

  • Something like this

    dhcp-host=98:90:96:BE:89:52,set:persica,192.168.1.31,persica1,5m
    # one dhcp-host line per host
    dhcp-boot=tag:persica,grub/grubx64.efi,illustrious.thighhighs.top,192.168.1.12

  • Run pihole restartdns after making changes

• PXE boot for kickstart install, which will hit calico for DHCP, then illustrious for the boot image and kickstart config
• tftpd-hpa is running on illustrious

  • Upstream repo mirror: https://repo.almalinux.org/almalinux/9/BaseOS/x86_64/os/EFI/BOOT/

  • Drop that content in /srv/tftp/

    root@illustrious:/srv/tftp# tree
    .
    ├── BOOTX64.EFI
    ├── default.efi
    ├── grub
    │   ├── grub.cfg
    │   ├── grub.cfg-01-98-90-96-be-89-52
    │   └── grubx64.efi
    ├── images
    │   └── Alma-9.1
    │       ├── initrd.img
    │       └── vmlinuz
    ├── ipxe.efi
    └── shimx64.efi

  • Add a grub config fragment for the host's MAC address: grub.cfg-01-xx-xx-xx-xx-xx-xx

  • Make sure the grub config has the correct URL for its kickstart config

• kickstart file served from /data/www/illustrious/ks: https://illustrious.thighhighs.top/ks/persica1.ks.cfg

  • Make sure your per-host config file has the correct name
• KS references:

  • Reference manual: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/performing_an_advanced_rhel_9_installation/kickstart-commands-and-options-reference_installing-rhel-as-an-experienced-user#keyboard-required_kickstart-commands-for-system-configuration

  • Generator tool: https://access.redhat.com/labs/kickstartconfig/

• k8s doesn't play well with swap so we need to disable it. Provision a minimal swap volume of 1gb, then disable it later

This was useful for figuring out the TFTP stuff for the first time: https://askubuntu.com/questions/1183487/grub2-efi-boot-via-pxe-load-config-file-automatically

Paths are hardcoded into the grubx64.efi binary, meaning HDD and PXE versions aren't the same. Make sure you put all the grub stuff in a grub/ directory. Check the $prefix to see where it's searching:

UEFI settings

Get to the UEFI

• Probably get stuck in windows for first boot
• Win, then "UEFI", get to advanced startup options
• Boot with Advanced Boot Options
• Troubleshoot, Advanced Options, UEFI Firmware Settings, Restart

Record details

• Get the LOM MAC Address from Settings, General, System Info

Change settings

• General
  • Boot Sequence
    • Select UEFI boot list
  • Advanced Boot Options
    • Disable Legacy OPROMs
  • UEFI Boot Path Security
    • Set to Never
  • Date/Time
    • Set clock to approx correct for UTC time
• System Configuration
  • Integrated NIC
    • Enable UEFI Network Stack
    • Enabled w/ PXE
  • SATA Operation
    • AHCI
  • SMART Reporting
    • Disabled, we don't need it
  • Audio
    • Disable all audio, we don't need it
• Security
  • TPM Security
    • Check everything except Clear
    • Activated
  • CPU XD support
    • Enabled
• Secure Boot
  • Secure Boot Enable
    • Disabled
• Performance
  • Multi-core support: All
  • Speedstep: Enabled
  • C-states: Enabled
  • Limit CPUID: Disabled
  • Turboboost: Enabled
• Power Management
  • AC Recovery: Power On
  • Deep Sleep Control: Disabled
  • USB Wake Support: Enable USB wake from Standby
  • Wake on LAN/WLAN: LAN with PXE Boot
  • Block Sleep: Enable blocking of sleep
• POST Behaviour
  • Keyboard Errors: Disable error detection
• Virtualisation support
  • Enable VT
  • Enable VT-d
  • Enable Trusted Execution

Reboot and go back in again.

• Boot only from IPv4 with NIC (PXE boot)

Ansible management after kickstart build

I should ansible'ise everything, making minimal assumptions about the kickstart part of the process.

I'm keeping a simple ansible repo in ~/git/persica-ansible/