persica cluster

This is a cluster of three identical nodes, named persica1/2/3

• Alma Linux 9.1 x64
• Dell Optiplex 9020 Micro
  • Intel Core i5-4590T @ 2.00 GHz
  • 16gb DDR3-1600
  • 128gb SSD

k8s notes

• Make a simple 3-node cluster
• Single-node control plane will run externally, on illustrious

• Use kubeadm to build the cluster: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

• Selected containerd as the container runtime
• Will use Flannel as the networking plugin
• Allocated IPs:
  • persica1 / 192.168.1.31
  • persica2 / 192.168.1.32
  • persica3 / 192.168.1.33
• Ingress: undecided so far
• Cgroup driver: let's use systemd
• k8s version: whatever is latest right now (2023-04-04)

Build notes

Per node

• Set BIOS to full UEFI mode, no legacy
• We'll be using DHCP, so find the MAC address so we can give it a consistent IP address when it boots
• Add the MAC address and IP assignment to dnsmasq on calico (a pihole box)

  • /etc/dnsmasq.d/02-pihole-dhcp-persica-cluster.conf

  • Something like this

    dhcp-host=98:90:96:BE:89:52,set:persica,192.168.1.31,persica1,5m
    # one dhcp-host line per host
    dhcp-boot=tag:persica,grub/grubx64.efi,illustrious.thighhighs.top,192.168.1.12

  • Run pihole restartdns after making changes

• PXE boot for kickstart install, which will hit calico for DHCP, then illustrious for the boot image and kickstart config
• tftpd-hpa is running on illustrious

  • Upstream repo mirror: https://repo.almalinux.org/almalinux/9/BaseOS/x86_64/os/EFI/BOOT/

  • Drop that content in /srv/tftp/

    root@illustrious:/srv/tftp# tree
    .
    ├── BOOTX64.EFI
    ├── default.efi
    ├── grub
    │   ├── grub.cfg
    │   ├── grub.cfg-01-98-90-96-be-89-52
    │   └── grubx64.efi
    ├── images
    │   └── Alma-9.1
    │       ├── initrd.img
    │       └── vmlinuz
    ├── ipxe.efi
    └── shimx64.efi

  • Add a grub config fragment for the host's MAC address: grub.cfg-01-xx-xx-xx-xx-xx-xx

  • Make sure the grub config has the correct URL for its kickstart config

• kickstart file served from /data/www/illustrious/ks: https://illustrious.thighhighs.top/ks/persica1.ks.cfg

  • Make sure your per-host config file has the correct name
• KS references:

  • Reference manual: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/performing_an_advanced_rhel_9_installation/kickstart-commands-and-options-reference_installing-rhel-as-an-experienced-user#keyboard-required_kickstart-commands-for-system-configuration

  • Generator tool: https://access.redhat.com/labs/kickstartconfig/

• k8s doesn't play well with swap so we need to disable it. Provision a minimal swap volume of 1gb, then disable it later

This was useful for figuring out the TFTP stuff for the first time: https://askubuntu.com/questions/1183487/grub2-efi-boot-via-pxe-load-config-file-automatically

Paths are hardcoded into the grubx64.efi binary, meaning HDD and PXE versions aren't the same. Make sure you put all the grub stuff in a grub/ directory. Check the $prefix to see where it's searching:

Ansible management after kickstart build

I should ansible'ise everything, making minimal assumptions about the kickstart part of the process.

I'm keeping a simple ansible repo in ~/git/persica-ansible/