= persica cluster =

This is a cluster of three identical nodes, named `persica1/2/3`

 * Alma Linux 9.1 x64
 * Dell Optiplex 9020 Micro
  * Intel Core i5-4590T @ 2.00 GHz
  * 16gb DDR3-1600
  * 128gb SSD

<<TableOfContents>>

== k8s notes ==

 * Make a simple 3-node cluster
 * Single-node control plane will run externally, on illustrious
 * Use kubeadm to build the cluster: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
 * Selected containerd as the container runtime
 * Will use Flannel as the networking plugin
 * Allocated IPs:
  * persica1 / 192.168.1.31
  * persica2 / 192.168.1.32
  * persica3 / 192.168.1.33
 * Ingress: undecided so far
 * Cgroup driver: let's use systemd
 * k8s version: whatever is latest right now (2023-04-04)

== Build notes ==

=== Per node ===

 * Update the BIOS using this guide: https://www.dell.com/support/kbdoc/en-au/000131486/update-the-dell-bios-in-a-linux-or-ubuntu-environment#updatebios2015
  * Despite the usual Dell docs saying you need to make a DOS boot disk and run the flash updater app from there, it turns out that the BIOS Flash Update target (mash F12 to get the one-time boot menu) can read the `9020MA19.exe` file from a FAT32 filesystem on a USB stick just fine
  * Not sure if this only works in UEFI mode or not, but I kinda don't care because we ''want'' to be in UEFI mode
  * This applies to systems made from 2015 or later
  * The latest BIOS update for the Optiplex 9020M is version A19, released 
 * Set BIOS to full UEFI mode, no legacy
 * We'll be using DHCP, so find the MAC address so we can give it a consistent IP address when it boots
 * Add the MAC address and IP assignment to dnsmasq on calico (a pihole box)
  * `/etc/dnsmasq.d/02-pihole-dhcp-persica-cluster.conf`
  * Something like this {{{
dhcp-host=98:90:96:BE:89:52,set:persica,192.168.1.31,persica1,5m
# one dhcp-host line per host
dhcp-boot=tag:persica,grub/grubx64.efi,illustrious.thighhighs.top,192.168.1.12
}}}
  * Run `pihole restartdns` after making changes
 * PXE boot for kickstart install, which will hit calico for DHCP, then illustrious for the boot image and kickstart config
 * tftpd-hpa is running on illustrious
  * Upstream repo mirror: https://repo.almalinux.org/almalinux/9/BaseOS/x86_64/os/EFI/BOOT/
  * Drop that content in `/srv/tftp/` {{{
root@illustrious:/srv/tftp# tree
.
├── BOOTX64.EFI
├── default.efi
├── grub
│   ├── grub.cfg
│   ├── grub.cfg-01-98-90-96-be-89-52
│   └── grubx64.efi
├── images
│   └── Alma-9.1
│       ├── initrd.img
│       └── vmlinuz
├── ipxe.efi
└── shimx64.efi
}}}
  * Add a grub config fragment for the host's MAC address: `grub.cfg-01-xx-xx-xx-xx-xx-xx`
  * Make sure the grub config has the correct URL for its kickstart config
 * kickstart file served from `/data/www/illustrious/ks`: https://illustrious.thighhighs.top/ks/persica1.ks.cfg
  * Make sure your per-host config file has the correct name
 * KS references:
  * Reference manual: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/performing_an_advanced_rhel_9_installation/kickstart-commands-and-options-reference_installing-rhel-as-an-experienced-user#keyboard-required_kickstart-commands-for-system-configuration
  * Generator tool: https://access.redhat.com/labs/kickstartconfig/
 * k8s doesn't play well with swap so we need to disable it. Provision a minimal swap volume of 1gb, then disable it later

This was useful for figuring out the TFTP stuff for the first time: https://askubuntu.com/questions/1183487/grub2-efi-boot-via-pxe-load-config-file-automatically

Paths are hardcoded into the `grubx64.efi` binary, meaning HDD and PXE versions aren't the same. Make sure you put all the grub stuff in a `grub/` directory. Check the `$prefix` to see where it's searching: 

=== UEFI settings ===

Get to the UEFI
 * Probably get stuck in windows for first boot
 * Win, then "UEFI", get to advanced startup options
 * Boot with Advanced Boot Options
 * Troubleshoot, Advanced Options, UEFI Firmware Settings, Restart

Record details
 * Get the LOM MAC Address from Settings, General, System Info

Change settings
 * General
  * Boot Sequence
   * Select UEFI boot list
  * Advanced Boot Options
   * Disable Legacy OPROMs
  * UEFI Boot Path Security
   * Set to Never
  * Date/Time
   * Set clock to approx correct for UTC time
 * System Configuration
  * Integrated NIC
   * Enable UEFI Network Stack
   * Enabled w/ PXE
  * SATA Operation
   * AHCI
  * SMART Reporting
   * Disabled, we don't need it
  * Audio
   * Disable all audio, we don't need it
 * Security
  * TPM Security
   * Check everything except Clear
   * Activated
  * CPU XD support
   * Enabled
 * Secure Boot
  * Secure Boot Enable
   * Disabled
 * Performance
  * Multi-core support: All
  * Speedstep: Enabled
  * C-states: Enabled
  * Limit CPUID: Disabled
  * Turboboost: Enabled
 * Power Management
  * AC Recovery: Power On
  * Deep Sleep Control: Disabled
  * USB Wake Support: Enable USB wake from Standby
  * Wake on LAN/WLAN: LAN with PXE Boot
  * Block Sleep: Enable blocking of sleep
 * POST Behaviour
  * Keyboard Errors: Disable error detection
 * Virtualisation support
  * Enable VT
  * Enable VT-d
  * Enable Trusted Execution

Reboot and go back in again.
 * Boot only from IPv4 with NIC (PXE boot) 


=== Ansible management after kickstart build ===

I should ansible'ise everything, making minimal assumptions about the kickstart part of the process.

I'm keeping a simple ansible repo in `~/git/persica-ansible/`