Differences between revisions 4 and 8 (spanning 4 versions)

persica cluster

This is a cluster of three identical nodes, named persica1/2/3

Alma Linux 9.1 x64
Dell Optiplex 9020 Micro
- Intel Core i5-4590T @ 2.00 GHz
- 16gb DDR3-1600
- 128gb SSD

Contents

persica cluster
1. k8s notes
2. Build notes

k8s notes

Make a simple 3-node cluster
Single-node control plane will run externally, on illustrious
Use kubeadm to build the cluster: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
Selected containerd as the container runtime
Will use Flannel as the networking plugin
Allocated IPs:
- persica1 / 192.168.1.31
- persica2 / 192.168.1.32
- persica3 / 192.168.1.33
Ingress: undecided so far
Cgroup driver: let's use systemd
k8s version: whatever is latest right now (2023-04-04)

Build notes

Per node

Set BIOS to full UEFI mode, no legacy
We'll be using DHCP, so find the MAC address so we can give it a consistent IP address when it boots
Add the MAC address and IP assignment to dnsmasq on calico (a pihole box)
- /etc/dnsmasq.d/02-pihole-dhcp-persica-cluster.conf
- Something like this
```
dhcp-host=98:90:96:BE:89:52,set:persica,192.168.1.31,persica1,5m
# one dhcp-host line per host
dhcp-boot=tag:persica,grub/grubx64.efi,illustrious.thighhighs.top,192.168.1.12
```
- Run pihole restartdns after making changes
PXE boot for kickstart install, which will hit calico for DHCP, then illustrious for the boot image and kickstart config

tftpd-hpa is running on illustrious

Upstream repo mirror: https://repo.almalinux.org/almalinux/9/BaseOS/x86_64/os/EFI/BOOT/

Drop that content in /srv/tftp/

root@illustrious:/srv/tftp# tree
.
├── BOOTX64.EFI
├── default.efi
├── grub
│   ├── grub.cfg
│   ├── grub.cfg-01-98-90-96-be-89-52
│   └── grubx64.efi
├── images
│   └── Alma-9.1
│       ├── initrd.img
│       └── vmlinuz
├── ipxe.efi
└── shimx64.efi

Add a grub config fragment for the host's MAC address: grub.cfg-01-xx-xx-xx-xx-xx-xx
Make sure the grub config has the correct URL for its kickstart config

kickstart file served from /data/www/illustrious/ks: https://illustrious.thighhighs.top/ks/persica1.ks.cfg
- Make sure your per-host config file has the correct name
KS references:
- Reference manual: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/performing_an_advanced_rhel_9_installation/kickstart-commands-and-options-reference_installing-rhel-as-an-experienced-user#keyboard-required_kickstart-commands-for-system-configuration
- Generator tool: https://access.redhat.com/labs/kickstartconfig/
k8s doesn't play well with swap so we need to disable it. Provision a minimal swap volume of 1gb, then disable it later

This was useful for figuring out the TFTP stuff for the first time: https://askubuntu.com/questions/1183487/grub2-efi-boot-via-pxe-load-config-file-automatically

Paths are hardcoded into the grubx64.efi binary, meaning HDD and PXE versions aren't the same. Make sure you put all the grub stuff in a grub/ directory. Check the $prefix to see where it's searching:

UEFI settings

Get to the UEFI

Probably get stuck in windows for first boot
Win, then "UEFI", get to advanced startup options
Boot with Advanced Boot Options
Troubleshoot, Advanced Options, UEFI Firmware Settings, Restart

Record details

Get the LOM MAC Address from Settings, General, System Info

Change settings

General
- Boot Sequence
  - Select UEFI boot list
- Advanced Boot Options
  - Disable Legacy OPROMs
- UEFI Boot Path Security
  - Set to Never
- Date/Time
  - Set clock to approx correct for UTC time
System Configuration
- Integrated NIC
  - Enable UEFI Network Stack
  - Enabled w/ PXE
- SATA Operation
  - AHCI
- SMART Reporting
  - Disabled, we don't need it
- Audio
  - Disable all audio, we don't need it
Security
- TPM Security
  - Check everything except Clear
  - Activated
- CPU XD support
  - Enabled
Secure Boot
- Secure Boot Enable
  - Disabled
Performance
- Multi-core support: All
- Speedstep: Enabled
- C-states: Enabled
- Limit CPUID: Disabled
- Turboboost: Enabled
Power Management
- TBC

Ansible management after kickstart build

I should ansible'ise everything, making minimal assumptions about the kickstart part of the process.

I'm keeping a simple ansible repo in ~/git/persica-ansible/

-  ⇤ ← Revision 4 as of 2023-04-04 14:19:05 → 
  Size: 2547
  Editor: furinkan
  Comment: k8s doesn't like having swap enabled
+   ← Revision 8 as of 2023-04-10 08:48:15 → ⇥
  Size: 4674
  Editor: furinkan
  Comment: UEFI settings
-Deletions are marked like this.
+Additions are marked like this.
 Line 30:
- * Full UEFI mode
 * PXE boot for kickstart install
 * tftpd-hpa running on illustrious
+=== Per node ===

 * Set BIOS to full UEFI mode, no legacy
 * We'll be using DHCP, so find the MAC address so we can give it a consistent IP address when it boots
 * Add the MAC address and IP assignment to dnsmasq on calico (a pihole box)
  * `/etc/dnsmasq.d/02-pihole-dhcp-persica-cluster.conf`
  * Something like this {{{
dhcp-host=98:90:96:BE:89:52,set:persica,192.168.1.31,persica1,5m
# one dhcp-host line per host
dhcp-boot=tag:persica,grub/grubx64.efi,illustrious.thighhighs.top,192.168.1.12
}}}
  * Run `pihole restartdns` after making changes
 * PXE boot for kickstart install, which will hit calico for DHCP, then illustrious for the boot image and kickstart config
 * tftpd-hpa is running on illustrious
-Line 34:
+Line 45:
+  * Drop that content in `/srv/tftp/` {{{
root@illustrious:/srv/tftp# tree
.
├── BOOTX64.EFI
├── default.efi
├── grub
│   ├── grub.cfg
│   ├── grub.cfg-01-98-90-96-be-89-52
│   └── grubx64.efi
├── images
│   └── Alma-9.1
│       ├── initrd.img
│       └── vmlinuz
├── ipxe.efi
└── shimx64.efi
}}}
  * Add a grub config fragment for the host's MAC address: `grub.cfg-01-xx-xx-xx-xx-xx-xx`
  * Make sure the grub config has the correct URL for its kickstart config
-Line 35:
+Line 64:
+  * Make sure your per-host config file has the correct name
-Line 40:
+Line 70:
-This was useful for figuring out the TFTP stuff for the first time. Paths are hardcoded into the `grubx64.efi` binary, meaning HDD and PXE versions aren't the same. Make sure you put all the grub stuff in a `grub/` directory. Check the `$prefix` to see where it's searching: https://askubuntu.com/questions/1183487/grub2-efi-boot-via-pxe-load-config-file-automatically
+This was useful for figuring out the TFTP stuff for the first time: https://askubuntu.com/questions/1183487/grub2-efi-boot-via-pxe-load-config-file-automatically
-Line 42:
+Line 72:
-I should ansible'ise everything. Can I start with this?
{{{
AlmaLinux 9 - AppStream                                                                                                                                                                          3.0 MB/s | 3.1 kB     00:00
Importing GPG key 0xB86B3716:
 Userid     : "AlmaLinux OS 9 <packager@almalinux.org>"
 Fingerprint: BF18 AC28 7617 8908 D6E7 1267 D36C B86C B86B 3716
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-AlmaLinux-9
Is this ok [y/N]: y
Key imported successfully
}}}
+Paths are hardcoded into the `grubx64.efi` binary, meaning HDD and PXE versions aren't the same. Make sure you put all the grub stuff in a `grub/` directory. Check the `$prefix` to see where it's searching: 

=== UEFI settings ===

Get to the UEFI
 * Probably get stuck in windows for first boot
 * Win, then "UEFI", get to advanced startup options
 * Boot with Advanced Boot Options
 * Troubleshoot, Advanced Options, UEFI Firmware Settings, Restart

Record details
 * Get the LOM MAC Address from Settings, General, System Info

Change settings
 * General
  * Boot Sequence
   * Select UEFI boot list
  * Advanced Boot Options
   * Disable Legacy OPROMs
  * UEFI Boot Path Security
   * Set to Never
  * Date/Time
   * Set clock to approx correct for UTC time

 * System Configuration
  * Integrated NIC
   * Enable UEFI Network Stack
   * Enabled w/ PXE
  * SATA Operation
   * AHCI
  * SMART Reporting
   * Disabled, we don't need it
  * Audio
   * Disable all audio, we don't need it

 * Security
  * TPM Security
   * Check everything except Clear
   * Activated
  * CPU XD support
   * Enabled

 * Secure Boot
  * Secure Boot Enable
   * Disabled

 * Performance
  * Multi-core support: All
  * Speedstep: Enabled
  * C-states: Enabled
  * Limit CPUID: Disabled
  * Turboboost: Enabled

 * Power Management
  * TBC
 


=== Ansible management after kickstart build ===

I should ansible'ise everything, making minimal assumptions about the kickstart part of the process.

I'm keeping a simple ansible repo in `~/git/persica-ansible/`

Useful(?) links

Navigation