|
Size: 2423
Comment: k8s provisioning notes
|
Size: 3271
Comment: note about running pihole restartdns after making changes
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 30: | Line 30: |
| * Full UEFI mode * PXE boot for kickstart install * tftpd-hpa running on illustrious |
=== Per node === * Set BIOS to full UEFI mode, no legacy * We'll be using DHCP, so find the MAC address so we can give it a consistent IP address when it boots * Add the MAC address and IP assignment to dnsmasq on calico (a pihole box) * `/etc/dnsmasq.d/02-pihole-dhcp-persica-cluster.conf` * Something like this {{{ dhcp-host=98:90:96:BE:89:52,set:persica,192.168.1.31,persica1,5m # one dhcp-host line per host dhcp-boot=tag:persica,grub/grubx64.efi,illustrious.thighhighs.top,192.168.1.12 }}} * Run `pihole restartdns` after making changes * PXE boot for kickstart install, which will hit calico for DHCP, then illustrious for the boot image and kickstart config * tftpd-hpa is running on illustrious |
| Line 34: | Line 45: |
| * Drop that content in `/srv/tftp/` {{{ root@illustrious:/srv/tftp# tree . ├── BOOTX64.EFI ├── default.efi ├── grub │ ├── grub.cfg │ ├── grub.cfg-01-98-90-96-be-89-52 │ └── grubx64.efi ├── images │ └── Alma-9.1 │ ├── initrd.img │ └── vmlinuz ├── ipxe.efi └── shimx64.efi }}} |
|
| Line 38: | Line 65: |
| * k8s doesn't play well with swap so we need to disable it. Provision a minimal swap volume of 1gb, then disable it later | |
| Line 39: | Line 67: |
| This was useful for figuring out the TFTP stuff for the first time. Paths are hardcoded into the `grubx64.efi` binary, meaning HDD and PXE versions aren't the same. Make sure you put all the grub stuff in a `grub/` directory. Check the `$prefix` to see where it's searching: https://askubuntu.com/questions/1183487/grub2-efi-boot-via-pxe-load-config-file-automatically | This was useful for figuring out the TFTP stuff for the first time: https://askubuntu.com/questions/1183487/grub2-efi-boot-via-pxe-load-config-file-automatically |
| Line 41: | Line 69: |
| I should ansible'ise everything. Can I start with this? {{{ AlmaLinux 9 - AppStream 3.0 MB/s | 3.1 kB 00:00 Importing GPG key 0xB86B3716: Userid : "AlmaLinux OS 9 <packager@almalinux.org>" Fingerprint: BF18 AC28 7617 8908 D6E7 1267 D36C B86C B86B 3716 From : /etc/pki/rpm-gpg/RPM-GPG-KEY-AlmaLinux-9 Is this ok [y/N]: y Key imported successfully }}} |
Paths are hardcoded into the `grubx64.efi` binary, meaning HDD and PXE versions aren't the same. Make sure you put all the grub stuff in a `grub/` directory. Check the `$prefix` to see where it's searching: === Ansible management after kickstart build === I should ansible'ise everything, making minimal assumptions about the kickstart part of the process. I'm keeping a simple ansible repo in `~/git/persica-ansible/` |
persica cluster
This is a cluster of three identical nodes, named persica1/2/3
- Alma Linux 9.1 x64
- Dell Optiplex 9020 Micro
- Intel Core i5-4590T @ 2.00 GHz
- 16gb DDR3-1600
- 128gb SSD
k8s notes
- Make a simple 3-node cluster
- Single-node control plane will run externally, on illustrious
Use kubeadm to build the cluster: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
- Selected containerd as the container runtime
- Will use Flannel as the networking plugin
- Allocated IPs:
- persica1 / 192.168.1.31
- persica2 / 192.168.1.32
- persica3 / 192.168.1.33
- Ingress: undecided so far
- Cgroup driver: let's use systemd
- k8s version: whatever is latest right now (2023-04-04)
Build notes
Per node
- Set BIOS to full UEFI mode, no legacy
- We'll be using DHCP, so find the MAC address so we can give it a consistent IP address when it boots
- Add the MAC address and IP assignment to dnsmasq on calico (a pihole box)
/etc/dnsmasq.d/02-pihole-dhcp-persica-cluster.conf
Something like this
dhcp-host=98:90:96:BE:89:52,set:persica,192.168.1.31,persica1,5m # one dhcp-host line per host dhcp-boot=tag:persica,grub/grubx64.efi,illustrious.thighhighs.top,192.168.1.12
Run pihole restartdns after making changes
- PXE boot for kickstart install, which will hit calico for DHCP, then illustrious for the boot image and kickstart config
- tftpd-hpa is running on illustrious
Upstream repo mirror: https://repo.almalinux.org/almalinux/9/BaseOS/x86_64/os/EFI/BOOT/
Drop that content in /srv/tftp/
root@illustrious:/srv/tftp# tree . ├── BOOTX64.EFI ├── default.efi ├── grub │ ├── grub.cfg │ ├── grub.cfg-01-98-90-96-be-89-52 │ └── grubx64.efi ├── images │ └── Alma-9.1 │ ├── initrd.img │ └── vmlinuz ├── ipxe.efi └── shimx64.efi
kickstart file served from /data/www/illustrious/ks: https://illustrious.thighhighs.top/ks/persica1.ks.cfg
- KS references:
Generator tool: https://access.redhat.com/labs/kickstartconfig/
- k8s doesn't play well with swap so we need to disable it. Provision a minimal swap volume of 1gb, then disable it later
This was useful for figuring out the TFTP stuff for the first time: https://askubuntu.com/questions/1183487/grub2-efi-boot-via-pxe-load-config-file-automatically
Paths are hardcoded into the grubx64.efi binary, meaning HDD and PXE versions aren't the same. Make sure you put all the grub stuff in a grub/ directory. Check the $prefix to see where it's searching:
Ansible management after kickstart build
I should ansible'ise everything, making minimal assumptions about the kickstart part of the process.
I'm keeping a simple ansible repo in ~/git/persica-ansible/