## page was renamed from servers/asval = azusa = * RPi Model 3B+ (aarch64, 4-core, 1gb RAM) * RPi OS Lite 64-bit (Bookworm) * LAN MAC `b8:27:eb:8c:f4:f8` * located at home <> == Build notes == This is the most generic stuff to do for initial setup, before tweaking it to a specific use. === OS imaging === Using the Raspberry Pi Imager app, start with the current '''RPi OS Lite 64-bit''', which is Debian bookwork 12.2, suitable for the RPi 3B+ It lets you make some customisations before flashing, which is really nice: * Set hostname to azusa * Enable SSH * Password auth (I would use SSH keys but it didn't work right for me and I couldn't sudo later) * Set username and password * `furinkan // ` * No WLAN * Set locale to Australia/Sydney, us keyboard * Disable telemetry Prepare DHCP server with static address for the LAN MAC address (should already be in place). Put in the card and let it boot, should be fairly quick. === First login === 1. Login as `furinkan@azusa` and copy your SSH key there {{{ ssh-keygen -t ed25519 # Enter 3 times touch ~/.ssh/authorized_keys chmod 0600 ~/.ssh/authorized_keys vi ~/.ssh/authorized_keys }}} 1. sudo up and copy your SSH key to root's account as well, use the same commands again 1. Login again directly as root 1. Install base packages {{{ apt install -y vim git screen ack }}} 1. Edit `/etc/pam.d/sshd` and remove `user_readenv=1`, this will keep the logs tidy 1. Configure vim {{{ cat < ~/.vimrc set nocompatible syntax on set background=dark set hlsearch set modeline set scrolloff=3 EOF }}} 1. Configure shell * Edit `/root/.bashrc` to enable colours * Set the default editor to vim.basic: {{{ update-alternatives --config editor }}} === Disable wifi and bluetooth and other stuff === I'm using azusa as a network appliance, so I don't need the radios: https://sleeplessbeastie.eu/2018/12/31/how-to-disable-onboard-wifi-and-bluetooth-on-raspberry-pi-3/ Also see the notes for RPi4, because the OS has changed in the meantime too: https://sleeplessbeastie.eu/2022/06/01/how-to-disable-onboard-wifi-and-bluetooth-on-raspberry-pi-4/ 1. Add dtoverlays to your `/boot/config.txt` to disable the radios: {{{ cat <> /boot/config.txt dtoverlay=disable-wifi dtoverlay=disable-bt EOF }}} 1. Let's also disable sound {{{ sed -r -i 's,^dtparam=audio=on$,dtparam=audio=off,' /boot/config.txt }}} 1. Disable bluetooth and modem services {{{ systemctl disable --now hciuart systemctl disable --now bluetooth.target systemctl disable --now bluetooth.service systemctl disable --now ModemManager.service }}} 1. Nuke the software packages as well {{{ apt purge -y bluez bluez-firmware wpasupplicant rm -rfv /etc/wpa_supplicant }}} 1. No keyboard means no hotkeys needed {{{ apt purge -y triggerhappy }}} 1. We don't need avahi functionality either {{{ apt purge -y avahi-daemon }}} 1. We don't need tty1 console config, maybe this'll help her boot faster {{{ apt purge -y console-setup-linux }}} 1. Clean up leftover packages {{{ apt autoremove -y }}} 1. Reboot === Fix IPv6 SLAAC address === WhyTF am I not getting an EUI-64-based IPv6 SLAAC address now? It's worked every time before. Looks like our network config uses !NetworkManager, so we need to configure that. It seems like it's not using privacy addresses, but it ''is'' doing stable-privacy now, which I don't want. {{{ cat < /etc/NetworkManager/conf.d/ip6-privacy.conf [connection] ipv6.ip6-privacy=0 ipv6.addr-gen-mode=0 EOF }}} Then reboot again. The mode is now "default" instead of "eui64" as I would've expected (`nmcli connection show Wired\ connection\ 1 | grep addr-gen`), and I've no idea what that default is, but I don't care because it works now. Read here for references: * https://github.com/coreos/fedora-coreos-tracker/issues/907 * https://developer-old.gnome.org/NetworkManager/stable/settings-ipv6.html * https://askubuntu.com/questions/1268900/what-is-setting-my-ipv6-addr-gen-mode === Other tweaks === 1. Using `raspi-config`: * System -> Audio -> pass out through HDMI * Display -> Screen blanking -> Disable it * Then exit and let it reboot 1. Configure screen {{{ curl -o ~/.screenrc https://gist.githubusercontent.com/barneydesmond/d16c5201ed9d2280251dfca7c620bb86/raw/.screenrc }}} 1. Configure `top`: `z x c s 1.5 e 0 1 W q` === Configure hardware RTC === I've installed the Jaycar XC-9044 RPi realtime clock RTC, it's apparently a good clock chip with a little battery (or something). Most models using this chip have a spot for a watch battery, but this one has a tiiiiny little thing soldered on the board. I hope it's decent. 1. Physically install the module on the 3V3 plus I2C pins 1. Enable i2c with `raspi-config`, it's in `Interface Options -> I2C -> Enable` * You can also do it yourself if you want: 1. Comment out any blacklist entries for `i2c[-_]bcm2708` in `/etc/modprobe.d/raspi-blacklist.conf` 1. Load the module at boot: {{{ echo i2c-dev >> /etc/modules }}} 1. Uncomment/add `dtparam=i2c_arm=on` in `/boot/config.txt` 1. Activate it now: {{{ modprobe i2c-dev }}} 1. Reboot now, it can't hurt 1. Install i2c tools {{{ apt install -y i2c-tools }}} 1. Detect the device on i2c bus: `i2cdetect -y 1` * Should appear at 0x68 1. Enable the kernel driver for it, or something, by adding a devicetree overlay {{{ echo "dtoverlay=i2c-rtc,ds3231" >> /boot/config.txt }}} 1. Reboot again to load the device tree overlay that we just configured 1. Again detect the device on i2c bus: `i2cdetect -y 1` * Should appear at 0x68, BUT with "UU" at the address this time 1. Remove the fake hardware clock {{{ systemctl disable fake-hwclock --now apt purge -y fake-hwclock }}} 1. In theory everything just works now thanks to a udev rule: https://www.raspberrypi.org/forums/viewtopic.php?t=209700 {{{ root@azusa:~# cat /lib/udev/rules.d/85-hwclock.rules # Set the System Time from the Hardware Clock and set the kernel's timezone # value to the local timezone when the kernel clock module is loaded. KERNEL=="rtc0", RUN+="/usr/lib/udev/hwclock-set $root/$name" }}} 1. Install chrony so it manages the hardware clock {{{ apt install -y chrony }}} It'll do the rest once it's installed and synced. Try some commands to see how it's fairing: {{{ chronyc sources chronyc tracking }}} === Full system update === Do a full update to make sure everything is current {{{ apt full-upgrade -y }}} Clean up leftover packages {{{ apt autoremove -y }}} === Save a copy === Optionally take a backup image of the fully configured system, by putting the card in another machine and shrinking the filesystem {{{ e2fsck -f /dev/sdb2 resize2fs -M /dev/sdb2 # These numbers are assuming that the filesystem is a bit under 4GB in size, which in this case it was. # We capture the first 4GB of the SD card to be sure. dd bs=4M count=1024 if=/dev/sdb | pv -br | pigz --fast > "$(date +%Y-%m-%d)_azusa_pristine_config.img.gz" }}} == Configure services == === Install useful tools === {{{ apt install -y bind9-host bind9-dnsutils inetutils-telnet }}} === TFTP server === 1. Install the daemon {{{ apt install -y tftpd-hpa }}} 1. Add some verbosity to the logs {{{ # Edit /etc/default/tftpd-hpa and update the options: TFTP_OPTIONS="--secure --verbosity 3" }}} This is still a legacy initd service, but you can get view logs with {{{ journalctl -u tftpd-hpa.service -f }}} 1. Copy your stuff into `/src/tftp` {{{ rsync -avx root@illustrious:/srv/tftp/ /srv/tftp/ }}} === HTTP server === We need to serve the kickstart files via HTTP. 1. Install package {{{ apt install -y micro-httpd }}} 1. '''SKIP THIS, IT'S ONLY NEEDED IF YOU NEED TO LISTEN ON SOMETHING OTHER THAN PORT 80''' {{{ systemctl edit micro-httpd.socket # Put this is there when prompted [Socket] ListenStream= ListenStream=0.0.0.0:8080 # Just to be sure systemctl restart micro-httpd.socket }}} 1. Create the httpd docroot {{{ mkdir /var/www/html/ks }}} 1. Copy the kickstart files in there {{{ rsync -avx illustrious:/data/www/illustrious/ks/ /var/www/html/ks/ }}} === DHCP config === Now go tell [[servers/helian]] to refer to azusa as the PXE boot target. == Read-only optimisation == This is something I experimented with before on `makarov`, the situation is even easier now and it seems pretty viable. The page I read before was: https://medium.com/@andreas.schallwig/make-your-raspberry-pi-file-system-read-only-raspbian-buster-c558694de79 It seems like now you'd probably just... * There's no syslog any more, so just set journald to not use disk at all {{{ echo -e "[Journal]\nStorage=volatile" > /etc/systemd/journald.conf.d/mem-only.conf systemctl restart systemd-journald.service }}} * Don't use swap {{{ systemctl stop dphys-swapfile.service apt purge -y dphys-swapfile }}} 1. Clean up leftover packages {{{ apt autoremove -y }}} 1. Reboot There's more to it as I didn't follow all the steps there, but it's a start. You'd need to see what your various services do, and probably bump them to a separate filesystem so that it can be safely corrupted and leave the OS partition alone. == Adding PXE boot targets == We've got Alma 9.2 but now we want to get Alma 9.3 which just came out. 1. Create a directory for the PXE kernel and initrd {{{ mkdir /srv/tftp/images/Alma-9.3 cd /srv/tftp/images/Alma-9.3/ }}} 1. Grab the PXE files and download them to that directory {{{ wget \ https://almalinux.mirror.digitalpacific.com.au/9.3/BaseOS/x86_64/os/images/pxeboot/vmlinuz \ https://almalinux.mirror.digitalpacific.com.au/9.3/BaseOS/x86_64/os/images/pxeboot/initrd.img }}} 1. Fix ownership {{{ chown -R root:nogroup /srv/tftp/images/Alma-9.3 }}} 1. Update the grub configs in `/srv/tftp/grub` to refer to the new path == Ansible management for targets == azusa will be the ansible master, we'll configure the persica cluster hosts before building a k8s cluster with them. * Prep ansible repo {{{ mkdir -p ~/git/ansible mkdir -p ~/git/k8s cd ~/git/ansible/ }}} * Grab the repo from wherever you've stashed it * Configure `~/.gitconfig` from an existing machine * Configure `~/.gitconfig-usesmain` from an existing machine At this point you should be able to `make persica` and the nodes will be prepared, rancher controller and worker nodes alike.