• servers
• azusa

azusa

• RPi Model 3B+ (aarch64, 4-core, 1gb RAM)
• RPi OS Lite 64-bit (Bookworm)

• LAN MAC b8:27:eb:8c:f4:f8

• located at home

Build notes

This is the most generic stuff to do for initial setup, before tweaking it to a specific use.

OS imaging

Using the Raspberry Pi Imager app, start with the current RPi OS Lite 64-bit, which is Debian bookwork 12.2, suitable for the RPi 3B+

It lets you make some customisations before flashing, which is really nice:

• Set hostname to azusa
• Enable SSH
  • Password auth (I would use SSH keys but it didn't work right for me and I couldn't sudo later)
• Set username and password

  • furinkan // <something new>

• No WLAN
• Set locale to Australia/Sydney, us keyboard
• Disable telemetry

Prepare DHCP server with static address for the LAN MAC address (should already be in place).

Put in the card and let it boot, should be fairly quick.

First login

1. Login as furinkan@azusa and copy your SSH key there

   ssh-keygen -t ed25519
   # Enter 3 times
   
   touch ~/.ssh/authorized_keys
   chmod 0600 ~/.ssh/authorized_keys
   vi ~/.ssh/authorized_keys

2. sudo up and copy your SSH key to root's account as well, use the same commands again
3. Login again directly as root

4. Install base packages

   apt install -y vim git screen ack

5. Edit /etc/pam.d/sshd and remove user_readenv=1, this will keep the logs tidy

6. Configure vim

   cat <<EOF > ~/.vimrc
   set nocompatible
   syntax on
   set background=dark
   set hlsearch
   set modeline
   set scrolloff=3
   EOF

7. Configure shell

   • Edit /root/.bashrc to enable colours

   • Set the default editor to vim.basic:

     update-alternatives --config editor

Disable wifi and bluetooth and other stuff

I'm using azusa as a network appliance, so I don't need the radios: https://sleeplessbeastie.eu/2018/12/31/how-to-disable-onboard-wifi-and-bluetooth-on-raspberry-pi-3/

Also see the notes for RPi4, because the OS has changed in the meantime too: https://sleeplessbeastie.eu/2022/06/01/how-to-disable-onboard-wifi-and-bluetooth-on-raspberry-pi-4/

1. Add dtoverlays to your /boot/config.txt to disable the radios:

   cat <<EOF >> /boot/config.txt
   dtoverlay=disable-wifi
   dtoverlay=disable-bt
   EOF

2. Let's also disable sound

   sed -r -i 's,^dtparam=audio=on$,dtparam=audio=off,' /boot/config.txt

3. Disable bluetooth and modem services

   systemctl disable --now hciuart
   systemctl disable --now bluetooth.target
   systemctl disable --now bluetooth.service
   systemctl disable --now ModemManager.service

4. Nuke the software packages as well

   apt purge -y bluez bluez-firmware wpasupplicant
   rm -rfv /etc/wpa_supplicant

5. No keyboard means no hotkeys needed

   apt purge -y triggerhappy

6. We don't need avahi functionality either

   apt purge -y avahi-daemon

7. We don't need tty1 console config, maybe this'll help her boot faster

   apt purge -y console-setup-linux

8. Clean up leftover packages

   apt autoremove -y

9. Reboot

Fix IPv6 SLAAC address

WhyTF am I not getting an EUI-64-based IPv6 SLAAC address now? It's worked every time before. Looks like our network config uses NetworkManager, so we need to configure that. It seems like it's not using privacy addresses, but it is doing stable-privacy now, which I don't want.

cat <<EOF > /etc/NetworkManager/conf.d/ip6-privacy.conf
[connection]
ipv6.ip6-privacy=0
ipv6.addr-gen-mode=0
EOF

Then reboot again.

The mode is now "default" instead of "eui64" as I would've expected (nmcli connection show Wired\ connection\ 1 | grep addr-gen), and I've no idea what that default is, but I don't care because it works now.

Read here for references:

• https://github.com/coreos/fedora-coreos-tracker/issues/907

• https://developer-old.gnome.org/NetworkManager/stable/settings-ipv6.html

• https://askubuntu.com/questions/1268900/what-is-setting-my-ipv6-addr-gen-mode

Other tweaks

1. Using raspi-config:

   • System -> Audio -> pass out through HDMI

   • Display -> Screen blanking -> Disable it

   • Then exit and let it reboot

2. Configure screen

   curl -o ~/.screenrc https://gist.githubusercontent.com/barneydesmond/d16c5201ed9d2280251dfca7c620bb86/raw/.screenrc

3. Configure top: z x c s 1.5 <Enter> e 0 1 W q

Configure hardware RTC

I've installed the Jaycar XC-9044 RPi realtime clock RTC, it's apparently a good clock chip with a little battery (or something). Most models using this chip have a spot for a watch battery, but this one has a tiiiiny little thing soldered on the board. I hope it's decent.

1. Physically install the module on the 3V3 plus I2C pins

2. Enable i2c with raspi-config, it's in Interface Options -> I2C -> Enable

   • You can also do it yourself if you want:

     1. Comment out any blacklist entries for i2c[-_]bcm2708 in /etc/modprobe.d/raspi-blacklist.conf

     2. Load the module at boot:

        echo i2c-dev >> /etc/modules

     3. Uncomment/add dtparam=i2c_arm=on in /boot/config.txt

     4. Activate it now:

        modprobe i2c-dev

3. Reboot now, it can't hurt

4. Install i2c tools

   apt install -y i2c-tools

5. Detect the device on i2c bus: i2cdetect -y 1

   • Should appear at 0x68

6. Enable the kernel driver for it, or something, by adding a devicetree overlay

   echo "dtoverlay=i2c-rtc,ds3231" >> /boot/config.txt

7. Reboot again to load the device tree overlay that we just configured

8. Again detect the device on i2c bus: i2cdetect -y 1

   • Should appear at 0x68, BUT with "UU" at the address this time

9. Remove the fake hardware clock

   systemctl disable fake-hwclock --now
   apt purge -y fake-hwclock

10. In theory everything just works now thanks to a udev rule: https://www.raspberrypi.org/forums/viewtopic.php?t=209700

    root@azusa:~# cat /lib/udev/rules.d/85-hwclock.rules
    # Set the System Time from the Hardware Clock and set the kernel's timezone
    # value to the local timezone when the kernel clock module is loaded.
    KERNEL=="rtc0", RUN+="/usr/lib/udev/hwclock-set $root/$name"

11. Install chrony so it manages the hardware clock

    apt install -y chrony

    It'll do the rest once it's installed and synced. Try some commands to see how it's fairing:

    chronyc sources
    chronyc tracking

Full system update

Do a full update to make sure everything is current

apt full-upgrade -y

Clean up leftover packages

apt autoremove -y

Save a copy

Optionally take a backup image of the fully configured system, by putting the card in another machine and shrinking the filesystem

e2fsck -f /dev/sdb2
resize2fs -M /dev/sdb2
# These numbers are assuming that the filesystem is a bit under 4GB in size, which in this case it was.
# We capture the first 4GB of the SD card to be sure.
dd bs=4M count=1024 if=/dev/sdb  | pv -br | pigz --fast > "$(date +%Y-%m-%d)_azusa_pristine_config.img.gz"

Configure services

Install useful tools

apt install -y bind9-host bind9-dnsutils inetutils-telnet

TFTP server

1. Install the daemon

   apt install -y tftpd-hpa

2. Add some verbosity to the logs

   # Edit /etc/default/tftpd-hpa and update the options:
   TFTP_OPTIONS="--secure --verbosity 3"

   This is still a legacy initd service, but you can get view logs with

   journalctl -u tftpd-hpa.service -f

3. Copy your stuff into /src/tftp

   rsync -avx root@illustrious:/srv/tftp/ /srv/tftp/

HTTP server

We need to serve the kickstart files via HTTP.

1. Install package

   apt install -y micro-httpd

2. SKIP THIS, IT'S ONLY NEEDED IF YOU NEED TO LISTEN ON SOMETHING OTHER THAN PORT 80

   systemctl edit micro-httpd.socket
   
   # Put this is there when prompted
   [Socket]
   ListenStream=
   ListenStream=0.0.0.0:8080
   
   # Just to be sure
   systemctl restart micro-httpd.socket

3. Create the httpd docroot

   mkdir /var/www/html/ks

4. Copy the kickstart files in there

   rsync -avx illustrious:/data/www/illustrious/ks/ /var/www/html/ks/

DHCP config

Now go tell servers/helian to refer to azusa as the PXE boot target.

Read-only optimisation

This is something I experimented with before on makarov, the situation is even easier now and it seems pretty viable. The page I read before was: https://medium.com/@andreas.schallwig/make-your-raspberry-pi-file-system-read-only-raspbian-buster-c558694de79

It seems like now you'd probably just...

• There's no syslog any more, so just set journald to not use disk at all

  echo -e "[Journal]\nStorage=volatile" > /etc/systemd/journald.conf.d/mem-only.conf
  systemctl restart systemd-journald.service

• Don't use swap

  systemctl stop dphys-swapfile.service
  apt purge -y dphys-swapfile

• Clean up leftover packages

  apt autoremove -y

• Reboot

There's more to it as I didn't follow all the steps there, but it's a start. You'd need to see what your various services do, and probably bump them to a separate filesystem so that it can be safely corrupted and leave the OS partition alone.

Adding PXE boot targets

We've got Alma 9.2 but now we want to get Alma 9.3 which just came out.

1. Create a directory for the PXE kernel and initrd

   mkdir /srv/tftp/images/Alma-9.3
   cd /srv/tftp/images/Alma-9.3/

2. Grab the PXE files and download them to that directory

   wget \
   https://almalinux.mirror.digitalpacific.com.au/9.3/BaseOS/x86_64/os/images/pxeboot/vmlinuz \
   https://almalinux.mirror.digitalpacific.com.au/9.3/BaseOS/x86_64/os/images/pxeboot/initrd.img

3. Fix ownership

   chown -R root:nogroup /srv/tftp/images/Alma-9.3

4. Update the grub configs in /srv/tftp/grub to refer to the new path

Ansible management for targets

azusa will be the ansible master, we'll configure the persica cluster hosts before building a k8s cluster with them.

• Prep ansible repo

  mkdir -p ~/git/ansible
  mkdir -p ~/git/k8s
  
  cd ~/git/ansible/

• Grab the repo from wherever you've stashed it

• Configure ~/.gitconfig from an existing machine

• Configure ~/.gitconfig-usesmain from an existing machine

At this point you should be able to make persica and the nodes will be prepared, rancher controller and worker nodes alike.