• servers
• illustrious

• Running at home, general purpose server
• Intel NUC (BOXNUC7i3BNH)
  • Core i3-7100U at 2.40GHz
  • 500GB WD Blue SN550 NVMe M.2
  • 8gb DDR4 2400MHz Kingston KVR24S17S8/8
• Ubuntu 21.10

users

• furinkan

services

• ssh
• http/s

  • https://rant.meidokon.net/furinkan/

• postgres 14

• Smokeping (available on internal IP, and IPv6)

build notes

On 2022-01-02, FDE on LVM, Ubuntu server 21.10

Had some problem with the curtin unpack of the base image, I changed the Ubuntu archive URL to Datamossa in AU and hey presto it worked. Guess something was corrupted, shrug.

• Mostly default settings
• Enable full disk encryption with LVM, default settings
• Enable sshd during install

basic env

• Install your authorized_keys into root and furinkan user

• Enable NOPASSWD sudo

  %sudo   ALL=(ALL:ALL) NOPASSWD: ALL

• Set hostname:

  hostnamectl set-hostname illustrious.thighhighs.top

• Set timezone

  timedatectl set-timezone Australia/Sydney

• Set editor

  echo "export EDITOR=vim" > /etc/profile.d/editor-vim.sh

• Disable HashKnownHosts

  echo -e "Host *\n    HashKnownHosts no" > /etc/ssh/ssh_config.d/99-global.conf

• Configure screen

  curl -o ~/.screenrc https://gist.githubusercontent.com/barneydesmond/d16c5201ed9d2280251dfca7c620bb86/raw/.screenrc

• Configure top by entering this cheatcode

  z x c b s 1.5 <Enter>
  e <zero> 1 W q

• Fix locales, select en_AU.UTF-8: dpkg-reconfigure locales

• Disable console blanking, seems this is already done by default: cat /sys/module/kernel/parameters/consoleblank

  • Already set to zero means it shouldn't blank

• Disable wifi and bluetooth, we don't need them and it slows down boot

  systemctl disable wpa_supplicant.service --now
  systemctl disable bluetooth.target --now

• Install useful packages

  apt update
  apt install -y vim screen bash-completion lsof tcpdump netcat strace nmap less bsdmainutils tzdata whiptail netbase wget curl python-is-python3 net-tools ack jq make elinks nmap whois ethtool bind9-dnsutils apt-utils man-db plocate

• Do a full upgrade, index the system for locate, then reboot

  apt full-upgrade
  updatedb
  reboot

Configure networking

Use netplan for this, it's convenient and easy.

cd /etc/netplan/
mv 00-installer-config.yaml 00-installer-config.yaml.disabled
vim 10-thighhighs.yaml

network:
    version: 2

    ethernets:
        eno1:
            critical: true
            dhcp-identifier: mac
            dhcp4: false
            dhcp4-overrides:
                use-dns: false
            dhcp6: true
            dhcp6-overrides:
                use-dns: false
            ipv6-privacy: false
            addresses:
                - "192.168.1.12/24"
                # :12 for the .1.12 IPv4
                - "2404:e80:42e3:0:12:0:0:12/64"
            routes:
                - to: 0.0.0.0/0
                  via: 192.168.1.1
                  on-link: true
            nameservers:
                addresses:
                    - 192.168.1.20
                    - 192.168.1.24
                    - fe80::e65f:1ff:fe1c:c6ea
                    - fe80::ba27:ebff:fe8c:f4f8
                search:
                    - thighhighs.top

Try applying it with netplan try, see if your SSH session still works, then go ahead and reboot if it's good.

Setup clevis for automated decrypt on boot

• apt install clevis-luks

• Bind the volume to the Tang servers

  clevis luks bind -d /dev/nvme0n1p3 sss '{"t": 1, "pins": {"tang": [{"url": "http://ocular.thighhighs.top:8888"},{"url": "http://funicular.thighhighs.top:8888"}]}}'

• apt install clevis-initramfs

Test by rebooting.

Docker Engine for services

Run with the official docs: https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository

Prep repo

apt install ca-certificates curl gnupg lsb-release
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" > /etc/apt/sources.list.d/docker.list

Install packages

apt update
apt install docker-ce docker-ce-cli containerd.io

Test that it's working

docker run hello-world

Setup log rotation in /etc/docker/daemon.json

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "50m",
    "max-file": "10"
  }
}

Prepare space for volumes

We'll use this later for dockerised apps.

mkdir /data

Reverse proxy for services

Proxy software

Let's try out Caddy, I've been curious for a while now and it might meet all my needs.

Use official docs for a repo-packaged version: https://caddyserver.com/docs/install

apt install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf https://dl.cloudsmith.io/public/caddy/stable/gpg.key        > /etc/apt/trusted.gpg.d/caddy-stable.asc
curl -1sLf https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt > /etc/apt/sources.list.d/caddy-stable.list

apt update
apt install caddy

This uses a systemwide config in /etc/caddy/Caddyfile, and acts as a generic HTTP server initially. It's serving up a Caddy landing page from /usr/share/caddy at http://illustrious.thighhighs.top/

SSL cert

Pop it in /etc/ssl like usual.

cd /etc/ssl/
# This is a one-time action
openssl dhparam -out dhparams.pem 4096

# Then copy the cert and key and intermediate CA chain here
cp KEY CERT /etc/ssl/
chgrp caddy /etc/ssl/STAR_*

Smokeping

Run using the docker container, it's more convenient and separates config+data from the installation.

https://hub.docker.com/r/linuxserver/smokeping

Prepare space for data and config using a logical volume

lvcreate -L 1G -n smokeping ubuntu-vg
mkfs.ext4 /dev/ubuntu-vg/smokeping
mkdir /data/smokeping

# Add to fstab
# Smokeping config and data
/dev/disk/by-uuid/a40142d8-06e0-44d7-b8bc-a3e20662cde2 /data/smokeping ext4 defaults 0 1

mount /data/smokeping
mkdir /data/smokeping/config
mkdir /data/smokeping/data
chown -R 1000:1000 /data/smokeping/config /data/smokeping/data

Run the container:

docker run -d \
  --name=smokeping \
  -e PUID=1000 \
  -e PGID=1000 \
  -e TZ=Australia/Sydney \
  -p 127.0.0.1:8000:80 \
  -v /data/smokeping/config:/config \
  -v /data/smokeping/data:/data \
  --restart unless-stopped \
  lscr.io/linuxserver/smokeping

Map it through with some caddy config

smokeping.thighhighs.top {
        reverse_proxy localhost:8000

        tls /etc/ssl/STAR_thighhighs_top.crtbundled /etc/ssl/STAR_thighhighs_top.key
}

Reload the config, and you should have a working smokeping again! \o/

systemctl reload caddy.service

Unifi controller

Move it to a slightly beefier machine, running it via Docker for cleanliness. Their insistence on only supporting Java 8 runtime is a nightmare, but perfect for Docker abstraction.

Prepare space

This is a somewhat bigger system that needs more diskspace

lvcreate -L 4G -n unifi ubuntu-vg
mkfs.ext4 /dev/ubuntu-vg/unifi
mkdir /data/unifi

## Add to fstab, use blkid to find the UUID
# Unifi controller data
/dev/disk/by-uuid/0a13b90e-904a-4803-896f-0f82e4a36518 /data/unifi ext4 defaults 0 1

mount /data/unifi
mkdir /data/unifi/config
chown -R 1000:1000 /data/unifi/config

Run container

docker run -d \
  --name=unifi-controller \
  -e PUID=1000 \
  -e PGID=1000 \
  -e MEM_LIMIT=1024 `#optional` \
  -e MEM_STARTUP=1024 `#optional` \
  -p 192.168.1.13:3478:3478/udp \
  -p 192.168.1.13:10001:10001/udp \
  -p 192.168.1.13:8080:8080 \
  -p 192.168.1.13:8443:8443 \
  -p 192.168.1.13:1900:1900/udp `#optional` \
  -p 192.168.1.13:8843:8843 `#optional` \
  -p 192.168.1.13:8880:8880 `#optional` \
  -p 192.168.1.13:6789:6789 `#optional` \
  -p 192.168.1.13:5514:5514/udp `#optional` \
  -v /data/unifi/config:/config \
  --restart unless-stopped \
  lscr.io/linuxserver/unifi-controller

Migration

Import the backup from old controller, then on the old controller switch the inform URL to the new IP address. We'll fix up DNS afterwards.

https://community.ui.com/questions/Replace-my-RPi-Controller-to-UC-CK-G2-plus-and-odd-behavior-in-the-Topology-view/36f5b1eb-ca15-46f2-a64b-c69d9628857e

Set the inform URL again on the new controller, because you've just restored a backup with the old IP.

TLS cert for unifi

Faff with the keystore so you can jam in your publicly signed cert. This is a script that I found and adapted.

# From https://util.wifi.gl/unifi-import-cert.sh which is now dead
# Modified by Barney Desmond on 2021-04-20 to just use a normal static paid-for cert.

# Author: Frank Gabriel, 01.01.2019
# Credits Kalle Lilja, @SprockTech and others
# Script location: /etc/letsencrypt/renewal-hooks/post/unifi-import-cert.sh (important for auto renewal)
# Tested with Debian 9 and UniFi 5.8.28, 5.9.22 and 5.9.32 - should work with any recent Unifi and Ubuntu/Debian releases

# This is the host-side of a Docker volume, you need to run this inside the
# container unless you have keytool installed outside.
UNIFI_DATADIR=/data/unifi/config/data

# XXX: this is the path inside the container
UNIFI_DATADIR=/config/data

# Backup previous keystore
cp -a "${UNIFI_DATADIR}/keystore" "${UNIFI_DATADIR}/keystore.backup.$(date +%F_%R)"
#cp -a /var/lib/unifi/keystore /var/lib/unifi/keystore.backup.$(date +%F_%R)

# XXX: do this manually on the host, outside of the container.
# Convert cert to PKCS12 format
# Ignore warnings
#openssl pkcs12 -export \
#       -inkey /etc/ssl/STAR_thighhighs_top.key \
#       -in /etc/ssl/STAR_thighhighs_top.crt \
#       -out /etc/ssl/STAR_thighhighs_top.p12 \
#       -name unifi -password pass:unifi

# Install certificate
# Ignore warnings
keytool -importkeystore \
        -deststorepass aircontrolenterprise \
        -destkeypass aircontrolenterprise \
        -destkeystore "${UNIFI_DATADIR}/keystore" \
        -srckeystore STAR_thighhighs_top.p12 \
        -srcstoretype PKCS12 \
        -srcstorepass unifi \
        -alias unifi \
        -noprompt

# Restart UniFi controller
#systemctl restart unifi