MeidokonWiki:

persica cluster

This is a cluster of three identical nodes, named persica1/2/3

I also later picked up an extra node to run as the controller, kalina, and whatever other services it might need. The controller is fairly heavyweight, so the Raspberry Pi won't cut it, and I don't want to jam it on the machine hosting my webserver at the same time.

I've also added a dedicated Mikrotik hEX S router (model RB760iGS) to the setup, it gives a dedicated /24 subnet to the cluster, routed to the rest of my LAN but without using NAT. Now I get to learn OSPF, and BGP once I add MetalLB to the cluster for ingress.

This has been such a chore getting things working smoothly, it's just so damn finicky and it makes my notes a mess. I've tried to clean them up, but I'll keep all the failure notes in a section at the end.

  • Doing it mostly raw with kubeadm sucked, the docs are completely unopinionated and give you every option at every instance of there being a choice. Great if you know what you're doing, but if you know what you're doing then you don't really need those docs

  • This guide looks like an improvement, something with actual explanations and a few opinions: https://github.com/hobby-kube/guide

  • So many guides assume you're doing this in the cloud, which is a fair assumption for starting as a beginner with no infra, but they make too many logical leaps that you have to fill in the gaps yourselves, or just can't be applied on your own baremetal

Contents

  1. persica cluster
    1. Intro
    2. Hardware prep for the cluster nodes
    3. Prepare azusa for PXE services
      1. ansible management for the cluster
      2. Have nice SSH config so azusa can connect to each k8s node easily
    4. Prepare kalina controller node
    5. Run Rancher on kalina
    6. Build the k8s nodes
    7. Ansible-ise the k8s nodes
    8. Stand up the cluster
    9. Install kubectl on controller kalina
    10. Install Longhorn cluster storage manager
      1. Backups
    11. Prepare dummy DNS records so we can test ingress and load balancing
    12. Load balancing with MetalLB
      1. Configure BGP
      2. Try MetalLB in Layer 2 mode first
      3. Try it in BGP mode next
    13. Making ingress work - was this for the kubeadm method?
    14. Things that suck
      1. cgroups
      2. Networking kernel modules
      3. Firewalls
      4. Cleanup and try again
      5. Installing older versions of kubectl
      6. You can't use selinux

Intro

So here's the gist of this setup, third or fourth attempt now:

Hardware prep for the cluster nodes

Setup each new node like so, it's stuff that we just need to do one time when we receive the hardware:

Prepare azusa for PXE services

This is needed so we can build kalina and the persica nodes consistently and easily. It can be used for other systems on the LAN as well, it's not just for this cluster.

Build servers/azusa as the network services node, directions on how to configure these components are on her page.

ansible management for the cluster

azusa will also host the ansible repo for managing the cluster.

Once a node is built with kickstart and online, we'll run an ansible playbook against it to get it up to spec. Make minimal assumptions about the kickstart part of the process, let ansible do the rest.

Have nice SSH config so azusa can connect to each k8s node easily

Make yourself a little config in ~/.ssh/config 

Host *
User root
IdentityFile ~/git/ansible/sshkey_ed25519

Prepare kalina controller node

Now build kalina:

  1. Kickstart-build kalina using the configs on azusa
  2. Run ansible against kalina, this will configure the OS and install docker.

  3. Check that docker works 

    docker run hello-world

  4. Push the certs from illustrious to kalina, we're using real publicly trusted CA-signed certs: https://ranchermanager.docs.rancher.com/pages-for-subheaders/rancher-on-a-single-node-with-docker#option-c-bring-your-own-certificate-signed-by-a-recognized-ca

    • On illustrious: 

      cd /etc/ssl/

rsync -avx \
  STAR_thighhighs_top.key \
  STAR_thighhighs_top.crtbundled \
  STAR_thighhighs_top.key.2023 \
  STAR_thighhighs_top.crtbundled.2023 \
  root@kalina:/etc/ssl/

    • Then on kalina: 

      chown root:root /etc/ssl/STAR_thighhighs_top.*

Run Rancher on kalina

If you're doing this on an ARM system follow this guide, it just tells you to specify an exact version so you know it's built with arm64 support: https://ranchermanager.docs.rancher.com/how-to-guides/advanced-user-guides/enable-experimental-features/rancher-on-arm64

Well I'm on x86 now so that doesn't matter, but I'm still going to specify an exact version because I'm sensible and want a repeatable build with no surprises. 

docker run -d --restart=unless-stopped \
  -p 443:443 \
  -v /etc/ssl/STAR_thighhighs_top.crtbundled:/etc/rancher/ssl/cert.pem \
  -v /etc/ssl/STAR_thighhighs_top.key:/etc/rancher/ssl/key.pem \
  --privileged \
  rancher/rancher:v2.6.6 \
  --no-cacerts

It'll take some time to start. Then you can try hitting the Rancher web UI: https://kalina.thighhighs.top/

Login with the local user password as directed, then let it set the new admin password. Record it somewhere safe, and set the server URL to https://persica.thighhighs.top because that's how we're going to access the cluster once we're done.

Build the k8s nodes

Manually kick the BIOS of each node to do a one-time PXE boot (mash F12 during POST), then let it do its thing.

Ansible-ise the k8s nodes

On azusa, run ansible against the hosts to configure the OS and install docker. 

make persica

Stand up the cluster

We're following these instructions: https://ranchermanager.docs.rancher.com/pages-for-subheaders/use-existing-nodes

  1. From the Dashboard click the Create button

  2. Select Use existing nodes and create a cluster using RKE

  3. Fill in the details
    • cluster name: persica
    • leave most options as default

    • I'm picking k8s version v1.20.15-rancher2-2 so it matches what we run at work, and I can test upgrades at home

    • set the docker root directory to /persist/docker because we're moving to a disk with plenty of space, separate to the OS

    • Allow unsupported versions of Docker is already enabled; we need this because we're using a much newer distro and docker version

    • Hit Next to go to the next page
  4. Check the boxes for all three cluster roles, all nodes will perform all roles

  5. Go ahead and run the supplied command on each node. I like to do it one at a time so I can watch it 

    docker run -d --privileged --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run  rancher/rancher-agent:v2.6.6 --server https://persica.thighhighs.top --token lx5qjbl4dn7zkpbmt5qqz8qfdvtgsl2x5ft95j8lh785bxrjjccq2t --etcd --controlplane --worker

docker logs recursing_proskuriakova -f

Give it like 10min, eventually the containers logs that you're following will die, because the container terminates once all the k8s components are up and running.

Install kubectl on controller kalina

This friggen sucks for older version, no package management for you!

https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/#install-kubectl-binary-with-curl-on-linux

Once you've got it installed, go to Rancher and Explore the persica cluster (https://persica.thighhighs.top/dashboard/c/c-gfnh7/explorer#cluster-events), then copy the kubeconfig to your clipboard with the button in the toolbar at the top of the screen.

Go paste that into ~/.kube/config in your account on kalina, now you can run kubectl there!

Add this to your ~/.bashrc for cool tab-completion: 

if hash kubectl 2>/dev/null ; then
    source <(kubectl completion bash)
fi

Install Longhorn cluster storage manager

This is done from the builtin Helm charts, let it go to work. It's a couple of simple clicks: https://persica.thighhighs.top/dashboard/c/c-gfnh7/apps/charts?category=Storage

For some reason the predefined things you can configure on the helm chart don't include the local path to the disk on each node. Which is pretty bloody obvious you'd think, but no. It'll default to /var/lib/longhorn or something unless you override it.

  1. Install into the System project

  2. Do customise helm options before install

  3. Go to the Edit YAML page and change the defaultDataPath to /persist/longhorn/ instead

  4. Now you can run the install

I tried out this dude's demo app that uses flask and redis to deploy a trivial website, that was a nifty test of all the bits working together as expected:

Blessedly the ingress just works. No idea what to do yet to make a service that presents itself on public IPs.

Backups

I tried using Backblaze B2 as an S3 backend for backups, but couldn't get it working. 

error listing backup volume names: failed to execute: /var/lib/longhorn/engine-binaries/rancher-mirrored-longhornio-longhorn-engine-v1.3.3/longhorn [backup ls --volume-only s3.us-west-001.backblazeb2.com], output driver is not supported , stderr, time="2024-07-08T12:33:34Z" level=error msg="driver is not supported" , error exit status 1

https://forums.rancher.com/t/longhorn-ibm-object-storage-backup-configuration/19175

Here, the backup target format should be s3://<your-bucket-name>@<your-aws-region>/mypath/

s3://persica-longhorn-backups@s3.us-west-001.backblazeb2.com/

https://github.com/longhorn/longhorn/issues/1552#issuecomment-678389544

error listing backup volume names: failed to execute: /var/lib/longhorn/engine-binaries/rancher-mirrored-longhornio-longhorn-engine-v1.3.3/longhorn [backup ls --volume-only s3://persica-longhorn-backups@s3.us-west-001.backblazeb2.com/], output failed to list objects with param: { Bucket: "persica-longhorn-backups", Delimiter: "/", Prefix: "/" } error: AWS Error: MissingEndpoint 'Endpoint' configuration is required for this service <nil> , stderr, time="2024-07-08T12:44:07Z" level=error msg="Failed to list s3" error="failed to list objects with param: {\n Bucket: \"persica-longhorn-backups\",\n Delimiter: \"/\",\n Prefix: \"/\"\n} error: AWS Error: MissingEndpoint 'Endpoint' configuration is required for this service <nil>\n" pkg=s3 time="2024-07-08T12:44:07Z" level=error msg="failed to list objects with param: {\n Bucket: \"persica-longhorn-backups\",\n Delimiter: \"/\",\n Prefix: \"/\"\n} error: AWS Error: MissingEndpoint 'Endpoint' configuration is required for this service <nil>\n" , error exit status 1

https://persica-longhorn-backups.s3.us-west-001.backblazeb2.com

So I gave up and used NFS served up by servers/iowa, which worked straight away.

Prepare dummy DNS records so we can test ingress and load balancing

Apps need ingress, and ingress means you need hostnames to refer to stuff. Let's add these to our zone: 

# Dodgy roundrobin for "load balancing" or ingress connections, which are terminated by a proxy on any node
persicanodes 300 IN A 192.168.3.3
persicanodes 300 IN A 192.168.3.4
persicanodes 300 IN A 192.168.3.5

# Now some unique names for all the apps we're going to try
app1.persica 300 IN CNAME persicanodes
app2.persica 300 IN CNAME persicanodes
app3.persica 300 IN CNAME persicanodes
app4.persica 300 IN CNAME persicanodes
app5.persica 300 IN CNAME persicanodes

# These will be BGP or Layer2 MetalLB IPs
lb1.persica 300 IN A 192.168.3.65
lb2.persica 300 IN A 192.168.3.66
lb3.persica 300 IN A 192.168.3.67
lb4.persica 300 IN A 192.168.3.68
lb5.persica 300 IN A 192.168.3.69

Load balancing with MetalLB

I thought I wouldn't need it, but it looks like I do, if I want sensible useful functionality. Here's an explanation of why I want to use Metal LB, and it's not just for BGP-based configs: https://github.com/kubernetes/ingress-nginx/blob/main/docs/deploy/baremetal.md

Install it:

  1. RTFM: https://metallb.universe.tf/installation/

  2. Grab the manifest and pull it into the repo, I'm using this one as it's similar to work: https://github.com/metallb/metallb/blob/v0.9/manifests/metallb.yaml

  3. Create the namespace first, I'm putting it into the System project: 

    apiVersion: v1
kind: Namespace
metadata:
  name: metallb-system
  # This is the System project on the prod cluster
  annotations:
    field.cattle.io/projectId: c-gfnh7:p-db8t4
  labels:
    app: metallb

  4. Create the metallb resources: kubectl apply -f 01-metallb.yaml

  5. Create the memberlist secret that the nodes need to communicate: kubectl -n metallb-system create secret generic memberlist --from-literal=secretkey="$$(openssl rand -base64 128)"

  6. Setup the configmap to configure its behaviour, they have a fully documented example here: https://github.com/metallb/metallb/blob/v0.9/manifests/example-config.yaml

  7. Apply the config: kubectl apply -f 02-config.yaml

Configure BGP

https://metallb.universe.tf/configuration/#bgp-configuration

Let's go for an iBGP design here - we both belong to the same private AS, number 64520

On helena: 

/routing/bgp/connection/add name=persica1 remote.address=192.168.3.3 as=64520 local.role=ibgp
/routing/bgp/connection/add name=persica2 remote.address=192.168.3.4 as=64520 local.role=ibgp
/routing/bgp/connection/add name=persica3 remote.address=192.168.3.5 as=64520 local.role=ibgp

And in metallb we drop this config in: 

data:
  config: |
    peers:
      - peer-address: 192.168.3.1
        peer-asn: 64520
        my-asn: 64520
    address-pools:
      - name: persica-lb
        protocol: bgp
        addresses:
          - 192.168.3.64/26
        avoid-buggy-ips: true
        auto-assign: false
        bgp-advertisements:
          - aggregation-length: 32
            localpref: 100
            communities:
              - no-export
    bgp-communities:
      # "Do not advertise this route to external BGP peers"
      no-export: 65535:65281
      # "Do not advertise this route to any peer"
      no-advertise: 65535:65282

The moment I apply this, helena sees a connection from the persica nodes, awesome.

When we just need to define a loadbalanced service in k8s, and they'll start advertising the address.

With a bit of faffing, it does just that. Had to force it to pick the IP I wanted, it uses .64 initially which I don't want. Our version doesn't respect the request by annotation, but spec.loadbalancerIP works (though it's deprecated).

Try MetalLB in Layer 2 mode first

NB: this is old

I'll use it in L2 mode with ARP/NDP I think. Just need to dedicate a bunch of IPs to it so it can manage the traffic to them.

Holy crap I think I got it working.

It's really as simple as adding type: LoadBalancer, then MetalLB selects the next free IP itself and binds it.

Try it in BGP mode next

TBC

Making ingress work - was this for the kubeadm method?

I don't understand this well enough, but I want to use ingress-nginx. Here's a page about it, albeit not using raw kubectl: https://kubernetes.github.io/ingress-nginx/kubectl-plugin/

Maybe this one too: https://medium.com/tektutor/using-nginx-ingress-controller-in-kubernetes-bare-metal-setup-890eb4e7772

Things that suck

cgroups

Alma9 introduces cgroups v2, which weren't a thing on Centos 7. That means you have to deal with them now. They tend to break docker a lot, so just revert back to v1 cgroups.

How it manifests:

Fix it:

Networking kernel modules

The problem: you fixed cgroups but now you get an error like this when Rancher starts up: 

I1125 03:57:50.129406      93 network_policy_controller.go:163] Starting network policy controller
F1125 03:57:50.130225      93 network_policy_controller.go:404] failed to run iptables command to create KUBE-ROUTER-FORWARD chain due to running [/usr/bin/iptables -t filter -S KUBE-ROUTER-FORWARD 1 --wait]: exit status 3: iptables v1.8.8 (legacy): can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
panic: F1125 03:57:50.130225      93 network_policy_controller.go:404] failed to run iptables command to create KUBE-ROUTER-FORWARD chain due to running [/usr/bin/iptables -t filter -S KUBE-ROUTER-FORWARD 1 --wait]: exit status 3: iptables v1.8.8 (legacy): can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.

And then it explodes and the container dies.

Turns out you need some iptables modules loaded. This fixed it the first time: 

[root@kalina ~]# modprobe iptable_nat
[root@kalina ~]# modprobe br_netfilter

But it happened again the next time I rebuilt the cluster. You gotta make it stick by adding config fragments to /etc/modules-load.d

Explanations:

Firewalls

Now whyTF can't persica2 and persica3 contact services on persica1..? Aha, firewalld is running on persica1, and it shouldn't be. Need to disable it using ansible as well. 

systemctl disable firewalld.service --now

Yeah that's jank, but hey it's what they tell you to do! https://ranchermanager.docs.rancher.com/how-to-guides/advanced-user-guides/open-ports-with-firewalld

"We recommend disabling firewalld. For Kubernetes 1.19.x and higher, firewalld must be turned off."

Cleanup and try again

Find that it doesn't work and you can't make it work, awesome. Tear it all down and start again, killing every container, nuking files, and starting from scratch: https://ranchermanager.docs.rancher.com/how-to-guides/new-user-guides/manage-clusters/clean-cluster-nodes#directories-and-files

Eventually, you get a cluster with three working nodes in it!!

Installing older versions of kubectl

Running an older version of k8s and need an older version of kubectl to go with it? You're shit out of luck, my friend!

https://kubernetes.io/blog/2023/08/15/pkgs-k8s-io-introduction/

They moved to new package repos in 2023, and as of early 2024 the old repos are gone! The new repos only have v1.24 and newer, so if you need anything older it's just not there.

Looks like our last option is: "You can directly download binaries instead of using packages. As an example, see Without a package manager instructions in "Installing kubeadm" document.

And you end up here: https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/#install-kubectl-binary-with-curl-on-linux

Here's a modern way of defining the repo on debian-type systems btw: 

# Our cluster is k8s v1.23 so we can use kubectl as late as 1.24

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.24/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

cat <<EOF > /etc/apt/sources.list.d/kubernetes.sources
X-Repolib-Name: Kubernetes
Enabled: yes
Types: deb
URIs: https://pkgs.k8s.io/core:/stable:/v1.24/deb/
Suites: /
Architectures: arm64
Signed-By: /etc/apt/keyrings/kubernetes-apt-keyring.gpg
EOF

apt update
apt install -y kubectl

You can't use selinux

It just breaks way too much shit, it's not worth it. Install something new and it doesn't work? You'll forever be wondering "is it selinux" immediately after it fails.

MeidokonWiki: servers/persica (last edited 2024-07-12 16:04:10 by furinkan)