• furinkan
• sysadmin
• Clevis_and_Tang

Clevis and Tang are two tools that work together to provide policy-based unlocking for encrypted disks. One such usage is Network-Bound Disk Encryption.

Here's a more human guide as well: https://www.reddit.com/r/CentOS/comments/8my171/followup_luks_network_bound_disk_encryption/

NBDE addresses a specific threat model and use-case. It roughly assumes:

• You need to use full-disk encryption
• This means a password is required for boot to proceed
• You can't or don't want to type a password at the local machine when the server needs to boot, such as after routine patching
• Your primary concern is someone physically stealing the server or stealing the disks out of the server
• This is not concerned with data security in-transit on the wire, or with unauthorised access to the data via apps/network paths, etc.

Tang

Tang is a lightweight daemon that provides asymmetric crypto over the network. Tang has one or more keypairs, and publishes the public key component. Anyone can encrypt data to one of the public keys, then later request for it to be decrypted. If the client can reach the Tang daemon over the network, it's considered to be duly authorised.

The data that we encrypt with Tang is the decryption key to unlock an encrypted volume.

Nice new tooling in RHEL8's version: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/configuring-automated-unlocking-of-encrypted-volumes-using-policy-based-decryption_security-hardening

Clevis

Clevis is the client-side counterpart to Tang, it handles the policy for Tang-assisted volume decryption.

Technically what Clevis is unlocking is a key to unlock the volume's master key. in LUKS terms, we're using one of the key slots to hold a Clevis-managed key. Tang can unlock that slot, letting us retrieve the master encryption key, and mount the volume.

Installation

This stuff is fairly new, and also has backing from Redhat, so it's not quite as mature on Debian-type systems.

You can get Tang from the usual Debian packages, but the dependency libjansson4 is too old unless you're on Debian 10 Buster or later. On Debian 9 Stretch you can use the unstable channel to work around it.

tang

This uses systemd for management and it's actually pretty elegant. It's essentially run like an inetd service, starting a new instance for each incoming connection.

• Install tang and its dependencies

• Override the listening port using systemctl edit tangd.socket if you want to, as it defaults to tcp/80

• Enable and start tangd.socket

clevis

Install the base package, along with clevis-luks. We would use the clevis-dracut package, but Debian doesn't use Dracut for building its initramfs, so that won't help us. For the record we'll do something similar to it though.

Messing around

These are unstructured notes.

# test encryption, works good:
echo hi | clevis encrypt tang '{"url": "http://tang.thighhighs.top:8264"}' > hi.jwe

# try binding tang to the luks partition
apt install clevis-luks
clevis luks bind -d /dev/sda3 tang '{"url": "http://tang.thighhighs.top:8264"}'

root@illustrious:~# clevis luks bind -d /dev/sda3 tang '{"url": "http://tang.thighhighs.top:8264"}'
The advertisement contains the following signing keys:

1XVz7r6j7V5DwogkcAQdk927nig

Do you wish to trust these keys? [ynYN] y
You are about to initialize a LUKS device for metadata storage.
Attempting to initialize it may result in data loss if data was
already written into the LUKS header gap in a different format.
A backup is advised before initialization is performed.

Do you wish to initialize /dev/sda3? [yn] y
Enter existing LUKS password: <the password>

root@illustrious:~# luksmeta show -d /dev/sda3
0   active empty
1   active cb6e8904-81ff-40da-a84a-07ab9ab5715e
2 inactive empty
3 inactive empty
4 inactive empty
5 inactive empty
6 inactive empty
7 inactive empty

man clevis-luks-unlockers


apt install clevis-dracut
update-initramfs -u


# Steal the scripts from here and drop them in /etc/initramfs tools, one hook and two scripts.
https://github.com/latchset/clevis/pull/101/files

# Modify the scripts to have a wait before trying to hit the Tang server

In the end I had to add a sleep 5 to the local-top/clevis script, but it does work nicely. The machine boots, gets online with DHCP, then clevis kicks in and fetches the decryption key for slot 1.

Trying this in 2020 on a modern system

I'm using pop_OS! (Ubuntu 20.04 under the hood) to try this, it's a fairly standard out of the box install, which will keep things sane.

It looks like there's now packages for Debian-type systems, and there's initramfs integration too! And it works with LUKS2 metadata which is fantastic (but the docs don't know it yet).

1. apt install clevis-luks

2. clevis luks bind -d /dev/nvme0n1p3 tang '{"url": "http://tang.thighhighs.top:8264"}'

The tang binding now gets stored as a Token, which is pinned to a Keyslot.

Then I followed this person's guide to get Wifi working in the initramfs environment: http://www.marcfargas.com/posts/enable-wireless-debian-initramfs/

However I can't check that it's working because my workstation doesn't like displaying video during boot time, which is... annoying.

I think I then need to add the unlock scripts manually from /usr/share/initramfs-tools or so.

Ubuntu in 2021

Packages for LUKS and initramfs! It only mentions LUKS v1 but I'm pretty sure it'll work on v2 as well. I've rebuilt illustrious so let's see if we can make it work again.

1. apt install clevis-luks

2. clevis luks bind -d /dev/nvme0n1p3 tang '{"url": "http://tang.thighhighs.top:8264"}'

3. clevis luks bind -d /dev/nvme0n1p3 tpm2 '{"pcr_ids":"7"}'

Seems to work great! I don't think my TPM is secure, but it does mean the encryption is still active, even if the unlocking isn't secure right now (until I get Tang working again).

On a DAP-1330 running Openwrt

I thought I might buy the DAP-1330 just to use its PSU and casing for mains power, but it seems the power supply puts out 3.3V and not 5V that I was hoping to serve as USB.

But even better, it turns out that openwrt has a package for tang, awesome!

This is for Tang version 6-1 or so. https://github.com/openwrt/packages/issues?q=is%3Aissue+tang

 * Setup networking, disable router actions, make it a client
 * Install tang package
  * tang is on port 8888 now by default
 * Generate keys
    /usr/libexec/tangd-keygen /usr/share/tang/db
    /usr/libexec/tangd-update /usr/share/tang/db /usr/share/tang/cache
 * Reboot and the service should be running on its own

Now you can try an encrypt, this will get the advert and do a transaction: echo "hello" | clevis encrypt tang '{"url":"http://tang.local:8888"}'

Go ahead and enrol a LUKS volume as normal.

The key generation stuff will change in future, but not yet: https://github.com/latchset/tang/pull/53

Take note of /lib/upgrade/keep.d/tang, looks like it'll keep the DB in future across upgrades, but will need the tangd-update run again to finish generation (unless v10 arrives sooner).

Tang server on plugin appliances

Tang will run on any linux box, but even better is something embedded that just runs and needs pretty much no maintenance. I have discovered that OpenWRT is pretty much ideal for this, as tang is already packaged for it.

Get the cheapest acceptable device you can find, because it doesn't need to be good or fast, it just needs to run. It'll probably be a wifi range extender, but you might find an access point even cheaper.

The TP-Link TL-WA850RE almost looks perfect for the job, except that it's a 4/32MB device, which means the flash ROM is too small to hold a current version of OpenWRT. From the forums it sounds like it's possible to run v18.06, but it may require a custom image build, which is a pain in the arse. But it's about half the price of a decent option (35-40 AUD vs 80-90 AUD), so it's kinda compelling.

There are a few options that are definitely good and supported though, by the current 19.07 version:

These are all plugin boxes with a pair of rabbit ears on them, and will look pretty innocuous. They can run version 19.07.7 of openwrt, which is good enough.

Openwrt config

Setup the device out of the box using eth0 (wired). Set a fixed static address in your regular LAN range, and don't start it on boot (we don't want it to be the default gateway). You'll only use this interface if something is broken.

Use wlan1 for the dedicated tang subnet (described below), and let it use DHCP to get its address from wifi. If the wifi is down due to someone attacking the network, your encrypted hosts can't be unlocked. Not being attached by an ethernet cable means you can stuff the device anywhere in radio range, nothing to attract attention. Being physically hard to find makes it harder to steal as well, if someone wants the keys. The device won't be invisible to someone sniffing around the wifi traffic, but it's a good start.

Remove all the firewall zones, you won't be using them.

Don't bridge the wifi and wired, that'll cause problems.

Add your SSH keys for root access.

Disable DHCP and dnsmasq on the device, this is strictly a client device.

Openwrt config dump

This assumes you're now wired directly to the plug, hitting 192.168.1.1 with root // <empty>

Set new root password

Networking config page
        empty ULA prefix IPv6
        disable bridging, select eth0
        protocol static
        bring up on boot
        set address on regular LAN
        IP settings as normal
        DNS = 8.8.8.8
        IPv6 length /64
        IPv6 hint = 0
        IPv6 suffix = eui64

        builtin v6 management
        don't force link

        fw zone unspecified

        DHCP server general: tick ignore interface
        DHCP server v6 settings: all disabled/blank

Save changes and commit them, get kicked out and go use the new IP address you just assigned.

Fix the gateway address if it didn't stick earlier.

Setup WLAN interface now.

Network -> Wireless
        Remove existing networks, that'll be the plug acting as an AP which we don't want
        Select the 2.4G radio and Scan on it, join the network

        Do replace wireless config
        New name is DungeonOfThings
        Don't lock BSSID
        Assign to unspecified fw zone

        Advanced settings
                Country AU
                Disallow legacy rates

Save and Apply

Set a static DHCP address in Unifi controller or the Pihole server.

Restart the WLAN interface to get the new fixed IP.

System config menu
        set hostname and timezone
        sync with ntp
        enable ntp client, use dhcp advertised servers
        ntp candidates: [0123].openwrt.pool.ntp.org

Add custom FW rules to allow only your admin machines to access it.

Install packages: tang

# This command sets bring-up-at-boot?
uci del network.lan.auto

Device hardening

Once you have things installed, it's best to lock them down as much as possible. This is applicable for regular servers as well, but for an appliance we have the freedom to lock it down much harder, because usability is less of a concern than security.

• Set a strong, unique root password
• Put them in a separate subnet in their own VLAN. This means you can do firewalling on the router to limit access, and possibly detect suspicious activity.
• This makes them less discoverable by casual scanning tools, which denies information to potential attackers
• Use openwrt's firewall for further protection, allowing only known workstations to reach the device
  • Set iptables policies like so. Usually you drop unwanted traffic, but I wanted to be able to confirm the devices are up while denying traffic
    • INPUT reject
    • OUTPUT accept
    • FORWARD reject

  • Use the custom ruleset to make allowances

    # Add a copy of these rules for each trusted workstation.
    # 22 for SSH, 80 for the LuCI web interface
    # 8888 for the tang daemon
    iptables -A input_rule -s 192.168.1.1234 -p tcp --dport 22 -j ACCEPT
    iptables -A input_rule -s 192.168.1.1234 -p tcp --dport 80 -j ACCEPT
    iptables -A input_rule -s 192.168.1.1234 -p tcp --dport 8888 -j ACCEPT

• Disable password SSH auth for root if you feel confident enough. Or disable passwords entirely.

Redundant tang servers

Once you have a couple running, you can allow clients to unlock their slot using any of the available servers.

As described here: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/configuring-automated-unlocking-of-encrypted-volumes-using-policy-based-decryption_security-hardening#high_available_nbde_using_shamir_s_secret_sharing

clevis luks bind -d /dev/nvme0n1p3 sss '{"t":1,"pins":{"tang":[{"url":"http://ocular.thighhighs.top:8888"},{"url":"http://funicular.thighhighs.top:8888"}]}}'

You'll get asked to accept each key in turn, then the bind will be completed once you enter an existing-good LUKS key.

Key rotation

Roughly as described here, except that our tool versions are a bit older, so there's more legwork: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/configuring-automated-unlocking-of-encrypted-volumes-using-policy-based-decryption_security-hardening#rotating-tang-keys_configuring-automated-unlocking-of-encrypted-volumes-using-policy-based-decryption

1. Generate new keys in the Tang DB dir using tangd-keygen

2. Update the cache, unless you have a new-enough version that does it for you
3. Check your key advertisements, you should see two of them now

4. Rename the old keys to have a leading . then re-update the cache

5. Check the advertisements and see that the old one has disappeared
6. Satisfy yourself that the old key is still usable, by inspecting the Tang cache dir

7. On the clients, use clevis-luks-report if you've got it. Or you can just unbind the volume and then rebind it to the same key spec again

8. Delete the old keys from the tang server and update the cache dir again if you want