Hardening Short.io’s bare metal nodes: from encrypted disks to verified boot

Bare metal gives us direct control over the hardware that runs Short.io. It also gives us responsibility for details that are easy to overlook when provisioning a cloud instance: how disks are encrypted, who can unlock them, and what happens when someone changes the software that runs before Linux starts.

We experimented with a bare metal provider to answer a practical question: could we operate these servers while maintaining the controls behind our SOC 2 and ISO 27001 commitments? Our process started with receiving instances with AlmaLinux preinstalled. We immediately wiped those installations and rebuilt the servers with encrypted storage, before allowing any customer data onto them.

The most interesting part turned out to be booting an encrypted server without having an engineer type a password after every restart.

Unencrypted disks were a compliance blocker

The preinstalled AlmaLinux instances came with unencrypted disks. Every one of those disks failed our encryption-at-rest requirements. Under Short.io’s security standards, not a single byte of customer data could reach those servers in that state—not for a test, not temporarily, and not while we finished configuring them.

Disk encryption was a hard prerequisite for bringing these nodes into our compliant environment. Without it, we could not meet the controls required by our SOC 2 and ISO 27001 compliance program. The servers had to remain outside the customer-data path until that prerequisite was satisfied.

A Linux login password does not protect an unencrypted disk from someone who can remove it or boot another operating system. Filesystem permissions are enforced by the running OS; a different OS can simply read the underlying storage.

We needed encryption for the operating system and application storage, including places where credentials, logs, temporary files, and database data might end up. Encrypting only the database volume would leave too many opportunities for sensitive data to land elsewhere.

Encryption was necessary, and the broader compliance work still included provider assessment, access management, monitoring, and evidence. ISO 27001 addresses the information security management system and risk treatment; SOC 2 examines controls within a defined scope. Our encryption requirement was a mandatory part of that wider program. ISO 27001 overview, AICPA’s SOC 2 guide.

For this experiment, the immediate engineering requirement was clear: data-bearing storage must be encrypted, and the provider must not need our disk passphrase to operate the hardware.

Reinstalling from the running OS

We kept AlmaLinux as the operating system, but replaced the provider’s installation with our own LUKS-encrypted setup. The preinstalled OS served only as the starting point for launching the installer; the reinstall wiped and replaced it. Encryption was in place from the beginning of the node’s service life.

One useful provisioning technique is to download the kernel and initial RAM filesystem normally used for PXE installation, then launch them from the existing Linux system with kexec. The installer runs from memory, allowing it to repartition the disk that previously held the OS. This uses PXE installer files without requiring a PXE server. Upstream kexec manual.

The following illustrates the process on a freshly delivered AlmaLinux x86_64 host with DHCP and a working remote console. It replaces the existing installation and interrupts access to the old OS. Our nodes had no customer workloads or customer data at this stage. If adapting this example to an existing server, drain its workloads and preserve needed data first.

# Run as root on the node being reprovisioned.
dnf install -y kexec-tools curl

install -d -m 0700 /root/alma-installer
cd /root/alma-installer

# Select a release and architecture supported by the target hardware.
INSTALL_TREE='https://repo.almalinux.org/almalinux/10.2/BaseOS/x86_64/os'

curl --fail --location --proto '=https' --proto-redir '=https' \
  "$INSTALL_TREE/images/pxeboot/vmlinuz" -o vmlinuz
curl --fail --location --proto '=https' --proto-redir '=https' \
  "$INSTALL_TREE/images/pxeboot/initrd.img" -o initrd.img

# Authenticate these artifacts before loading them; see below.
kexec --kexec-file-syscall --load ./vmlinuz \
  --initrd=./initrd.img \
  --command-line="ip=dhcp rd.neednet=1 inst.repo=$INSTALL_TREE console=tty0"

# Ends the current session and boots the installer.
systemctl kexec

This is a provisioning example, not a universal installer script. Static addressing, bonds, VLANs, and serial consoles require corresponding boot arguments. The installation tree must contain the matching installer runtime and package metadata. The installer will not inherit the old OS’s WireGuard session. Red Hat’s network installation documentation.

Authenticate the installation artifacts before execution—for example, compare them against files extracted from an AlmaLinux ISO whose checksum and signature you have verified. HTTPS alone does not establish an independently verified release. The file-based kexec interface allows kernel signature enforcement on supported locked-down systems, but that is not authentication of a separately downloaded initramfs. If the platform rejects the load, use a supported installation boot path rather than weakening the final boot policy.

In the installer, choose the intended disks explicitly and enable LUKS encryption for the data-bearing volumes. The EFI System Partition remains readable so firmware can start the machine. That distinction becomes important later: some boot files must be readable, but they must also be protected against unauthorized changes.

Encryption introduced a reboot problem

With passphrase-based LUKS unlocking, a restart stops at a password prompt. That is manageable for a single machine and inconvenient for a fleet expected to recover automatically after maintenance or power loss.

Clevis can automate unlocking using the server’s TPM. Its TPM2 integration can bind access to a secret to the values of Platform Configuration Registers, or PCRs. These registers record measurements associated with the boot process. Clevis TPM2 documentation.

An illustrative binding to PCR 7 looks like this:

# /dev/sda3 is an example: identify the actual LUKS device first.
clevis luks bind -d /dev/sda3 tpm2 \
  '{"pcr_bank":"sha256","pcr_ids":"7"}'

Root-volume unlocking also needs Clevis and TPM support in the initramfs, typically through the distribution’s clevis-dracut integration followed by rebuilding the initramfs. A successful bind alone does not prove that the next boot will unlock automatically. Red Hat’s automated unlocking guide.

PCR 7 reflects Secure Boot policy and the authorities involved in the boot chain. With Secure Boot properly enabled before enrollment, changing that policy—for example, disabling Secure Boot—changes the expected measurement and prevents automatic unlocking. PCR 7 does not identify every byte of the kernel, initramfs, or command line. systemd-cryptenroll’s PCR reference.

That made PCR 7 a useful step, but it left another question: could a machine still unlock after someone changed how it boots?

The gap: changing the kernel command line

PCR 7 alone does not protect the kernel command line. Someone who can edit the boot arguments can add init=/bin/bash. On a boot path that accepts this argument, Clevis can still unlock the disk automatically, after which Linux starts a root shell instead of the normal init process. The attacker gets access to the decrypted filesystem without entering the disk passphrase. The Secure Boot policy has not changed, so the PCR 7 check does not catch the edit. systemd’s PCR reference.

Four steps left to right: boot arguments edited to init=/bin/bash; Secure Boot policy unchanged, so PCR 7 still matches; the TPM releases the key and Clevis unlocks LUKS automatically; a root shell instead of init, on the decrypted filesystem.

The final design: signed Unified Kernel Images

A Unified Kernel Image, or UKI, packages the kernel, initramfs, and embedded kernel command line into one EFI executable. Signing that image allows Secure Boot to authenticate the bundle, including the early userspace that performs disk unlocking.

With systemd-stub, Secure Boot enabled, and an embedded command line, an external command line cannot simply replace the embedded one. That closes the familiar route of changing boot arguments to request a different startup program. systemd-stub documentation.

Our final design pairs the signed UKI with TPM unlocking tied to the approved boot image. In the systemd UKI flow, PCR 11 records measurements of the image’s contents and boot phases. PCR 7 and PCR 11 therefore serve different purposes: one anchors the Secure Boot policy; the other helps identify the approved software being started. systemd’s PCR reference.

One signed EFI executable containing the kernel, the initramfs and the embedded command line, with an external init=/bin/bash struck out and rejected. Alongside: PCR 7 measures the Secure Boot policy and the authorities in the chain, PCR 11 the contents of this image and the boot phases it passes through.

This requires more than changing a Clevis setting from 7 to 11. The unlock policy must match the measurements at the point where the disk is unlocked, and it must accommodate authorized updates. systemd supports signed PCR policies, allowing approved future images to unlock without manually pinning every node to one current measurement. systemd-cryptenroll documentation.

The signing keys belong outside the nodes they authorize. Recovery passphrases need separate storage. Superseded vulnerable images need an explicit retirement policy, because an old valid signature does not become invalid merely because a newer image exists. Any earlier, weaker automatic-unlock slot must also be removed after the replacement has been tested; adding a stronger path does not disable the old one.

This design addresses boot tampering under the assumption that the firmware, TPM, and signature verification remain trustworthy. It does not protect data from a compromised running kernel or eliminate the need to assess the provider’s physical and management-plane security.

Network encryption was much easier

Compared with deciding when a TPM should release a disk-unlocking secret, protecting traffic was straightforward.

We route inter-node and administrative traffic through WireGuard. Public application traffic reaches the nodes through Cloudflare Tunnels. Direct public access to application and management services is disabled; authorized administration stays inside the encrypted network.

WireGuard encrypts traffic between authenticated peers. Cloudflare Tunnel establishes encrypted outbound connections from cloudflared to Cloudflare, so an application origin does not need an exposed inbound service port. WireGuard documentation, Cloudflare Tunnel documentation.

The boundary matters: if a tunnel connector forwards a request to a service on another physical host, that hop must also use an encrypted path. We do not treat a provider’s private VLAN as a substitute for encryption. Likewise, disabling direct public service access still permits the transport needed for WireGuard and established tunnel connections.

The result is a node design with two explicit conditions: network access arrives through authenticated, encrypted paths, and automatic disk unlocking depends on an approved boot environment. Bare metal made us implement those conditions ourselves—and made it necessary to understand exactly what each one protects.