There were some good points about vendor lock in and the (supposed) death of general purpose computing. There are always tradeoffs. But the truth is that for the most part you don’t want a 100% general purpose computer. You want a device that does exactly what you want it to do, and not much else.

I don’t want my router to DDOS the internet.

I don’t want my phone or other devices to spy on me (any more then they already do!)

I don’t want my laptop to encrypt all my files and extort me for the decryption key.

And at work, I want my servers to run production workloads, not malware.

So how do we make that happen? It takes a lot more then just remote attestation. That was just the first step.

I want the train to stay on the tracks.

If you haven’t read the previous post about remote attestation, a lot of this post will not make sense. So go read that first.

So now that the host was booted in a way we can trust, how do we maintain that trust? How do we prove at any given moment that a host should still be trusted?

We can’t.

Once a computer has booted it can do more or less anything, and the problem space rapidly expands to be unmanageable. How can I scan every byte of memory, every file, every packet, and prove that a host isn’t compromised? It is an intractable problem.

I can’t just grab a host of the rack and prove that it should be trusted. But I can start a host in a way that it is significantly harder for it to go off the rails later, and then prove at boot that the host has those guardrails.

Some example guardrails:

• Only trusted code can be run in the kernel (signed kernel modules)

• Only trusted code can have root, or maybe only trusted code can run at all (signed binaries)

• System files can’t be overwritten (immutable filesystem)

• The EDR (Endpoint Detect and Respond) is running

• My LSM (Linux Security Module) rules of choice are engaged

• It goes on…

If we can lock down enough of the different directions a system can go in, we make our attacker’s life that much harder. We have this one shot, at the beginning, where we have the crypto that backs up a claim about the state of the system. If that claim includes a bunch of security features, then we know that (at least at one point in time), those security features where on, and that an attacker’s life is that much harder.

Also all of our logs can be signed with the LDevID we minted as part of remote attestation, which makes them unspoofable, which is a neat property.

dm-verity

The best guardrail is to make things immutable. If state simply can’t be changed after boot then we have nothing to worry about. The train can’t leave the tracks.

dm-verity is kernel level block device verification, originally developed for android. It makes whatever filesystem it is applied to immutable and allows recording a hash of the filesystem contents so the kernel can detect tampering. If the filesystem is changed (even out of band, say by someone writing to the device directly) the kernel can detect that modification.

This verification is done efficiently using a Merkle tree. TLDR a Merkle tree speeds up hashing by constructing a tree of blocks that contain the raw bytes of the image. Instead of a naive hash which has time complexity N with respect to the number of blocks (it has to touch each one), hashing a Merkle tree has time complexity log N to validate the contents of a block. The root hash is what we save/sign, and what will be changed if any block in the tree is modified.

The way to think about it is like this. I can’t efficiently record the hash of every block. I want just a single hash that tells me “was the filesystem changed or not.” We want a single hash for the whole filesystem. And we can’t just hash the whole filesystem on each read for obvious reasons. Instead we store the root hash of the Merkle tree, and walk the tree from leaf to root every time a block is read, and check that the root hash computes correctly.

The reason this works is that each node’s hash is the hash of child nodes. Walking up the tree from one leaf to the root, only sibling nodes need to be explored. This sounds expensive, and is by some standards (performance numbers vary heavily by workload), but the kernel caches pages in the page cache, and the tree itself, keeping things fast in most cases.

The point of this exercise is that any mutation invalidates the root hash. We can keep that root hash somewhere we attested (like the kernel command line) or sign it, if the dm-verity partition is not included in the measured boot.

Upgrading Immutable Systems

dm-verity makes your filesystem completely immutable. If you are using dm-verity with your root filesystem, then you can’t update it without a reboot, which also involves rolling over your boot measurements. This is unfortunate, but we can at least mitigate the issue of always needing to roll over your boot measurements to upgrade parts of the system. In modern deployments the actual product/business logic is deployed in a container, which we will ignore for now, except to state that the host can measure this container and the container can measure the host, to ensure that both sides of the equation check out. Also if a host doesn’t have valid certificates from the CA (which it needs to attest to get) the container orchestrator should refuse to schedule jobs on the host.

But how do we upgrade everything on the system (everything outside the container)? The usual solution is to have multiple dm-verity devices. The root/boot device is minimal and contains things that are rarely upgraded anyway (like systemd and glibc). Then we install additional dm-verity images. The root hash of each image is signed, with a public key we trust. So we only attested the root image and the public keys we accept. All other components are signed not attested, making life easier.

In order to have a cohesive system, each image is unpacked, then using overlayfs we layer the different dm-verity images together to create a cohesive virtual filesystem (VFS). overlayfs takes a lower and upper directory and combines them to form a virtual directory where the contents of upperdir are overlayed on lowerdir. This allows us to have, say, a single /usr/local/bin, that combines binaries from many different signed, dm-verity images.

Systemd can manage this for us (because of course it can). systemd-sysext exists to manage binary dm-verity images and systemd-confext exists to manage system configuration images (/etc). The main difference is that confext leaves /etc with a writeable overlay on top since some applications expect /etc to be writeable, whereas you don’t expect binaries to be mutable.

You still need some other mutability, usually /var and /tmp is left writeable for applications that need logging, storing certificates, tmp files, and of course containers generally need storage as well. We encrypt /var with a TPM sealed key and /tmp is ephemeral, so if boot artifacts are changed a reboot bricks the system and access to any data is blocked.

We can mount things that are mutable as NOEXEC to prevent people from circumventing this system to some degree. Ideally, we want only binaries on signed images to be executed. But an attacker can still drop scripts and execute them through an approved interpreter (say shell scripts). There is work in the kernel for a cooperative system where scripting interpreters would treat scripts like binaries and use a special execveat flag to check if they can be executed. I wouldn’t rely on something like this since it is cooperative (though if you only install scripting languages that cooperate then it works). Instead, I prefer writing (arguably highly invasive) LSM rules which I’ll talk about more in later posts. SELinux in particular can be used to lock this kind of thing down effectively.

fs-verity

fs-verity gets an honorable mention here, since it will come up in later posts. It is the exact same idea as dm-verity but for specific files on an otherwise mutable filesystem. Not all filesystems support it, but many major ones do.

The nice thing about fs-verity is that it very quickly gives you a Merkle tree hash of a file that you can either store somewhere or sign. The built-in fs-verity signing mechanism is imperfect but works fine for many cases, it uses public keys stored in a specific keyring to validate a root hash. But when the official kernel docs caution you against using this signature method, you should take note. They recommend IMA or userspace validation, I personally like using eBPF LSM (yet another post there).

fs-verity is for filesystems where you can’t make the whole image immutable. You can still insist on key files being immutable and signed.

Signed Kernel Modules

A malicious kernel module breaks all security measures in a system instantly. Since the kernel provides many of these security primitives, a kernel module can trivially bypass such measures.

Many kernel modules are packaged in the initrd/initramfs of a system. These are attested as part of boot and we don’t have to worry about those. But often there are additional kernel modules that need to be loaded during runtime. They can be signed and you should enforce that only signed modules can be loaded.

LSM Rules

There are a lot of LSM things out there, and this post is starting to get long. Long story short, if you enable your LSM rules in some attested artifact, then you know they were engaged and those LSM rules can prevent themselves from being disabled. There are a couple of interesting LSMs that are commonly used:

• Lockdown — protects the kernel from many tampering avenues: kernel modules, debugging, raw memory etc.

• Yama — Enforces a more restrictive hierarchy on ptrace and friends, signals, etc. Normally all processes from the same user can ptrace each other leading to excessive lateral movement.

• IMA — TPM PCR measurement and immutability of chosen files and directories. As well as an immutable log of what was executed.

• AppArmor — Sandboxing/privilege reduction of services based on paths. Easy to use, but has some known methods of abuse. Heavily used in Ubuntu.

• Landlock — Newer, filesystem sandboxing, more features are being added all the time.

• SELinux — By far the most comprehensive LSM out there (developed by the NSA). Be warned, it is extremely complicated. On one project I was on, 2 people out of a team of 15 worked more or less full time on the SELinux policy. But it allows you to express almost anything you could want in LSM rules. More associated with Redhat/CentOS.

• eBPF — You can write arbitrary eBPF LSM programs. This has been my primary occupation for the last few years, so there will be a lot more on this in later posts.

Regardless of what LSMs you are using, you are going to pick the ones that provide the properties you want and then attest that they were engaged. Once active, they provide additional guardrails to keep the system in line with what is expected.

You need to use both Mandatory Access Control (MAC) like an LSM and Discretionary Access Control (DAC) like Unix users/groups to get defense in depth on a system. I’m not touching DAC since that’s pretty well understood and a lot has been written about it. One thing that is rarely mentioned is that ephemeral users per-service or per-container can be far more effective then the usual shared user/group setup (and thus the need for an LSM for Yama).

Wrap Up

This is not at all a comprehensive post of immutable systems/post-attestation guardrails. But hopefully it gives an idea of what is possible. Properly constructed, you can drastically narrow the post-exploit movement of an attacker that has a foothold on a system.

I haven’t touched on networking restrictions, but those are possible as well. I also haven’t talked about EDRs (security monitoring and telemetry), most companies buy those though some open source ones like Falco are available.

I’m surprised ChatGPT agreed to make that one.

But seriously. How do you know that your data isn’t sitting on a compromised stack? How do you know that you aren’t trying to schedule workloads on an attacker controlled machine? How do you know any of your security controls are actually in place on a host? And once an attacker has taken over a host, there are endless places to hide and persist through upgrades and reboots.

Most setups have a lot of faith in hosts once they have been provisioned. Hosts are placed in the trust boundary and then just remain there, regardless of whatever has happened in the mean time. If they even should have been there in the first place.

Enter the wonderful world of remote attestation which (mostly) solves this problem. Using a TPM, we can remotely, cryptographically prove a couple of things:

• The host has the expected hardware

• The host has the expected firmware

• The host has the expected kernel and init image

• The host has the expected root filesystem (depends on the setup)

• The host passed arbitrary checks that we define

This means that after a reboot, we know the exact state a host is in. Without extensive physical modifications and tampering, a fresh boot either puts us in a trusted state or fails and breaks trust in the host.

The measure boot cycle is different then trusted boot, it relies on signed measurements, not signed artifacts like secure boot. It offers a much bigger coverage surface at the cost of considerable complexity. Even malicious, signed, drivers and rootkits can be caught.

But what’s the point of measuring if we don’t block boot like secure boot does? What does measurement get us?

This gets into how you trust hosts. You can use your TPM to encrypt all or part of your root-fs (where all your keys and data are), so that a host is functionally inoperable without the measured boot succeeding. Your RA can refuse to issue certificates to a host that doesn’t provide correct measurements (you are using mTLS right?). You can even back TLS x509 certs with the TPM so an incorrectly rebooted host can’t authenticate over mTLS at all. This offers extremely strong guarantees about the initial state of hosts, so you can trust that your workloads and data are in an environment you trust.

This isn’t a perfect solution. Physical attacks (like memory taps) are still a problem. But coupled with robust EDR and a good LSM policy, you have no closed the door on a huge number of issues, in particular nasty kernel and driver supply chain attacks that bypass all of your nice userland defenses.

But what about attacks after boot?

That’s your EDR’s problem. Trusted boot provides the bedrock to build a bunch of other primitives on top of. Including cryptographic proof your EDR is installed and running (at boot), immutable filesystems (verified at boot), signed upgrades, confidential computing, etc. Without it you can’t trust your hosts themselves and can’t make further security guarantees. Houses built on sand and all that.

Just use remote attestation, it is easy

Unfortunately, remotely measuring every step of your boot process, for every host in your organization, is a tremendous undertaking. It goes into how you build and distribute firmware, init images, kernels. On the plus side, it will give you a chance to clean up a bunch of legacy cruft and also probably reveal a bunch of supply chain issues you weren’t thinking about.

How it works

TPM magic more or less. TPMs expose a limited interface of cryptographic operations to the host, and that interface is locked down sufficiently that it controls your access to cryptographic material. They are also resistant to physical attacks, but the main function is to mitigate software attacks, not physical. Even kernel code can’t dig into a TPM and extract private material.

The TPM comes with a set of PCRs (Platform Configuration Registers). Their only function is to store hashes. They only go forward, you can measure new hashes into a PCR and it hashes the previous value with the new one, yielding a unique result. No going backwards. Different artifacts get measured into the different PCRs during boot and init, and similar to secure boot, each phase measures the next phase.

Breaking this chain anywhere will taint the PCR values.

The TPM cares about these values. You can do a couple thing based on them:

• TPM sealing: material can be stored in the TPM and only unlocked if the PCRs (the ones you chose) check out.

• TPM quotes: the values can be signed with a key resident to the TPM. Your RA knows the providence of that key (more in a moment) and this offers strong proof that the measurements are correct.

The full ritual is outside the scope of a blog post like this (ask Claude), but it goes something like this:

When a host is first be provisioned the TPM has an Endorsement Key (EK) which is a decryption key burned in by the manufacturer. The EK comes with a x509 cert signed by the manufacture’s PKI. So the EK proves the TPM is legit.

You then mint an Attestation Key (AK), derived from the EK. The AK is a restricted signing key, it can only sign TPM quotes, nothing else (what would the point of this be if you could make your own quote and sign it) and you generally persist the AK for subsequent reboots.

But wait, how do we know that an AK is tied to an EK? The EK is a decryption key, it doesn’t sign anything. The attester just gets two public keys, the EK and AK public keys and the EK x509 leaf cert. The full answer involves more crypto math then I am comfortable trying to explain. TLDR the verifier issues a challenge: encrypting a credential with the EK public key and sending it back. The device decrypts the credential with the EK private key (which never leaves the TPM), and sends the credential to the attester, proving that it is talking to a device that has control of the EK private key.

The AK can only sign quotes, and the attester has a set of permissible PCR values for each register. So an AK proves the host booted the way we expect, the “measured boot” side of the story. But need a third key, the LDevID (Locally-scoped Device ID), which represents a node identity.

The LDevID is a child of the AK. Importantly, the LDevID can sign anything, not just quotes. Anything signed by the LDevID is proven to have been signed with this specific TPM. Importantly, since it can sign anything we need to restrict when signing can take place — otherwise an attacker can gain control of it. Frequently it is sealed to golden PCR values, so a reboot into a bad state cannot use it.

Once the whole ritual is done the host can actually prove itself to be legitimate. This does not ensure runtime security, that part is on you. This is only about boot. If this is all set up correctly, attacks will not survive a reboot. Or at least can only brick the host.

TPM sealing runs into problems with upgrades. New firmware has different measurements and will break attestation. The solution is to authorize a policy (TPM2_PolicyAuthorize), signed by an AuthKey, that authorizes the TPM to accept new PCR values. The AuthKey can be held by the node itself, in which case it needs to vet updates and determine whether or not to accept them, or an authority hold it and distributes the public key to the device/TPM. It just depends if you want to validate upgrades on device or elsewhere in your infra.

What does this let you do?

If your infra consistently enforces mTLS, you can reject hosts that were not attested, since the RA won’t give them certs. If you use TPM backed certs, tampered hosts naturally lose access to the rest of production.

Every host has providence. Data signed by the LDevID (whether TLS or something else), you can prove is from a specific host.

Workloads can refuse to run on hosts that were not attested. Your scheduler can demand cryptographic proof before allowing jobs to run on a host. Your jobs themselves can challenge a host before pulling in data.

Like I said above, if you are confident in your boot security you can build your runtime security appropriately. In the next post I’ll get into dm-verity and immutable root images with systemd.