Security After Remote Attestation
For what happens after boot
In the previous post I talked about how we can remotely attest and trust hosts. This led to a spirited discussion on hacker news.
There were some good points about vendor lock in and the (supposed) death of general purpose computing. There are always tradeoffs. But the truth is that for the most part you don’t want a 100% general purpose computer. You want a device that does exactly what you want it to do, and not much else.
I don’t want my router to DDOS the internet.
I don’t want my phone or other devices to spy on me (any more then they already do!)
I don’t want my laptop to encrypt all my files and extort me for the decryption key.
And at work, I want my servers to run production workloads, not malware.
So how do we make that happen? It takes a lot more then just remote attestation. That was just the first step.
I want the train to stay on the tracks.
If you haven’t read the previous post about remote attestation, a lot of this post will not make sense. So go read that first.
So now that the host was booted in a way we can trust, how do we maintain that trust? How do we prove at any given moment that a host should still be trusted?
We can’t.
Once a computer has booted it can do more or less anything, and the problem space rapidly expands to be unmanageable. How can I scan every byte of memory, every file, every packet, and prove that a host isn’t compromised? It is an intractable problem.
I can’t just grab a host of the rack and prove that it should be trusted. But I can start a host in a way that it is significantly harder for it to go off the rails later, and then prove at boot that the host has those guardrails.
Some example guardrails:
Only trusted code can be run in the kernel (signed kernel modules)
Only trusted code can have root, or maybe only trusted code can run at all (signed binaries)
System files can’t be overwritten (immutable filesystem)
The EDR (Endpoint Detect and Respond) is running
My LSM (Linux Security Module) rules of choice are engaged
It goes on…
If we can lock down enough of the different directions a system can go in, we make our attacker’s life that much harder. We have this one shot, at the beginning, where we have the crypto that backs up a claim about the state of the system. If that claim includes a bunch of security features, then we know that (at least at one point in time), those security features where on, and that an attacker’s life is that much harder.
Also all of our logs can be signed with the LDevID we minted as part of remote attestation, which makes them unspoofable, which is a neat property.
dm-verity
The best guardrail is to make things immutable. If state simply can’t be changed after boot then we have nothing to worry about. The train can’t leave the tracks.
dm-verity is kernel level block device verification, originally developed for android. It makes whatever filesystem it is applied to immutable and allows recording a hash of the filesystem contents so the kernel can detect tampering. If the filesystem is changed (even out of band, say by someone writing to the device directly) the kernel can detect that modification.
This verification is done efficiently using a Merkle tree. TLDR a Merkle tree speeds up hashing by constructing a tree of blocks that contain the raw bytes of the image. Instead of a naive hash which has time complexity N with respect to the number of blocks (it has to touch each one), hashing a Merkle tree has time complexity log N to validate the contents of a block. The root hash is what we save/sign, and what will be changed if any block in the tree is modified.
The way to think about it is like this. I can’t efficiently record the hash of every block. I want just a single hash that tells me “was the filesystem changed or not.” We want a single hash for the whole filesystem. And we can’t just hash the whole filesystem on each read for obvious reasons. Instead we store the root hash of the Merkle tree, and walk the tree from leaf to root every time a block is read, and check that the root hash computes correctly.
The reason this works is that each node’s hash is the hash of child nodes. Walking up the tree from one leaf to the root, only sibling nodes need to be explored. This sounds expensive, and is by some standards (performance numbers vary heavily by workload), but the kernel caches pages in the page cache, and the tree itself, keeping things fast in most cases.
The point of this exercise is that any mutation invalidates the root hash. We can keep that root hash somewhere we attested (like the kernel command line) or sign it, if the dm-verity partition is not included in the measured boot.
Upgrading Immutable Systems
dm-verity makes your filesystem completely immutable. If you are using dm-verity with your root filesystem, then you can’t update it without a reboot, which also involves rolling over your boot measurements. This is unfortunate, but we can at least mitigate the issue of always needing to roll over your boot measurements to upgrade parts of the system. In modern deployments the actual product/business logic is deployed in a container, which we will ignore for now, except to state that the host can measure this container and the container can measure the host, to ensure that both sides of the equation check out. Also if a host doesn’t have valid certificates from the CA (which it needs to attest to get) the container orchestrator should refuse to schedule jobs on the host.
But how do we upgrade everything on the system (everything outside the container)? The usual solution is to have multiple dm-verity devices. The root/boot device is minimal and contains things that are rarely upgraded anyway (like systemd and glibc). Then we install additional dm-verity images. The root hash of each image is signed, with a public key we trust. So we only attested the root image and the public keys we accept. All other components are signed not attested, making life easier.
In order to have a cohesive system, each image is unpacked, then using overlayfs we layer the different dm-verity images together to create a cohesive virtual filesystem (VFS). overlayfs takes a lower and upper directory and combines them to form a virtual directory where the contents of upperdir are overlayed on lowerdir. This allows us to have, say, a single /usr/local/bin, that combines binaries from many different signed, dm-verity images.
Systemd can manage this for us (because of course it can). systemd-sysext exists to manage binary dm-verity images and systemd-confext exists to manage system configuration images (/etc). The main difference is that confext leaves /etc with a writeable overlay on top since some applications expect /etc to be writeable, whereas you don’t expect binaries to be mutable.
You still need some other mutability, usually /var and /tmp is left writeable for applications that need logging, storing certificates, tmp files, and of course containers generally need storage as well. We encrypt /var with a TPM sealed key and /tmp is ephemeral, so if boot artifacts are changed a reboot bricks the system and access to any data is blocked.
We can mount things that are mutable as NOEXEC to prevent people from circumventing this system to some degree. Ideally, we want only binaries on signed images to be executed. But an attacker can still drop scripts and execute them through an approved interpreter (say shell scripts). There is work in the kernel for a cooperative system where scripting interpreters would treat scripts like binaries and use a special execveat flag to check if they can be executed. I wouldn’t rely on something like this since it is cooperative (though if you only install scripting languages that cooperate then it works). Instead, I prefer writing (arguably highly invasive) LSM rules which I’ll talk about more in later posts. SELinux in particular can be used to lock this kind of thing down effectively.
fs-verity
fs-verity gets an honorable mention here, since it will come up in later posts. It is the exact same idea as dm-verity but for specific files on an otherwise mutable filesystem. Not all filesystems support it, but many major ones do.
The nice thing about fs-verity is that it very quickly gives you a Merkle tree hash of a file that you can either store somewhere or sign. The built-in fs-verity signing mechanism is imperfect but works fine for many cases, it uses public keys stored in a specific keyring to validate a root hash. But when the official kernel docs caution you against using this signature method, you should take note. They recommend IMA or userspace validation, I personally like using eBPF LSM (yet another post there).
fs-verity is for filesystems where you can’t make the whole image immutable. You can still insist on key files being immutable and signed.
Signed Kernel Modules
A malicious kernel module breaks all security measures in a system instantly. Since the kernel provides many of these security primitives, a kernel module can trivially bypass such measures.
Many kernel modules are packaged in the initrd/initramfs of a system. These are attested as part of boot and we don’t have to worry about those. But often there are additional kernel modules that need to be loaded during runtime. They can be signed and you should enforce that only signed modules can be loaded.
LSM Rules
There are a lot of LSM things out there, and this post is starting to get long. Long story short, if you enable your LSM rules in some attested artifact, then you know they were engaged and those LSM rules can prevent themselves from being disabled. There are a couple of interesting LSMs that are commonly used:
Lockdown — protects the kernel from many tampering avenues: kernel modules, debugging, raw memory etc.
Yama — Enforces a more restrictive hierarchy on ptrace and friends, signals, etc. Normally all processes from the same user can ptrace each other leading to excessive lateral movement.
IMA — TPM PCR measurement and immutability of chosen files and directories. As well as an immutable log of what was executed.
AppArmor — Sandboxing/privilege reduction of services based on paths. Easy to use, but has some known methods of abuse. Heavily used in Ubuntu.
Landlock — Newer, filesystem sandboxing, more features are being added all the time.
SELinux — By far the most comprehensive LSM out there (developed by the NSA). Be warned, it is extremely complicated. On one project I was on, 2 people out of a team of 15 worked more or less full time on the SELinux policy. But it allows you to express almost anything you could want in LSM rules. More associated with Redhat/CentOS.
eBPF — You can write arbitrary eBPF LSM programs. This has been my primary occupation for the last few years, so there will be a lot more on this in later posts.
Regardless of what LSMs you are using, you are going to pick the ones that provide the properties you want and then attest that they were engaged. Once active, they provide additional guardrails to keep the system in line with what is expected.
You need to use both Mandatory Access Control (MAC) like an LSM and Discretionary Access Control (DAC) like Unix users/groups to get defense in depth on a system. I’m not touching DAC since that’s pretty well understood and a lot has been written about it. One thing that is rarely mentioned is that ephemeral users per-service or per-container can be far more effective then the usual shared user/group setup (and thus the need for an LSM for Yama).
Wrap Up
This is not at all a comprehensive post of immutable systems/post-attestation guardrails. But hopefully it gives an idea of what is possible. Properly constructed, you can drastically narrow the post-exploit movement of an attacker that has a foothold on a system.
I haven’t touched on networking restrictions, but those are possible as well. I also haven’t talked about EDRs (security monitoring and telemetry), most companies buy those though some open source ones like Falco are available.