Resources

People often ask me "How did you learn how to hack?" The answer: by reading. This page is a collection of the blog posts and other articles that I have accumulated over the years of my journey. Enjoy!

Cloudflare Containers run workloads in multi-tenant infrastructure. The researchers of this post identified a way to break the isolation between customers.

Cloudflare uses a thin Linux device mapper provisioning system to provide each container with a writable root risk. This container lives in a dedicated virtual machine via Firecracker, with the disk at /dev/vdc. The provisioning layer allocates physical storage when a virtual disk writes to an unmapped region. When the volume was deleted, the physical blocks were returned to the pool to be used again later.

Here's the bug: the physical devices were only cleared if the setting skip_block_zeroing was enabled. In other cases, the volumes could contain uninitialized memory from previous users. This allowed information to leak from other users.

It's crazy to me this was found in a black box review... why would you think that turning off a default setting would leak information from another user? Crazy! From the data, they saw database pages, directory structures, and other sensitive information. They only reviewed a small amount of data to prove the impact before submitting the bug.

An attacker couldn't select a particular user, couldn't modify data, and didn't appear to be exploited. To remediate the bug, they just zeroed all new allocations. Because OCI containers contain caching, they had to retire all running container disks, and rebuild all OCI images for their users. This was down in off-peak hours to reduce friction.

As part of the response, they reviewed the historical disk I/O telemetry for potential exploitation. With this, they had a signature of reusd KiB storage blocks to read the data with zeroing disabled. They identified zero real-world exploits.

Overall, a great blog post, and a fun bug! I really enjoy the dissection of the mitigation and the checking of exploitation. Mitigations are never as simple as they seem, as demonstrated here with the OCI container images that would have had the cached file data.