Resources

People often ask me "How did you learn how to hack?" The answer: by reading. This page is a collection of the blog posts and other articles that I have accumulated over the years of my journey. Enjoy!

The author has been around the vulnerability research scene since the beginning. In the beginning, it was fun and motivating; they trafficked hidden knowledge, trying to understand obscure things. Within a decade, the mood shifted. Exploitation techniques like C++ vtables were common discussion points. The in-memory layouts of fonts became widely discussed because they are exploited in memory corruption attacks.

They make two claims. First, vulnerabilities hide in non-obvious areas that don't have anything to do with security. The entry point for hackers is where they have control of the data and where the code is bad. Second, the cost of being an elite security researcher is high, making it very difficult for many people to be good at it. It requires attention from elite people

Because of these two claims, they say that LLMs are perfect for exploitation research. Vulnerabilities are found by pattern-matching bug classes, which the LLM was a complete library of. It has great constraint-solving capabilities, like reachability and exploitability concerns. LLMs can write code and test exploitation as well, creating a great feedback loop.

Nick Carlini works on Anthropic's Frontier Red Team. They claimed to have generated 500 high-severity vulnerabilities with a super-simple set of operations. For each source file in the repo, Claude was asked to find an exploitable vulnerability in that project. With a lot of vulnerabilities, they would give Claude the reports and tell Claude to triage them. Accordingly, it had almost a 100% success rate. This process is simple, very focused on a specific file, and gets deep coverage.

The simple approach is weird to me. I would think that human intuition and guidance would be good for the LLM. The reality is much different: just let the LLM cook. Richard Sutton's bitter lesson realized that the important thing is data and computation. Everything else doesn't matter.

So, what's the future hold then? The claim is that vulnerabilities will come in large waves. The cost is apparently going down to do this. The author is worried that legislators will try to fix the issue by preventing LLMs from conducting security research.

They finally claim that there's room for human vulnerability research but only at the highest end. Not everything is documented, and humans come up with new tricks all the time. LLMs are great at pattern-matching and pulling concepts together across the realm, but NOT creating new ones. Still, this is a small subset of people. The post is very dystopian but potentially true. I think LLMs tend to find the same bugs, but I'm doubtful it's finding ALL of the bugs.