As a pentester, how often do you think to yourself "Why haven't they gotten back to me on my bug report?" Or how about "Why isn't this bug fixed yet?" I have frequently had these sorts of thoughts toward developers as a security researcher. For us, it's fairly common to find a vulnerability, report it, and throw it over a wall to the developers, never to see anything about the bug ever again.

Previously, I worked at Security Innovation doing application security assessments and did a decent amount of bug bounty hunting on the side. I’m currently at Asymmetric Research, where one of my roles is acting as a member of the internal security team for the cross-chain bridging product Wormhole. With billions of dollars in assets at stake, we perform security reviews on new products, design new security features, handle the bug bounty program, and many other security-related tasks. The thoughts that follow in this blog post are entirely my own and do not reflect the opinions of my current or previous employers.

I've been on both sides of the vulnerability wall: reporter and reportee. I've thrown things over and been hit by heavy reports over the years. From being on both sides, I finally understand Why You Can't Fix This Bug Faster.

Warning: What is below is somewhat of a letter to my former self, trying to remedy my past mistakes. Vulnerability reporting is difficult to get correct, especially when we normally only see one side of it. If at any point you feel offended, know that this was written with love to give a better picture of how the software development lifecycle works to improve as a security researcher. Enjoy!

Attitude

Before we go step by step into the full reporting to fixing process, we need to talk about the general attitude of security folks towards developers. Early in my career, I approached this relationship with a distinct us vs. them mindset. This perspective isn’t uncommon in the security world either. A recent blog post even stated, "Actually, it’s our opinion that they prioritize a lot over code security" when discussing vulnerabilities they had found in a product. Unfortunately, this kind of comment isn’t an isolated case—it reflects a prevalent attitude among many security researchers toward developers and their work.

Since working more closely with developers in my current role, I’ve reevaluated this mindset and deemed it unproductive and wrong. The biggest realization? I wouldn’t have wanted to work with the version of myself that held this attitude. Would you want to collaborate with someone who spoke so critically and negatively about your work? I know I wouldn’t. While honesty is important, how we communicate matters. These developers are smart people trying extraordinarily hard, but it's impossible to be perfect in security. There is a reason that cybersecurity has turned into its own niche: it's incredibly hard to write secure code! We should be proud of our expertise but not at the expense of others.

In reality, this negative attitude is the opposite of what security folks should be doing. We should be trying to partner with the developers to improve the product. This means it's a with and not an against mentality.

A positive attitude towards the product will go a long way when a bug is found. Even a single compliment about the code will help tremendously. What's the benefit? Developers will listen to you more. Collaboration will improve. The willingness to change their mind will go up. Morale towards security will even improve, leading to faster remediation and larger payouts. All sorts of things come from having a better view of the company and the product.

Risk assessment

Context Matters

Bug hunters commonly have a very narrow perspective of the product. So, when a vulnerability is found, the actual impact is commonly out of sync with the reported impact. This is because context matters.

I once found what I perceived to be a horrible access control vulnerability in an application - a horizontal privilege escalation to act as any user. From my context, it was an absolute game-over bug in the platform. From the company's context, it was not, though... The application was only available in a kiosk mode on company-owned computers. Although my vulnerability felt impactful, exploiting it was impractical because a user would need to break out of the kiosk to exploit the issue. It turns out that the kiosk mode was actually a core part of the product's security model. I later tested this kiosk mode on a separate project and found it solid.

The point is that you may be missing the forest for the trees. Try to reconcile the entire product's security model before jumping to severity conclusions.

Differing Threat Models

Security researchers tend to get excited about their findings, which is fine. It's hard to find bugs, so we should be excited when we find one. However, this often affects our opinion of how impactful a vulnerability is. We look for every small piece of its impact without considering what the owner cares about.

During some research I did years ago on the Zyxel NAS, I found CVE-2019-10633, a gnarly RCE bug. To me, this was the bug of all bugs and had to be fixed immediately. To them, the severity was not very high because it required authentication to trigger. In their threat model, if you had credentials to the NAS, you could view all files on the device anyway. This led to them not caring too much about the vulnerability.

Although this sucks as a researcher and I disagreed at the moment, I understand where they are coming from now. Their threat model involved unrestricted access to files, which I did not have. By understanding the owner's threat model, you can position yourself to clearly articulate the issue to make it impactful to the owners. You can even use their own threat model against them. For instance, on the Critical Thinking bug bounty podcast, they recommend discussing HIPAA, SOX, and other compliance mechanisms.

On the other hand, this trust in the owner's threat model can and will be abused at times. Some owners do not want any vulnerabilities in their product because it looks bad on the outside, or they are cheap and do not want to pay for the bug. Try to understand the owner's threat model. However, if it becomes ridiculous or clear that they are playing you, call out their ridiculousness and explain the perspective of the real impact. As a word of caution, be gentle with this because changing someone's threat model is not easy.

Overblown Reports

From the researcher's perspective, bug bounty aims to find bugs and make money. A pentester aims to discover many impactful vulnerabilities to encourage clients to sign more work. Both incentivize overrating vulnerabilities. This can make each security researcher into the boy or girl who cried "wolf!" If everything is a critical, then nothing is a critical. This leads developers to take the researcher's feedback less seriously. You have to understand the actual impact.

We have all seen the reports of missing security headers or 500 errors being submitted as high to programs unnecessarily. Another example with a little more nuance is a cross-site scripting (XSS) vulnerability. To the researcher, this is a great find. However, the bug exists on a random subdomain in this case and is only reflected in XSS. Many people would report this as a critical/high-severity vulnerability without considering its actual impact to try to get a bigger payout. Users aren't losing money, data, or anything else since it's on a random subdomain. Meanwhile, the developers do not see the XSS on a random subdomain as an immediate threat. So, we now have a discrepancy on the real impact, partially because of the conflicting interests of the bug reporter and developer, leading to frustration on both ends.

Time Allocations

Businesses focus their time and energy where the money is. And can you really blame them? Tech folks commonly change jobs to make more money. Security researchers hunt on bug bounty programs with higher payouts. It's okay for money to be an incentive. So, I will be blunt: fixing security bugs does not increase the bottom line for companies. New features and new clients bring in money. Management doesn't care about tech debt or an exploitable bug unless it affects the bottom line. Management cares about their big deal with the new company, bringing in a big paycheck.

We are starting to see the tide turn on this some. Although fixing these bugs does not directly bring in money, it helps keep money. Equifax had to pay $575M in damages and TMobile had to pay $350M in damages for their data breach. Still though, many companies are willing to take the risk of putting security second to pay for the slap on the wrist as a fine later. It’s like saving money by parking illegally; if you get fined less than you would have to pay, it's worth parking illegally.

We have established that fixing bugs is not the priority in many places. To the end user, this sucks if it affects you. Until end users care about security or fines become gigantic for companies, this will not change though. In practice, this means the developers are not given time, people-power, or budget to fix vulns; it's not where the money is. The Agile Sprints do not include a block for "fix bug from a project we built 3 years ago". Because fixing a bug takes time away from other priorities.

Root Causing

Seeing a vulnerability in action and trying to fix it are entirely different things. Fixing a vulnerability could be as simple as fixing an off-by-one error in an if statement, a configuration issue, or the need to add some code for input validation. Other times, the bug could be very subtle, in code not owned by the team or in very old code that the current developers don't understand. Root-causing a vulnerability can be difficult, depending on the bug.

One major example of the difficulty of root causing comes to mind: Github Session Handling Race Condition. Randomly, a user on Github.com was suddenly authenticated as another user. After days of effort, they found the root cause, which was extremely complicated. The HTTP server Unicorn does not create new and separate environment objects for each request. When paired with two authenticated users alongside an anonymous user making a request, session cookies from threads were being placed back into the requests incorrectly. This led to the anonymous user being given active session tokens for an authenticated user. This took a sequence of 8 events in a very specific order to trigger. How does one even begin the process of root causing this?

The developer's job isn't just fixing one instance of the reported bug either. There are likely other variants of the same bug on the platform. Especially on an open-source project, you want to kill all variants of this bug before releasing the fix for one. Otherwise, you may get pwned by a similar bug. Doing this requires much time and effort.

Fixing the Vulnerability

I once found a vulnerability that was a simple input validation issue. I thought I could fix it and get it merged in a day. It took me two weeks to get the fix merged into the repository. This time was spent between designing and implementing the fix, writing new tests and multiple rounds of code review. Remember, these things take time.

Context Strikes Again

Once the bug has been identified, it can be fixed. This is the hardest part of the process, where most things can go wrong. To the hacker, fixing should sometimes be a few lines of code, but much more needs to go into depending on the business use case of the issue.

Again, context is key. Let's consider an XSS vulnerability again. Sometimes, the fix is literally just not using dangerouslySetInnerHTML in React. Other times, the feature was designed to use this. So, something more complicated, like integrating client-side HTML filtering on the input, may need to be done. In the worst-case scenario, an entire redesign may be required, such as this Cosmos SDK RCE that I found. In these scenarios, the fixes may take weeks or months to come out with because of the amount of time development takes.

Test Test Test

The code needs to have unit tests for the functionality to evaluate whether the fix works as intended. In addition to unit tests, integration tests are necessary to ensure it works with the rest of the system. Many developers also write regression tests to validate that this vulnerability does not occur again.

Everything must work before releasing the code. These tests are necessary to confirm that the functionality works as expected. Writing tests can be incredibly time-consuming, but it is a required part of the release cycle. Otherwise, you may be working on a Saturday to patch a bug that was added to production. A quick fix can turn into a nightmare scenario very quickly.

Code Review

Most people do not like doing code reviews, but it's a required part of the software development process. In many companies and projects, all code is reviewed before production. Since these reviews don't happen instantaneously, there is a lag between code complete and launch. At this stage, changes could be required by the reviewers, creating even more lag.

Finally, we can deploy the code to the test environment then to production if everything works. We have crossed the finish line!

Takeaways

The next time you report a vulnerability, I hope you appreciate the entire process of fixing it. The development lifecycle has much more to deal with than just one bug report. Additionally, deploying a proper fix to the vulnerability simply takes time to get correct. From my perspective, here is what we can do better as security researchers:

• Understand context more. Take time to understand the product and the sensitive parts of it. In reality, you probably don't know as much as you think about it. Ask for their opinion on the risk and why they see it this way to build a better picture.
• Your threat model is not the developer's threat model. You're better off framing it from their threat model than trying to recreate it for them. If their perspective is ridiculous, try to call out specific wrong parts of it to bolster your argument, but tread lightly on this ground.
• Don't overhype your vulns. If everything is critical then nothing is.
• Be patient. Developers have a lot of things on their plate. Vulnerabilities can take a long time to fix properly. Give them the benefit of the doubt.

Conclusion

I hope you understand why fixes can take so long from a developer's perspective now. I have taken the developer's side to show you why these fixes take time. But it should be noted that sometimes the developers are wrong. Some vulnerabilities need to be fixed faster than 3 years, and some bugs are not evaluated for severity properly. But I want people to see the other side of the wall regarding vulnerability hunting. Thanks for listening to me ramble for a while, and I hope this was valuable to you!

Feel free to contact me (contact information is in the footer) if you have any questions or comments about this article or anything else. Cheers from Maxwell "ꓘ" Dulin.