NaN Of Your Business - My Favorite Unintended CTF Solution

1/21/2025

Most CTF challenges are created to have a logical set of steps to solve by its creators. Typically, this means there is a single solution to the challenge in mind. With hundreds of creative people looking at the same challenge, unexpected paths to the end goal are bound to happen. CTF goers love a good unintended solution! In a way, you hacked the hacker.

I host a CTF oriented at college and high school students called the Spokane Cyber Cup (SCC). Since I've been doing application security professionally for about 5 years, I would expect these students never to find an unintended solution. In reality, there are always one or two interesting workarounds for the challenges we prepare. This post concerns one of my favorite challenges and hilarious tale around the unintended solution for it- using NaN to break the bank. I hope you enjoy!

Floating Bank Challenge

The challenge is a C program that is a simple bank deposit and withdraw system. It's a standard-looking pwnable that has four options, as shown in Figure 1:

1. Deposit Money
2. Withdraw Money
3. Show Money
4. Show Stolen Money

The Deposit option is capped to a total of 1e10, and you cannot Withdraw more than you currently have. The deposit amount is a 32-bit float. Floats are complicated, as documented by Bruce Dawson in his series on floating point numbers that inspired this challenge.

I'm guessing the option that tickled your fancy was Show Stolen Money. How does it keep track of this? While the actual amount in the bank is a float, there is a separate long long used on the Withdraw option that is used to track the amount stolen.

Intended Solution

Why do we keep track of one value as a float and another as an integer? Shouldn't these values be the same? No, they are not! The solution is the difference between these two data types.

Floating Point Rounding

Unlike integers, which can represent all integers in their range, not all floats can be represented within the IEEE 754 format. The further a number gets from zero, the more significant the gap between valid numbers become. When a number is between two valid representations, it goes to the nearest number. For example, if the two valid numbers are 1.3 and 1.6, then 1.4 will go to 1.3.

How big of a deal is this rounding? Are we talking 0.00000001 or something smaller? Well, it depends! The maximum deposit number 1e10 was chosen for a specific reason. The gap between valid numbers becomes about 1024.00; this is very large! This is the price we may pay for being able to represent the numbers in a different way; we lose precision.

Solving the Challenge

The completion criteria is stealing more than $1500. To do this, we must abuse upward rounding in floats when using Withdraw at the bank. If the current deposit is large enough and the withdrawal is small enough, no money will be removed from our account on the amount float. Yet, we will still withdraw the funds ourselves.

To solve the challenge, do the following steps:

1. Deposit 1e10. Balance is now 1e10. Funds stolen is 0.
2. Withdraw 500. Balance is now 1e10. Funds stolen is 500. Notice that the balance did not go down! This is because the nearest float of 1e10-500 is still 1e10.
3. Withdraw 500. Balance is now 1e10. The funds stolen are $1000.
4. Withdraw 500. Balance is now 1e10. The funds stolen are $1500.
5. Challenge is now complete!

The floating bank was a fun challenge to teach students about floats and some of their interesting properties. Now, onto the real shenanigans!

Unintended Solution

The Spokane Cyber Cup has a mix of high school and college students. In this case, the student who found the unintended solution was an underclassman in high school named Seth Quast.

The solution is to use NaN as the deposit amount. Why? We'll get to that, as it's extremely unintuitive. When I asked how the high schooler solved the challenge, he casually said "I read the docs" and he had the web version of the man pages open. RTFM bro! Let's explore how and why NaN broke the program.

Not a Number (NaN)

Not a Number (NaN) is a valid value in the IEEE 754 specification. It is used in cases where the mathematical value is impossible, such as 0.0/0.0. Unlike integers, which will crash the program when this happens, floats opt to set NaN and continue execution.

NaN has a weird property similar to undefined in JavaScript: anything compared with it is always false. Even NaN == NaN is false! This is relevant later for the logic of the program.

Breaking Withdraw's Guard Statement

When the user wants to deposit or withdraw funds from the bank, the program retrieves it as a float using the function atof on a string.

Surprisingly, atof accepts more than just numbers. It can use inf for infinity and NaN for Not a Number. This function will convert the input string to the IEEE 754 specification, including for NaN/Inf. This is not something that I knew about at the time and, hence was not checking for. This is bug #1 - bad input validation.

When programming, I write guard statements for input validation and situational conditions, as seen in Figure 3. In particular, I place all checks at the beginning of functionality in if statements. If the conditional is true, then the rest of the call will fail. However, this is a fail open design, which probably wasn't the wisest decision in hindsight.

The check in Figure 3 is used in the Withdraw functionality to prevent withdrawing more funds than the user is currently has. If the user attempts to withdraw more funds than they should, the if statement is hit and aborts withdrawing the funds. However, if the amount is set to NaN, then the conditional will be false every time. Remember, NaN compared with anything is always false! Since this fails open, the code after it will continue to execute the withdrawal.

The over-withdrawal check is now completely null and void. NaN!

The Finishing Touch

The next magic happens in the code above in Figure 4. amount is currently NaN. Converting NaN to a long long results in 0! So, this code turns into 0 - (0 - LARGE_NUMBER). When simplifying, this turns into 0 + LARGE_NUMBER after the subtraction of a negative number. Funnily enough, these parentheses are wrong, but they don't matter in the case of the happy path. So, I never noticed until writing this post! To make this more clear, the red was added to the misplaced parentheses in Figure 4. This leads to amount_stolen becoming the amount we asked to Withdraw without any other sanity checks. We have now stolen all the funds necessary to complete the challenge!

Here is the full input to solve the challenge using Seth's NaN trick:

1. Deposit NaN. Balance is now NaN. Funds stolen is 0.
2. Withdraw 10000000. Balance is still NaN. Funds stolen is 10000000. Craziness.
3. Challenge is now complete!

Takeaways

Triaging how this worked took me a long while to do. I had no idea converting NaN to a long long would result in it being 0. I also had no idea that NaN is always false when doing numerical comparisons. The casting seemed fine while testing, but does weird things when the numbers are negative. It was super fun figuring all of this out back then and now!

After running the event, having a student find the unintended solution, and having years to reflect on it, here is what I have learned:

• Different minds find different bugs. Don't underestimate people, especially when crowd-sourcing is involved. I'm still amazed a high schooler found this!
• RTFM. Jokes aside, reading documentation is a great way to learn about potential edge cases of a program to try.
• NaN-related vulns are real. Admittedly, they are not very common. The only real instance I have found is Jack Baker's speed hack using NaN on an RPC request for Universal UE4. Regardless, it's another tool in the belt.

Conclusion

I'm always pleasantly surprised by the college and high school student's capabilities. Just about every year, there is an unintended solution or two at the SCC, but this is definitely my favorite. I hope Seth is kicking ass in whatever he is doing now! I hope you enjoyed learning about float rounding, NaN, and a funny CTF challenge workaround. If you want to try this out for yourself, the challenge is publicly available and Dockerized for an easy setup.

Feel free to contact me (contact information is in the footer) if you have any questions or comments about this article or anything else. If you have any fun or crazy unintended solutions, feel free to post them alongside this post; I would love to hear about them! Cheers from Maxwell "ꓘ" Dulin.