The root cause of many vulnerabilities are from the mishandling of numbers. The standard int type can go from 0x7FFFFFFF all the way to -0x80000000 (notice the negative) with an integer overflow. Or, it can be truncated and change the number from positive to negative. Integers can be a nightmare in C and have caused many memory corruption vulnerabilities over the years.

Recently quite a few findings have caught my attention that fall into the integer bug classes. A few to call out are the integer truncation bug in the Linux kernel found by Qualys, the signedness conversion bug (issue 2) in the BSD hypervisor by GHSL and an integer overflow in the Kindle on a buffer allocation size found by Checkpoint. Then, my hacking buddy seiraib found an integer overflow bug fuzzing but we could not find where the overflow actually occurred at. All of this got me wondering: "Can these bug classes be discovered at compilation or immediately crash at runtime?"

Yes, yes they can! GCC and Clang have compilation flags to find several bug classes at compile time. Additionally, there are some runtime protections that can cause crashes to make root causing MUCH easier. This blog post is about finding vulnerabilities via compiler warnings and dynamic instrumentation. All of the source code snippets with extra examples can be found at mdulin2/integer_ compilation_flags.

Bug Classes

The goal is to change a number in a way that the program does not expect to cause memory corruption down the road. There are three number (but mainly integer) vulnerability classes in C that will be focused on:

• Overflow/underflow: Integers will simply wrap around if you go past the maximum or minimum value of the type in C. For instance, an overflow of a signed integer would go from 0x7FFFFFFF all the way to -0x80000000. The underflow goes in the opposite direction and happens when subtracting values instead.
• Truncation: Shrinking the storage capacity of a number. For instance, going from uint64_t to uint32_t would cut the storage capacity of the number in half. This has the potential to drastically change the value and signedness of the number.
• Signedness conversion: Numbers are either signed and unsigned (unsigned int and int for example). Going between these can have terrible consequences when converting from a negative signed number to an unsigned number or a very large unsigned number to a signed number. For instance, an unsigned integer of 0xFFFFFFFF converted to a signed integer would be -1 instead of a very large number.

Static Analysis

Two of these bug classes can be determined at compile time via specific compilation flags. Although there will likely be an exceptional amount of unexploitable occurrences of this, there are wonderful leads for finding vulnerabilities.

Truncation

Using the Wconversion flag during compilation will output the warning "implicit conversions that may alter a value". This is directly referencing truncation and conversion bugs!

For example, code for the truncation case can seen in Figure 1. This code has a type of long long that is being converted to an int. Because this changes the storage capacity from 64-bit to 32-bit, this has the potential to cause corruption. When this code is compiled with the Wconversion flag, a warning message will appear mentioning a truncation issue! This warning can be seen in Figure 3. This exact error message could have been used to find the Linux Kernel kernel bug mentioned above, when the conversion from size_t (unsigned long long) to int was performed.

Figure 3 - Truncation warning message

Another interesting item to consider is the case with float and double. Since a double is 2x the size of a float, the same type of truncation can be detected by the same compilation flag. An example of this in code is shown here.

Signedness

The Wconversion flag can also be used in the detection of signedness conversion bugs. This happens when converting from a signed to an unsigned or converting from an unsigned integer to a signed integer; both of these issues are shown in Figure 2 above. An example of the compile time error message is shown in Figure 4 below.

Figure 4 - Signedness conversion warning message

Static Analysis Wrap Up

These flags only check for implicit conversions. Sometimes, a value needs to be converted from an unsigned integer to a signed integer in order to do some math, which is valid and expected C. When a number is casted explicitly, these error messages will not show up, as a result. In order to find explicit casts, using something like CodeQL, manual review or dynamic testing is the way to go.

Although we have only been using Wconversion for the static analysis so far, there are a plethora of flags that actually make up this single flag. For instance, Wsign-conversion flag warns for implicit conversions that may change the sign of an integer value. For more information on these, visit the GCC documentation. But, the tldr; is to just use Wconversion to catch all of these issues at compilation time.

Dynamic Instrumentation

The static analysis is likely to pull up mostly false positives with a few real bugs. However, dynamic analysis, will always find real bugs, if the code path can be triggered. Below, we will show instrumentation options to crash/notify on integer related bugs. Instead of discussing bug classes, we are going to talk about a few specific flags in GCC and Clang.

ftrapv

This option generates traps for signed overflows on addition, subtraction and multiplication operations. Again, since this is dynamic testing, this will cause the program to crash whenever this occurs! An example of this code can be seen below:

int main(){
	int a = 0x7FFFFFFF; 
	a = a + 1; 
	printf("Value: %d\n", a);
}

The code above will cause an integer overflow when the line a = a + 1; is executed since the maximum size of an integer in C is 0x7FFFFFFF. What happens? The program is aborted and never reaches the print statement. By forcing the crash when the overflow occurs, we guarantee that the bug is detected and the cause of the overflow is much easier to find. This is extremely useful when fuzzing and trying to root cause a crash!

It should also be noted that this flag detects signed integer underflows as well.

fsanitize=integer

The UBSAN (Uninitialized Behavior Sanitizer) for finding integer bugs is amazing! This flag is specific to Clang, for the integer related bugs.

While the ftrapv catches only signed integer overflows, fsanitize=integer will crash on unsigned and signed integer overflows (signed-integer-overflow and unsigned-integer-overflow). This means that all integer overflows, regardless of sign, via addition, subtraction or multiplication will be caught at runtime! Damn, this is a major improvement.

Besides the discovering of overflow/underflows in programs, we can find the two other bug classes mentioned with this flag: truncation (implicit-signed-integer-truncation & implicit-unsigned-integer-truncation) and conversion (implicit-integer-sign-change). Unlike the static method above, a bad math operation must occur to trigger UBSAN to crash the program.

Let's see this in action though! We will use the original signedness issue (Figure 1). When we run the code, the long long with the value LONG_MAX ( 0x7FFFFFFFFFFFFFFF) will be cut in half (truncated) because of the conversion to an integer. As a result, this will be 0xFFFFFFFF or -1 as a signed integer. Because of the extra instrumentation added, the program crashes upon this truncation happening! Nice for us, the instrumentation does not require a sign change; it checks that the value inside of the long long will not fit into the int. This crash can be seen in Figure 5 below.

Figure 5 - Truncation Conversion Crash

Are We Missing Anything?

We have mentioned dynamic and static checks for integer overflows/underflows, integer truncation's and signedness conversion issues. However, there is one thing missing from the dynamic instrumentation: floating point math.

When a float is overflowed in C, it goes to infinity or inf. Oddly enough, the float never truly wraps around because of how floating point math handles precision; it simply just goes to infinity! An example of this overflow can be seen at here.

An additional uncaught bug is float truncation. For instance, the conversion from a double to a float is not caught at runtime. There is a misleading UBSAN flag (-fsanitize=float-cast-overflow) that only finds bad double/float to integer conversions but NOT truncation bugs between floating point numbers. An example of this can be seen here.

Knowing about floats becoming inf and NaN may be useful to know since some crazy issues, such as Jack Bakers NaN propagation bug in the Unreal Gaming Engine do happen. However, there is currently no detection on the overflowing, underflowing, truncation or NaN/Inf usage of floating point numbers in C at runtime. A deep dive into the craziness of floating point math in C is out of the scope for this article but it is really interesting and worth an afternoon of learning about, as it is easy to make mistakes with.

Conclusion

When trying to find vulnerabilities, any help from automated tools or instrumentation is a huge win. Besides the integer vulnerability classes mentioned above, there are loads of other interesting flags and instrumentation, such as the well known ASAN (Address Sanitizer) to help find use after free bugs or the lesser known use after scope vulnerability on stack memory. There are many other compilation flags and instrumentation options to help find specific bugs classes as well; this article only focuses on finding integer related vulnerability classes.

I hope the knowledge of these compilation warnings and dynamic instrumentation helps you find many bugs in the future! Feel free to reach out to me (contact information is in the footer) if you have any questions or comments about this article. Cheers from Maxwell "ꓘ" Dulin.