Stack smashing prevention

Buffer overflows are common attack vectors that leverage missing checks of input data to overwrite program instructions. A rewarding program location to overwrite is a function’s return address on the call stack, as it allows the attacker to divert control flow to an arbitrary target. The following little program demonstrates this by overwriting the return address of function foo with the address of the function itself, thereby creating a recursive call.

#include <stdio.h>
#include <stdlib.h>

void foo(int dummy) {
    static int i = 0;
    if (i++ < 10) {
        printf("Call # %d\n", i);
    } else {
        exit(0);
    }
    (&dummy)[-1] = (unsigned int)foo;
}

int main() {
    foo(333);
    printf("If this line is printed, something went wrong...\n");
}

When executed, function foo is called 10 times before the program exits (note, that the program assumes an IA-32 memory layout and will not show the same behavior when compiled as an IA-64 binary).

To prevent the call stack from being modified during the function’s execution, the program needs to be compiled with a corresponding compiler flag that introduces some extra code for checking the stack’s integrity:

gcc smash.c -o smash -std=c99 -m32 -fstack-protector-all

Executing the compiled binary now shows a different behavior, as foo is only called once and the program exits afterwards. Using the -fstack-protector-all flag will make the output file a few bytes larger, a direct result of the introduced checks. You can make these checks visible by using the objdump utility:

objdump -d smash | grep __stack_chk_fail

Looking at the full output of objdump shows that the checking code is introduced at the ends of both the foo and the main function.