What is a Stack Smashing Attack?
Stack smashing is a form of vulnerability where the stack of a computer application or OS is forced to overflow. This may lead to subverting the program/system and crashing it. A stack, a first-in last-out (FILO) circuit, is a form of buffer holding intermediate results of operations within it. Stack smashing involves putting more data into a stack than its holding capacity.1 Skilled hackers can deliberately introduce excessive data into the stack. The excessive data might be stored in other stack variables, including the function return address, and when the function returns, it jumps to the malicious code on the stack, which might corrupt the entire system. The adjacent data on the stack is affected and forces the program to crash.
If the program affected by stack smashing accepts data from untrusted networks and runs with special privileges, this is a potential security vulnerability. If the buffer contains data provided by an untrusted user, the stack may be corrupted by injecting executable code into the program, thus gaining unauthorized access to a computer. A cyberattacker can also overwrite control flow information stored in the stack.
Stack smashing is a serious vulnerability. The following approaches can be used to protect yourself against stack smashing2:
Stack canaries. Stack canaries are used to detect a stack buffer overflow before the execution of malicious code can occur. This method works by placing a small integer with a random value in memory just before the stack return pointer. Since memory is typically overwritten in buffer overflows from lower to higher memory addresses, the canary value must also be overwritten in order to overwrite the return pointer and so take over the process. The most common types of stack canaries are Null canary, Terminator canary, Random canary, Random XOR canary, 64-bit canary, and Custom canary.3
Address Space Layout Randomization (ASLR). ASLR is a technique that randomizes the memory layout of a process, making it difficult for a cyberattacker to predict the location of specific code or data in memory. Since the cyberattacker can no longer rely on the memory layout being the same every time that the program runs, it makes it much more difficult for them to exploit vulnerabilities in a program.
Nonexecutable Stacks. This approach to preventing stack overflow involves enforcing a memory policy on the memory region of the stack that disallows execution from the stack. To execute the shellcode from the stack, the cyberattacker will either have to find a way to disable the execution protection from the stack’s memory, or to find a way to put their shellcode payload in an unprotected region of the memory. This method has become more popular since hardware support for the no-execute flag became available.
1 Rouse, 2011, “Stack Smashing”
2 Neagu, 2023, “What is Stack Smashing?”
3 Lemmens, 2021, “Stack Canaries – Gingerly Sidestepping the Cage”