Session 3: Memory Safety¶

What is a buffer overflow?¶

A program reserves a fixed block of memory for input — for example, 64 bytes for a name. If you type more than 64 bytes and the program does not check the length, the extra bytes spill into adjacent memory. This is called a stack buffer overflow. The memory next to your buffer might hold another variable, a function address, or the return address that tells the CPU where to go after a function finishes. Overflowing into any of those lets an attacker change program behaviour without ever having access to the source code.

The exercises use Protostar, a virtual machine built specifically for learning exploits. It runs as a 32-bit Linux VM inside QEMU on the local machine.

Setup used: - Host: macOS, QEMU 11.0.2 - Guest: Protostar (Linux 2.6.32, i686) - Access: ssh -o HostKeyAlgorithms=+ssh-rsa -p 2223 user@localhost

Stack Zero — Changing a variable by overflowing a buffer¶

What the program does¶

The program creates a 64-byte buffer and an integer called modified next to each other in memory. It reads input with gets() — a C function that copies bytes into the buffer with no length check. If modified is non-zero at the end, it prints success.

char buffer[64];
int modified;
gets(buffer);

How does gets(buffer) change modified?¶

Memory on the stack is laid out sequentially. buffer gets 64 bytes; modified sits in the 4 bytes immediately after. gets() does not know where buffer ends — it keeps writing as long as there is input. Typing 65 characters fills the 64-byte buffer and then writes one extra byte directly into modified. The CPU does not object; it simply writes to the next address.

python -c "print('A' * 65)" | /opt/protostar/bin/stack0

Result¶

you have changed the 'modified' variable

What does “Segmentation fault” mean?¶

A segmentation fault happens when a program tries to access a memory address it is not allowed to touch — for example, jumping to a garbage address, or reading from a region that belongs to a different process. The operating system detects this and kills the program. In Stack Zero we only touched a valid stack address (modified), so there was no crash. A segfault would happen if the overflow reached the saved return address and the CPU tried to jump somewhere invalid.

Stack One — Writing a specific value into a variable¶

What the program does¶

Same layout as Stack Zero, but now modified must equal exactly 0x61626364. Any non-zero value is not enough — the bytes written must match precisely.

What does “little endian” mean?¶

x86 processors store numbers in little-endian order: the smallest byte goes first in memory. The number 0x61626364 has four bytes: 61, 62, 63, 64. In little-endian, the byte 64 goes to the lowest address, then 63, 62, 61. So the bytes we must write are \x64\x63\x62\x61 — the reverse of how you would write the number on paper.

/opt/protostar/bin/stack1 $(python -c "print('A' * 64 + '\x64\x63\x62\x61')")

64 A bytes fill the buffer; the next 4 bytes write 0x61626364 into modified in the correct order.

Result¶

you have correctly got the variable to the right value

Why no segmentation fault?¶

The payload was exactly 68 bytes: 64 to fill the buffer, 4 to overwrite modified. It did not reach the saved return address further down the stack. The program continued running normally, checked modified, and printed the success message.

Stack Three — Redirecting execution to a hidden function¶

What the program does¶

Instead of an integer, the variable next to the buffer is a function pointer — a variable that holds the address of a function. If it is non-null, the program calls it. There is a function called win() in the binary that is never called in the normal code path. The goal is to overwrite the function pointer with win()’s address so the program jumps there.

How to input hex values that don’t correspond to keyboard letters¶

The address of win() contains bytes like \x24 and \x08 that have no keyboard equivalent. The solution is to use Python to generate the exact bytes and pipe them directly to the program:

python -c "print('A' * 64 + '\x24\x84\x04\x08')" | /opt/protostar/bin/stack3

python -c runs a one-liner. Python treats \x24 as the raw byte with value 0x24 — not the two characters \ and x. The pipe (|) sends those bytes straight to the program’s input.

Finding the address of win()¶

objdump -d /opt/protostar/bin/stack3 | grep win

Output:

08048424 <win>:

win() is at address 0x08048424. In little-endian: \x24\x84\x04\x08.

Exploit¶

python -c "print('A' * 64 + '\x24\x84\x04\x08')" | /opt/protostar/bin/stack3

Result¶

calling function pointer, jumping to 0x08048424
code flow successfully changed

Key Takeaways¶

A buffer overflow lets an attacker write into memory they were never supposed to touch — just by typing more input than the program expected.
The three levels show a progression: first change any value, then change a specific value, then redirect where the program goes next.
gets() is the dangerous function here — it copies input with no length limit. Modern C compilers refuse to use it; safe alternatives like fgets() require a length argument.
Stack canaries, ASLR, and non-executable stack are mitigations that make real-world exploitation much harder. Protostar disables them so you can see the raw mechanics.
Languages like Rust and Go prevent this class of bug entirely by enforcing memory safety at the language level.

Level	What was overwritten	What was required	How
Stack Zero	Integer `modified`	Any non-zero value	65 bytes of padding
Stack One	Integer `modified`	Exact value `0x61626364`	64 bytes + target in little-endian
Stack Three	Function pointer `fp`	Address of `win()`	64 bytes + address from `objdump`

AI Assistance¶

I used Claude AI (claude-sonnet-4-6) as a learning assistant throughout this session. I ran all commands myself inside the Protostar VM via SSH over QEMU running locally on my Mac. Key prompts I wrote: provided the homework requirements, ran each binary, used objdump to find the win() function address, and built the exploit payloads step by step.

Claude helped me understand why the stack layout causes the overflow, work out the correct payload size and byte order for each level, and explain the concepts (segfault, little endian, function pointers) in my own words. All outputs shown are real results from the Protostar VM.