Memory Safety

Bits, bytes, hex, and the CPU

Bits. The smallest piece of information a computer can store is a bit — a single binary digit, either 0 or 1. Physically it is a circuit that holds a tiny electric charge (1) or nothing (0). A single bit can only distinguish two states, which is not very useful on its own.

Bytes. Eight bits grouped together form a byte. With 8 bits you can represent 2⁸ = 256 distinct values, from 0 to 255. Every piece of information your computer handles — a letter, a pixel, a machine instruction, a memory address — is ultimately stored as one or more bytes. When people say a file is "40 kilobytes," they mean it takes up roughly 40,000 bytes of storage.

Hexadecimal. Writing memory contents in decimal is unwieldy. Programmers almost always use hexadecimal (base 16, abbreviated hex) because one byte maps to exactly two hex digits. Hex uses the digits 0–9 for values zero through nine, then the letters A–F for ten through fifteen. The prefix 0x signals a hex literal. The table below shows the most commonly seen values; memorise these two: 0x41 is ‘A’ (attackers fill buffers with ‘A’ bytes to spot them in dumps) and 0x00 is the null terminator that ends every C string.

RAM. RAM (Random Access Memory) is your computer’s working memory — where programs and their data live while they run. Think of it as an enormous array of numbered slots, where each slot holds exactly one byte. The slot number is the address. A 32-bit process can address up to 2³² = 4,294,967,296 slots (4 GB of address space), regardless of how much physical RAM is actually installed.

The CPU and the fetch-decode-execute cycle. The CPU runs programs by repeatedly performing three steps: (1) Fetch — read the bytes at the current instruction address; (2) Decode — interpret those bytes as an instruction (add, subtract, jump…); (3) Execute — carry out the instruction and advance to the next one. The address of the next instruction is held in a dedicated register called the instruction pointer.

Registers. Registers are a handful of tiny, extremely fast storage locations built directly into the CPU — not in RAM. Accessing a register is hundreds of times faster than accessing RAM. The most important registers for exploit analysis are listed below.

Figure 1. RAM as a flat byte array (left) and the CPU’s key registers (right). EIP currently points to address 0x1003, so the CPU will fetch and execute the byte at that address next.

Virtual address space

Every process runs inside its own virtual address space — an illusion the operating system and CPU hardware create together. Rather than accessing physical RAM directly, your program uses virtual addresses. The CPU’s Memory Management Unit (MMU) silently translates each virtual address to a physical one on every memory access.

Why bother? Three reasons. (1) Isolation: process A cannot read or corrupt process B’s memory even though they share the same physical RAM. (2) Simplicity: every program acts as if it owns the entire address space, so there is no need to coordinate base addresses with other running programs. (3) Security: the OS can attach fine-grained permission flags to each page, telling the hardware which accesses to allow and which to fault.

Pages and permission bits. The virtual address space is divided into fixed-size chunks called pages — typically 4,096 bytes (4 KB). Every page has three permission bits set by the OS and enforced in hardware:

Violating a permission causes a hardware fault immediately — the OS kills the process with a segmentation fault. This enforcement is the foundation of Non-Executable Memory (NX), discussed in Section 10.

The virtual address space of a 32-bit Linux process is partitioned into fixed-purpose segments, arranged as shown below. Memorise this layout — it underpins every concept that follows.

Figure 2. Virtual address space of a 32-bit Linux process (not to scale). Stack lives near the top; executable code lives near the bottom.

The stack: push, pop, and ESP

The call stack (or just "the stack") is a region of memory that holds the live state of every currently executing function. It works on a last-in, first-out principle — like a stack of plates: you always add to and remove from the top. Adding an item is called a push; removing one is called a pop.

A crucial oddity: the stack grows downward. When you push something onto the stack, the memory address goes down (toward lower addresses). "Top of stack" in x86 means the lowest occupied address. The Stack Pointer register ESP always holds this lowest-occupied address — the address of the most recently pushed value.

Push in detail. The instruction push VAL does two things atomically: (1) decrement ESP by 4 (for a 32-bit value), making room at the lower address; (2) write VAL to the memory location now pointed to by ESP. In one instruction, ESP moves down and the value appears at the new top.

Pop in detail. The instruction pop REG does the reverse: (1) read 4 bytes from the address currently in ESP into REG; (2) increment ESP by 4, "freeing" that slot. The bytes are not erased; they just become unreachable from normal code until the stack grows again and overwrites them.

Figure 3. Stack before and after push 42 on x86-32. ESP decrements by 4 and the value 42 (= 0x2A) appears at the new top-of-stack slot.

Function calls and stack frames

Every function call creates a stack frame (also called an activation record): a block of stack memory holding that function’s arguments, saved registers, local variables, and the critical return address — the address the CPU should jump back to when the function finishes.

Here is the six-step sequence the x86-32 cdecl calling convention follows for a call like foo(42):

Steps 1–2 (caller sets up the call): (1) Push arguments right-to-left: push 42. (2) Execute call foo: the CPU automatically pushes the address of the instruction immediately after the call (this is the return address), then jumps to foo.

Steps 3–4 (callee prologue): (3) Save the caller’s frame pointer: push ebp. (4) Set up a new frame pointer: mov ebp, esp. EBP now anchors this frame; [EBP+8] is the argument, [EBP+4] is the return address, [EBP+0] is the saved EBP. (5) Allocate local variables by decrementing ESP: sub esp, 20 reserves 20 bytes (e.g., a 16-byte buffer plus a 4-byte int).

Steps 5–6 (epilogue and return): (5) Restore ESP to EBP (mov esp, ebp or leave). (6) Restore caller’s EBP (pop ebp). Then ret pops the return address off the stack into EIP, jumping back to the caller.

; Generated prologue for: void foo(int n) { char buf[16]; int x; }
foo:
    push  ebp          ; push caller's EBP  → [ESP] = old EBP
    mov   ebp, esp     ; EBP anchors frame at saved-EBP slot
    sub   esp, 20      ; carve out 20 bytes: buf[16] + int x[4]

    ; Local layout relative to EBP:
    ;   [EBP-20 .. EBP-5]  = char buf[16]
    ;   [EBP-4]            = int x
    ;   [EBP+0]            = saved EBP  ← overflow crosses here
    ;   [EBP+4]            = return address  ← primary target
    ;   [EBP+8]            = argument n

    ...

    mov   esp, ebp     ; restore ESP (deallocate locals)
    pop   ebp          ; restore caller's EBP
    ret               ; pop return address into EIP → jump back

Figure 4. Stack frame of foo(42) after the prologue. The buffer lives at the lowest addresses. Bytes written past the end of buf travel upward through int x, the saved EBP, and finally reach the return address at EBP+4.

C strings and dangerous library functions

To understand buffer overflows you must understand how C represents strings. In C, a string is simply a contiguous array of char bytes followed by a null terminator — the byte value 0x00, written '\0' in source. There is no embedded length field. Functions that operate on strings scan forward from the start until they encounter that 0x00 byte.

Consider the string "Hello": it occupies six bytes in memory — H(0x48) e(0x65) l(0x6C) l(0x6C) o(0x6F) \0(0x00). The function strlen("Hello") works by starting at H, counting bytes until it reaches 0x00, and returning 5. If that 0x00 byte is overwritten (perhaps by an overflow), strlen will keep reading past the end of the buffer indefinitely — a buffer over-read.

/* How a 6-byte C string sits in memory */
char *s = "Hello";

/* Memory:  addr+0  addr+1  addr+2  addr+3  addr+4  addr+5
            0x48    0x65    0x6C    0x6C    0x6F    0x00
              'H'     'e'     'l'     'l'     'o'    NUL  */

Why C has no bounds checking. C was designed in the early 1970s for systems programming on tiny computers. Adding automatic bounds checking to every array access would have made programs too slow. The philosophy was to trust programmers to manage memory correctly. Decades later, this remains the root cause of a massive fraction of all exploitable security vulnerabilities.

The standard C library ships several functions that accept a destination buffer and source data but never check whether the source fits in the destination. These are known collectively as dangerous or unsafe string functions:

/* Safe string handling — always pass the buffer size */
char name[64];

/* WRONG — no length limit */
gets(name);
scanf("%s", name);
strcpy(name, user_input);

/* CORRECT — always bound the write */
fgets(name, sizeof(name), stdin);
scanf("%63s", name);          // 63 = sizeof(name)-1
strncpy(name, user_input, sizeof(name) - 1);
name[sizeof(name) - 1] = '\0'; // always null-terminate

The hacker’s spatial reasoning

An attacker studying a C program thinks in terms of memory layout. Given a function with a local buffer, they ask: what lies in memory just above this buffer? Because the stack grows toward lower addresses, "above" (in the direction of increasing addresses) means: other locals, the saved EBP, and then the return address. Any write that overruns the buffer will travel upward through each of those fields in order.

Here is the attacker’s mental model, step by step, for the function greet() below:

Step 1. Identify the buffer: char buf[16] lives at EBP−16. It is 16 bytes wide.

Step 2. Identify the write primitive: strcpy(buf, name) — no length check. As long as name contains no null byte before position 16, bytes will travel past the end of buf.

Step 3. Compute the offset to the return address: sizeof(buf) = 16 bytes, then 4 bytes for the saved EBP, then 4 bytes for the return address. Total padding needed to reach the return address: 16 + 4 = 20 bytes. The return address occupies bytes 21–24 of the input.

Step 4. Choose a target address. Classic options: (a) the start of the shellcode placed at the beginning of the buffer; (b) a gadget in libc (see ROP, Section 11); (c) a known function like system() in libc (ret2libc).

Step 5. Craft the payload: [16 bytes padding][4 bytes overwrite SFP][4 bytes = target address].

Step 6. Deliver the payload through the channel the program accepts (stdin, network socket, file, command-line argument).

/* Vulnerable function — 16-byte buffer, unbounded copy */
void greet(char *name) {
    char buf[16];
    strcpy(buf, name);  // ← overflow here if strlen(name) >= 16
    printf("Hello, %s!\n", buf);
}

/* Attacker supplies this 24-byte string:
   "AAAAAAAAAAAAAAAA" + "BBBB" + "\x34\x12\xBF\xFF"
    └─── 16 bytes ────┘ └SFP─┘  └── return address ──┘
   The bytes 0x34, 0x12, 0xBF, 0xFF (little-endian: 0xFFBF1234)
   are loaded into EIP on ret ─── CPU jumps to 0xFFBF1234.     */

Buffer overflow mechanics — interactive

Let’s make it concrete with the vulnerable function below. The stack frame has six 4-byte rows (24 bytes from the start of the buffer to the end of the return address slot). Drag the slider to feed bytes in one at a time and watch what happens to each row.

void vulnerable(char *input) {
    char buf[16];          // 16-byte local buffer
    strcpy(buf, input);   // no length check
}

/* Stack layout (bytes 0–27, relative to start of buf):
     0  ..  3 : buf[0..3]   ← bytes 0..3   arrive here
     4  ..  7 : buf[4..7]
     8  .. 11 : buf[8..11]
    12  .. 15 : buf[12..15]
    16  .. 19 : saved EBP    ← bytes 16..19 overwrite SFP
    20  .. 23 : return addr  ← bytes 20..23 control execution  */

The three phases:

Bytes 1–16 (normal). Writes stay inside buf[16]. No overflow yet. The function will execute and return normally.

Bytes 17–20 (saved EBP corrupted). The overflow crosses into the saved EBP (SFP). On most programs this only causes a crash when the caller tries to use EBP after this function returns — the saved EBP is no longer a valid frame pointer. Many older exploits tolerated this because they only cared about controlling the return address.

Bytes 21–24 (return address overwritten). The most important moment. The attacker now fully controls where ret jumps. On x86-32 this is 4 bytes of their choice, loaded directly into EIP.

Return address overwrite in detail

Let’s trace a concrete overflow at the byte level. Suppose buf starts at address 0xBFFF0004, and the function was called from address 0x0804853C (the instruction to resume after the call).

Before the overflow — the stack looks like this:

After supplying 24 bytes: 0x41 * 16 (fill buf) + 0x41 * 4 (overwrite SFP) + 0x34 0x12 0xBF 0xFF (attacker address, little-endian):

When vulnerable() executes its ret instruction, the CPU pops the 4 bytes at 0xBFFF0018 into EIP. EIP is now 0xFFBF1234 — the attacker’s chosen address. The very next fetch-decode-execute cycle begins at whatever bytes live at that address.

Control flow hijacking and code injection

Redirecting EIP is only meaningful if the attacker knows what to point it at. The classical answer, dominant until the mid-2000s, was shellcode injection: embed a hand-crafted sequence of machine instructions directly inside the input buffer, then overwrite the return address to point back at the start of that buffer. When ret fires, the CPU starts executing the attacker’s code.

What is shellcode? Shellcode is a compact blob of machine instructions designed to run in an unknown context — no libraries, no operating system support, just raw bytes. The name comes from its most common payload: spawning a shell (/bin/sh), giving the attacker interactive command access. Modern shellcode is increasingly diverse: it might add a user account, install a backdoor, or pivot to another network host.

Constraints on shellcode. Because the input often goes through a function like strcpy() that stops at null bytes, shellcode must usually avoid 0x00 bytes. This is why shellcode uses instruction encodings that produce the same effect without null bytes — for example, xor eax, eax zeros EAX without containing a 0x00 byte in its encoding, while mov eax, 0 would.

/* Classic x86 execve("/bin/sh") shellcode — 23 bytes, no null bytes */
unsigned char sc[] =
    "\x31\xc0"              // xor  eax, eax      ; eax = 0
    "\x50"                  // push eax            ; push null terminator
    "\x68\x2f\x2f\x73\x68"  // push "//sh"         ; push /bin//sh
    "\x68\x2f\x62\x69\x6e"  // push "/bin"
    "\x89\xe3"              // mov  ebx, esp       ; ebx = ptr to "/bin//sh"
    "\x50"                  // push eax            ; push NULL (argv[1])
    "\x53"                  // push ebx            ; push filename ptr (argv[0])
    "\x89\xe1"              // mov  ecx, esp       ; ecx = argv array
    "\xb0\x0b"              // mov  al,  11        ; syscall 11 = execve
    "\xcd\x80";             // int  0x80           ; trigger kernel → /bin/sh!

/* Complete 24-byte exploit payload for buf[16]:
   [ shellcode (23 B) ][ pad (1 B) ][ &buf = 0xBFFF0004 (4 B LE) ]
   The last 4 bytes overwrite the return address, pointing it
   back to byte 0 of buf where the shellcode begins.              */

Figure 5. Left: normal control flow — vulnerable() returns to main(). Right: overwritten return address makes ret jump to shellcode at the start of buf.

Non-executable memory (NX / DEP / W⊕X)

Code injection requires the CPU to execute bytes the attacker placed in a writable region — the stack, heap, or any data segment. The straightforward hardware defence is to ensure that writable pages are not executable, so that landing on attacker-controlled data causes a fault rather than executing shellcode.

History. Early Intel and AMD CPUs lacked any distinction between writable and executable memory — every page the CPU could read it could also execute. PaX (a Linux kernel patch) implemented software-based NX in the late 1990s. AMD introduced the NX bit (bit 63 of the page table entry) in hardware in 2003 with its 64-bit processors. Intel followed with its Execute Disable (XD) bit in 2004. Windows XP SP2 (2004) introduced Data Execution Prevention (DEP) as a mainstream feature. Linux gained kernel NX support in 2.6.8 (2004) and later added hardware enforcement across all architectures.

The W⊕X rule. W⊕X (Writable XOR Executable, pronounced “W-xor-X”) is the policy that no page may be both writable and executable simultaneously. The stack and heap are R+W but not executable. The code segment is R+X but not writable. Any attempt to execute code on the stack now causes an immediate hardware fault.

Figure 6. W⊕X policy. Left: executable pages (code + libraries). Right: writable pages (stack, heap, data). Any attempt to execute code on the right side triggers a hardware fault.

Return-oriented programming (ROP)

NX defeats code injection, but notice what it does not prevent: the overflow itself still works, the return address is still overwritten, and the CPU still jumps to the attacker-supplied address. NX only adds a rule — the target must be in an already-executable page. The attacker’s insight is to point the CPU at code that was already in the binary or in libc.

What is a gadget? A ROP gadget is any short sequence of instructions that ends in a ret instruction, already present in the executable pages. A common gadget might be:

; gadget found at address 0x401234 in libc:
pop rdi       ; pop one value from the stack into RDI
ret           ; pop next value from stack into RIP (instruction pointer)

Why is this useful? Because ret is equivalent to pop rip — it pops the top of the stack into the instruction pointer and jumps there. By laying out a sequence of gadget addresses and data values on the stack, the attacker can chain gadgets together: each gadget executes a few instructions, then its ret pops the next gadget’s address from the stack and jumps to it. The CPU ends up executing a sequence the attacker chose entirely from existing bytes in executable memory.

ret2libc: the historical precursor. Before the full formalisation of ROP (Shacham et al., 2007), attackers used a simpler technique: overwrite the return address with the address of system() in libc, and place the string "/bin/sh" on the stack as its argument. No custom code at all — just repurposing an existing library function.

/* ret2libc payload layout (x86-32):
   [ padding 20B ] [ &system() ] [ fake return addr ] [ &"/bin/sh" ]
    buf + SFP        overwrites     (don't care)          argument   */

unsigned long system_addr  = 0x77E60000; // libc system() address
unsigned long binsh_addr   = 0x77F91B12; // "/bin/sh" in libc data

/* When vulnerable() returns, ESP points here:
     [0x77E60000]   ← ret jumps here (= system())
     [0xDEADBEEF]  ← "return address" for system() (don't care)
     [0x77F91B12]  ← argument to system() = "/bin/sh"
   system() executes /bin/sh — shell spawned.              */

A full ROP chain. On 64-bit, function arguments go in registers (RDI, RSI, RDX) not on the stack, so you need gadgets to load them:

/* x86-64 ROP chain to execve("/bin/sh", NULL, NULL) — syscall #59 */
// Gadgets (addresses found with ROPgadget --binary /lib/x86_64-linux-gnu/libc.so.6):
// 0x401234: pop rdi ; ret     ← loads first arg
// 0x401678: pop rsi ; ret     ← loads second arg
// 0x40189A: pop rdx ; ret     ← loads third arg
// 0x401890: pop rax ; ret     ← loads syscall number
// 0x40200A: syscall ; ret     ← triggers kernel

/* Crafted stack (RSP points here after overflow):
    0x0000000000401234   ← 1st gadget: pop rdi ; ret
    0x<addr of "/bin/sh">          ← value popped into RDI
    0x0000000000401678   ← 2nd gadget: pop rsi ; ret
    0x0000000000000000   ← NULL into RSI
    0x000000000040189A   ← 3rd gadget: pop rdx ; ret
    0x0000000000000000   ← NULL into RDX
    0x0000000000401890   ← 4th gadget: pop rax ; ret
    0x000000000000003b   ← 59 = execve syscall number
    0x000000000040200A   ← 5th gadget: syscall ; ret → shell! */

Figure 7. Three ROP gadgets (top, in existing code) and the attacker’s crafted stack (bottom). Each ret pops the next address, chaining gadgets into arbitrary computation without injecting a single new byte.

Defenses: canaries, ASLR, CFI, and safe languages

No single mitigation stops all attacks, but layering them together raises the cost of exploitation dramatically. We cover the four most widely deployed categories in order of historical introduction.

1. Stack canaries

Introduced by StackGuard (1998) and adopted into GCC as -fstack-protector (2003), a stack canary is a secret random value placed between the local variables and the saved EBP at function entry. Just before executing ret, compiler-inserted code checks that the canary is unchanged. An overflow that reaches the return address must pass through the canary slot first, corrupting it and triggering an abort.

/* What GCC inserts around your code with -fstack-protector: */
void foo(char *input) {
    /* prologue addition: */
    unsigned long canary = __stack_chk_guard;  // read thread-local secret

    char buf[16];
    strcpy(buf, input);  // potential overflow

    /* epilogue addition (before ret): */
    if (canary != __stack_chk_guard)
        __stack_chk_fail();  // abort() — crash rather than proceed
}

Figure 8. Stack frame with canary. An overflow travelling from buf to the return address must corrupt the canary first, triggering __stack_chk_fail() before ret executes.

2. Address Space Layout Randomisation (ASLR)

Even with a canary defeated, the attacker still needs to know where to redirect execution — what address contains the shellcode, or useful gadgets, or system(). ASLR removes this certainty by loading the stack, heap, and shared libraries at random base addresses every time the program starts. Each run, the attacker faces a different address space.

ASLR was first added to the Linux kernel in version 2.6.12 (2005). Combined with PIE (Position-Independent Executables), even the main binary’s .text and .data segments are randomised. On modern 64-bit Linux, ASLR randomises 28–32 bits of the load address, making brute-force impractical (2²⁸ = 268 million possibilities).

# Demonstrating ASLR on Linux — stack base changes every run:
$ cat /proc/self/maps | grep stack
bfa6d000-bfa8f000 rwxp [stack]

$ cat /proc/self/maps | grep stack
b7f32000-b7f54000 rwxp [stack]   # completely different each run

$ cat /proc/self/maps | grep stack
bfabc000-bfade000 rwxp [stack]   # and again different

Figure 9. Without ASLR (left), the stack and libc load at the same address every run. With ASLR (right), they move to random bases, invalidating hardcoded addresses.

3. Control Flow Integrity (CFI)

Stack canaries protect the return address slot. ASLR hides addresses. CFI takes a different approach: it instruments every indirect transfer of control (indirect calls, indirect jumps, and returns) to verify at runtime that the target is a legitimate destination according to the program’s intended control-flow graph. Even if the attacker controls the address in the return slot, CFI can prevent the jump if the destination is not an approved target.

Shadow stacks. Intel CET (Control-flow Enforcement Technology, shipped in Ice Lake CPUs, 2020) adds a hardware-managed shadow stack: a separate, CPU-protected copy of return addresses. Every call pushes the return address to both the normal stack and the shadow stack. Every ret pops both and compares them; a mismatch causes a fault. Because the shadow stack is mapped read-only from user space, an attacker who overwrites the return address on the normal stack will trigger a mismatch and abort. ARM equivalents include PAC (Pointer Authentication Codes) on ARMv8.3+ and BTI (Branch Target Identification).

CFI is deployed in Clang/LLVM (since 3.9), Microsoft CFGuard (Windows 10), and the Linux kernel IBT/SHSTK support (merged in 5.18).

4. Memory-safe languages

All the defences above are mitigations — they make exploitation harder without removing the underlying vulnerability. The root cause is that C and C++ allow arbitrary pointer arithmetic and array writes without bounds checking. Memory-safe languages eliminate entire classes of bugs at compile time or through runtime enforcement.

Rust uses an ownership and borrow-checking type system enforced entirely at compile time. The compiler proves that every reference is valid and every array access is in-bounds — or it refuses to compile the code. No garbage collector is required; safety is achieved through static analysis. Code that violates the rules must be placed inside an unsafe block, making the risky surface area explicit and auditable.

// Rust — bounds enforcement at compile time and runtime
fn greet(name: &str) {
    let mut buf = [0u8; 16];
    // This panics if name.len() > 16 — never silently overflows:
    let n = name.len().min(16);
    buf[..n].copy_from_slice(&name.as_bytes()[..n]);
    // Compiler proves the slice [..n] cannot exceed buf's length.
    // No buffer overflow is possible in safe Rust.
}

Other safe languages. Go, Swift, Java, C#, and Python all enforce bounds checking on every array access (raising an exception on violation) and manage memory through garbage collection, preventing use-after-free and double-free bugs. The trade-off is runtime overhead (GC pauses, bounds-check cost) and reduced control over memory layout.

Industry adoption. Google reports that around 70% of its severe security bugs are memory-safety issues. Microsoft reports similar numbers. Both companies are actively migrating new code to Rust where performance and safety are both required. The Linux kernel has accepted Rust as a second implementation language since 2022 (kernel 5.20). Android, Windows, Chrome, and Firefox all contain Rust components as of 2024.