Session 3: Software security and Memory safety¶

Link to the video

“AI (Claude) has been very helpfull for this assignment, this was out of my confort zone. I used it to understand the code (“what does this code ?”), to design a strategy (“Identify weaknesses, from cybersecurity point of view, how could you exploit those weaknesses? “) and to understand step by step why that strategy works (“Explain step by step your strategy”). All demos are my own. I understand what it does and why”

Part 1 - Setup the environment¶

Option 1 - using the provided VM¶

Download the Protostar .iso image from https://exploit.education/protostar/

From the desktop terminal, run the virtual machine with QEMU; typically:

qemu-system-x86_64 -boot d -cdrom ~/Downloads/exploit-exercises-protostar-2.iso -m 512

Option 2 - using my own infrastructure¶

Download the Protostar .iso image from https://exploit.education/protostar/

In Proxmox, create a VM

Start, login, done !

Part 2 - Protostar challenge (stack0)¶

The code¶

#include <stdlib.h>
#include <unistd.h>
#include <stdio.h>

int main(int argc, char   *argv)
{
    volatile int modified;
    char buffer[64];

    modified = 0;
    gets(buffer);

    if(modified != 0) {
        printf("you have changed the 'modified' variable\n");
    } else {
        printf("Try again?\n");
    }
}

The hack (try until you get “you have changed the ‘modified’ variable”)¶

In order to achieve the expected result, the system needs to run the true part of the “if’ statement. It occurs when “modified” local variable is not equal to zero

What happened ?¶

We need some piece of code to run that C program. There are many ways to do that, it could be a command in a shell or… a call from a Python program. The Python -c parameter tells the system to run the code just after. In our example “print something”.

Then the output of our very minimalistic Python program goes to the next player, our code under test, along with the parameters. (that’s what the | does, passing things to the next)

Here we are, our program runs its main() function

How is it that providing an input in the line that reads “gets(buffer)” can change the value of the variable “modified”?¶

gets() is a very bad boy, he literaly does’t know its limits. The value of the parameter we pass is larger than 64 bytes. No worry, gets() will accept it and store it in memory but.. somewhere he is not supposed to write to.

And guess what, the nearest neigbour in memory is very delicate boolean variable (well, an int in fact, that’s another debate) named “modified” living in a true/false simple world.

The memory space occupied by “modified” is overwritten, with an “h” letter. Is “h” true or false from the “modified” fragile creature point of view ? Well, since it is not a zero, he will take no chance and considered it as .. false.

Done !

What does “Segmentation fault” mean?¶

A segmentation fault (segfault) is a runtime error that occurs when a program tries to access memory it’s not allowed to access.

If the bad memory access happens to land on memory your process does own (e.g. a nearby stack variable or heap block), the OS won’t intervene and no segfault occurs. Instead you get silent memory corruption — often considered worse, because the bug shows up far from its cause.

So, I’m confused here: there is a “segmentation fault> message printed BUT the code does not stop (we still get the “you have changed the ‘modified’ variable” message)

The reason is : the OS only intervenes when main() tries to return to a corrupted/invalid address — and by then, the printf has already run.

Part 3 - Protostar challenge (stack1)¶

The code¶

#include <stdlib.h>
#include <unistd.h>
#include <stdio.h>
#include <string.h>

int main(int argc, char   *argv)
{
    volatile int modified;
    char buffer[64];

    if(argc == 1) {
        errx(1, "please specify an argument\n");
    }

    modified = 0;
    strcpy(buffer, argv[1]);

    if(modified == 0x61626364) {
        printf("you have correctly got the variable to the right value\n");
    } else {
        printf("Try again, you got 0x%08x\n", modified);
    }
}

The hack (try until you get “you have correctly got the variable to the right value”)¶

In order to achieve the expected result, the system needs to run the true part of the “if’ statement. It occurs when “modified” local variable is not equel to zero. This time, our little friend is not considered as a boolean anymore. Want some specific value !

What happened ?¶

We passed 64 bytes of any garbage, followed by a sequence of 4 letters: “d”, “c”, “b” and “a”.

How is this in any kind related to 0x61626364 ? Well, 0x61626364 is 0x61 followed by 0x62 and then 0x63 and 0x64. Those are the hexadecimal values of a,b, c and d lowercase letters. (see https://lc11asciicode.weebly.com/differences.html)

But.. the order is wrong ! Keep reading..

What does “little endian” mean?¶

You mean that guy ?

Seriously, little endian is a method computers use to store multi-byte numbers in memory. In a little-endian system, the least significant byte (the “little end”) is stored at the lowest memory address, and the most significant byte is stored at the highest address.

Why does little endian exist? To make you confused !

In fact, it’s actually useful for CPUs — when you read a multi-byte number starting at its lowest address, you get the least significant byte first, which makes certain arithmetic operations (like addition with carry) simpler in hardware since you process the lowest-value part first.

Basically all desktop/laptop CPUs, and ARM in its default mode. BUT network protocols (TCP/IP uses “network byte order” = big endian). I see trouble coming…

Why didn’t you get “Segmentation fault”?¶

We got a silent memory corruption, i.e. we touched very delicately something very close to the memory location occupied by the buffer.

Part 4 - Protostar challenge (stack3)¶

The code¶

#include <stdlib.h>
#include <unistd.h>
#include <stdio.h>
#include <string.h>

void win() {
    printf("code flow successfully changed\n");
}

int main(int argc, char   *argv) {
    volatile int (*fp)();
    char buffer[64];

    fp = 0;

    gets(buffer);

    if(fp) {
        printf("calling function pointer, jumping to 0x%08x\n", fp);
        fp();
    }
}

The hack (Try until you get this message “code flow successfully changed”)¶

In order to achieve the expected result, we need to execute the win() function. But there is no call to that function in our program, how could we ???

There is another call to a function in our program, the fp(). We don’t know what it is and what it does, we just know the adress of that function (volatile int (*fp)();)

What if we could overwrite that address in memory ? Then, it would jump potentialy anywhere and that anywhere could be the place where fp() is bootstrapped in memory.

What happened ?¶

First, we need to determine the fp() address. Now, the tricky part (without AI, it would have been very difficult for me to figure out this)

objdump -d /opt/protostar/bin/stack3

The value is 0x08048424

python -c "import struct; print 'A'*64 + struct.pack('<I', 0x08048424)" | /opt/protostat/bin//stack3

Let’s examine this step by step:

struct is a Python package
pack is a function in Python’s struct module that takes a Python value and converts it into raw bytes the way C would store it in memory.
’<’ stands for “little endian”
I stands for “integer”

So basicaly, it says “build a sequence starting with 64 times the uppercase A letter (it could be anything else, it does’t matter for this challenge) followed by a 4 bytes integer with a value of 0x08048424 and put it the right way, we are on x86, so… little endian”

The complete paramter passed to main() is

0 to 62 0x41 (‘A’)
63 0x41 (‘A’) ← end of buffer[64]
64 0x24 ← fp byte 0 (LSB of 0x08048424)
65 0x84 ← fp byte 1
66 0x04 ← fp byte 2
67 0x08 ← fp byte 3 (MSB of 0x08048424)

How can you input hex values that don’t correspond to keyboard letters?¶

Just prefix then with 0x followed by the hexa code

Early computers and their programmers had to deal with raw binary constantly — toggling switches, reading printouts, debugging memory dumps. When IBM introduced the System/360 in 1964, it standardized the 8-bit byte as the fundamental unit of data Decimal feels natural to humans, but it has a fatal flaw for computing: it doesn’t map cleanly onto binary. There’s no simple relationship between decimal digits and bits, so converting between them requires actual arithmetic

Hex sits in a sweet spot:

Compact enough to be readable (unlike binary)
Structured enough to reflect the underlying bits (unlike decimal)
Small enough that humans can memorize the 16-digit table (unlike base-32 or base-64 for casual use)

It’s worth emphasizing: computers never use hex internally. Everything is binary at the hardware level. Hex is purely a display and input convention — a lens that lets humans read and write binary data without going insane. It was invented because the alternative (staring at raw binary) was error-prone and exhausting for engineers debugging real systems.

Now, what about ASCII ?

As I wrote previously, computers only understand numbers (binary). ASCII assigns a number (0–127) to each character — letters, digits, punctuation, and control codes — so that hardware and software from different manufacturers could exchange text reliably. It was standardized in 1963.

So, 7 bits are enough ?

No, ASCII only covers English. The 8th bit was later used by various “extended ASCII” schemes (like ISO-8859-1) to add accented letters and other symbols, but these were incompatible with each other. That chaos eventually led to Unicode, which can represent over 1 million characters from virtually every writing system. UTF-8, the dominant encoding today, is a superset of ASCII — the first 128 UTF-8 code points are identical to ASCII, so plain ASCII text is valid UTF-8.

So, as an human being, do you prefer ..

Hello Fab Futures or
As a sequence: 72 101 108 108 111 32 70 97 98 32 70 117 116 117 114 101 115 or
In hex: 48 65 6C 6C 6F 20 46 61 62 20 46 75 74 75 72 65 73 or
In binary: 01001000 01100101 01101100 01101100 01101111 00100000 01000110 01100001 01100010 00100000 01000110 01110101 01110100 01110101 01110010 01100101 01110011

Part 5 - Final race¶

Is it that bad, are with living in a collapsing world dominated by hackers ?

Yes.. and no. Most of the challenges we faced in Protostar have been addressed, at least in well maintained systems.

Here are list of countermeasures:

Stack buffer overflow → Stack canaries (1998/GCC 2005)¶

Blocks buffer overflows that overwrite the return address by placing a random canary value between local variables and the saved return address.

Shellcode on stack → NX/DEP (2004)¶

Blocks shellcode injection (Stack5/Stack6 style — jump to shellcode on the stack) Marks memory regions with hardware-enforced permissions using the CPU’s NX bit (AMD) / XD bit (Intel)

Return-to-libc → ASLR (2001, mainstream 2005)¶

Blocks hardcoded addresses — you can’t jump to win() or system() if you don’t know where they are, by randomizing the base address of the stack, the heap, the shared libraries (libc, etc.) and the executable itself (when combined with PIE)

ROP chains → CET / Shadow Stack (2020)¶

ROP was invented specifically to bypass NX — instead of injecting code, attackers chain existing executable snippets (“gadgets”) ending in ret.

Defenses against ROP:

Shadow Stack (Intel CET — Control-flow Enforcement Technology)
Hardware maintains a separate, protected stack that only stores return addresses
On ret, CPU compares against shadow stack copy; mismatch → fault
Available since Intel Tiger Lake (2020); Linux kernel support added in 6.x
Windows has supported CET since Windows 10 20H1
IBT (Indirect Branch Tracking) — also part of CET
Every valid indirect call/jump target must be preceded by an ENDBR64 instruction
Eliminates jumping to arbitrary gadgets mid-function
SafeStack (Clang/LLVM)
Splits the stack: return addresses go on a safe stack (not adjacent to buffers), local variables on a separate unsafe stack
Software-only, no hardware required
Stack Pivot Detection
Heuristics in the kernel/hypervisor to detect RSP being moved to unexpected memory regions

Fixed binary addresses → PIE (widespread ~2014)¶

Blocks hardcoded addresses within the binary itself. ASLR randomizes libraries but not the main executable unless it’s compiled as PIE:

GOT overwrites → Full RELRO (widespread ~2014)¶

Blocks GOT/PLT overwrite attacks. The Global Offset Table is a writable structure used for dynamic linking. Attackers who can write to it can hijack calls to libc functions.

Partial RELRO: Reorders ELF sections to reduce exposure; GOT still writable
Full RELRO: The entire GOT is remapped read-only after startup, before main() runs

Heap metadata attacks → Safe unlinking, safe-linking (2004–2020)¶

Heap canaries - Canary in chunk headers; detects corruption
Safe unlinking - Verifies forward/back pointers before free
Tcache hardening - Pointer mangling in tcache (glibc 2.32+)
Safe-linking - XORs freelist pointers with a secret (glibc 2.32+)
Guard pages - Unmapped pages between heap regions
jemalloc / mimalloc - Alternative allocators with stronger isolation by design

Format string attacks → Fortify Source (2004)¶

Blocks unsafe function calls like gets(), strcpy(), sprintf()
Replaces dangerous functions with checked versions at compile time:

Modern binaries compiled with full protections (-fstack-protector-all -fPIE -pie -D_FORTIFY_SOURCE=2 -Wl,-z,relro,-z,now) with ASLR enabled make the Protostar-style attacks essentially impossible without an information leak vulnerability to defeat ASLR first — which is why modern exploit research focuses heavily on chaining multiple bugs together rather than single clean overflows.

But the final race isn’t over yet !