1 of 1

Compiling, Customising and booting the Kernel

Instructions for compiling the kernel with your own settings, as well as compiling kernel modules for a specific kernel version.

This isn't necessary for learning how to write kernel exploits - all the important parts will be provided! This is just to help those hoping to write challenges of their own, or perhaps set up their own VMs for learning purposes.

Prerequisites

$ apt-get install flex bison libelf-dev

There may be other requirements, I just already had them. Check here for the full list.

The Kernel

Cloning

git clone https://github.com/torvalds/linux --depth=1

Use --depth 1 to only get the last commit.

Customise

Remove the current compilation configurations, as they are quite complex for our needs

$ cd linux
$ rm -f .config

Now we can create a minimal configuration, with almost all options disabled. A .config file is generated with the least features and drivers possible.

$ make allnoconfig
  YACC    scripts/kconfig/parser.tab.[ch]
  HOSTCC  scripts/kconfig/lexer.lex.o
  HOSTCC  scripts/kconfig/menu.o
  HOSTCC  scripts/kconfig/parser.tab.o
  HOSTCC  scripts/kconfig/preprocess.o
  HOSTCC  scripts/kconfig/symbol.o
  HOSTCC  scripts/kconfig/util.o
  HOSTLD  scripts/kconfig/conf
#
# configuration written to .config
#

We create a kconfig file with the options we want to enable. An example is the following:

CONFIG_64BIT=y
CONFIG_SMP=y
CONFIG_PRINTK=y
CONFIG_PRINTK_TIME=y

CONFIG_PCI=y

# We use an initramfs for busybox with elf binaries in it.
CONFIG_BLK_DEV_INITRD=y
CONFIG_RD_GZIP=y
CONFIG_BINFMT_ELF=y
CONFIG_BINFMT_SCRIPT=y

# This is for /dev file system.
CONFIG_DEVTMPFS=y

# For the power-down button (triggered by qemu's `system_powerdown` command).
CONFIG_INPUT=y
CONFIG_INPUT_EVDEV=y
CONFIG_INPUT_KEYBOARD=y

CONFIG_MODULES=y

CONFIG_KPROBES=n
CONFIG_LTO_NONE=y
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_EMBEDDED=n
CONFIG_TMPFS=y

CONFIG_RELOCATABLE=y
CONFIG_RANDOMIZE_BASE=y

CONFIG_USERFAULTFD=y

Explanation of Options

CONFIG_64BIT - compiles the kernel for 64-bit
CONFIG_SMP - simultaneous multiprocessing; allows the kernel to run on multiple cores
CONFIG_PRINTK, CONFIG_PRINTK_TIME - enables log messages and timestamps
CONFIG_PCI - enables support for loading an initial RAM disk
CONFIG_RD_GZIP - enables support for gzip-compressed initrd images
CONFIG_BINFMT_ELF - enables support for executing ELF binaries
CONFIG_BINFMT_SCRIPT - enables executing scripts with a shebang (#!) line
CONFIG_DEVTMPFS - Enables automatic creation of device nodes in /dev at boot time using devtmpfs
CONFIG_INPUT - enables support for the generic input layer required for input device handling
CONFIG_INPUT_EVDEV - enables support for the event device interface, which provides a unified input event framework
CONFIG_INPUT_KEYBOARD - enables support for keyboards
CONFIG_MODULES - enables support for loading and unloading kernel modules
CONFIG_KPROBES - disables support for kprobes, a kernel-based debugging mechanism. We disable this because ... TODO
CONFIG_LTO_NONE - disables Link Time Optimization (LTO) for kernel compilation. This is to allow better debugging
CONFIG_SERIAL_8250, CONFIG_SERIAL_8250_CONSOLE - TODO
CONFIG_EMBEDDED - disables optimizations/features for embedded systems
CONFIG_TMPFS - enables support for the tmpfs in-memory filesystem
CONFIG_RELOCATABLE - builds a relocatable kernel that can be loaded at different physical addresses
CONFIG_RANDOMIZE_BASE - enables KASLR support
CONFIG_USERFAULTFD - enables support for the userfaultfd system call, which allows handling of page faults in user space

In order to update the minimal .config with these options, we use the provided merge_config.sh script:

$ scripts/kconfig/merge_config.sh .config ../kconfig

Building

$ make -j4

That takes a while, but eventually builds a kernel in arch/x86/boot/bzImage. This is the same bzImage that you get in CTF challenges.

Kernel Modules

When we compile kernel modules for our own kernel, we use the following Makefile structure:

all:
    make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules

To compile it for a different kernel, all we do is change the -C flag to point to the newly-compiled kernel rather than the system's:

all:
    make -C /home/ir0nstone/linux M=$(PWD) modules

The module is now compiled for the specific kernel version!

Booting the Kernel in a Virtual Machine

References

Creating the File System and Executables

We now have a minimal kernel bzImage and a kernel module that is compiled for it. Now we need to create a minimal VM to run it in.

To do this, we use busybox, an executable that contains tiny versions of most Linux executables. This allows us to have all of the required programs, in as little space as possible.

We will download and extract busybox; you can find the latest version here.

$ curl https://busybox.net/downloads/busybox-1.36.1.tar.bz2 | tar xjf -

We also create an output folder for compiled versions.

$ mkdir busybox_compiled

Now compile it statically. We're going to use the menuconfig option, so we can make some choices.

$ cd busybox-1.36.1
$ make O=../busybox_compiled menuconfig

Once the menu loads, hit Enter on Settings. Hit the down arrow key until you reach the option Build static binary (no shared libs). Hit Space to select it, and then Escape twice to leave. Make sure you choose to save the configuration.

Now, make it with the new options

$ cd ../busybox_compiled
$ make -j
$ make install

Now we make the file system.

$ cd ..
$ mkdir initramfs
$ cd initramfs
$ mkdir -pv {bin,dev,sbin,etc,proc,sys/kernel/debug,usr/{bin,sbin},lib,lib64,mnt/root,root}
$ cp -av ../busybox_compiled/_install/* .
$ sudo cp -av /dev/{null,console,tty,sda1} dev/

The last thing missing is the classic init script, which gets run on system load. A provisional one works fine for now:

#!/bin/sh
 
mount -t proc none /proc
mount -t sysfs none /sys
 
echo -e "\nBoot took $(cut -d' ' -f1 /proc/uptime) seconds\n"
 
exec /bin/sh

Make it executable

$ chmod +x init

Finally, we're going to bundle it into a cpio archive, which is understood by QEMU.

find . -not -name *.cpio | cpio -o -H newc > initramfs.cpio

The -not -name *.cpio is there to prevent the archive from including itself
You can even compress the filesystem to a .cpio.gz file, which QEMU also recognises

If we want to extract the cpio archive (say, during a CTF) we can use this command:

$ cpio -i -F initramfs.cpio

Loading it with QEMU

Put bzImage and initramfs.cpio into the same folder. Write a short run.sh script that loads QEMU:

#!/bin/sh

qemu-system-x86_64 \
    -kernel bzImage \
    -initrd initramfs.cpio \
    -append "console=ttyS0 quiet loglevel=3 oops=panic" \
    -monitor /dev/null \
    -nographic \
    -no-reboot

Explanation of Flags

-kernel bzImage - sets the kernel to be our compiled bzImage
-initrd initramfs.cpio - provide the file system
-append ... - basic features; in the future, this flag is also used to set protections
- console=ttyS0 - Directs kernel messages to the first serial port (ttyS0)
- quiet - Only showing critical messages from the kernel
- loglevel=3 - Only show error messages and higher-priority messages
- oops=panic - Make the kernel panic immediately on an oops (kernel error)
-monitor /dev/null - Disable the QEMU monitor
-nographic - Disable GUI, operate in headless mode (faster)
no-reboot - Do not automatically restart the VM when encountering a problem (useful for debugging and working out why it crashes, as the crash logs will stay).

Once we make this executable and run it, we get loaded into a VM!

User Accounts

Right now, we have a minimal linux kernel we can boot, but if we try and work out who we are, it doesn't act quite as we expect it to:

~ # whoami
whoami: unknown uid 0

This is because /etc/passwd and /etc/group don't exist, so we can just create those!

/etc/passwd

root:x:0:0:root:/root:/bin/sh
user:x:1000:1000:User:/home/user:/bin/sh

/etc/group

root:x:0:
user:x:1000:

Loading the Kernel Module

The final step is, of course, the loading of the kernel module. I will be using the module from my Double Fetch section for this step.

First, we copy the .ko file to the filesystem root. Then we modify the init script to load it, and also set the UID of the loaded shell to 1000 (so we are not root!).

#!/bin/sh

insmod /double_fetch.ko
mknod /dev/double_fetch c 253 0
chmod 666 /dev/double_fetch

mount -t proc none /proc
mount -t sysfs none /sys

mknod -m 666 /dev/ttyS0 c 4 64

setsid /bin/cttyhack setuidgid 1000 /bin/sh

Here I am assuming that the major number of the double_fetch module is 253.

Why am I doing that?

If we load into a shell and run cat /proc/devices, we can see that double_fetch is loaded with major number 253 every time. I can't find any way to load this in without guessing the major number, so we're sticking with this for now - please get in touch if you find one!

Compiling a Different Kernel Version

If we want to compile a kernel version that is not the latest, we'll dump all the tags:

$ git fetch --tags

It takes ages to run, naturally. Once we do that, we can check out a specific version of choice:

$ git checkout v5.11

We then continue from there.

Some tags seem to not have the correct header files for compilation. Others, weirdly, compile kernels that build, but then never load in QEMU. I'm not quite sure why, to be frank.

Compiling, Customising and booting the Kernel

Instructions for compiling the kernel with your own settings, as well as compiling kernel modules for a specific kernel version.

Prerequisites

$ apt-get install flex bison libelf-dev

There may be other requirements, I just already had them. Check here for the full list.

The Kernel

Cloning

git clone https://github.com/torvalds/linux --depth=1

Use --depth 1 to only get the last commit.

Customise

Remove the current compilation configurations, as they are quite complex for our needs

$ cd linux
$ rm -f .config

Now we can create a minimal configuration, with almost all options disabled. A .config file is generated with the least features and drivers possible.

$ make allnoconfig
  YACC    scripts/kconfig/parser.tab.[ch]
  HOSTCC  scripts/kconfig/lexer.lex.o
  HOSTCC  scripts/kconfig/menu.o
  HOSTCC  scripts/kconfig/parser.tab.o
  HOSTCC  scripts/kconfig/preprocess.o
  HOSTCC  scripts/kconfig/symbol.o
  HOSTCC  scripts/kconfig/util.o
  HOSTLD  scripts/kconfig/conf
#
# configuration written to .config
#

We create a kconfig file with the options we want to enable. An example is the following:

CONFIG_64BIT=y
CONFIG_SMP=y
CONFIG_PRINTK=y
CONFIG_PRINTK_TIME=y

CONFIG_PCI=y

# We use an initramfs for busybox with elf binaries in it.
CONFIG_BLK_DEV_INITRD=y
CONFIG_RD_GZIP=y
CONFIG_BINFMT_ELF=y
CONFIG_BINFMT_SCRIPT=y

# This is for /dev file system.
CONFIG_DEVTMPFS=y

# For the power-down button (triggered by qemu's `system_powerdown` command).
CONFIG_INPUT=y
CONFIG_INPUT_EVDEV=y
CONFIG_INPUT_KEYBOARD=y

CONFIG_MODULES=y

CONFIG_KPROBES=n
CONFIG_LTO_NONE=y
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_EMBEDDED=n
CONFIG_TMPFS=y

CONFIG_RELOCATABLE=y
CONFIG_RANDOMIZE_BASE=y

CONFIG_USERFAULTFD=y

Explanation of Options

CONFIG_64BIT - compiles the kernel for 64-bit
CONFIG_SMP - simultaneous multiprocessing; allows the kernel to run on multiple cores
CONFIG_PRINTK, CONFIG_PRINTK_TIME - enables log messages and timestamps
CONFIG_PCI - enables support for loading an initial RAM disk
CONFIG_RD_GZIP - enables support for gzip-compressed initrd images
CONFIG_BINFMT_ELF - enables support for executing ELF binaries
CONFIG_BINFMT_SCRIPT - enables executing scripts with a shebang (#!) line
CONFIG_DEVTMPFS - Enables automatic creation of device nodes in /dev at boot time using devtmpfs
CONFIG_INPUT - enables support for the generic input layer required for input device handling
CONFIG_INPUT_EVDEV - enables support for the event device interface, which provides a unified input event framework
CONFIG_INPUT_KEYBOARD - enables support for keyboards
CONFIG_MODULES - enables support for loading and unloading kernel modules
CONFIG_KPROBES - disables support for kprobes, a kernel-based debugging mechanism. We disable this because ... TODO
CONFIG_LTO_NONE - disables Link Time Optimization (LTO) for kernel compilation. This is to allow better debugging
CONFIG_SERIAL_8250, CONFIG_SERIAL_8250_CONSOLE - TODO
CONFIG_EMBEDDED - disables optimizations/features for embedded systems
CONFIG_TMPFS - enables support for the tmpfs in-memory filesystem
CONFIG_RELOCATABLE - builds a relocatable kernel that can be loaded at different physical addresses
CONFIG_RANDOMIZE_BASE - enables KASLR support
CONFIG_USERFAULTFD - enables support for the userfaultfd system call, which allows handling of page faults in user space

In order to update the minimal .config with these options, we use the provided merge_config.sh script:

$ scripts/kconfig/merge_config.sh .config ../kconfig

Building

$ make -j4

That takes a while, but eventually builds a kernel in arch/x86/boot/bzImage. This is the same bzImage that you get in CTF challenges.

Kernel Modules

When we compile kernel modules for our own kernel, we use the following Makefile structure:

all:
    make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules

To compile it for a different kernel, all we do is change the -C flag to point to the newly-compiled kernel rather than the system's:

all:
    make -C /home/ir0nstone/linux M=$(PWD) modules

The module is now compiled for the specific kernel version!

Booting the Kernel in a Virtual Machine

References

Creating the File System and Executables

We now have a minimal kernel bzImage and a kernel module that is compiled for it. Now we need to create a minimal VM to run it in.

To do this, we use busybox, an executable that contains tiny versions of most Linux executables. This allows us to have all of the required programs, in as little space as possible.

We will download and extract busybox; you can find the latest version here.

$ curl https://busybox.net/downloads/busybox-1.36.1.tar.bz2 | tar xjf -

We also create an output folder for compiled versions.

$ mkdir busybox_compiled

Now compile it statically. We're going to use the menuconfig option, so we can make some choices.

$ cd busybox-1.36.1
$ make O=../busybox_compiled menuconfig

Now, make it with the new options

$ cd ../busybox_compiled
$ make -j
$ make install

Now we make the file system.

$ cd ..
$ mkdir initramfs
$ cd initramfs
$ mkdir -pv {bin,dev,sbin,etc,proc,sys/kernel/debug,usr/{bin,sbin},lib,lib64,mnt/root,root}
$ cp -av ../busybox_compiled/_install/* .
$ sudo cp -av /dev/{null,console,tty,sda1} dev/

The last thing missing is the classic init script, which gets run on system load. A provisional one works fine for now:

#!/bin/sh
 
mount -t proc none /proc
mount -t sysfs none /sys
 
echo -e "\nBoot took $(cut -d' ' -f1 /proc/uptime) seconds\n"
 
exec /bin/sh

Make it executable

$ chmod +x init

Finally, we're going to bundle it into a cpio archive, which is understood by QEMU.

find . -not -name *.cpio | cpio -o -H newc > initramfs.cpio

The -not -name *.cpio is there to prevent the archive from including itself
You can even compress the filesystem to a .cpio.gz file, which QEMU also recognises

If we want to extract the cpio archive (say, during a CTF) we can use this command:

$ cpio -i -F initramfs.cpio

Loading it with QEMU

Put bzImage and initramfs.cpio into the same folder. Write a short run.sh script that loads QEMU:

#!/bin/sh

qemu-system-x86_64 \
    -kernel bzImage \
    -initrd initramfs.cpio \
    -append "console=ttyS0 quiet loglevel=3 oops=panic" \
    -monitor /dev/null \
    -nographic \
    -no-reboot

Explanation of Flags

-kernel bzImage - sets the kernel to be our compiled bzImage
-initrd initramfs.cpio - provide the file system
-append ... - basic features; in the future, this flag is also used to set protections
- console=ttyS0 - Directs kernel messages to the first serial port (ttyS0)
- quiet - Only showing critical messages from the kernel
- loglevel=3 - Only show error messages and higher-priority messages
- oops=panic - Make the kernel panic immediately on an oops (kernel error)
-monitor /dev/null - Disable the QEMU monitor
-nographic - Disable GUI, operate in headless mode (faster)
no-reboot - Do not automatically restart the VM when encountering a problem (useful for debugging and working out why it crashes, as the crash logs will stay).

Once we make this executable and run it, we get loaded into a VM!

User Accounts

Right now, we have a minimal linux kernel we can boot, but if we try and work out who we are, it doesn't act quite as we expect it to:

~ # whoami
whoami: unknown uid 0

This is because /etc/passwd and /etc/group don't exist, so we can just create those!

/etc/passwd

root:x:0:0:root:/root:/bin/sh
user:x:1000:1000:User:/home/user:/bin/sh

/etc/group

root:x:0:
user:x:1000:

Loading the Kernel Module

The final step is, of course, the loading of the kernel module. I will be using the module from my Double Fetch section for this step.

First, we copy the .ko file to the filesystem root. Then we modify the init script to load it, and also set the UID of the loaded shell to 1000 (so we are not root!).

#!/bin/sh

insmod /double_fetch.ko
mknod /dev/double_fetch c 253 0
chmod 666 /dev/double_fetch

mount -t proc none /proc
mount -t sysfs none /sys

mknod -m 666 /dev/ttyS0 c 4 64

setsid /bin/cttyhack setuidgid 1000 /bin/sh

Here I am assuming that the major number of the double_fetch module is 253.

Why am I doing that?

Compiling a Different Kernel Version

If we want to compile a kernel version that is not the latest, we'll dump all the tags:

$ git fetch --tags

It takes ages to run, naturally. Once we do that, we can check out a specific version of choice:

$ git checkout v5.11

We then continue from there.

Some tags seem to not have the correct header files for compilation. Others, weirdly, compile kernels that build, but then never load in QEMU. I'm not quite sure why, to be frank.