This setup has been tested on Ubuntu 20.04.

The Buildroot build is already broken on Ubuntu 21.04 onwards: #155, so just do this from inside a 20.04 Docker instead as shown in the Docker host setup setup. We could fix the build on Ubuntu 21.04, but it will break again inevitably later on.

For other host operating systems see: [supported-hosts].

Reserve 12Gb of disk and run:

You don’t need to clone recursively even though we have .git submodules: download-dependencies fetches just the submodules that you need for this build to save time.

If something goes wrong, see: [common-build-issues] and use our issue tracker: https://github.com/cirosantilli/linux-kernel-module-cheat/issues

The initial build will take a while (30 minutes to 2 hours) to clone and build, see [benchmark-builds] for more details.

If you don’t want to wait, you could also try the following faster but much more limited methods:

• Prebuilt setup

• Host kernel module setup

but you will soon find that they are simply not enough if you anywhere near serious about systems programming.

After ./run, QEMU opens up leaving you in the /lkmc/ directory, and you can start playing with the kernel modules inside the simulated system:

This should print to the screen:

which are printk messages from init and cleanup methods of those modules.

Sources:

• kernel_modules/hello.c

• kernel_modules/hello2.c

Quit QEMU with:

See also: Section 14.1.1, “Quit QEMU from text mode”.

All available modules can be found in the kernel_modules directory.

It is super easy to build for different CPU architectures, just use the --arch option:

To avoid typing --arch aarch64 many times, you can set the default arch as explained at: [default-command-line-arguments]

I now urge you to read the following sections which contain widely applicable information:

• [run-command-after-boot]

• [clean-the-build]

• Build the documentation

• Linux kernel

  • printk

  • Kernel command line parameters

Once you use GDB step debug and tmux, your terminal will look a bit like this:

Besides a seamless initial build, this project also aims to make it effortless to modify and rebuild several major components of the system, to serve as an awesome development setup.

2.2.2.1. Your first Linux kernel hack

vim submodules/linux/init/main.c

pr_info("I'VE HACKED THE LINUX KERNEL!!!");

./build-linux
./run

<6>[    0.000000] I'VE HACKED THE LINUX KERNEL!!!

./build --dry-run

2.2.2.2. Your first kernel module hack

pr_info("hello init hacked\n");

./build-modules

insmod /mnt/9p/out_rootfs_overlay/lkmc/hello.ko

ls "$(./getvar out_rootfs_overlay_dir)"

./build-modules
./build-buildroot
./run --eval-after 'insmod hello.ko'

./build --dry-run

./run --kvm

2.2.2.3. Your first glibc hack

./build-buildroot -- glibc-reconfigure

diff --git a/libio/ioputs.c b/libio/ioputs.c
index 706b20b492..23185948f3 100644
--- a/libio/ioputs.c
+++ b/libio/ioputs.c
@@ -38,8 +38,9 @@ _IO_puts (const char *str)
   if ((_IO_vtable_offset (_IO_stdout) != 0
        || _IO_fwide (_IO_stdout, -1) == -1)
       && _IO_sputn (_IO_stdout, str, len) == len
+      && _IO_sputn (_IO_stdout, " hacked", 7) == 7
       && _IO_putc_unlocked ('\n', _IO_stdout) != EOF)
-    result = MIN (INT_MAX, len + 1);
+    result = MIN (INT_MAX, len + 1 + 7);

   _IO_release_lock (_IO_stdout);
   return result;

./run --eval-after './c/hello.out'

hello hacked

./run --userland userland/c/hello.c

2.2.2.4. Your first Binutils hack

./build-userland

binutils_hack.c:8: Error: no such instruction: `myinc %rax'

vim submodules/binutils-gdb/opcodes/i386-tbl.h

diff --git a/opcodes/i386-tbl.h b/opcodes/i386-tbl.h
index af583ce578..3cc341f303 100644
--- a/opcodes/i386-tbl.h
+++ b/opcodes/i386-tbl.h
@@ -1502,6 +1502,19 @@ const insn_template i386_optab[] =
     { { { 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
 	  0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0,
 	  1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0 } } } },
+  { "myinc", 1, 0xfe, 0x0, 1,
+    { { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 } },
+    { 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+      0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,
+      0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+      0, 0, 0, 0, 0, 0 },
+    { { { 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+	  0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0,
+	  1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0 } } } },
   { "sub", 2, 0x28, None, 1,
     { { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,

./build-buildroot -- host-binutils-rebuild
./build-userland --static
./run --static --userland userland/arch/x86_64/binutils_hack.c

2.2.2.5. Your first GCC hack

./build-userland --static
./run --static --userland userland/c/gcc_hack.c

i = 2
j = 0

vim submodules/gcc/gcc/c/c-parser.c

diff --git a/gcc/c/c-parser.c b/gcc/c/c-parser.c
index 101afb8e35f..89535d1759a 100644
--- a/gcc/c/c-parser.c
+++ b/gcc/c/c-parser.c
@@ -8529,7 +8529,7 @@ c_parser_postfix_expression_after_primary (c_parser *parser,
 		expr.original_type = DECL_BIT_FIELD_TYPE (field);
 	    }
 	  break;
-	case CPP_PLUS_PLUS:
+	case CPP_MINUS_MINUS:
 	  /* Postincrement.  */
 	  start = expr.get_start ();
 	  finish = c_parser_peek_token (parser)->get_finish ();
@@ -8548,7 +8548,7 @@ c_parser_postfix_expression_after_primary (c_parser *parser,
 	  expr.original_code = ERROR_MARK;
 	  expr.original_type = NULL;
 	  break;
-	case CPP_MINUS_MINUS:
+	case CPP_PLUS_PLUS:
 	  /* Postdecrement.  */
 	  start = expr.get_start ();
 	  finish = c_parser_peek_token (parser)->get_finish ();

./build-buildroot -- host-gcc-final-rebuild
./build-userland --static
./run --static --userland userland/c/gcc_hack.c

i = 2
j = 0

What QEMU and Buildroot are:

• [introduction-to-buildroot]

• Introduction to QEMU

This is our reference setup, and the best supported one, use it unless you have good reason not to.

It was historically the first one we did, and all sections have been tested with this setup unless explicitly noted.

Read the following sections for further introductory material:

• Introduction to QEMU

• [introduction-to-buildroot]

./run --dry-run --userland userland/c/hello.c

+ /path/to/linux-kernel-module-cheat/out/qemu/default/opt/x86_64-linux-user/qemu-x86_64 \
  -L /path/to/linux-kernel-module-cheat/out/buildroot/build/default/x86_64/target \
  -r 5.2.1 \
  -seed 0 \
  -trace enable=load_file,file=/path/to/linux-kernel-module-cheat/out/run/qemu/x86_64/0/trace.bin \
  -cpu max \
  /path/to/linux-kernel-module-cheat/out/userland/default/x86_64/c/hello.out \
;

cat "$(./getvar run_dir)/run.sh"

cat out/run.sh

This setup is like the QEMU Buildroot setup, but it uses gem5 instead of QEMU as a system simulator.

QEMU tries to run as fast as possible and give correct results at the end, but it does not tell us how many CPU cycles it takes to do something, just the number of instructions it ran. This kind of simulation is known as functional simulation.

The number of instructions executed is a very poor estimator of performance because in modern computers, a lot of time is spent waiting for memory requests rather than the instructions themselves.

gem5 on the other hand, can simulate the system in more detail than QEMU, including:

• simplified CPU pipeline

• caches

• DRAM timing

and can therefore be used to estimate system performance, see: Section 24.2, “gem5 run benchmark” for an example.

The downside of gem5 much slower than QEMU because of the greater simulation detail.

See gem5 vs QEMU for a more thorough comparison.

For the most part, if you just add the --emulator gem5 option or *-gem5 suffix to all commands and everything should magically work.

If you haven’t built Buildroot yet for QEMU Buildroot setup, you can build from the beginning with:

If you have already built previously, don’t be afraid: gem5 and QEMU use almost the same root filesystem and kernel, so ./build will be fast.

Remember that the gem5 boot is considerably slower than QEMU since the simulation is more detailed.

If you have a relatively new GCC version and the gem5 build fails on your machine, see: [gem5-build-broken-on-recent-compiler-version].

To get a terminal, either open a new shell and run:

You can quit the shell without killing gem5 by typing tilde followed by a period:

If you are inside tmux, which I highly recommend, you can both run gem5 stdout and open the guest terminal on a split window with:

See also: Section 3.3.1, “tmux gem5”.

At the end of boot, it might not be very clear that you have the shell since some printk messages may appear in front of the prompt like this:

but if you look closely, the PS1 prompt marker # is there already, just hit enter and a clear prompt line will appear.

If you forgot to open the shell and gem5 exit, you can inspect the terminal output post-mortem at:

More gem5 information is present at: Section 24, “gem5”

Good next steps are:

• gem5 run benchmark: how to run a benchmark in gem5 full system, including how to boot Linux, checkpoint and restore to skip the boot on a fast CPU

• m5out directory: understand the output files that gem5 produces, which contain information about your run

• m5ops: magic guest instructions used to control gem5

• [add-new-files-to-the-buildroot-image]: how to add your own files to the image if you have a benchmark that we don’t already support out of the box (also send a pull request!)

sudo apt-get install docker
./setup
./run-docker create && \
./run-docker sh -- ./build --download-dependencies qemu-buildroot
./run-docker

./run

./run-docker

./run-docker sh -- ./run-gdb start_kernel

./run --graphic --vnc

sudo apt-get install vinagre
./vnc

This setup uses prebuilt binaries that we upload to GitHub from time to time.

We don’t currently provide a full prebuilt because it would be too big to host freely, notably because of the cross toolchain.

Our prebuilts currently include:

• QEMU Buildroot setup binaries

  • Linux kernel

  • root filesystem

• Baremetal setup binaries for QEMU

For more details, see our our release procedure.

Advantage of this setup: saves time and disk space on the initial install, which is expensive in largely due to building the toolchain.

The limitations are severe however:

• can’t GDB step debug the kernel, since the source and cross toolchain with GDB are not available. Buildroot cannot easily use a host toolchain: [prebuilt-toolchain].

  Maybe we could work around this by just downloading the kernel source somehow, and using a host prebuilt GDB, but we felt that it would be too messy and unreliable.

• you won’t get the latest version of this repository. Our [travis] attempt to automate builds failed, and storing a release for every commit would likely make GitHub mad at us anyway.

• gem5 is not currently supported. The major blocking point is how to avoid distributing the kernel images twice: once for gem5 which uses vmlinux, and once for QEMU which uses arch/* images, see also:

  • #79

  • vmlinux vs bzImage vs zImage vs Image.

This setup might be good enough for those developing simulators, as that requires less image modification. But once again, if you are serious about this, why not just let your computer build the full featured setup while you take a coffee or a nap? :-)

Checkout to the latest tag and use the Ubuntu packaged QEMU to boot Linux:

You have to checkout to the latest tag to ensure that the scripts match the release format: https://stackoverflow.com/questions/1404796/how-to-get-the-latest-tag-name-in-current-branch-in-git

This is known not to work for aarch64 on an Ubuntu 16.04 host with QEMU 2.5.0, presumably because QEMU is too old, the terminal does not show any output. I haven’t investigated why.

Or to run a baremetal example instead:

Be saner and use our custom built QEMU instead:

To build the kernel modules as in Your first kernel module hack do:

TODO: for now the only way to test those modules out without building Buildroot is with 9p, since we currently rely on Buildroot to manipulate the root filesystem.

Command explanation:

• modules_prepare does the minimal build procedure required on the kernel for us to be able to compile the kernel modules, and is way faster than doing a full kernel build. A full kernel build would also work however.

• --gcc-which host selects your host Ubuntu packaged GCC, since you don’t have the Buildroot toolchain

• --no-modules-install is required otherwise the make modules_install target we run by default fails, since the kernel wasn’t built

To modify the Linux kernel, build and use it as usual:

./build-modules --host

./build-modules --host -- hello hello2

./build-modules \
  --gcc-which host \
  --host \
  -- \
  kernel_modules/hello.c \
  kernel_modules/hello2.c \
;

cd "$(./getvar kernel_modules_source_dir)"
mv broken.c broken.c~

cd "$(./getvar kernel_modules_build_host_subdir)"
sudo insmod hello.ko

# Our module is there.
sudo lsmod | grep hello

# Last message should be: hello init
dmesg -T

sudo rmmod hello

# Last message should be: hello exit
dmesg -T

# Not present anymore
sudo lsmod | grep hello

Minimal host build system example:

In order to test the kernel and emulators, userland content in the form of executables and scripts is of course required, and we store it mostly under:

• userland/

• [rootfs-overlay]

• [add-new-buildroot-packages]

When we started this repository, it only contained content that interacted very closely with the kernel, or that had required performance analysis.

However, we soon started to notice that this had an increasing overlap with other userland test repositories: we were duplicating build and test infrastructure and even some examples.

Therefore, we decided to consolidate other userland tutorials that we had scattered around into this repository.

Notable userland content included / moving into this repository includes:

• [userland-assembly]

• [c]

• [cpp]

• [posix]

• [algorithms]

There are several ways to run our [userland-content], notably:

• natively on the host as shown at: Section 2.8.2.1, “Userland setup getting started natively”

  Can only run examples compatible with your host CPU architecture and OS, but has the fastest setup and runtimes.

• from user mode simulation with:

  • the host prebuilt toolchain: Section 2.8.2.2, “Userland setup getting started with prebuilt toolchain and QEMU user mode”

  • the Buildroot toolchain you built yourself: Section 11.1, “QEMU user mode getting started”

  This setup:

  • can run most examples, including those for other CPU architectures, with the notable exception of examples that rely on kernel modules

  • can run reproducible approximate performance experiments with gem5, see e.g. [bst-vs-heap-vs-hashmap]

• from full system simulation as shown at: Section 2.2.1, “QEMU Buildroot setup getting started”.

  This is the most reproducible and controlled environment, and all examples work there. But also the slower one to setup.

2.8.2.1. Userland setup getting started natively

sudo apt-get install gcc
cd userland
./build c/hello
./c/hello.out
./build --clean

cd userland
./build c
./test c

cd userland/c
./build
./test

cd linux-kernel-module-cheat
./setup
./build --download-dependencies userland-host

./build --package-all
./test --package-all

./build --ccflags='-foptimize-sibling-calls -foptimize-strlen' --force-rebuild

./build --optimization-level 3 --force-rebuild

./build-userland-in-tree
./build-userland-in-tree userland/c
./build-userland-in-tree userland/c/hello.c

./build-userland --gcc-which host --in-tree userland/c

./build-userland --gcc-which host --userland-build-id host
./run --emulator native --userland userland/c/hello.c --userland-build-id host

./run --debug-vm --emulator native --userland userland/c/hello.c --userland-build-id host

2.8.2.2. Userland setup getting started with prebuilt toolchain and QEMU user mode

sudo apt-get install gcc-aarch64-linux-gnu qemu-system-aarch64
./build-userland \
  --arch aarch64 \
  --gcc-which host \
  --userland-build-id host \
;
./run \
  --arch aarch64 \
  --qemu-which host \
  --userland-build-id host \
  --userland userland/c/command_line_arguments.c \
  --cli-args 'asdf "qw er"' \
;

./test-executables \
  --arch aarch64 \
  --gcc-which host \
  --qemu-which host \
  --userland-build-id host \
;

./run \
  --arch aarch64 \
  --gdb \
  --gcc-which host \
  --qemu-which host \
  --userland-build-id host \
  --userland userland/c/command_line_arguments.c \
  --cli-args 'asdf "qw er"' \
;

2.8.2.3. Userland setup getting started full system

./build-userland

/mnt/9p/out_rootfs_overlay/c/hello.out

./build-buildroot

./hello.out

This setup does not use the Linux kernel nor Buildroot at all: it just runs your very own minimal OS.

x86_64 is not currently supported, only arm and aarch64: I had made some x86 bare metal examples at: https://github.com/cirosantilli/x86-bare-metal-examples but I’m lazy to port them here now. Pull requests are welcome.

The main reason this setup is included in this project, despite the word "Linux" being on the project name, is that a lot of the emulator boilerplate can be reused for both use cases.

This setup allows you to make a tiny OS and that runs just a few instructions, use it to fully control the CPU to better understand the simulators for example, or develop your own OS if you are into that.

You can also use C and a subset of the C standard library because we enable Newlib by default. See also:

• https://electronics.stackexchange.com/questions/223929/c-standard-libraries-on-bare-metal/400077#400077

• https://stackoverflow.com/questions/13063055/does-a-libc-os-exist/59771531#59771531

Our C bare-metal compiler is built with crosstool-NG. If you have already built Buildroot previously, you will end up with two GCCs installed. Unfortunately I don’t see a solution for this, since we need separate toolchains for Newlib on baremetal and glibc on Linux: https://stackoverflow.com/questions/38956680/difference-between-arm-none-eabi-and-arm-linux-gnueabi/38989869#38989869

Every .c file inside baremetal/ and .S file inside baremetal/arch/<arch>/ generates a separate baremetal image.

For example, to run baremetal/arch/aarch64/dump_regs.c in QEMU do:

And the terminal prints the values of certain system registers. This example prints registers that are only accessible from EL1 or higher, and thus could not be run in userland.

In addition to the examples under baremetal/, several of the userland examples can also be run in baremetal! This is largely due to the awesomeness of Newlib.

The examples that work include most C examples that don’t rely on complicated syscalls such as threads, and almost all the [userland-assembly] examples.

The exact list of userland programs that work in baremetal is specified in [path-properties] with the baremetal property, but you can also easily find it out with a baremetal test dry run:

For example, we can run the C hello world userland/c/hello.c simply as:

and that outputs to the serial port the string:

which QEMU shows on the host terminal.

To modify a baremetal program, simply edit the file, e.g.

and rebuild:

./build qemu-baremetal that we run previously is only needed for the initial build. That script calls build-baremetal for us, in addition to building prerequisites such as QEMU and crosstool-NG.

./build-baremetal uses crosstool-NG, and so it must be preceded by build-crosstool-ng, which ./build qemu-baremetal also calls.

Now let’s run userland/arch/aarch64/add.S:

This time, the terminal does not print anything, which indicates success: if you look into the source, you will see that we just have an assertion there.

You can see a sample assertion fail in userland/c/assert_fail.c:

and the terminal contains:

and the exit status of our script is 1:

You can run all the baremetal examples in one go and check that all assertions passed with:

To use gem5 instead of QEMU do:

and then as usual open a shell with:

Or as usual, tmux users can do both in one go with:

TODO: the carriage returns are a bit different than in QEMU, see: [gem5-baremetal-carriage-return].

Note that ./build-baremetal requires the --emulator gem5 option, and generates separate executable images for both, as can be seen from:

This is unlike the Linux kernel that has a single image for both QEMU and gem5:

The reason for that is that on baremetal we don’t parse the device tress from memory like the Linux kernel does, which tells the kernel for example the UART address, and many other system parameters.

gem5 also supports the RealViewPBX machine, which represents an older hardware compared to the default VExpress_GEM5_V1:

see also: Section 24.18, “gem5 ARM platforms”.

This generates yet new separate images with new magic constants:

But just stick to newer and better VExpress_GEM5_V1 unless you have a good reason to use RealViewPBX.

When doing baremetal programming, it is likely that you will want to learn userland assembly first, see: [userland-assembly].

For more information on baremetal, see the section: [baremetal].

The following subjects are particularly important:

• Tracing

• [baremetal-gdb-step-debug]

asciidoctor README.adoc
xdg-open README.html

./setup
./build --download-dependencies docs

./build-doc

xdg-open out/README.html

./run --gdb-wait

./run-gdb start_kernel

./run-gdb init/main.c:1088

list
next
continue

Just don’t forget to pass --arch to ./run-gdb, e.g.:

and:

https://stackoverflow.com/questions/29151235/how-to-de-optimize-the-linux-kernel-to-and-compile-it-with-o0

O=0 is an impossible dream, O=2 being the default.

So get ready for some weird jumps, and <value optimized out> fun. Why, Linux, why.

The -O level of some other userland content can be controlled as explained at: [optimization-level-of-a-build].

./run

./count.sh

./run-gdb

Ctrl-C
break __x64_sys_write
continue
continue
continue

rbreak .*sys_write

tmux

./run --gdb

./run --gdb-wait --tmux --tmux-args start_kernel

man tmux

./run --gdb

./run --gdb-wait
./run-gdb start_kernel

./run --gdb --tmux-args=--no-continue

If you are using gem5 instead of QEMU, --tmux has a different effect by default: it opens the gem5 terminal instead of the debugger:

To open a new pane with GDB instead of the terminal, use:

which is equivalent to:

--tmux-program implies --tmux, so we can just write:

If you also want to see both GDB and the terminal with gem5, then you will need to open a separate shell manually as usual with ./gem5-shell.

From inside tmux, you can create new terminals on a new window with Ctrl-B C split a pane yet again vertically with Ctrl-B % or horizontally with Ctrl-B ".

./run

insmod timer.ko

./run-gdb

scanning for modules in /root/linux-kernel-module-cheat/out/kernel_modules/x86_64/kernel_modules
loading @0xffffffffc0000000: /root/linux-kernel-module-cheat/out/kernel_modules/x86_64/kernel_modules/timer.ko

break lkmc_timer_callback
continue
continue
continue

TODO on arm 51e31cdc2933a774c2a0dc62664ad8acec1d2dbe it does not always work, and lx-symbols fails with the message:

Can’t reproduce on x86_64 and aarch64 are fine.

It is kind of random: if you just insmod manually and then immediately ./run-gdb --arch arm, then it usually works.

But this fails most of the time: shell 1:

shell 2:

then hit Ctrl-C on shell 2, and voila.

Then:

says that the load address is:

so it is close to the failing 0xbf0000cc.

readelf:

does not give any interesting hits at cc, no symbol was placed that far.

TODO find a more convenient method. We have working methods, but they are not ideal.

This is not very easy, since by the time the module finishes loading, and lx-symbols can work properly, module_init has already finished running!

Possibly asked at:

• https://stackoverflow.com/questions/37059320/debug-a-kernel-module-being-loaded

• https://stackoverflow.com/questions/11888412/debug-the-init-module-call-of-a-linux-kernel-module

3.4.2.1. GDB module_init step into it

./run

./run-gdb do_one_initcall

833         ret = fn();

./run-gdb init/main.c:833

3.4.2.2. GDB module_init calculate entry address

./run --eval-after './pr_debug.sh;insmod fops.ko;./linux/poweroff.out'

0xffffffffc0000000 .text

./run-toolchain readelf -- \
  -s "$(./getvar kernel_modules_build_subdir)/fops.ko" | \
  grep myinit

    30: 0000000000000240    43 FUNC    LOCAL  DEFAULT    2 myinit

0xffffffffc0000000 + 0x240 = 0xffffffffc0000240

./run --eval 'insmod fops.ko;./linux/poweroff.out' --gdb-wait

./run-gdb '*0xffffffffc0000240'

3.4.2.3. GDB module_init break at the end of sys_init_module

b sys_finit_module

n

b do_init_module

fin

3.4.2.4. GDB module_init add trap instruction

asm( " int $3");

Useless, but a good way to show how hardcore you are. Disable lx-symbols with:

From inside guest:

as mentioned at:

• https://stackoverflow.com/questions/6384605/how-to-get-address-of-a-kernel-module-loaded-using-insmod/6385818

• https://unix.stackexchange.com/questions/194405/get-base-address-and-size-of-a-loaded-kernel-module

This will give a line of form:

And then tell GDB where the module was loaded with:

Alternatively, if the module panics before you can read /proc/modules, there is a pr_debug which shows the load address:

And then search for a line of type:

Tested on 4f4749148273c282e80b58c59db1b47049e190bf + 1.

./run-gdb --no-continue

TODO https://stackoverflow.com/questions/2589845/what-are-the-first-operations-that-the-linux-kernel-executes-on-boot

As mentioned at: GDB step debug early boot, the very first kernel instructions executed appear to be placed into memory at a different location than that of the kernel ELF section.

As a result, we are unable to break on early symbols such as:

gem5 ExecAll trace format>> however does show the right symbols however! This could be because gem5 uses vmlinux to boot, which QEMU uses the compressed version, and as mentioned on the Stack Overflow answer, the entry point is actually a tiny decompresser routine.

I also tried to hack run-gdb with:

and no I do have the symbols from arch/arm/boot/compressed/vmlinux', but the breaks still don’t work.

v4.19 also added a CONFIG_HAVE_KERNEL_UNCOMPRESSED=y option for having the kernel uncompressed which could make following the startup easier, but it is only available on s390. aarch64 however is already uncompressed by default, so might be the easiest one. See also: Section 17.20.1, “vmlinux vs bzImage vs zImage vs Image”.

You then need the associated KERNEL_UNCOMPRESSED to enable it if available:

3.5.1.1. arm64 secondary CPU entry point

static int smp_spin_table_cpu_prepare(unsigned int cpu)
{
	__le64 __iomem *release_addr;

	if (!cpu_release_addr[cpu])
		return -ENODEV;

	/*
	 * The cpu-release-addr may or may not be inside the linear mapping.
	 * As ioremap_cache will either give us a new mapping or reuse the
	 * existing linear mapping, we can use it to cover both cases. In
	 * either case the memory will be MT_NORMAL.
	 */
	release_addr = ioremap_cache(cpu_release_addr[cpu],
				     sizeof(*release_addr));
	if (!release_addr)
		return -ENOMEM;

	/*
	 * We write the release address as LE regardless of the native
	 * endianess of the kernel. Therefore, any boot-loaders that
	 * read this address need to convert this address to the
	 * boot-loader's endianess before jumping. This is mandated by
	 * the boot protocol.
	 */
	writeq_relaxed(__pa_symbol(secondary_holding_pen), release_addr);
	__flush_dcache_area((__force void *)release_addr,
			    sizeof(*release_addr));

	/*
	 * Send an event to wake up the secondary CPU.
	 */
	sev();

static int smp_spin_table_cpu_init(unsigned int cpu)
{
	struct device_node *dn;
	int ret;

	dn = of_get_cpu_node(cpu, NULL);
	if (!dn)
		return -ENODEV;

	/*
	 * Determine the address from which the CPU is polling.
	 */
	ret = of_property_read_u64(dn, "cpu-release-addr",
				   &cpu_release_addr[cpu]);

start_kernel is the first C function to be executed basically: https://stackoverflow.com/questions/18266063/does-kernel-have-main-function/33422401#33422401

For the earlier arch-specific entry point, see: Linux kernel entry point.

When booting Linux on a slow emulator like gem5, what you observe is that:

• first nothing shows for a while

• then at once, a bunch of message lines show at once followed on aarch64 Linux 5.4.3 by:

  [    0.081311] printk: console [ttyAMA0] enabled

This means of course that all the previous messages had been generated earlier and stored, but were only printed to the terminal once the terminal itself was enabled.

Notably for example the very first message:

happens very early in the boot process.

If you get a failure before that, it will be hard to see the print messages.

One possible solution is to parse the dmesg buffer, gem5 actually implements that: gem5 m5out/system.workload.dmesg file.

This is the userland debug setup most likely to work, since at init time there is only one userland executable running.

For executables from the userland/ directory such as userland/posix/count.c:

• Shell 1:

  ./run --gdb-wait --kernel-cli 'init=/lkmc/posix/count.out'

• Shell 2:

  ./run-gdb --userland userland/posix/count.c main

  Alternatively, we could also pass the full path to the executable:

  ./run-gdb --userland "$(./getvar userland_build_dir)/posix/count.out" main

  Path resolution is analogous to that of ./run --baremetal.

Then, as soon as boot ends, we are left inside a debug session that looks just like what gdbserver would produce.

BusyBox custom init process:

• Shell 1:

  ./run --gdb-wait --kernel-cli 'init=/bin/ls'

• Shell 2:

  ./run-gdb --userland "$(./getvar buildroot_build_build_dir)"/busybox-*/busybox ls_main

This follows BusyBox' convention of calling the main for each executable as <exec>_main since the busybox executable has many "mains".

BusyBox default init process:

• Shell 1:

  ./run --gdb-wait

• Shell 2:

  ./run-gdb --userland "$(./getvar buildroot_build_build_dir)"/busybox-*/busybox init_main

init cannot be debugged with gdbserver without modifying the source, or else /sbin/init exits early with:

Non-init process:

• Shell 1:

  ./run --gdb-wait

• Shell 2:

  ./run-gdb --userland userland/linux/rand_check.c main

• Shell 1 after the boot finishes:

  ./linux/rand_check.out

This is the least reliable setup as there might be other processes that use the given virtual address.

3.6.3.1. GDB step debug userland non-init without --gdb-wait

Cannot access memory at address 0x10604

./run-toolchain --arch arm readelf -- \
  -s "$(./getvar --arch arm userland_build_dir)/linux/myinsmod.out" | \
  grep main

info line main

000105fc

b *0x000105fc

>>> call printk(0, "asdf")
Could not fetch register "orig_rax"; remote failure reply 'E14'
>>> b printk
Breakpoint 2 at 0xffffffff81091bca: file kernel/printk/printk.c, line 1824.
>>> call fdget_pos(fd)
No symbol "fdget_pos" in current context.
>>> b fdget_pos
Breakpoint 3 at 0xffffffff811615e3: fdget_pos. (9 locations)
>>>

581 SYSCALL_DEFINE3(write, unsigned int, fd, const char __user *, buf,
582         size_t, count)
583 {
584     struct fd f = fdget_pos(fd);

Could not fetch register "orig_rax"; remote failure reply 'E14'

fin

./run --cpus 2 --eval-after './linux/sched_getaffinity.out'

sched_getaffinity = 1 1
sched_getcpu = 1
sched_getaffinity = 1 0
sched_getcpu = 0

./build-buildroot \
  --config 'BR2_PACKAGE_UTIL_LINUX=y' \
  --config 'BR2_PACKAGE_UTIL_LINUX_SCHEDUTILS=y' \
;
./run --eval-after 'taskset -c 1,1 ./linux/sched_getaffinity.out'

sched_getaffinity = 0 1
sched_getcpu = 1
sched_getaffinity = 1 0
sched_getcpu = 0

./run \
  --cpus 2 \
  --eval-after 'i=0; while true; do taskset -c $i,$i ./linux/sched_getaffinity.out; i=$((! $i)); done' \
  --gdb-wait \
;

./run-gdb --userland "$(./getvar userland_build_dir)/linux/sched_getaffinity.out" main

(gdb) info threads
  Id   Target Id         Frame
* 1    Thread 1 (CPU#0 [running]) main () at sched_getaffinity.c:30
  2    Thread 2 (CPU#1 [halted ]) native_safe_halt () at ./arch/x86/include/asm/irqflags.h:55
(gdb) c
(gdb) info threads
  Id   Target Id         Frame
  1    Thread 1 (CPU#0 [halted ]) native_safe_halt () at ./arch/x86/include/asm/irqflags.h:55
* 2    Thread 2 (CPU#1 [running]) main () at sched_getaffinity.c:30
(gdb) c

./run --cpus 2 --eval-after './linux/sched_getaffinity_threads.out'

./run-gdb --userland "$(./getvar userland_build_dir)/linux/sched_getaffinity_threads.out"

b main_thread_0

lx-dmesg

lx-cmdline

lx-fdtdump
pwd

lx-lsmod

Address            Module                  Size  Used by
0xffffff80006d0000 hello                  16384  0

List all processes:

Sample output:

The second and third fields are obviously PID and process name.

The first one is more interesting, and contains the address of the task_struct in memory.

This can be confirmed with:

which contains the correct PID for all threads I’ve tried:

TODO get the PC of the kthreads: https://stackoverflow.com/questions/26030910/find-program-counter-of-process-in-kernel Then we would be able to see where the threads are stopped in the code!

On ARM, I tried:

but task_pt_regs is a #define and GDB cannot see defines without -ggdb3: https://stackoverflow.com/questions/2934006/how-do-i-print-a-defined-constant-in-gdb which are apparently not set?

Bibliography:

• https://stackoverflow.com/questions/9561546/thread-aware-gdb-for-kernel

• https://wiki.linaro.org/LandingTeams/ST/GDB

• https://events.static.linuxfound.org/sites/events/files/slides/Debugging%20the%20Linux%20Kernel%20with%20GDB.pdf presentation: https://www.youtube.com/watch?v=pqn5hIrz3A8

3.10.1.1. CONFIG_PID_IN_CONTEXTIDR

./build-linux --arch aarch64 --linux-build-id CONFIG_PID_IN_CONTEXTIDR --config 'CONFIG_PID_IN_CONTEXTIDR=y'
# Checkpoint run.
./run --arch aarch64 --emulator gem5 --linux-build-id CONFIG_PID_IN_CONTEXTIDR --eval './gem5.sh'
# Trace run.
./run \
  --arch aarch64 \
  --emulator gem5 \
  --gem5-readfile 'posix/getpid.out; posix/getpid.out' \
  --gem5-restore 1 \
  --linux-build-id CONFIG_PID_IN_CONTEXTIDR \
  --trace FmtFlag,ExecAll,-ExecSymbol \
;

pid=44
pid=45

cut -d' ' -f4 "$(./getvar --arch aarch64 --emulator gem5 trace_txt_file)" | uniq -c

  97227 A39
 147476 A38
 222052 A40
      1 terminal
1117724 A40
  27529 A31
  43868 A40
  27487 A31
 138349 A40
  13781 A38
 231246 A40
  25536 A38
  28337 A40
 214799 A38
 963561 A41
  92603 A38
  27511 A31
 224384 A38
 564949 A42
 182360 A38
 729009 A43
   8398 A23
  20200 A10
 636848 A43
 187995 A44
  27529 A31
  70071 A44
  16981 A0
 623806 A44
  16981 A0
 139319 A44
  24487 A0
 174986 A44
  25420 A0
  89611 A44
  16981 A0
 183184 A44
  24728 A0
  89608 A44
  17226 A0
 899075 A44
  24974 A0
 250608 A44
 137700 A43
1497997 A45
 227485 A43
 138147 A38
 482646 A46

./run --arch aarch64 --emulator gem5 --baremetal baremetal/arch/aarch64/contextidr_el1.c --trace-insts-stdout

  31500: system.cpu: A0 T0 : @main+12    :   ldr   x0, [sp, #12]      : MemRead :  D=0x0000000000000001 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsLoad)
  32000: system.cpu: A1 T0 : @main+16    :   msr   contextidr_el1, x0 : IntAlu :  D=0x0000000000000001  flags=(IsInteger|IsSerializeAfter|IsNonSpeculative)
  32500: system.cpu: A1 T0 : @main+20    :   ldr   x0, [sp, #12]      : MemRead :  D=0x0000000000000001 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsLoad)
  33000: system.cpu: A1 T0 : @main+24    :   add   w0, w0, #1         : IntAlu :  D=0x0000000000000002  flags=(IsInteger)
  33500: system.cpu: A1 T0 : @main+28    :   str   x0, [sp, #12]      : MemWrite :  D=0x0000000000000002 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsStore)
  34000: system.cpu: A1 T0 : @main+32    :   ldr   x0, [sp, #12]      : MemRead :  D=0x0000000000000002 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsLoad)
  34500: system.cpu: A1 T0 : @main+36    :   subs   w0, #9            : IntAlu :  D=0x0000000000000000  flags=(IsInteger)
  35000: system.cpu: A1 T0 : @main+40    :   b.le   <main+12>         : IntAlu :   flags=(IsControl|IsDirectControl|IsCondControl)
  35500: system.cpu: A1 T0 : @main+12    :   ldr   x0, [sp, #12]      : MemRead :  D=0x0000000000000002 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsLoad)
  36000: system.cpu: A2 T0 : @main+16    :   msr   contextidr_el1, x0 : IntAlu :  D=0x0000000000000002  flags=(IsInteger|IsSerializeAfter|IsNonSpeculative)
  36500: system.cpu: A2 T0 : @main+20    :   ldr   x0, [sp, #12]      : MemRead :  D=0x0000000000000002 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsLoad)
  37000: system.cpu: A2 T0 : @main+24    :   add   w0, w0, #1         : IntAlu :  D=0x0000000000000003  flags=(IsInteger)
  37500: system.cpu: A2 T0 : @main+28    :   str   x0, [sp, #12]      : MemWrite :  D=0x0000000000000003 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsStore)
  38000: system.cpu: A2 T0 : @main+32    :   ldr   x0, [sp, #12]      : MemRead :  D=0x0000000000000003 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsLoad)
  38500: system.cpu: A2 T0 : @main+36    :   subs   w0, #9            : IntAlu :  D=0x0000000000000000  flags=(IsInteger)
  39000: system.cpu: A2 T0 : @main+40    :   b.le   <main+12>         : IntAlu :   flags=(IsControl|IsDirectControl|IsCondControl)
  39500: system.cpu: A2 T0 : @main+12    :   ldr   x0, [sp, #12]      : MemRead :  D=0x0000000000000003 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsLoad)
  40000: system.cpu: A3 T0 : @main+16    :   msr   contextidr_el1, x0 : IntAlu :  D=0x0000000000000003  flags=(IsInteger|IsSerializeAfter|IsNonSpeculative)

./run --debug
./run-gdb --before '-ex "set remotetimeout 99999" -ex "set debug remote 1"' start_kernel

This error means that the GDB server, e.g. in QEMU, sent more registers than the GDB client expected.

This can happen for the following reasons:

• you set the architecture of the client wrong, often 32 vs 64 bit as mentioned at: https://stackoverflow.com/questions/4896316/gdb-remote-cross-debugging-fails-with-remote-g-packet-reply-is-too-long

• there is a bug in the GDB server and the XML description does not match the number of registers actually sent

• the GDB server does not send XML target descriptions and your GDB expects a different number of registers by default. E.g., gem5 d4b3e064adeeace3c3e7d106801f95c14637c12f does not send the XML files

The XML target description format is described a bit further at: https://stackoverflow.com/questions/46415059/how-to-observe-aarch64-system-registers-in-qemu/53043044#53043044

insmod timer.ko
./kgdb.sh

break lkmc_timer_callback
continue
continue
continue

./run --kdb

[0]kdb> go

./count.sh &
./kgdb.sh

[0]kdb> bp __x64_sys_write
[0]kdb> go
[0]kdb> go
[0]kdb> go
[0]kdb> go

[0]kdb> help

You can also use KDB directly from the graphic window with:

This setup could be used to debug the kernel on machines without serial, such as modern desktops.

This works because --graphics adds kbd (which stands for KeyBoarD!) to kgdboc.

TODO neither arm and aarch64 are working as of 1cd1e58b023791606498ca509256cc48e95e4f5b + 1.

arm seems to place and hit the breakpoint correctly, but no matter how many go commands I do, the count.sh stdout simply does not show.

aarch64 seems to place the breakpoint correctly, but after the first go the kernel oopses with warning:

and stack trace:

My theory is that every serious ARM developer has JTAG, and no one ever tests this, and the kernel code is just broken.

./gdbserver.sh ls

./run-gdb --gdbserver --userland "$(./getvar buildroot_build_build_dir)"/busybox-*/busybox ls_main

./gdbserver.sh ./posix/count.out

./run-gdb --gdbserver --userland userland/posix/count.c main

break sleep
continue

step

set sysroot ${common_buildroot_build_dir}/staging

This example illustrates how reading from the x86 control registers with mov crX, rax can only be done from kernel land on ring0.

From kernel land:

works and output the registers, for example:

However if we try to do it from userland:

stdout gives:

and dmesg outputs:

Sources:

• kernel_modules/ring0.c

• lkmc/ring0.h

• userland/arch/x86_64/ring0.c

In both cases, we attempt to run the exact same code which is shared on the ring0.h header file.

Bibliography:

• https://stackoverflow.com/questions/7415515/how-to-access-the-control-registers-cr0-cr2-cr3-from-a-program-getting-segmenta/7419306#7419306

• https://stackoverflow.com/questions/18717016/what-are-ring-0-and-ring-3-in-the-context-of-operating-systems/44483439#44483439

TODO Can you run arm executables in the aarch64 guest? https://stackoverflow.com/questions/22460589/armv8-running-legacy-32-bit-applications-on-64-bit-os/51466709#51466709

I’ve tried:

but it fails with:

./run --kernel-cli 'init=/lkmc/count.sh'

./run --eval 'echo "asdf qwer";insmod hello.ko;./linux/poweroff.out'

./run --kernel-cli 'init=/lkmc/eval_base64.sh - lkmc_eval="insmod hello.ko;./linux/poweroff.out"'

echo '
cd /lkmc
insmod hello.ko
./linux/poweroff.out
' > data/gitignore.sh
./run --eval "$(cat data/gitignore.sh)"

echo '#!/bin/sh
cd /lkmc
insmod hello.ko
./linux/poweroff.out
' > rootfs_overlay/lkmc/gitignore.sh
chmod +x rootfs_overlay/lkmc/gitignore.sh
./build-buildroot
./run --kernel-cli 'init=/lkmc/gitignore.sh'

Just using BusyBox' poweroff at the end of the init does not work and the kernel panics:

because BusyBox' poweroff tries to do some fancy stuff like killing init, likely to allow userland to shutdown nicely.

But this fails when we are init itself!

BusyBox' poweroff works more brutally and effectively if you add -f:

but why not just use our minimal ./linux/poweroff.out and be done with it?

Source: userland/linux/poweroff.c

This also illustrates how to shutdown the computer from C: https://stackoverflow.com/questions/28812514/how-to-shutdown-linux-using-c-or-qt-without-call-to-system

I dare you to guess what this does:

Source: userland/posix/sleep_forever.c

This executable is a convenient simple init that does not panic and sleeps instead.

Get a reasonable answer to "how long does boot take in guest time?":

Source: userland/linux/time_boot.c

That executable writes to dmesg directly through /dev/kmsg a message of type:

which tells us that boot took 2.188242 seconds based on the dmesg timestamp.

Bibliography: https://stackoverflow.com/questions/12683169/measure-time-taken-for-linux-kernel-from-bootup-to-userpace/46517014#46517014

./run --eval-after 'echo asdf;ls /proc;ls /sys;echo qwer'

./run --kernel-cli-after-dash 'lkmc_eval="insmod hello.ko;./linux/poweroff.out;"'

cp rootfs_overlay/etc/init.d/S98 rootfs_overlay/etc/init.d/S99.gitignore
vim rootfs_overlay/etc/init.d/S99.gitignore
./build-buildroot
./run

./run --kernel-cli 'init=/lkmc/linux/init_env_poweroff.out' --kernel-cli-after-dash 'asdf=qwer zxcv'

init=/lkmc/linux/init_env_poweroff.out console=ttyS0 - lkmc_home=/lkmc asdf=qwer zxcv

args:
/lkmc/linux/init_env_poweroff.out
-
zxcv

env:
HOME=/
TERM=linux
lkmc_home=/lkmc
asdf=qwer

<6>[    0.309984] Run /sbin/init as init process
<7>[    0.309991]   with arguments:
<7>[    0.309997]     /sbin/init
<7>[    0.310004]     nokaslr
<7>[    0.310010]     -
<7>[    0.310016]   with environment:
<7>[    0.310022]     HOME=/
<7>[    0.310028]     TERM=linux
<7>[    0.310035]     earlyprintk=pl011,0x1c090000
<7>[    0.310041]     lkmc_home=/lkmc

The annoying dash - gets passed as a parameter to init, which makes it impossible to use this method for most non custom executables.

Arguments with dots that come after - are still treated specially (of the form subsystem.somevalue) and disappear, from args, e.g.:

outputs:

so see how a.b is gone.

The simple workaround is to just create a shell script that does it, e.g. as we’ve done at: rootfs_overlay/lkmc/gem5_exit.sh.

Wait, where do HOME and TERM come from? (greps the kernel). Ah, OK, the kernel sets those by default: https://github.com/torvalds/linux/blob/94710cac0ef4ee177a63b5227664b38c95bbf703/init/main.c#L173

On top of the Linux kernel, the BusyBox /bin/sh shell will also define other variables.

We can explore the shenanigans that the shell adds on top of the Linux kernel with:

From there we observe that:

gives:

therefore adding SHLVL and PWD to the default kernel exported variables.

Furthermore, to increase confusion, if you list all non-exported shell variables https://askubuntu.com/questions/275965/how-to-list-all-variables-names-and-their-current-values with:

then it shows more variables, notably:

7.4.3.1. BusyBox shell initrc files

/etc/profile
$HOME/.profile

cat /proc/$$/cmdline

initrd /initrd.img-4.4.0-108-generic

./build-buildroot --initramfs
./build-linux --initramfs
./run --initramfs

cat "$(./getvar run_dir)/run.sh"

ls -lh "$(./getvar linux_image)"

./build-linux
./run

./build-buildroot --initramfs
./build-linux --initramfs --linux-build-id initramfs
./run --initramfs --linux-build-id

See: Section 8.3, “rootfs”

src/arch/arm/linux/atag.hh

VFS: Cannot open root device "sda" or unknown-block(8,0): error -5

"$(./getvar buildroot_host_dir)"/bin/dtc -I dtb -O dts -o a.dts a.dtb
"$(./getvar buildroot_host_dir)"/bin/dtc -I dts -O dtb -o a.dtb a.dts

sudo apt-get install device-tree-compiler

/dts-v1/;

/ {
    a;
};

dtc a.dts

/dts-v1/;

/ {
    a;
};

/ {
    b;
};

/dts-v1/;

/ {
        a;
        b;
};

./build-buildroot \
  --arch aarch64 \
  --config 'BR2_PACKAGE_DTC=y' \
  --config 'BR2_PACKAGE_DTC_PROGRAMS=y' \
;

./run --arch aarch64

dtc -I fs -O dts /sys/firmware/devicetree/base

        cpus {
                #address-cells = <0x1>;
                #size-cells = <0x0>;

                cpu@0 {
                        compatible = "arm,cortex-a57";
                        device_type = "cpu";
                        reg = <0x0>;
                };
        };

./run --arch aarch64 --cpus 2

                cpu@0 {
                cpu@1 {

./run --arch aarch64 -- -machine dumpdtb=dtb.dtb

sudo apt-get install git
git clone https://github.com/cirosantilli/linux-kernel-module-cheat
cd linux-kernel-module-cheat
./setup -y

panic: KVM: Failed to enter virtualized mode (hw reason: 0x80000021)

./build user-mode-qemu
./run \
  --userland userland/c/command_line_arguments.c \
  --cli-args='asdf "qw er"' \
;

/path/to/linux-kernel-module-cheat/out/userland/default/x86_64/c/command_line_arguments.out
asdf
qw er

./build-userland

./build-qemu --mode userland

--mode userland

It’s nice when the obvious just works, right?

and on another shell:

Or alternatively, if you are using tmux, do everything in one go with:

To stop at the very first instruction of a freestanding program, just use --no-continue. A good example of this is shown at: [freestanding-programs].

./build --all-archs test-executables-userland
./test-executables --all-archs --all-emulators

./build --all-archs test-executables-userland-qemu
./test-executables --all-archs --emulator qemu

./run \
  --userland "$(./getvar buildroot_target_dir)/bin/echo" \
  --cli-args='asdf' \
;

./run \
  --arch aarch64 \
  --userland "$(./getvar --arch aarch64 buildroot_target_dir)/bin/sh" \
  --terminal \
;

./run \
  --arch aarch64 \
  --userland "$(./getvar --arch aarch64 buildroot_target_dir)/bin/sh" \
  --cli-args='-c "uname -a && pwd"' \
;

glibc has a check for kernel version, likely obtained from the uname syscall, and if the kernel is not new enough, it quits.

Both gem5 and QEMU however allow setting the reported uname version from the command line for User mode simulation, which we do to always match our toolchain.

QEMU by default copies the host uname value, but we always override it in our scripts.

Determining the right number to use for the kernel version is of course highly non-trivial and would require an extensive userland test suite, which most emulators don’t have.

Source: userland/posix/uname.c.

The QEMU source that does this is at: https://github.com/qemu/qemu/blob/v3.1.0/linux-user/syscall.c#L8931 The default ID is just hardcoded on the source.

Bibliography:

• https://stackoverflow.com/questions/48959349/how-to-solve-fatal-kernel-too-old-when-running-gem5-in-syscall-emulation-se-m

• https://stackoverflow.com/questions/53085048/how-to-compile-and-run-an-executable-in-gem5-syscall-emulation-mode-with-se-py/53085049#53085049

• https://gem5-review.googlesource.com/c/public/gem5/+/15855

For some reason QEMU / glibc x86_64 picks up the host libc, which breaks things.

Other archs work as they different host libc is skipped. User mode static executables also work.

We have worked around this with with https://bugs.launchpad.net/qemu/+bug/1701798/comments/12 from the thread: https://bugs.launchpad.net/qemu/+bug/1701798 by creating the file: rootfs_overlay/etc/ld.so.cache which is a symlink to a file that cannot exist: /dev/null/nonexistent.

Reproduction:

Outcome:

To get things working again, restore ld.so.cache with:

I’ve also tested on an Ubuntu 16.04 guest and the failure is different one:

A non-QEMU-specific example of stack smashing is shown at: https://stackoverflow.com/questions/1345670/stack-smashing-detected/51897264#51897264

Tested at: 2e32389ebf1bedd89c682aa7b8fe42c3c0cf96e5 + 1.

./build-userland \
  --arch aarch64 \
  --static \
;
./run \
  --arch aarch64 \
  --static \
  --userland userland/c/command_line_arguments.c \
  --cli-args 'asdf "qw er"' \
;

One limitation of static executables is that Buildroot mostly only builds dynamic versions of libraries (the libc is an exception).

So programs that rely on those libraries might not compile as GCC can’t find the .a version of the library.

For example, if we try to build [blas] statically:

it fails with:

11.5.1.1. C++ static and pthreads

./run --userland userland/cpp/atomic/std_atomic.cpp --static

qemu-x86_64: /path/to/linux-kernel-module-cheat/submodules/qemu/accel/tcg/cpu-exec.c:700: cpu_exec: Assertion `!have_mmap_lock()' failed.
qemu-x86_64: /path/to/linux-kernel-module-cheat/submodules/qemu/accel/tcg/cpu-exec.c:700: cpu_exec: Assertion `!have_mmap_lock()' failed.

./run --arch aarch64 --userland userland/cpp/atomic/fail.cpp --static

terminate called after throwing an instance of 'std::system_error'
  what():  Unknown error 16781344
qemu: uncaught target signal 6 (Aborted) - core dumped

-pthread -Wl,--whole-archive -lpthread -Wl,--no-whole-archive

./run --userland userland/c/getchar.c

enter a character:

you entered: a

printf a > f.tmp
./run --userland userland/c/getchar.c < f.tmp

printf a > f.tmp
./run --emulator gem5 --userland userland/c/getchar.c --stdin-file f.tmp

./build-userland --arch aarch64
./run \
  --arch aarch64 \
  --emulator gem5 \
  --userland userland/c/command_line_arguments.c \
  --cli-args 'asdf "qw er"' \
;

./run \
  --arch aarch64 \
  --emulator gem5 \
  --gdb-wait \
  --userland userland/c/command_line_arguments.c \
  --cli-args 'asdf "qw er"' \
;
./run-gdb \
  --arch aarch64 \
  --emulator gem5 \
  --userland userland/c/command_line_arguments.c \
  main \
;

Support for dynamic linking was added in November 2019:

• https://stackoverflow.com/questions/50542222/how-to-run-a-dynamically-linked-executable-syscall-emulation-mode-se-py-in-gem5/50696098#50696098

• https://stackoverflow.com/questions/64547306/cannot-open-lib-ld-linux-aarch64-so-1-in-qemu-or-gem5/64551313#64551313

Note that as shown at [benchmark-emulators-on-userland-executables], the dynamic version runs 200x more instructions, which might have an impact on smaller simulations in detailed CPUs.

As of gem5 7fa4c946386e7207ad5859e8ade0bbfc14000d91, the crappy se.py script does not forward the exit status of syscall emulation mode, you can test it with:

Source: userland/c/false.c.

Then manually run the generated gem5 CLI, and do:

and the output is always 0.

Instead, it just outputs a message to stdout just like for m5 fail:

which we parse in run and then exit with the correct result ourselves…​

Related thread: https://stackoverflow.com/questions/56032347/is-there-a-way-to-identify-if-gem5-run-got-over-successfully

Since gem5 has to implement syscalls itself in syscall emulation mode, it can of course clearly see which syscalls are being made, and we can log them for debug purposes with gem5 tracing, e.g.:

the trace as of f2eeceb1cde13a5ff740727526bf916b356cee38 + 1 contains:

so we see that two syscall lines were added for each syscall, showing the syscall inputs and exit status, just like a mini strace!

gem5 user mode multithreading has been particularly flaky compared to QEMU’s, but work is being put into improving it.

In gem5 syscall simulation, the fork syscall checks if there is a free CPU, and if there is a free one, the new threads runs on that CPU.

Otherwise, the fork call, and therefore higher level interfaces to fork such as pthread_create also fail and return a failure return status in the guest.

For example, if we use just one CPU for userland/posix/pthread_self.c which spawns one thread besides main:

fails with this error message coming from the guest stderr:

It works however if we add on extra CPU:

Once threads exit, their CPU is freed and becomes available for new fork calls: For example, the following run spawns a thread, joins it, and then spawns again, and 2 CPUs are enough:

because at each point in time, only up to two threads are running.

gem5 syscall emulation does show the expected number of cores when queried, e.g.:

outputs 1 and 2 respectively.

This can also be clearly by running sched_getcpu:

which necessarily produces an output containing the CPU numbers from 1 to 4 and no higher:

TODO why does the 2 come at the end here? Would be good to do a detailed assembly run analysis.

gem5 syscall emulation has the nice feature of allowing you to run multiple executables "at once".

Each executable starts running on the next free core much as if it had been forked right at the start of simulation: gem5 syscall emulation multithreading.

This can be useful to quickly create deterministic multi-CPU workload.

se.py --cmd takes a semicolon separated list, so we could do which LKMC exposes this by taking --userland multiple times as in:

We need at least one CPU per executable, just like when forking new processes.

The outcome of this is that we see two different pid messages printed to stdout:

since from [gem5-process] we can see that se.py sets up one different PID per executable starting at 100:

We can also see that these processes are running concurrently with gem5 tracing by hacking:

which starts with:

and therefore shows one instruction running on each CPU for each process at the same time.

11.7.5.1. gem5 syscall emulation --smt

--cpu 1 --cpu-type Derivo3CPU --caches

example/se.py:115:        assert(options.cpu_type == "DerivO3CPU")

gem5.opt: /path/to/linux-kernel-module-cheat/out/gem5/master3/build/ARM/cpu/o3/cpu.cc:205: FullO3CPU<Impl>::FullO3CPU(DerivO3CPUParams*) [with Impl = O3CPUImpl]: Assertion `params->numPhysVecPredRegs >= numThreads * TheISA::NumVecPredRegs' failed.
Program aborted at tick 0

At 8d8307ac0710164701f6e14c99a69ee172ccbb70 + 1, I noticed that if you run userland/posix/count.c:

it first waits for 3 seconds, then the program exits, and then it dumps all the stdout at once, instead of counting once every second as expected.

The same can be reproduced by copying the raw QEMU command and piping it through tee, so I don’t think it is a bug in our setup:

TODO: investigate further and then possibly post on QEMU mailing list.

11.8.1.1. QEMU user mode does not show errors

qemu-x86_64 asdf | cat

./run --eval-after 'insmod hello.ko'

# init_module
./linux/myinsmod.out hello.ko
# finit_module
./linux/myinsmod.out hello.ko "" 1
./linux/myrmmod.out hello

man init_module

ls /lib/modules/<kernel_version>

ls -l /bin/lsmod

lrwxrwxrwx 1 root root 4 Jul 25 15:35 /bin/lsmod -> kmod

dpkg -l | grep -Ei

ii  kmod                                        22-1ubuntu5                                         amd64        tools for managing Linux kernel modules

Name of a predecessor set of tools.

kmod’s modprobe can also load modules under different names to avoid conflicts, e.g.:

./build-disk2

mkdir /mnt/vdb
mount /dev/vdb /mnt/vdb
/mnt/vdb/lkmc/c/hello.out

umount /mnt/vdb

# Edit the file.
vim userland/c/hello.c
./build-userland
./build-disk2

./run --vnc

./vnc

https://superuser.com/questions/1087859/how-to-quit-the-qemu-monitor-when-not-using-a-gui

However, our QEMU setup captures Ctrl + C and other common signals and sends them to the guest, which makes it hard to quit QEMU for the first time since there is no GUI either.

The simplest way to quit QEMU, is to do:

Alternative methods include:

• quit command on the QEMU monitor

• pkill qemu

./run --graphic

./qemu-monitor info qtree

cat /dev/urandom > /dev/fb0

Scroll up in QEMU graphic mode:

but I never managed to increase that buffer:

• https://askubuntu.com/questions/709697/how-to-increase-scrollback-lines-in-ubuntu14-04-2-server-edition

• https://unix.stackexchange.com/questions/346018/how-to-increase-the-scrollback-buffer-size-for-tty

The superior alternative is to use text mode and GNU screen or tmux.

14.2.2.1. QEMU graphic mode arm terminal

./run --arch aarch64 --graphic

cat /dev/urandom > /dev/fb0

cat: write error: No space left on device

14.2.2.2. QEMU graphic mode arm terminal implementation

-device virtio-gpu-pci

CONFIG_DRM=y
CONFIG_DRM_VIRTIO_GPU=y

14.2.2.3. QEMU graphic mode arm VGA

-device VGA

# We use virtio-gpu because the legacy VGA framebuffer is
# very troublesome on aarch64, and virtio-gpu is the only
# video device that doesn't implement it.

./build-linux \
  --arch arm \
  --custom-config-file-gem5 \
  --linux-build-id gem5-v4.15 \
;
./run --arch arm --emulator gem5 --linux-build-id gem5-v4.15

vinagre localhost:5900

[    0.152755] [drm] found ARM HDLCD version r0p0
[    0.152790] hdlcd 2b000000.hdlcd: bound virt-encoder (ops 0x80935f94)
[    0.152795] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013).
[    0.152799] [drm] No driver support for vblank timestamp query.
[    0.215179] Console: switching to colour frame buffer device 240x67
[    0.230389] hdlcd 2b000000.hdlcd: fb0:  frame buffer device
[    0.230509] [drm] Initialized hdlcd 1.0.0 20151021 for 2b000000.hdlcd on minor 0

system.vncserver: Listening for connections on port 5900

info: VNC client attached

./run \
  --arch arm \
  --emulator gem5 \
  --linux-build-id gem5-v4.15 \
  -- --frame-capture \
;

frames_system.vncserver/fb.<frame-index>.<timestamp>.png.gz

kmscube[706]: unhandled level 2 translation fault (11) at 0x00000000, esr 0x92000006, in libgbm.so.1.0.0[7fbf6a6000+e000]

For aarch64 we also need to configure the kernel with linux_config/display:

This is because the gem5 aarch64 defconfig does not enable HDLCD like the 32 bit one arm one for some reason.

TODO get working. There is an unmerged patchset at: https://gem5-review.googlesource.com/c/public/gem5/+/11036/1

The DP650 is a newer display hardware than HDLCD. TODO is its interface publicly documented anywhere? Since it has a gem5 model and in-tree Linux kernel support, that information cannot be secret?

The key option to enable support in Linux is DRM_MALI_DISPLAY=y which we enable at linux_config/display.

Build the kernel exactly as for Graphic mode gem5 aarch64 and then run with:

We cannot use mainline Linux because the gem5 arm Linux kernel patches are required at least to provide the CONFIG_DRM_VIRT_ENCODER option.

gem5 emulates the HDLCD ARM Holdings hardware for arm and aarch64.

The kernel uses HDLCD to implement the DRM interface, the required kernel config options are present at: linux_config/display.

TODO: minimize out the --custom-config-file. If we just remove it on arm: it does not work with a failing dmesg:

So what other options are missing from gem5_defconfig? It would be cool to minimize it out to better understand the options.

./build-buildroot --config-fragment buildroot_config/x11
./run --graphic

startx

xcalc
xeyes

./build-buildroot --config-fragment buildroot_config/x11 -- xserver_xorg-server-reconfigure

[    2.809104] WARNING: CPU: 0 PID: 51 at drivers/gpu/drm/ttm/ttm_bo_vm.c:304 ttm_bo_vm_open+0x37/0x40

TODO 9076c1d9bcc13b6efdb8ef502274f846d8d4e6a1 I’m 100% sure that it was working before, but I didn’t run it forever, and it stopped working at some point. Needs bisection, on whatever commit last touched x11 stuff.

• https://askubuntu.com/questions/730891/how-can-i-get-a-mouse-cursor-in-qemu

• https://stackoverflow.com/questions/19665412/mouse-and-keyboard-not-working-in-qemu-emulator

-show-cursor did not help, I just get to see the host cursor, but the guest cursor still does not move.

Doing:

shows that interrupts do happen when mouse and keyboard presses are done, so I expect that it is some wrong either with:

• QEMU. Same behaviour if I try the host’s QEMU 2.10.1 however.

• X11 configuration. We do have BR2_PACKAGE_XDRIVER_XF86_INPUT_MOUSE=y.

/var/log/Xorg.0.log contains the following interesting lines:

The file /dev/inputs/mice does not exist.

Note that our current link:kernel_confi_fragment sets:

for gem5, so you might want to remove those lines to debug this.

On ARM, startx hangs at a message:

and nothing shows on the screen, and:

says:

A friend told me this but I haven’t tried it yet:

• xf86-video-modesetting is likely the missing ingredient, but it does not seem possible to activate it from Buildroot currently without patching things.

• xf86-video-fbdev should work as well, but we need to make sure fbdev is enabled, and maybe add some line to the Xorg.conf

ifup -a

wget google.com
cat index.html

ifdown -a

ping google.com

PING google.com (216.58.204.46): 56 data bytes

15.3.1.1. nc host to guest

nc -l -p 45455

echo asdf | nc localhost 45455

15.3.1.2. ssh into guest

./build-buildroot --config 'BR2_PACKAGE_OPENSSH=y'

./sshd.sh

ssh root@localhost -p 45456

15.3.1.3. gem5 host to guest networking

First Enable networking.

Then in the host, start a server:

And then in the guest, find the IP we need to hit with:

which gives:

so we use in the guest:

Bibliography:

• https://serverfault.com/questions/769874/how-to-forward-a-port-from-guest-to-host-in-qemu-kvm/951835#951835

• https://unix.stackexchange.com/questions/78953/qemu-how-to-ping-host-network/547698#547698

All of 9P and NFS (and sshfs) allow sharing directories between guest and host.

Advantages of 9P

• requires sudo on the host to mount

• we could share a guest directory to the host, but this would require running a server on the guest, which adds simulation overhead

  Furthermore, this would be inconvenient, since what we usually want to do is to share host cross built files with the guest, and to do that we would have to copy the files over after the guest starts the server.

• QEMU implements 9P natively, which makes it very stable and convenient, and must mean it is a simpler protocol than NFS as one would expect.

  This is not the case for gem5 7bfb7f3a43f382eb49853f47b140bfd6caad0fb8 unfortunately, which relies on the diod host daemon, although it is not unfeasible that future versions could implement it natively as well.

Advantages of NFS:

• way more widely used and therefore stable and available, not to mention that it also works on real hardware.

• the name does not start with a digit, which is an invalid identifier in all programming languages known to man. Who in their right mind would call a software project as such? It does not even match the natural order of Plan 9; Plan then 9: P9!

As usual, we have already set everything up for you. On host:

Guest:

Host:

The main ingredients for this are:

• 9P settings in our kernel configs

• 9p entry on our rootfs_overlay/etc/fstab

  Alternatively, you could also mount your own with:

  mkdir /mnt/my9p
  mount -t 9p -o trans=virtio,version=9p2000.L host0 /mnt/my9p

  where mount tag host0 is set by the emulator (mount_tag flag on QEMU CLI), and can be found in the guest with: cat /sys/bus/virtio/drivers/9pnet_virtio/virtio0/mount_tag as documented at: https://www.kernel.org/doc/Documentation/filesystems/9p.txt.

• Launch QEMU with -virtfs as in your run script

  When we tried:

  security_model=mapped

  writes from guest failed due to user mismatch problems: https://serverfault.com/questions/342801/read-write-access-for-passthrough-9p-filesystems-with-libvirt-qemu

Bibliography:

• https://superuser.com/questions/628169/how-to-share-a-directory-with-the-host-without-networking-in-qemu

• https://wiki.qemu.org/Documentation/9psetup

Is possible on aarch64 as shown at: https://gem5-review.googlesource.com/c/public/gem5/+/22831, and it is just a matter of exposing to X86 for those that want it.

Enable it by passing the --vio-9p option on the fs.py gem5 command line:

Then on the guest:

Yes, you have to pass the full path to the directory on the host. Yes, this is horrible.

The shared directory is:

so we can observe the file the guest wrote from the host with:

and vice versa:

is now visible from the guest:

Checkpoint restore with an open mount will likely fail because gem5 uses an ugly external executable to implement diod. The protocol is not very complex, and QEMU implements it in-tree, which is what gem5 should do as well at some point.

Also checkpoint without --vio-9p and restore with --vio-9p did not work either, the mount fails.

However, this did work, on guest:

then restore with the detalied CPU of interest e.g.

Tested on gem5 b2847f43c91e27f43bd4ac08abd528efcf00f2fd, LKMC 52a5fdd7c1d6eadc5900fc76e128995d4849aada.

TODO: get working.

9P is better with emulation, but let’s just get this working for fun.

First make sure that this works: Section 15.3.2, “Guest to host networking”.

Then, build the kernel with NFS support:

Now on host:

Now edit /etc/exports to contain:

and restart the server:

Now on guest:

TODO: failing with:

And now the /tmp directory from host is not mounted on guest!

If you don’t want to start the NFS server after the next boot automatically so save resources, do:

To modify a single option on top of our default kernel configs, do:

Kernel modules depend on certain kernel configs, and therefore in general you might have to clean and rebuild the kernel modules after changing the kernel config:

and then proceed as in Your first kernel module hack.

You might often get way without rebuilding the kernel modules however.

To use an extra kernel config fragment file on top of our defaults, do:

To use just your own exact .config instead of our defaults ones, use:

There is also a shortcut --custom-config-file-gem5 to use the gem5 arm Linux kernel patches.

The following options can all be used together, sorted by decreasing config setting power precedence:

• --config

• --config-fragment

• --custom-config-file

To do a clean menu config yourself and use that for the build, do:

But remember that every new build re-configures the kernel by default, so to keep your configs you will need to use on further builds:

So what you likely want to do instead is to save that as a new defconfig and use it later as:

You can also use other config generating targets such as defconfig with the same method as shown at: Section 17.1.3.1.1, “Linux kernel defconfig”.

Get the build config in guest:

or with our shortcut:

or to conveniently grep for a specific option case insensitively:

Source: rootfs_overlay/lkmc/conf.sh.

This is enabled by:

From host:

Just for fun https://stackoverflow.com/questions/14958192/how-to-get-the-config-from-a-linux-kernel-image/14958263#14958263:

although this can be useful when someone gives you a random image.

By default, build-linux generates a .config that is a mixture of:

• a base config extracted from Buildroot’s minimal per machine .config, which has the minimal options needed to boot as explained at: Section 17.1.3.1, “About Buildroot’s kernel configs”.

• small overlays put top of that

To find out which kernel configs are being used exactly, simply run:

and look for the merge_config.sh call. This script from the Linux kernel tree, as the name suggests, merges multiple configuration files into one as explained at: https://unix.stackexchange.com/questions/224887/how-to-script-make-menuconfig-to-automate-linux-kernel-build-configuration/450407#450407

For each arch, the base of our configs are named as:

e.g.: linux_config/buildroot-x86_64.

These configs are extracted directly from a Buildroot build with update-buildroot-kernel-configs.

Note that Buildroot can sed override some of the configurations, e.g. it forces CONFIG_BLK_DEV_INITRD=y when BR2_TARGET_ROOTFS_CPIO is on. For this reason, those configs are not simply copy pasted from Buildroot files, but rather from a Buildroot kernel build, and then minimized with make savedefconfig: https://stackoverflow.com/questions/27899104/how-to-create-a-defconfig-file-from-a-config

On top of those, we add the following by default:

• linux_config/min: see: Section 17.1.3.1.2, “Linux kernel min config”

• linux_config/default: other optional configs that we enable by default because they increase visibility, or expose some cool feature, and don’t significantly increase build time nor add significant runtime overhead

  We have since observed that the kernel size itself is very bloated compared to defconfig as shown at: Section 17.1.3.1.1, “Linux kernel defconfig”.

17.1.3.1. About Buildroot’s kernel configs

./build-linux \
  --linux-build-id defconfig \
  --custom-config-target defconfig \
  --config CONFIG_VIRTIO_PCI=y \
  --config CONFIG_VIRTIO_BLK=y \
;
./run --linux-build-id defconfig

du -h \
  "$(./getvar vmlinux)" \
  "$(./getvar --linux-build-id defconfig vmlinux)" \
;

360M    /path/to/linux-kernel-module-cheat/out/linux/default/x86_64/vmlinux
47M     /path/to/linux-kernel-module-cheat/out/linux/defconfig/x86_64/vmlinux

./build-linux \
  --linux-build-id defconfig \
  --custom-config-target defconfig \
;
./run --initrd --linux-build-id defconfig

./build-linux \
  --arch aarch64 \
  --linux-build-id defconfig \
  --custom-config-target defconfig \
;
./run \
  --arch aarch64 \
  --initrd \
  --linux-build-id defconfig \
  --memory 2G \
;

118M    /path/to/linux-kernel-module-cheat/out/linux/default/aarch64/vmlinux
240M    /path/to/linux-kernel-module-cheat/out/linux/defconfig/aarch64/vmlinux

./build-linux \
  --arch aarch64 \
  --config-fragment linux_config/min \
  --custom-config-file linux_config/buildroot-aarch64 \
  --linux-build-id min \
;

17.1.3.2. Notable alternate gem5 kernel configs

We try to use the latest possible kernel major release version.

In QEMU:

or in the source:

During update all you kernel modules may break since the kernel API is not stable.

They are usually trivial breaks of things moving around headers or to sub-structs.

The userland, however, should simply not break, as Linus enforces strict backwards compatibility of userland interfaces.

This backwards compatibility is just awesome, it makes getting and running the latest master painless.

This also makes this repo the perfect setup to develop the Linux kernel.

In case something breaks while updating the Linux kernel, you can try to bisect it to understand the root cause, see: [bisection].

17.2.2.1. Update the Linux kernel LKMC procedure