Falkervisor (grilled_cheese)
Follow me on Twitter, @gamozolabs Tweets go out before my reddit post spam, also I announce live streams there
See it in action
This was me using it to fuzz r2 back when this was under active development. This video should provide enough background to modify it for your own use.
https://www.youtube.com/watch?v=AqFMSI8e9Qo
History
This is the latest C version of my hypervisor and probably some of the best C code I've ever written (I've since switched to Rust, you should too). This was used roughly between 2015-2016, and replaced with a Rust version in late 2016.
Building
Make sure you have python, nasm, clang, and ld.lld in your path. Then just type python build.py. Use python build.py clean to clean.
It builds on Windows easily with the following (however there are no hard reqs on versions):
C:\dev\grilled_cheese>clang --version
clang version 7.0.0 (trunk)
Target: x86_64-pc-windows-msvc
Thread model: posix
InstalledDir: C:\Program Files\LLVM\bin
C:\dev\grilled_cheese>python --version
Python 3.6.5
C:\dev\grilled_cheese>nasm --version
NASM version 2.13.03 compiled on Feb 7 2018
C:\dev\grilled_cheese>ld.lld --version
LLD 7.0.0 (compatible with GNU linkers)
You might see some files with *.dc extensions. This because the build system builds any *.c file in the tree, thus to disable things I wasn't using I just threw a d in the extension.
Feature differences from brownie
See brownie here: https://github.com/gamozolabs/falkervisor_beta/
Improvements over brownie
- It was written in C
- Networking code was greatly improved
- Kernel was portable to Intel (hypervisor not though)
- DHCPv4 is used for getting an IP, no more hardcoded packets
- Remote mapping of files
- Generic VM support, designed to in theory support AMD, Intel, etc from one API
- Many design features were made to reduce the chances of heisenbugs. Full page heap, full ASLR, strict mapping requirements really helped here
- RSTATE error model allowed for human readable call stacks on errors without worrying about unwind info, symbols, or inlining.
Similarities to brownie
- Performance is about the same, there isn't room for growth in either tools
- Only supports x540 as a network device
- UDP only for networking
- Custom server for communicating over UDP
Cool features
Clang support
Early versions of my kernel used MSVC instead. Clang offers much better portability. By using clang and ld.lld this kernel is easily built on any system without special toolchain requirements. Further clang allows for inline assembly which is nice for kernel development, even though I keep usage to an absolute minimum.
Remote mapping
The remote mapping net_map_remote() allows for files to be mapped over the network and faulted in only when they are used. For VMs this is used to map the entire ~4 GiB snapshot. Combined with CoW, this meant that only pages that were ever touched during a fuzz case were present in memory, and only one copy. If memory was ever written during a fuzz case then this memory might be duplicated to each core.
Typically for a medium size target (1-2 second fuzz case maximum), this meant usually no more than 2-4 MiB was used per VM. Leading to running 64 4GiB VMs with using only a few hundred MiB of RAM!
Full ASLR
Every bit of every allocation is fully ASLRed. The kernel base and stacks are also fully ASLRed. This ASLR is 36-bits (meaning every bit that isn't a page offset is random). This means you'll get allocations in the 0xffff... range and the 0x0000... range.
Page heap
All allocations are page heaped, their addresses are fully deleted from page tables on free and overflows and underflows can only corrupt data inside the structure itself.
No identity/linear physical memory map
The machine I developed this for had 512 GiB of RAM. Most kernels have a linear mapping of ALL of physical memory somewhere in the kernel. When dealing with large amounts of memory this uses a huge amount of the virtual address space. This is not really a huge issue, however it makes it so there's a "decent chance" a random address might actually hit valid memory. This makes it possible for heisenbugs for things that "sometimes work".
Instead I use a design where the bootloader provides a 512-entry dynamic mapping mechanism. This consists of single page directory which is mapped at a random address. This means that using 4 KiB (PD) + 2 MiB of reserved page we are able to have a fast mechanism of accessing all physical memory. This greatly reduces the size of active addresses in the virtual address space, leaving more room for randomness and less chance of a random address hitting valid memory.
Performance is about 20% slower with this model over a linear mapping, which is really not that bad.
Generic VM model
The VM model was starting to be designed to be more generic. This was going to allow for an Intel hypervisor being added while still using the same API for managing both. In this version there are actually 2 VM models, one for AMD's SVM, and another using user-VM. User VM was a made up VM model that makes a semi-isolated guest by using ring3 rather than a whole VM itself. This was great for things like JITs which needed a unique address space but I wanted to support on Intel.
Rstate
You'll see that all error handling is done through RSTATES. This model allows for all errors to chain together the origin of the error all the way up to the top which paniced or handled it. This does not rely on symbols or unwind unwalking so it's much more simple. Further it works correctly regardless of optimization levels or inlining as the stack is managed in the C code itself. Getting clean errors out of this really helped keep code quality up and bug fix times to a minimum.