System Programming Roadmap
A roadmap to teach myself compiler dev, malware reverse engineering and kernel dev fundamentals. To be noted they are only for the fundamental knowledge and doesn't make you a master of any. I will pick one or more of the below mentioned fields for later research in some specific topics. Low Level Programming University also has a good list of resources to follow but this is my personal roadmap.
Topics to study here may or may not be in order and can be studied according to your preference, gievn that prerequisites are getting fulfilled for each one of them.
Prerequisites
I'm already assuming that you have basic understanding of computer architecture and experience with atleast one system programming language, some basics of how assembly works and familiar using any POSIX system. A good detailed look of how computers work at the electronics level can be found in the book Introduction to Digital Electronics by Agner Fog.
System Programming Languages
Learn any two of the given languages, make some basic projects to get yourself familiar with it, solve some programming exercises.
Learn some Computer Architecture
Learn Arm and RISCV based computer architecture to build an efficient and optimized approach towards solving the problems at hardware level
- David A. Patterson, John L. Hennessy "Computer Architecture: A Quantitative Approach"
- David A. Patterson, John L. Hennessy "Computer Organization and Design ARM Edition"
- David A. Patterson, John L. Hennessy "Computer Organization and Design RISC-V Edition"
- John Paul Shen, Mikko H. Lipasti "Modern Processor Design: Fundamentals of Superscalar Processors"
- CMU Computer Architecture by CMU Youtube
Learn some x86
Prerequisites: Learn about Digital Logic
If you are not familiar with assembly yet, I would recommend checking out some tutorials like-
- x86 quickstart[MASM]
- x86 quickstart [NASM]
- Another x86 quickstart[NASM]
- Introduction to x86 assembly language by Davy on youtube
- OMU x86_64 lessons
- https://asmtutor.com/
- x86 Asm
- The Art Of Asm
- Even Manuals are great sources of learning. The manuals for processors can easily be found using a Google search ("Intel Manuals," "ARM manuals)
- Compiler Explorer: Making C programs and reading the disassembly always helps to match patterns.
- Article by 0x41 reversing for dummies to be able to reverse basic crackmes.
After this, I would recommend solving easy crackmes for exercise. crackmes.one and tryhackme are places to find some of the easy ones. Hard ones still require some pwning knowledge which I'm gona discuss in the exploitation section.
Compilers
Prerequisites include experience creating projects in a system programming language and a deep understanding of memory and CPU.
- Read the Dragon Book.
- Crafting Interpreters is a good one for beginners.
- Language Implementation Patterns provides some good insights on the workings of compilers.
- Stanford Notes CS143 Good assignments and notes related to compiler design.
- CMU slides and Projects
- Awesome Compilers
- Make a Language in Rust
- Rust Parsing Basics
- Make a tree walk interpreted programming language.
- Also try to implement a bytecode engine for your interpreter, try out some optimizations and GC.
- You can also emulate machines like Chip8 or Nes.
- Emulation requires knowledge of VM internals and graphics programming.
- You can use SDL as an IO/graphics/sound engine.
- Try to make a compiled programming language targetting one architecture.
- Learn about the LLVM toolchain
- LLVM tutorial in Rust
- Try to follow the llvm tutorial to make your first programming language using llvm backend.
- Try to make a Just In Time Compiler around the bytecode engine, detect hot regions and JIT them.
- My discord server lang-dev section
Exploitation
Prerequisites include experience with assembly.
- pwn.college is the best learning resource I got so far for exploitation. From assembly to kernel exploitation, it covers it all.
- Introduction to exploit development
- Nightmare: Intro to binary exploitation based around CTFs.
- CS6265: Reverse Engineering and Binary Exploitation Lab
- OMU exploitation labs
- LiveOverflow's binexp series on youtube
- Tutorial by 0xinfection
- Exploit dev on the infosec reference
- ROP Emporium
- Windows Stuff
- After learning about some exploitation, you can solve CTFs now. Some of them include:
- pwnable.kr
- Exploit Education VMs
- Overthewire wargames covering exploitation
- HackTheBox challenges based on binary exploitation
Browser Hacking
Prerequisites include high level knowledge of VM internals, and solid understanding and experience with Compiler Engineering
- Development
- Create a basic html dom parser Rust
- Toy browser engine, Browser engine from scratch
- JavaScript bytecode VM Andreas Kling
- Browser Parsing & JS AST Andreas Kling
- Inside look at modern browser
- Blogs to follow: V8, MozHacks, Webkit
- Docs: Firefox, Chromium, Webkit Wiki
- Compiler Compiler: A Twitch series about working on a JavaScript engine
- Graphics: Choose a 2d graphics lib for your language or platform. You can surely use OpenGL or Vulkan?!? to render your innocent CSS but trust me it is not worth it.
- High-performance gc for V8
- Adventures in JIT compilation
- Speculation in JavaScriptCore
- Network Programming Rust Networking, Rust std::net, C
- After learning about parsing, rendering, and JIT, you can now make your own browser with basic APIs and minimal features, following the whatwg standards
- Exploitation: A great way to understand how a browser works is to try to hack it: (prerequisites include solid binary exploitation skills)
- Browser Exploition series by LiveOverflow | Written
- Web Assembly Hacking talk Black Hat
- Browser pwn on github
- Web Browser Exploitation- University of Florida
- Go through writeups of CVEs or CTF challenges based on browsers or runtime envs.
Malware
Prerequisites include a high-level understanding of windows and solid reverse engineering skills.
- Practical Malware Analysis
- Malware analysis bootcamp by hackersploit
- CS5138 Malware Analysis, UC
- Prelude's live streams
- Cr0w's Youtube Channel
- After learning the basics of malware reversing and behavior, you can now move to reverse some real samples of those.
- Labs by Malware Unicorn
- VX Underground - The largest collection of malware source code, samples, and papers on the internet.
- Malware section from the infosec reference
- Malware Bazar
OS Fundamentals
I'm not quite sure that I want to get into kernel development (yet) but the concepts seem cool and its a good idea for a vacation project or assignments for my OS classes in university. Make sure to read the requirements before getting started.
- OS Dev Wiki is the go-to place if you want to learn about OS. It's well documented and also helps eyes to bleed.
- Linux Kernel Labs
- Tutorials Section from awesome OS on github
- Broken Thorn's Tutorial
- OS in 3 pieces
- Little OS Book
- Blog OS: Writing an OS in Rust
- Bootlin Slides and Labs
- 539kernel: A Journey in Creating an OS Kernel
- Stuff to work on:
- Awesome OS on github
- My discord server's OS dev channel to get some more resources and books.
VM internals
Lists of VM internals to study while making progress in compiler engineering and Browser development:
- How to build a virtual machine
- JS internals, V8's bytecode
- Dart VM architecture
- JVM structure main, JVM internals I, JVM internals Beginners
Collective Courses
Collection of resources which includes 2 or more of the topics discussed above:
- Nand To Tetris A course to teach you about how to build a computer, OS and a compiler form stratch.
- Dive Into Systems A really good book to introduce you with systems programming.