This is a graduate class on software reverse engineering. The focus of the class is:
Understand how to write code that analyzes compiled programs.
This class covers three areas: 64-bit Intel assembly language (with some excursions into ARM), lots of tools for analyzing programs, and automated methods for reverse engineering software. There is some formal methods stuff in here, some Python programming, and some assembly development. Did you ever want to know way too much about how your computer works? This might be the class for you.
Want to know what it is like? The 2023 lecture series is on YouTube.
Every time this class is taught it changes a bit based on prior results, current research, and new technologies. For example, the prior class added a discussion of the various protection schemes (like stack canaries) built into modern executables.
What do you need?
You will need a computer on which you can run Ubuntu. This computer can be almost anything. The best case would be a modern 64-bit Intel-based computer running Linux, Windows, or macOS. You would then run Ubuntu under a virtual machine such as VirtualBox. If you have an M1 or M2 Mac, you have two options. The latest versions of VirtualBox support Apple silicon, and you can run Linux for free using a virtual machine: see the video How To Install Ubuntu 22.04 On M1 Mac for how to do this.
Do you need to know assembly?
No. But you must have strong programming skills, preferably in a low-level language like C. Understanding pointers and the difference between the stack and the heap is especially important.
The class opens with a quick course on the AMD64 instruction set architecture before turning to analytical techniques.
If you have, for some odd reason, learned 32-bit Intel assembly, that’s great, but the AMD64 architecture is very different.
What will we learn?
The class focuses on writing programs that analyze other programs. This is different from reverse engineering programs by examining them in a tool like Ghidra. This class will cover many of the algorithms that are used by Ghidra to analyze a program.
Some topics are control flow analysis, data flow analysis, liveness, slicing, and type recovery, building to using SMT solvers and concolic execution.
Time permitting we will cover a bit of ARM and also Windows.
To do all this, we are going to also have to learn some computer architecture and operating system details that you have probably ignored until now. You need a lower-level understanding of this stuff to understand what is going on.
We use Python in this class, so you might want to learn some Python 3.
Fall 2025
The next time I am scheduled to teach this class is in Fall 2025. More information will appear here as that gets closer.