A very simple hypervisor for learning experience.
This project is really about playing with intel hardware virtualization features, which for me helps understanding how they work.
Create a hypervisor project such that:
The project consists of several parts:
The hypervisor is a self contained ELF binary that aims to be cross platform. The Linux / Windows / UEFI loader drivers are there to load the hypervisor under Linux, Windows, and UEFI respectively.
The project is configured using the
environment.config file located at the project root.
This file configures important makefile variables for compiling and debugging environment.
# The supported architectures. SUPPORTED_ARCHITECTURES := x86_64 # Selected architecture to build. SELECTED_ARCHITECTURE := x86_64 # For SSH deployment, the SSH target, port, and password. SSH_TARGET := [email protected] SSH_PORT := 22 SSH_PASSWORD := password1 # For GDB debugging, the GDB server address. GDB_SERVER_ADDRESS := :1337 # The drivers to build, linux, windows, uefi and both. BUILD_DRIVERS := linux windows uefi # Whether the hypervisor is configured to wait for debugger. HYPERVISOR_WAIT_FOR_DEBUGGER := 0 # The linux kernel version for the linux driver. LINUX_KERNEL := 4.18.0-15-generic # Windows build dependencies can be referenced from either # the windows side, WSL, or an arbitrary linux machine with # access to the needed resources below. ROOT := ifeq ($(OS), Windows_NT) ROOT := C: else ROOT := /mnt/c endif # Windows build dependencies. VISUAL_STUDIO_ROOT := $(ROOT)/Program\ Files\ \(x86\)/Microsoft\ Visual\ Studio/2017/Community/VC/Tools/MSVC/14.16.27023 WINDOWS_KITS_ROOT := $(ROOT)/Program\ Files\ \(x86\)/Windows\ Kits/10 WINDOWS_KITS_VERSION := 10.0.18362.0 EDK2_ROOT := $(ROOT)/Temp/edk2-UDK2018 ANDROID_NDK_ROOT := $(ROOT)/CustomPrograms/android-ndk-r19b LLVM_ROOT := $(ROOT)/Program\ Files/LLVM
This section describes what is needed to compile the project and its subprojects.
The list of requirements varies between the host system that is used for the build and the loader drivers that are participating in the build.
Download / Install:
C:/Program Files/Git/usr/binas first directory.
Download / Install:
Make sure the
./environment.config file contains your correct paths and settings in
BUILD_DRIVERSconfiguration to build Linux/Windows/UEFI drivers or both. To compile just the hypervisor, leave
HYPERVISOR_WAIT_FOR_DEBUGGERto whether or not you wish the hypervisor to wait for debugging.
LINUX_KERNELvariable to control linux headers version.
EDK2_ROOTfield to the EDK2 directory.
ANDROID_NDK_ROOTto the LLVM installation folder and
After completing the above steps, just run
make -j or
make -j mode=release for to build the
project for debug or release configuration respectively.
The build output will be located at the
To load the hypervisor on a remote Linux machine, use the
./environment/linux_load.sh script that will
push the loader driver which has the hypervisor binary within and run it.
The loader driver will load the hypervisor and exit immediately afterwards, due to an intentional error code return to the Linux kernel, the error code is EPERM.
./environment/linux_load.sh script requires
sshpass to avoid having to type the password in SSH,
therefore it needs to be installed.
There is currently no script that automatically loads the hypervisor for Windows, thus, we have to load the driver manually.
To load the hypervisor on a Windows machine, move the driver located at
to the machine and load the driver using:
> sc create zpp_loader type=kernel binPath=C:/path/to/zpp_loader.sys > sc start zpp_loader
Remember to turn off integrity checks beforehand.
The windows driver returns an error code of
STATUS_INSUFFICIENT_POWER when it succeeds, to unload itself.
I am yet to be familiar with UEFI best practices. Until now I have used a rather violant
mount of the EFI partition and replaced the
*.efi image that I knew was the main boot selection.
Currently the UEFI support is really experimental, there are probably many issues that are invisible to me, some are multi core issues that I did not handle and yet to even have the knowlege to solve. I am planning to learn more about those issues and solve them eventually.
Debugging the project can be done using
gdb together with VMWare or Qemu-KVM that are configured
to allow nested virtualization and expose a
A friendly reminder for
Visual Studio users in Windows, is that it supports connecting to a
gdb stub allowing
pretty much the same debugging experience as any other application compiled in
The hypervisor can be configured using the
environment.config file to perform a busy loop until
a debugger is attached and changes the loop variable
gdb_attached, make sure to enable this and compile the
./hypervisor/src/hypervisor/main.cpp file again to enjoy the refreshed setting.
While the hypervisor is inside the busy loop, we typically have the instruction pointer within our module which makes it easy for our debugging scripts to find the module base and load symbol information, as well as getting out of the loop by changing the loop variable.
debugStub.listen.guest64.remote = "TRUE" debugStub.port.guest64 = "1337" debugStub.hideBreakpoints = "TRUE" monitor.debugOnStartGuest64 = "TRUE"
I recommend using the
hideBreakpoints configuration listed above,
which makes gdb uses hardware breakpoints instead of patching the code which has lead me to unpleasant pitfalls.
This configurtion however limits the amount of breakpoints to four, use wisely.
In addition, I recommend using the
debugOnStartGuest64 configuration listed above as well as it waits for you to attach to the VM before starting
to use it.
When you build the project, a directory named
environment will be created according to this configuration with
useful environmental scripts.
This is fairly simple, just add the following option to the qemu-kvm launch command line:
Once having the debug machine ready and waiting for connection, run the following command from the root directory of the project:
Once inside gdb, once your instruction pointer is within the hypervisor, use the
zstartl command that was added
to gdb in the command file given to it. This command will look for the ELF header of the hypervisor and load symbols.
Once having the debug machine ready and waiting for connection, launch the command window of Visual Studio using the Ctrl+Alt+A shortcut, and define the following alias:
>alias d Debug.MIDebugLaunch /Executable:C:/ /OptionsFile:C:/Projects/git/zpp_hypervisor/environment/options.xml
This alias will be used to attach to the target machine
Notice that the executable switch to the command is not used so it is safe to leave it as
C:/, as for the
to provide a full path to the
options.xml file inside the
environment/options.xml located at the project root.
Also, I recommend defining the following alias to easily execue
gdb commands within
>alias e Debug.MIDebugExec
Now, to attach to the target machine, execute the following command within the command window:
To load symbols once the instruction pointer is within the hypervisor, execute the following command in the command window:
I hope that you enjoy using this project and feel free to report any issues.