syzkaller is an unsupervised coverage-guided kernel fuzzer. Linux kernel fuzzing has the most support, akaros, freebsd, fuchsia, netbsd and windows are supported to varying degrees. Initially, syzkaller was developed with Linux kernel fuzzing in mind, but now itβs being extended to support other OS kernels as well. Most of the documentation at this moment is related to the Linux kernel. For other OS kernels check: Akaros , FreeBSD , Fuchsia , NetBSD , Windows .
The syz-manager process starts, monitors and restarts several VM instances, and starts a syz-fuzzer process inside of the VMs. It is responsible for persistent corpus and crash storage. As opposed to syz-fuzzer processes, it runs on a host with stable kernel which does not experience white-noise fuzzer load.
The syz-fuzzer process runs inside of presumably unstable VMs. The syz-fuzzer guides fuzzing process itself (input generation, mutation, minimization, etc) and sends inputs that trigger new coverage back to the syz-manager process via RPC. It also starts transient syz-executor processes.
Each syz-executor process executes a single input (a sequence of syscalls). It accepts the program to execute from the syz-fuzzer process and sends results back. It is designed to be as simple as possible (to not interfere with fuzzing process), written in C++, compiled as static binary and uses shared memory for communication.
The syz-fuzzer
process generates programs to be executed by syz-executor
based on syscall descriptions described here .
When syzkaller finds a crasher, it saves information about it into workdir/crashes directory. The directory contains one subdirectory per unique crash type. Each subdirectory contains a description file with a unique string identifying the crash (intended for bug identification and deduplication); and up to 100 logN and reportN files, one pair per test machine crash:
- crashes/
- 6e512290efa36515a7a27e53623304d20d1c3e
- description
- log0
- report0
- log1
- report1
...
- 77c578906abe311d06227b9dc3bffa4c52676f
- description
- log0
- report0
...
Descriptions are extracted using a set of regular expressions . This set may need to be extended if you are using a different kernel architecture, or are just seeing a previously unseen kernel error messages.
logN
files contain raw syzkaller
logs and include kernel console output as well as programs executed before the crash. These logs can be fed to syz-repro
tool for crash location and minimization , or to syz-execprog
tool for manual localization . reportN
files contain post-processed and symbolized kernel crash reports (e.g. a KASAN report). Normally you need just 1 pair of these files (i.e. log0
and report0
), because they all presumably describe the same kernel bug. However, syzkaller
saves up to 100 of them for the case when the crash is poorly reproducible, or if you just want to look at a set of crash reports to infer some similarities or differences.
There are 3 special types of crashes:
no output from test machine
: the test machine produces no output whatsoeverlost connection to test machine
: the ssh connection to the machine was unexpectedly closedtest machine is not executing programs
: the machine looks alive, but no test programs were executed for long period of timeThe following components are needed to use syzkaller:
Syzkaller is a coverage-guided fuzzer and therefore it needs the kernel to be built with coverage support, which requires a recent GCC version. Coverage support was submitted to GCC in revision 231296
, released in GCC v6.0.
Besides coverage support in GCC, you also need support for it on the kernel side. KCOV was committed upstream in Linux kernel version 4.6 and can be enabled by configuring the kernel with CONFIG_KCOV=y
. For older kernels you need to backport commit kernel: add kcov code coverage .
To enable more syzkaller features and improve bug detection abilities, itβs recommended to use additional config options.
Syzkaller performs kernel fuzzing on slave virtual machines or physical devices. These slave enviroments are referred to as VMs. Out-of-the-box syzkaller supports QEMU, kvmtool and GCE virtual machines, Android devices and Odroid C2 boards.
These are the generic requirements for a syzkaller VM:
syz-manager
βs configuration. In other words, you should be able to do ssh -i $SSHID -p $PORT root@localhost
without being prompted for a password (where SSHID
is the SSH identification file and PORT
is the port that are specified in the syz-manager
configuration file)./sys/kernel/debug
.To use QEMU syzkaller VMs you have to install QEMU on your host system, see QEMU docs for details. The create-image.sh script can be used to create a suitable Linux image. Detailed steps for setting up syzkaller with QEMU on a Linux host are avaialble for x86-64 and arm64 kernels.
The syzkaller tools are written in Go , so a Go compiler (>= 1.8) is needed to build them.
$HOME/go
.GOROOT=$HOME/go
env var.PATH
, PATH=$HOME/go/bin:$PATH
.GOPATH
env var to some empty dir, say GOPATH=$HOME/gopath
.go get -u -d github.com/google/syzkaller/...
to checkout syzkaller sources.cd $GOPATH/src/github.com/google/syzkaller
and build with make
, which generates compiled binaries in the bin/
folder.Note: if you want to do cross-OS/arch testing, you need to specify TARGETOS
, TARGETVMARCH
and TARGETARCH
arguments to make
. See the Makefile for details.
Start the syz-manager
process as:
./bin/syz-manager -config my.cfg
The syz-manager
process will wind up VMs and start fuzzing in them. The -config
command line option gives the location of the configuration file, which is described here . Found crashes, statistics and other information is exposed on the HTTP address specified in the manager config.
Once syzkaller detected a kernel crash in one of the VMs, it will automatically start the process of reproducing this crash (unless you specified "reproduce": false
in the config). By default it will use 4 VMs to reproduce the crash and then minimize the program that caused it. This may stop the fuzzing, since all of the VMs might be busy reproducing detected crashes.
The process of reproducing one crash may take from a few minutes up to an hour depending on whether the crash is easily reproducible or reproducible at all. Since this process is not perfect, thereβs a way to try to manually reproduce the crash, as described here .
If a reproducer is successfully found, it can be generated in one of the two forms: syzkaller program or C program. Syzkaller always tries to generate a more user-friendly C reproducer, but sometimes fails for various reasons (for example slightly different timings). In case syzkaller only generated a syzkaller program, thereβs a way to execute them to reproduce and debug the crash manually.
github.com/google/syzkaller
github.com/google/syzkaller/blob/master/docs/akaros/README.md
github.com/google/syzkaller/blob/master/docs/configuration.md
github.com/google/syzkaller/blob/master/docs/executing_syzkaller_programs.md
github.com/google/syzkaller/blob/master/docs/freebsd.md
github.com/google/syzkaller/blob/master/docs/fuchsia.md
github.com/google/syzkaller/blob/master/docs/linux/setup_linux-host_qemu-vm_arm64-kernel.md
github.com/google/syzkaller/blob/master/docs/linux/setup_ubuntu-host_qemu-vm_x86-64-kernel.md
github.com/google/syzkaller/blob/master/docs/netbsd.md
github.com/google/syzkaller/blob/master/docs/reproducing_crashes.md
github.com/google/syzkaller/blob/master/docs/syscall_descriptions.md
github.com/google/syzkaller/blob/master/docs/windows.md
github.com/google/syzkaller/blob/master/Makefile
github.com/google/syzkaller/blob/master/pkg/report/report.go#L33
github.com/google/syzkaller/blob/master/tools/create-image.sh
github.com/torvalds/linux/commit/5c9a8750a6409c63a0f01d51a9024861022f6593