Lucene search

K
n0whereN0whereN0WHERE:172399
HistoryNov 14, 2017 - 8:21 p.m.

Unsupervised Coverage-Guided Kernel Fuzzer: syzkaller

2017-11-1420:21:34
n0where.net
176

syzkaller is an unsupervised coverage-guided kernel fuzzer. Linux kernel fuzzing has the most support, akaros, freebsd, fuchsia, netbsd and windows are supported to varying degrees. Initially, syzkaller was developed with Linux kernel fuzzing in mind, but now it’s being extended to support other OS kernels as well. Most of the documentation at this moment is related to the Linux kernel. For other OS kernels check: Akaros , FreeBSD , Fuchsia , NetBSD , Windows .

The syz-manager process starts, monitors and restarts several VM instances, and starts a syz-fuzzer process inside of the VMs. It is responsible for persistent corpus and crash storage. As opposed to syz-fuzzer processes, it runs on a host with stable kernel which does not experience white-noise fuzzer load.

The syz-fuzzer process runs inside of presumably unstable VMs. The syz-fuzzer guides fuzzing process itself (input generation, mutation, minimization, etc) and sends inputs that trigger new coverage back to the syz-manager process via RPC. It also starts transient syz-executor processes.

Each syz-executor process executes a single input (a sequence of syscalls). It accepts the program to execute from the syz-fuzzer process and sends results back. It is designed to be as simple as possible (to not interfere with fuzzing process), written in C++, compiled as static binary and uses shared memory for communication.

Syscall descriptions

The syz-fuzzer process generates programs to be executed by syz-executor based on syscall descriptions described here .

Crash reports

When syzkaller finds a crasher, it saves information about it into workdir/crashes directory. The directory contains one subdirectory per unique crash type. Each subdirectory contains a description file with a unique string identifying the crash (intended for bug identification and deduplication); and up to 100 logN and reportN files, one pair per test machine crash:

 - crashes/
   - 6e512290efa36515a7a27e53623304d20d1c3e
     - description
     - log0
     - report0
     - log1
     - report1
     ...
   - 77c578906abe311d06227b9dc3bffa4c52676f
     - description
     - log0
     - report0
     ...

Descriptions are extracted using a set of regular expressions . This set may need to be extended if you are using a different kernel architecture, or are just seeing a previously unseen kernel error messages.

logN files contain raw syzkaller logs and include kernel console output as well as programs executed before the crash. These logs can be fed to syz-repro tool for crash location and minimization , or to syz-execprog tool for manual localization . reportN files contain post-processed and symbolized kernel crash reports (e.g. a KASAN report). Normally you need just 1 pair of these files (i.e. log0 and report0 ), because they all presumably describe the same kernel bug. However, syzkaller saves up to 100 of them for the case when the crash is poorly reproducible, or if you just want to look at a set of crash reports to infer some similarities or differences.

There are 3 special types of crashes:

  • no output from test machine : the test machine produces no output whatsoever
  • lost connection to test machine : the ssh connection to the machine was unexpectedly closed
  • test machine is not executing programs : the machine looks alive, but no test programs were executed for long period of time

Install

The following components are needed to use syzkaller:

  • C compiler with coverage support
  • Linux kernel with coverage additions
  • Virtual machine or a physical device
  • syzkaller itself

C Compiler

Syzkaller is a coverage-guided fuzzer and therefore it needs the kernel to be built with coverage support, which requires a recent GCC version. Coverage support was submitted to GCC in revision 231296 , released in GCC v6.0.

Linux Kernel

Besides coverage support in GCC, you also need support for it on the kernel side. KCOV was committed upstream in Linux kernel version 4.6 and can be enabled by configuring the kernel with CONFIG_KCOV=y . For older kernels you need to backport commit kernel: add kcov code coverage .

To enable more syzkaller features and improve bug detection abilities, it’s recommended to use additional config options.

VM Setup

Syzkaller performs kernel fuzzing on slave virtual machines or physical devices. These slave enviroments are referred to as VMs. Out-of-the-box syzkaller supports QEMU, kvmtool and GCE virtual machines, Android devices and Odroid C2 boards.

These are the generic requirements for a syzkaller VM:

  • The fuzzing processes communicate with the outside world, so the VM image needs to include networking support.
  • The program files for the fuzzer processes are transmitted into the VM using SSH, so the VM image needs a running SSH server.
  • The VM’s SSH configuration should be set up to allow root access for the identity that is included in the syz-manager β€˜s configuration. In other words, you should be able to do ssh -i $SSHID -p $PORT root@localhost without being prompted for a password (where SSHID is the SSH identification file and PORT is the port that are specified in the syz-manager configuration file).
  • The kernel exports coverage information via a debugfs entry, so the VM image needs to mount the debugfs filesystem at /sys/kernel/debug .

To use QEMU syzkaller VMs you have to install QEMU on your host system, see QEMU docs for details. The create-image.sh script can be used to create a suitable Linux image. Detailed steps for setting up syzkaller with QEMU on a Linux host are avaialble for x86-64 and arm64 kernels.

Syzkaller

The syzkaller tools are written in Go , so a Go compiler (>= 1.8) is needed to build them.

  • Go distribution can be downloaded from https://golang.org/dl/ .
  • Unpack Go into a directory, say, $HOME/go .
  • Then, set GOROOT=$HOME/go env var.
  • Then, add Go binaries to PATH , PATH=$HOME/go/bin:$PATH .
  • Then, set GOPATH env var to some empty dir, say GOPATH=$HOME/gopath .
  • Then, run go get -u -d github.com/google/syzkaller/... to checkout syzkaller sources.
  • Then, cd $GOPATH/src/github.com/google/syzkaller and build with make , which generates compiled binaries in the bin/ folder.

Note: if you want to do cross-OS/arch testing, you need to specify TARGETOS , TARGETVMARCH and TARGETARCH arguments to make . See the Makefile for details.

Running

Start the syz-manager process as:

./bin/syz-manager -config my.cfg

The syz-manager process will wind up VMs and start fuzzing in them. The -config command line option gives the location of the configuration file, which is described here . Found crashes, statistics and other information is exposed on the HTTP address specified in the manager config.

Crashes

Once syzkaller detected a kernel crash in one of the VMs, it will automatically start the process of reproducing this crash (unless you specified "reproduce": false in the config). By default it will use 4 VMs to reproduce the crash and then minimize the program that caused it. This may stop the fuzzing, since all of the VMs might be busy reproducing detected crashes.

The process of reproducing one crash may take from a few minutes up to an hour depending on whether the crash is easily reproducible or reproducible at all. Since this process is not perfect, there’s a way to try to manually reproduce the crash, as described here .

If a reproducer is successfully found, it can be generated in one of the two forms: syzkaller program or C program. Syzkaller always tries to generate a more user-friendly C reproducer, but sometimes fails for various reasons (for example slightly different timings). In case syzkaller only generated a syzkaller program, there’s a way to execute them to reproduce and debug the crash manually.

Unsupervised Coverage-Guided Kernel Fuzzer: syzkaller Download

References