Fully [reproducible builds]() are important because they bridge the gap between auditable open source and convenient binary artifacts. Technologies like [TUF]() and [Binary Transparency]() provide accountability for what binaries are shipped to users, but that's of limited utility if there is no way (short of reverse engineering) of proving that the binary is in fact the result of compiling the intended source. That's why the Debian project is putting [tremendous effort]() into making packages reproducible. The good news is that **Go builds are reproducible by default**. ## Prerequisites There are a few common sense requirements. * Of course, the builds must be reproducible in the [weaker sense](): that means the source code must match perfectly. * This includes dependencies, so the project has to **vendor** them strictly. This is important beyond binary reproducibility: you don't want for "version 1.3" of a software to mean different things based on when it was built. * The compiler version must be the same. * GOPATH and GOROOT must match ([#16860]()), annoyingly, as they are all over the binary in debug file paths. * EDIT: As [Shawn Walker suggests on Twitter](), you can strip the paths with `-asmflags -trimpath`. (Of course only works if you control the original build.) * Note: the default GOROOT, the one that the compiler will use if the environment variable is not set, must also match, since it will be copied into binaries ([#17943]()). You can only change that by recompiling the toolchain in the right directory. * In cgo be dragons ([#15405](), [#19964](), [#9206]()), meaning that [it's possible to get reproducible builds since 1.7]() but it depends on the C linker. Interestingly, the build host architecture does not matter. In other words, **builds are reproducible across cross-compiling**. ## Reproducing rclone I picked [rclone]() for this exercise because it's a self-contained Go binary that vendors dependencies and [offers binary installs](). Here's the binaries we will try to reproduce. bfe0d7e041b4020001b6c48ff170e727243855cbb447f96d983e05b04c090ea8 rclone-v1.36-windows-386/rclone.exe 71827d554c5d860d302ec76d79dcd8433fe63065eac5df4d81b4d2bbefc760b3 rclone-v1.36-linux-amd64/rclone 61ab593c6a007e54c63e64ff2b6ee66dba77c40e12d8ca6b81cf50e8272f43b3 rclone-v1.36-openbsd-amd64/rclone ### Detecting parameters To start, we need to figure out the GOPATH and GOROOT values they were built with. This is easy to figure out [using `debug/gosym`]() and debug information to query the file path of known functions. (PE support is... left as an exercise to the reader.) $ go run gosym.go rclone-v1.36-linux-amd64/rclone /home/ncw/go/src/github.com/ncw/rclone/rclone.go /opt/go/go1.8/src/runtime/extern.go So the GOPATH is `/home/ncw/go` and the GOROOT is `/opt/go/go1.8`. For the compiler version I don't have a good solution (that will work even if DWARF is stripped), so I'll give you a bad one, that relies on the global variable backing `runtime.Version()`. $ egrep -a -o 'go[0-9\.]+' rclone-v1.36-linux-amd64/rclone go. go1.8 go1.8 Yes, it's literally `strings`. You're also on your own for the compiler's default GOROOT, but `strings` will bring it up. Finally, you might have to look at the project docs to find out what flags they use. [rclone uses `-s`, `-X` and `CGO_ENABLED=0`](). ### Reproducing it Since the host architecture does not matter but the environment does, we'll use Docker to do our build. FROM debian:jessie RUN apt-get update && apt-get install -y unzip wget tar ca-certificates git build-essential RUN wget https://storage.googleapis.com/golang/go1.8.linux-amd64.tar.gz RUN tar xvf go1.8.linux-amd64.tar.gz RUN mkdir -p /opt/go && cp -r go /opt/go/go1.8 RUN cd /opt/go/go1.8/src && GOROOT_BOOTSTRAP=/go ./make.bash ENV PATH "/opt/go/go1.8/bin:$PATH" RUN mkdir -p /home/ncw/go/src/github.com/ncw/ RUN cd /home/ncw/go/src/github.com/ncw && git clone https://github.com/ncw/rclone RUN cd /home/ncw/go/src/github.com/ncw/rclone && git checkout v1.36 ENV GOPATH /home/ncw/go ENTRYPOINT ["go"] $ docker run -it --rm -v $(pwd):$(pwd) -w $(pwd) -e CGO_ENABLED=0 4f6d1bc86d5e \ build --ldflags "-s -X github.com/ncw/rclone/fs.Version=v1.36" \ -o rclone-v1.36-linux-amd64/rclone.ours github.com/ncw/rclone To cross-compile, I just added the GOOS and GOARCH environment variables with `docker run -e`. ### Debugging Reproducing someone else's build is not always easy. And indeed, my rclone build mismatched. The first thing to look at is the Build ID. The Build ID is a hash of the filenames of the compiled files, plus the version of the compiler (and other things in zversion.go, like the default GOROOT). See [pkg.go](). You can read it with `readelf -x .note.go.buildid` or by [extracting it from the text section](). If the build ID does not match, the first thing you can compare are the paths of all symbols, again with gosym. Here's a slight patch to the gosym.go script we used above: for _, fu := range table.Funcs { path, _, _ := table.PCToLine(fu.Entry) fmt.Println(path) } If the build ID matches, then you're looking at compiler flags. Failing all that, strings and vbindiff are your friend. What got me with rclone was not rebuilding the compiler in the new location to get the right default GOROOT—the `make.bash` step of the Dockerfile. If you enjoy debugging, [here's the tootstorm on Mastodon](). ### Result bfe0d7e041b4020001b6c48ff170e727243855cbb447f96d983e05b04c090ea8 rclone-v1.36-windows-386/rclone.exe bfe0d7e041b4020001b6c48ff170e727243855cbb447f96d983e05b04c090ea8 rclone-v1.36-windows-386/rclone.ours 71827d554c5d860d302ec76d79dcd8433fe63065eac5df4d81b4d2bbefc760b3 rclone-v1.36-linux-amd64/rclone 71827d554c5d860d302ec76d79dcd8433fe63065eac5df4d81b4d2bbefc760b3 rclone-v1.36-linux-amd64/rclone.ours 61ab593c6a007e54c63e64ff2b6ee66dba77c40e12d8ca6b81cf50e8272f43b3 rclone-v1.36-openbsd-amd64/rclone 61ab593c6a007e54c63e64ff2b6ee66dba77c40e12d8ca6b81cf50e8272f43b3 rclone-v1.36-openbsd-amd64/rclone.ours So good news, rclone is not backdoored! If you enjoy these exercises, you can [follow me on Twitter]() or [Mastodon](). > Good news, rclone is not backdoored! Why reproducible builds matter, and how easy they are in Go. [pic.twitter.com/xsSqKvyX8w]() > > -- Filippo Valsorda (@FiloSottile) [23 April 2017]()

Reproducing Go binaries byte-by-byte