Benchmarking Tips for Go
Go’s tooling for benchmarking is quite powerful. Here are some tips for benchmarking Go code.
During my sabbatical at UC Berkeley in 2023, I was benchmarking my implementation of the bbhash minimal perfect hash function. I was using the Go benchmarking tool to compare the performance of different variants of my implementation.
To accomplish this, I used the benchstat tool to compare the results of different benchmarks.
You will need to install it as follows:
go install golang.org/x/perf/cmd/benchstat@latest
The most trivial use of the benchstat tool is the following:
benchstat old.txt new.txt
Where old.txt and new.txt are the benchmark results you want to compare.
However, you can do much more with benchstat.
Here is an example benchmark that I ran:
go test -run x -bench BenchmarkLookupChunkProofs -benchmem -count=20 -timeout=0 > wi.txt
Notice that this command runs the benchmark 20 times and writes the results to wi.txt.
Moreover, the benchmarks contains custom metrics that I wanted to compare, such as bits per key (bpk), memory bits per key (m_bpk), and protobuf bits time per key (p_bpk).
The output from my benchmarks looked like this:
BenchmarkChunkProofsWithIndex/chunks=1000/gamma=1.0/partitions=32-12 800 1515037 ns/op 13.45 bpk 15.33 m_bpk 16.85 p_bpk 1265072 B/op 5614 allocs/op
BenchmarkChunkProofsWithIndex/chunks=1000/gamma=1.0/partitions=32-12 778 1502760 ns/op 13.45 bpk 15.33 m_bpk 16.85 p_bpk 1265072 B/op 5614 allocs/op
BenchmarkChunkProofsWithIndex/chunks=1000/gamma=1.0/partitions=32-12 796 1509592 ns/op 13.45 bpk 15.33 m_bpk 16.85 p_bpk 1265078 B/op 5614 allocs/op
BenchmarkChunkProofsWithIndex/chunks=1000/gamma=1.0/partitions=32-12 783 1513837 ns/op 13.45 bpk 15.33 m_bpk 16.85 p_bpk 1265072 B/op 5614 allocs/op
Here are some examples that I used to extract a csv file from benchmark results in wi.txt:
benchstat -table "" -col "/chunks /gamma" -row /partitions -filter ".unit:ns/op" -format csv wi.txt > cpu-gamma.csv
benchstat -table "" -col "/chunks /gamma" -row /partitions -filter ".unit:bpk" -format csv wi.txt > bpk-gamma.csv
benchstat -table "" -col "/chunks /gamma" -row /partitions -filter ".unit:m_bpk" -format csv wi.txt > mbpk-gamma.csv
benchstat -table "" -col "/chunks /gamma" -row /partitions -filter ".unit:p_bpk" -format csv wi.txt > pbpk-gamma.csv
benchstat -table "" -col /chunks -row /partitions -filter "/gamma:1.0 .unit:ns/op" -format csv wi.txt > cpu-gamma10.csv
benchstat -table "" -col /chunks -row /partitions -filter "/gamma:1.5 .unit:ns/op" -format csv wi.txt > cpu-gamma15.csv
benchstat -table "" -col /chunks -row /partitions -filter "/gamma:2.0 .unit:ns/op" -format csv wi.txt > cpu-gamma20.csv
benchstat -table "" -col /gamma -row /partitions -filter "/chunks:1000 .unit:bpk" -format csv wi.txt > bpk-chunks1000.csv
The above are just some examples, and you’ll want to adapt them to your specific benchmarks.
The -col and -row refer to the columns and rows you want to extract from the benchmark results.
To add custom metrics, you can do stuff like this in your benchmark code:
b.Run(fmt.Sprintf("chunks=%d/gamma=%.1f/partitions=%d", sz, gamma, p), func(b *testing.B) {
b.ResetTimer()
for i := 0; i < b.N; i++ {
cp, _, err := prover.ChunkProofsWithIndex(nonce)
if err != nil {
b.Fatal(err)
}
b.StopTimer()
// Bits per key for the MPHFs only.
b.ReportMetric(cp.BitsPerKey(), "bpk")
// Bits for the protobuf message fields only.
msgFieldsOnlyBits := cp.Size() * 8
b.ReportMetric(float64(msgFieldsOnlyBits)/float64(sz), "m_bpk")
// Bits for the protobuf message (full overhead).
protoBits := proto.Size(cp) * 8
b.ReportMetric(float64(protoBits)/float64(sz), "p_bpk")
b.StartTimer()
}
})
For more information, you can check out the following resources:
I’m happy to consult or assist on the technicalities if that’s useful.
Hein Meling
Professor of Distributed Systems
My main interest is in secure and dependable distributed computing, including cryptocurrencies and blockchains.