# benchmarkjunit

`benchmarkjunit` is a go utility that runs go [`testing` package benchmarks]
with a command like `go test -v -bench=. <packages>` and translates the output
into JUnit XML.

## Usage Examples

Run `go run ./pkg/benchmarkjunit --help` to see all the available flags.

```shell
go run ./pkg/benchmarkjunit k8s.io/test-infra/experiment/dummybenchmarks
```

```shell
go run ./pkg/benchmarkjunit ./experiment/dummybenchmarks/...
```

```shell
go run ./pkg/benchmarkjunit \
-o junit_benchmark.xml -l testlog.txt \
--test-arg=-benchmem --bench=Core --pass-on-error \
./experiment/dummybenchmarks/...
```

With bazel:
```shell
bazel run //pkg/benchmarkjunit -- k8s.io/test-infra/experiment/dummybenchmarks
```

### Running as a ProwJob

`benchmarkjunit` can be used from a ProwJob to generate JUnit results for
consumption in Testgrid, Spyglass, Gubernator, etc. See the 
[`prowjob_example.yaml`](/pkg/benchmarkjunit/prowjob_example.yaml) file.

### Demo Job

The [`ci-test-infra-benchmark-demo`] periodic ProwJob provides a live example
of how to use `benchmarkjunit` and what the results look like. Examples:
- [Testgrid](https://testgrid.k8s.io/sig-testing-canaries#benchmark-demo&width=20&graph-metrics=avg%20op%20duration%20(ns%2Fop)&graph-metrics=test-duration-minutes&graph-metrics=op%20count&graph-metrics=MB%2Fs&graph-metrics=alloced%20B%2Fop&graph-metrics=allocs%2Fop),
- [Spyglass](https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-test-infra-benchmark-demo/1124364619008184320),
- [Raw results](https://gcsweb.k8s.io/gcs/kubernetes-jenkins/logs/ci-test-infra-benchmark-demo/1124364619008184320/artifacts/).

## Translation Example:

The following is the `go test` benchmark output and the corresponding JUnit XML
for the [`experiment/dummybenchmarks`](/experiment/dummybenchmarks) package.
This was generated by running 
`go run ./pkg/benchmarkjunit ./experiment/dummybenchmarks -l testlog.txt`.

#### Benchmark output:

```
goos: linux
goarch: amd64
pkg: k8s.io/test-infra/experiment/dummybenchmarks
BenchmarkCoreSimple-12            	 5000000	       231 ns/op
BenchmarkCoreAllocsAndBytes-12    	10000000	       232 ns/op	  85.84 MB/s	     112 B/op	       1 allocs/op
BenchmarkCoreParallel-12          	50000000	        28.9 ns/op
BenchmarkCoreLog-12               	10000000	       224 ns/op
--- BENCH: BenchmarkCoreLog-12
    dummy_test.go:53: About to DoTheThing() x1.
    dummy_test.go:53: About to DoTheThing() x100.
    dummy_test.go:53: About to DoTheThing() x10000.
    dummy_test.go:53: About to DoTheThing() x1000000.
    dummy_test.go:53: About to DoTheThing() x10000000.
--- SKIP: BenchmarkCoreSkip
    dummy_test.go:60: This Benchmark is skipped.
--- SKIP: BenchmarkCoreSkipNow
--- FAIL: BenchmarkCoreError
    dummy_test.go:68: Early Benchmark error.
    dummy_test.go:53: About to DoTheThing() x1.
--- FAIL: BenchmarkCoreFatal
    dummy_test.go:73: This Benchmark failed.
--- FAIL: BenchmarkCoreFailNow
BenchmarkCoreNestedShallow/simple-12         	 5000000	       236 ns/op
BenchmarkCoreNestedShallow/parallel-12       	50000000	        28.9 ns/op
Benchmark-12                                 	10000000	       226 ns/op
BenchmarkReportAllocs-12                     	10000000	       234 ns/op	     112 B/op	       1 allocs/op
BenchmarkSetBytes-12                         	 5000000	       240 ns/op	  83.00 MB/s
BenchmarkNestedDeep/1_simple-12              	 5000000	       228 ns/op
BenchmarkNestedDeep/1_parallel-12            	50000000	        30.5 ns/op
BenchmarkNestedDeep/3A_simple-12             	10000000	       225 ns/op
BenchmarkNestedDeep/3A-12                    	       1	2453493929 ns/op
BenchmarkNestedDeep/3B_parallel-12           	50000000	        33.1 ns/op
BenchmarkNestedDeep/3B-12                    	       1	1697677606 ns/op
BenchmarkNestedDeep/2-12                     	       1	4152090400 ns/op
BenchmarkNestedDeep/1-12                     	       1	7125087855 ns/op
FAIL
exit status 1
FAIL	k8s.io/test-infra/experiment/dummybenchmarks	25.419s
```

#### JUnit XML:

```xml
<testsuites>
    <testsuite name="k8s.io/test-infra/experiment/dummybenchmarks" time="25.419" failures="3" tests="20">
        <testcase name="BenchmarkCoreSimple-12" time="1.155" classname="dummybenchmarks">
            <properties>
                <property name="op count" value="5000000"></property>
                <property name="avg op duration (ns/op)" value="231"></property>
            </properties>
        </testcase>
        <testcase name="BenchmarkCoreAllocsAndBytes-12" time="2.32" classname="dummybenchmarks">
            <properties>
                <property name="op count" value="10000000"></property>
                <property name="avg op duration (ns/op)" value="232"></property>
                <property name="MB/s" value="85.84"></property>
                <property name="alloced B/op" value="112"></property>
                <property name="allocs/op" value="1"></property>
            </properties>
        </testcase>
        <testcase name="BenchmarkCoreParallel-12" time="1.445" classname="dummybenchmarks">
            <properties>
                <property name="op count" value="50000000"></property>
                <property name="avg op duration (ns/op)" value="28.9"></property>
            </properties>
        </testcase>
        <testcase name="BenchmarkCoreLog-12" time="2.24" classname="dummybenchmarks">
            <system-out>dummy_test.go:53: About to DoTheThing() x1.&#xA;dummy_test.go:53: About to DoTheThing() x100.&#xA;dummy_test.go:53: About to DoTheThing() x10000.&#xA;dummy_test.go:53: About to DoTheThing() x1000000.&#xA;dummy_test.go:53: About to DoTheThing() x10000000.&#xA;</system-out>
            <properties>
                <property name="op count" value="10000000"></property>
                <property name="avg op duration (ns/op)" value="224"></property>
            </properties>
        </testcase>
        <testcase name="BenchmarkCoreSkip" time="0" classname="dummybenchmarks">
            <skipped>dummy_test.go:60: This Benchmark is skipped.&#xA;</skipped>
        </testcase>
        <testcase name="BenchmarkCoreSkipNow" time="0" classname="dummybenchmarks">
            <skipped></skipped>
        </testcase>
        <testcase name="BenchmarkCoreError" time="0" classname="dummybenchmarks">
            <failure>dummy_test.go:68: Early Benchmark error.&#xA;dummy_test.go:53: About to DoTheThing() x1.&#xA;</failure>
            <properties>
                <property name="categorized_fail" value="dummy_test.go:68: Early Benchmark error.&#xA;dummy_test.go:53: About to DoTheThing() x1.&#xA;"></property>
            </properties>
        </testcase>
        <testcase name="BenchmarkCoreFatal" time="0" classname="dummybenchmarks">
            <failure>dummy_test.go:73: This Benchmark failed.&#xA;</failure>
            <properties>
                <property name="categorized_fail" value="dummy_test.go:73: This Benchmark failed.&#xA;"></property>
            </properties>
        </testcase>
        <testcase name="BenchmarkCoreFailNow" time="0" classname="dummybenchmarks">
            <failure></failure>
        </testcase>
        <testcase name="BenchmarkCoreNestedShallow/simple-12" time="1.18" classname="dummybenchmarks">
            <properties>
                <property name="op count" value="5000000"></property>
                <property name="avg op duration (ns/op)" value="236"></property>
            </properties>
        </testcase>
        <testcase name="BenchmarkCoreNestedShallow/parallel-12" time="1.445" classname="dummybenchmarks">
            <properties>
                <property name="op count" value="50000000"></property>
                <property name="avg op duration (ns/op)" value="28.9"></property>
            </properties>
        </testcase>
        <testcase name="Benchmark-12" time="2.26" classname="dummybenchmarks">
            <properties>
                <property name="op count" value="10000000"></property>
                <property name="avg op duration (ns/op)" value="226"></property>
            </properties>
        </testcase>
        <testcase name="BenchmarkReportAllocs-12" time="2.34" classname="dummybenchmarks">
            <properties>
                <property name="op count" value="10000000"></property>
                <property name="avg op duration (ns/op)" value="234"></property>
                <property name="alloced B/op" value="112"></property>
                <property name="allocs/op" value="1"></property>
            </properties>
        </testcase>
        <testcase name="BenchmarkSetBytes-12" time="1.2" classname="dummybenchmarks">
            <properties>
                <property name="op count" value="5000000"></property>
                <property name="avg op duration (ns/op)" value="240"></property>
                <property name="MB/s" value="83.00"></property>
            </properties>
        </testcase>
        <testcase name="BenchmarkNestedDeep/1_simple-12" time="1.14" classname="dummybenchmarks">
            <properties>
                <property name="op count" value="5000000"></property>
                <property name="avg op duration (ns/op)" value="228"></property>
            </properties>
        </testcase>
        <testcase name="BenchmarkNestedDeep/1_parallel-12" time="1.525" classname="dummybenchmarks">
            <properties>
                <property name="op count" value="50000000"></property>
                <property name="avg op duration (ns/op)" value="30.5"></property>
            </properties>
        </testcase>
        <testcase name="BenchmarkNestedDeep/3A_simple-12" time="2.25" classname="dummybenchmarks">
            <properties>
                <property name="op count" value="10000000"></property>
                <property name="avg op duration (ns/op)" value="225"></property>
            </properties>
        </testcase>
        <testcase name="BenchmarkNestedDeep/3A-12" time="2.453493929" classname="dummybenchmarks">
            <properties>
                <property name="op count" value="1"></property>
                <property name="avg op duration (ns/op)" value="2453493929"></property>
            </properties>
        </testcase>
        <testcase name="BenchmarkNestedDeep/3B_parallel-12" time="1.655" classname="dummybenchmarks">
            <properties>
                <property name="op count" value="50000000"></property>
                <property name="avg op duration (ns/op)" value="33.1"></property>
            </properties>
        </testcase>
        <testcase name="BenchmarkNestedDeep/3B-12" time="1.697677606" classname="dummybenchmarks">
            <properties>
                <property name="op count" value="1"></property>
                <property name="avg op duration (ns/op)" value="1697677606"></property>
            </properties>
        </testcase>
        <testcase name="BenchmarkNestedDeep/2-12" time="4.1520904" classname="dummybenchmarks">
            <properties>
                <property name="op count" value="1"></property>
                <property name="avg op duration (ns/op)" value="4152090400"></property>
            </properties>
        </testcase>
        <testcase name="BenchmarkNestedDeep/1-12" time="7.125087855" classname="dummybenchmarks">
            <properties>
                <property name="op count" value="1"></property>
                <property name="avg op duration (ns/op)" value="7125087855"></property>
            </properties>
        </testcase>
    </testsuite>
</testsuites>
```

<!-- links -->
[`testing` package benchmarks]: https://golang.org/pkg/testing/#hdr-Benchmarks
[`ci-test-infra-benchmark-demo`]: https://github.com/kubernetes/test-infra/blob/1423204f375ee7cf99f2a8a61a213562039b2ac7/config/jobs/kubernetes/test-infra/test-infra-canaries.yaml#L33-L51
