# Performance Testing Framework
The `perf` sub-module provides a framework for writing and running performance tests.

## Default Command Options

The flags below match the .NET `Azure.Test.Perf` runner (`PerfOptions.cs`) so
that perf-automation can drive both languages from a single matrix.

| Flag | Short | Default | Description |
| ---- | ----- | ------- | ----------- |
| `--duration` | `-d` | 10 | Measurement-phase duration in seconds. |
| `--warmup` | `-w` | 5 | Warmup duration in seconds. Zero skips warmup. |
| `--parallel` | `-p` | 1 | Number of goroutines executing the test concurrently. |
| `--iterations` | `-i` | 1 | How many times the measurement phase is repeated. |
| `--rate` | `-r` | 0 | Target throughput in ops/sec aggregated across workers. Zero = unlimited. |
| `--status-interval` |  | 1 | Seconds between live status lines. |
| `--latency` | `-l` | false | Track per-operation latency and print a percentile summary. |
| `--job-statistics` |  | false | Print the `#StartJobStatistics` / `#EndJobStatistics` block parsed by perf-automation. |
| `--no-cleanup` |  | false | Skip `Cleanup` / `GlobalCleanup` at end of run. |
| `--sync` |  | false | Accepted for CLI parity; no-op in Go. |
| `--insecure` |  | false | Accepted for CLI parity; the default transport already skips TLS verification for the test proxy. |
| `--test-proxies` | `-x` |  | Semicolon-separated list of test-proxy URLs. |
| `--results-file` |  |  | When combined with `--latency`, writes per-operation results as JSON. |
| `--output-file-prefix` |  |  | Writes run summary artifacts to `<prefix>.json/.csv/.txt/.md`. |
| `--resource-telemetry` |  | false | Print a `runtime.MemStats` / goroutine-count delta at end of run. |
| `--config` |  |  | Path to a workload-config JSON file. |
| `--workload` |  |  | Workload name to select from the config file. |
| `--debug` |  | false | Print extra debug output. |
| `--maxprocs` |  | `runtime.NumCPU()` | Override `GOMAXPROCS` for the run. |
| `--max-io-completion-threads` |  | 0 | Accepted for CLI parity with .NET ThreadPool tuning; no-op in Go. |
| `--max-worker-threads` |  | 0 | Accepted for CLI parity with .NET ThreadPool tuning; no-op in Go. |
| `--min-io-completion-threads` |  | 0 | Accepted for CLI parity with .NET ThreadPool tuning; no-op in Go. |
| `--min-worker-threads` |  | 0 | Accepted for CLI parity with .NET ThreadPool tuning; no-op in Go. |

The runner always samples process CPU and memory in the background; both are
shown live in the status line (`CPU`, `Memory(MiB)`) and as
`averageCpuPercent` / `averageMemoryBytes` in the run-summary artifacts.


## Adding Performance Tests to an SDK

1. Create a performance test directory at `testdata/perf` within your module. For example, the storage performance tests live in `sdk/storage/azblob/testdata/perf`.

2. Run `go mod init` to create a new module. Add the `sdk/internal` module and the module you are testing for to your go.mod file and run `go mod tidy`

3. Create a `struct` that maintains all global values for the performance test (ie. account name, blob name, etc.).

4. Implement `GlobalPerfTest` interface by adding the `NewPerfTest(context.Context, *PerfTestOptions) (PerfTest, error)` and the `GlobalCleanup(context.Context)` functions on the global struct. The `NewPerfTest` method creates a new struct that is responsible for running a performance test in its own `goroutine`. `GlobalCleanup` takes care of any session level cleanup.

5. Implement the `PerfTest` interface on the struct returned from the `NewPerfTest` method. Add a `Run(context.Context)` and a `Cleanup(context.Context)` function. The `Run` method is the method you want to measure the performance of. The `Cleanup` method is for cleaning up within a single `goroutine`, most of the time this will be empty.

6. Create a main package that uses the `perf.Run` function to provide all the global constructor methods. `perf.Run` takes a map[string]PerfMethods

6. (Optional): Create a method that registers local flags.

```golang
package main

import (
    "github.com/Azure/azure-sdk-for-go/sdk/internal/perf"
)

// uploadTestRegister registers flags for the "UploadBlobTest"
// This is optional and does not to be included for every test.
func uploadTestRegister() {
	flag.IntVar(&uploadTestOpts.size, "size", 10240, "Size in bytes of data to be transferred in upload or download tests. Default is 10240.")
}

func main() {
	perf.Run(map[string]perf.PerfMethods{
		"UploadBlobTest":   {Register: uploadTestRegister, New: NewUploadTest},
		"ListBlobTest":     {Register: listTestRegister, New: NewListTest},
		"DownloadBlobTest": {Register: downloadTestRegister, New: NewDownloadTest},
	})
}
```

### Writing a test
The following walks through the `azblob` Blob Upload performance test:

#### `GlobalPerfTest` Interface
This struct handles global set up for your account and spawning structs for each `goroutine`. Make sure to embed the `perf.PerfTestOptions` struct in your global struct. The `NewUploadTest` will be run once per process and is used to spawn each parallel test.
```go
type uploadTestGlobal struct {
	perf.PerfTestOptions
	containerName         string
	blobName              string
	globalContainerClient azblob.ContainerClient
}

// NewUploadTest is called once per process
func NewUploadTest(ctx context.Context, options perf.PerfTestOptions) (perf.GlobalPerfTest, error) {
	u := &uploadTestGlobal{
		PerfTestOptions: options,
		containerName:   "uploadcontainer",
		blobName:        "uploadblob",
	}

	connStr, ok := os.LookupEnv("AZURE_STORAGE_CONNECTION_STRING")
	if !ok {
		return nil, fmt.Errorf("the environment variable 'AZURE_STORAGE_CONNECTION_STRING' could not be found")
	}

	containerClient, err := azblob.NewContainerClientFromConnectionString(connStr, u.containerName, nil)
	if err != nil {
		return nil, err
	}
	u.globalContainerClient = containerClient
	_, err = u.globalContainerClient.Create(context.Background(), nil)
	if err != nil {
		return nil, err
	}

	return u, nil
}
```

The `GlobalCleanup` method cleans up the account, resetting it into the default.
```go
func (m *uploadTestGlobal) GlobalCleanup(ctx context.Context) error {
	connStr, ok := os.LookupEnv("AZURE_STORAGE_CONNECTION_STRING")
	if !ok {
		return fmt.Errorf("the environment variable 'AZURE_STORAGE_CONNECTION_STRING' could not be found")
	}

	containerClient, err := azblob.NewContainerClientFromConnectionString(connStr, m.containerName, nil)
	if err != nil {
		return err
	}

	_, err = containerClient.Delete(context.Background(), nil)
	return err
}
```

`NewPerfTest` is called once per `goroutine`, and creates a new `PerfTest` interface which will be used by each goroutine. This method should also include the setup for the eventual returned struct.
```go
// uploadPerfTest implements perf.PerfTest and is used to test performance of your event
type uploadPerfTest struct {
	*uploadTestGlobal
	perf.PerfTestOptions
	data       io.ReadSeekCloser
	blobClient azblob.BlockBlobClient
}

// NewPerfTest is called once per goroutine
func (g *uploadTestGlobal) NewPerfTest(ctx context.Context, options *perf.PerfTestOptions) (perf.PerfTest, error) {
	u := &uploadPerfTest{
		uploadTestGlobal: g,
		PerfTestOptions:  *options,
	}

	connStr, ok := os.LookupEnv("AZURE_STORAGE_CONNECTION_STRING")
	if !ok {
		return nil, fmt.Errorf("the environment variable 'AZURE_STORAGE_CONNECTION_STRING' could not be found")
	}

	containerClient, err := azblob.NewContainerClientFromConnectionString(
		connStr,
		u.uploadTestGlobal.containerName,
		&azblob.ClientOptions{
			Transporter: u.PerfTestOptions.Transporter,
		},
	)
	if err != nil {
		return nil, err
	}
	bc := containerClient.NewBlockBlobClient(u.blobName)
	u.blobClient = bc

	data, err := perf.NewRandomStream(uploadTestOpts.size)
	if err != nil {
		return nil, err
	}
	u.data = data

	return u, nil
}
```

#### `PerfTest` Interface
The `PerfTest` interface is an interface that is responsible for running a single performance test. Each performance test in run within a single `goroutine`, these `goroutine`s are created by the `perf` framework. The `Run` method is the method that is being measured and the `Cleanup` method is responsible for cleanup work after each goroutine completes or errors out.

```go
func (m *uploadPerfTest) Run(ctx context.Context) error {
	_, err := m.data.Seek(0, io.SeekStart) // rewind to the beginning
	if err != nil {
		return err
	}
	_, err = m.blobClient.Upload(ctx, m.data, nil)
	return err
}

func (m *uploadPerfTest) Cleanup(ctx context.Context) error {
	return nil
}
```

#### Constructor
Each method needs to have a constructor that returns an instantialized `perf.GlobalPerfTest`.
```go
// NewUploadTest is called once per process
func NewUploadTest(ctx context.Context, options perf.PerfTestOptions) (perf.GlobalPerfTest, error) {
	u := &uploadTestGlobal{
		PerfTestOptions: options,
		containerName:   "uploadcontainer",
		blobName:        "uploadblob",
	}

	connStr, ok := os.LookupEnv("AZURE_STORAGE_CONNECTION_STRING")
	if !ok {
		return nil, fmt.Errorf("the environment variable 'AZURE_STORAGE_CONNECTION_STRING' could not be found")
	}

	containerClient, err := azblob.NewContainerClientFromConnectionString(connStr, u.containerName, nil)
	if err != nil {
		return nil, err
	}
	_, err = containerClient.Create(context.Background(), nil)
	if err != nil {
		return nil, err
	}

	return u, nil
}
```

#### `main.go` file
The `main` function must have the `perf.Run` method with a map of each performance test constructor. Flag parsing, test selection, reporting, proxy interfaces, and all other portions are taken care of for the user in the `perf.Run` method.

```go
package main

import (
	"github.com/Azure/azure-sdk-for-go/sdk/internal/perf"
)

func main() {
	perf.Run(map[string]perf.PerfMethods{
		"UploadBlobTest":   {Register: uploadTestRegister, New: NewUploadTest},
		"ListBlobTest":     {Register: listTestRegister, New: NewListTest},
		"DownloadBlobTest": {Register: downloadTestRegister, New: NewDownloadTest},
	})
}
```

### Running with the test proxy

To run with the test proxy, configure your client to route requests to the test proxy using the `Transporter` parameters for the `Transport` in your client options. If the `--test-proxy` flag is not specified, `Transporter` is the same default HTTP client from `azcore`.

```go
func (k *keysPerfTest) GlobalSetup() error {
    ...
    options = &azkeys.ClientOptions{
        ClientOptions: azcore.ClientOptions{
            Transport: k.Transporter,
        },
    }

    client, err := azkeys.NewClient("<my-vault-url>", cred, options)
    if err != nil {return err}
    ...
}
```

### Registering Local Arguments
We use the `flag` library for argument parsing. Each perf test suite can have optional arguments and they are registered using the `perf.RegisterArguments` method, which takes a simple function that registers arguments using `flag.Int`, `flag.String`, etc. If you have no local arguments, you can skip this step.

```go
type uploadTestOptions struct {
	size int
}

var uploadTestOpts uploadTestOptions = uploadTestOptions{size: 10240}

// uploadTestRegister is called once per process
func uploadTestRegister() {
	flag.IntVar(&uploadTestOpts.size, "size", 10240, "Size in bytes of data to be transferred in upload or download tests.")
}
```

The `main.go` file changes to:
```go
package main

import "github.com/Azure/azure-sdk-for-go/sdk/internal/perf"

func main() {
	perf.Run(map[string]perf.PerfMethods{
		"UploadBlobTest":   {Register: uploadTestRegister, New: NewUploadTest},
		"ListBlobTest":     {Register: listTestRegister, New: NewListTest},
		"DownloadBlobTest": {Register: downloadTestRegister, New: NewDownloadTest},
	})
}
```

## Running Performance Tests

First, make sure you have go version 1.17 or later installed. You can install go from the [go.dev](https://go.dev/doc/install) site.

Navigate to your SDK performance test e.g., `sdk/security/keyvault/azkeys/testdata/perf` folder and run `go mod download` to download all the requirements for the performance framework.

To run a single performance test specify the test as the first argument:
```pwsh
go run . CreateEntityTest
```

To specify flags for a performance test, add them after the first argument:
```pwsh
go run . CreateEntityTest --duration 7 --test-proxies https://localhost:5001 --num-blobs 100
```

To run against multiple proxies and with multiple parallel tests running at the same time use the `-p/--parallel` flag
```pwsh
go run . CreateEntityTest --duration 7 --parallel 4 --test-proxies "https://localhost:5001;http://localhost:5000" --num-blobs 100
```

This runs four goroutines of the same test and splits the traffic to two different proxy addresses. The first and third routine will target `https://localhost:5001`, the second and fourth will use `http://localhost:5000`.

For help run `go run . --help`. A list of registered performance tests, global command flags, and local command flags will print out.
