# Performance measurements

This directory contains scripts that measure the start-up performance of the Theia frontend in both the browser and the Electron examples.

The frontend's start-up time is measured using the timestamp of the last recorded `Largest contentful paint (LCP)` candidate metric.
In addition, the scripts scrape Theia's Stopwatch log output to capture complementary metrics; see [Log-based metrics](#log-based-metrics) below.

## Running the browser start-up script

### Quick Start

Execute `npm run performance:startup:browser` in the root directory to startup the backend and execute the script.

### Prerequisites

To run the script the Theia backend needs to be started.
This can either be done with the `Launch Browser Backend` launch config or by running `npm run start` in the `examples/browser-app` directory.
For the per-contribution Stopwatch breakdown to be captured, the backend must be started at debug log level (e.g. `npm run start -- --log-level=debug`); otherwise only aggregate metrics will appear in the script's summary.
The bundled `npm run performance:startup:browser` flow already does this.

### Executing the script

The script can be executed using `node browser-performance.js` in this directory.

The script accepts the following optional parameters:

- `--name`: Specify a name for the current measurement (default: `Browser Frontend Startup`)
- `--url`: Point Theia to a url for example for specifying a specific workspace (default: `http://localhost:3000/#/<pathToMeasurementScript>/workspace`)
- `--folder`: Folder name for the generated tracing files in the `profiles` folder (default: `browser`)
- `--runs`: Number of runs for the measurement (default: `10`)
- `--headless`: Boolean, if the tests should be run in headless mode (default: `true`)

_**Note**: When multiple runs are specified the script will calculate the mean and the standard deviation of all values._

## Running the Electron start-up script

### Quick Start

Execute `yarn run performance:startup:electron` in the root directory to execute the script.

### Prerequisites

To run the script the Theia Electron example needs to be built. In the root directory:

```console
npm install
npm run build:electron
```

### Executing the script

The script can be executed using `node electron-performance.js` in this directory.

The script accepts the following optional parameters:

- `--name`: Specify a name for the current measurement (default: `Electron Frontend Startup`)
- `--folder`: Folder name for the generated tracing files in the `profiles` folder (default: `electron`)
- `--workspace`: Absolute path to a Theia workspace to open (default: an empty workspace folder)
- `--runs`: Number of runs for the measurement (default: `10`)
- `--debug`: Whether to log debug information to the console. Currently, this is only the standard error of the Electron app, which ordinarily is suppressed because the child process is detached

_**Note**: When multiple runs are specified the script will calculate the mean and the standard deviation of all values, except for any runs that failed to capture a measurement due to an exception._

It can happen that the Electron app does not start normally because the native browser modules are not properly built for the Electron target.
The symptom for this is usually an error about a module not self-registering; when this condition is detected, the script stops rather than print out an inevitable series of failures to measure the performance.

## Measure impact on startup performance of extensions

To measure the startup performance impact that extensions have on the application, another script is available, which uses the measurements from the `browser-performance.js` or `electron-performance.js` script.
The `extension-impact.js` script runs the measurement for a defined base application (`base-package.json` in this directory) and then measures the startup time when one of the defined extensions is added to the base application.
The script will then print a table (in CSV format) to the console (and store it in a file) which contains the mean, standard deviation (Std Dev) and coefficient of variation (CV) for each extensions run.
Additionally, each extensions entry will contain the difference to the base application time.

Example Table:

| Extension Name     | Mean (10 runs) (in s) | Std Dev (in s) | CV (%) | Delta (in s) |
| ------------------ | --------------------- | -------------- | ------ | ------------ |
| Base Theia         | 2.027                 | 0.084          | 4.144  | -            |
| @theia/core:1.19.0 | 2.103                 | 0.041          | 1.950  | 0.076        |

### Script usage

The script can be executed by running `node extension-impact.js` in this directory.

The following parameters are available:

- `--app`: The example app in which to measure performance, either `browser` or `electron` (default: `browser`)
- `--runs`: Specify the number of measurements for each extension (default: `10`)
- `--base-time`: Provide an existing measurement (mean) for the base Theia application. If none is provided it will be measured.
- `--extensions`: Provide a list of extensions (need to be locally installed) that shall be tested (default: all extensions in packages folder)

    _**Note**: Each entry should:_

  - _have the format {name}:{version}_
  - _not contain whitespaces_
  - _and be separated by whitespaces_

    _For example: `--extensions @theia/core:1.19.0 @theia/keymaps:1.19.0`_

- `--yarn`: Flag to trigger a full build at script startup (e.g. to build changes to extensions)
- `--url`: Specify a URL that Theia should be launched with (can be used to specify the workspace to be opened). _Applies only to the `browser` app_ (default: `http://localhost:3000/#/<GIT_ROOT>/scripts/performance/workspace`)
- `--workspace`: Specify a workspace on which to launch Theia. _Applies only to the `electron` app_ (default: `/<GIT_ROOT>/scripts/performance/workspace`)
- `--file`: Relative path to the main output file (default: `./script.csv`)
- `--detail-file`: Relative path to a second CSV that records per-contribution breakdowns (default: derived from `--file` by inserting `-details` before the `.csv` suffix)

_**Note**: If no extensions are provided all extensions from the `packages` folder will be measured._

### Per-contribution detail output

Alongside the main LCP comparison CSV, `extension-impact.js` emits a detail CSV with one row per (extension, metric) pair.
Metrics include the `All frontend/backend contributions settled` aggregate and every per-contribution timing that the Theia Stopwatch emitted during the runs.
This lets you attribute an extension's startup cost to the specific contribution(s) responsible.

Example detail table:

| Extension Name     | Metric                                        | Mean (10 runs) (in s) | Std Dev (in s) | CV (%) |
| ------------------ | --------------------------------------------- | --------------------- | -------------- | ------ |
| @theia/foo:1.19.0  | All frontend contributions settled            | 2.015                 | 0.032          | 1.588  |
| @theia/foo:1.19.0  | Frontend FooFrontendContribution.onStart      | 0.087                 | 0.006          | 6.897  |
| @theia/foo:1.19.0  | Frontend FooFrontendContribution.initialize   | 0.004                 | 0.001          | 25.000 |

## Log-based metrics

The Theia Stopwatch emits startup timing logs in a well-defined format that the performance scripts scrape:

```
<activity>: <ms> ms [<seconds> s since {backend process|frontend page} start]
```

Log lines the scripts currently capture:

- `All frontend contributions settled`: logged by the frontend once all contributions whose lifecycle methods returned promises have resolved (i.e. steady state, as opposed to first paint).
- `All backend contributions settled`: the backend equivalent.
- `Frontend <ContributionName>.<phase>`: per-contribution timing for `initialize`, `configure`, `onStart`, `initializeLayout`, and `onDidInitializeLayout`.
- `Backend <ContributionName>.<phase>`: per-contribution timing for `initialize`, `configure`, and `onStart`.
- `Frontend <ContributionName> settled` / `Backend <ContributionName> settled`: per-contribution settlement, logged only when a contribution returned promises from more than one lifecycle method.

Where these logs are visible:

| Metric                                     | Browser                  | Electron                   |
| ------------------------------------------ | ------------------------ | -------------------------- |
| LCP                                        | Chrome trace             | Chromium trace             |
| `All frontend contributions settled`       | Puppeteer page console   | Electron child stdout/stderr |
| `All backend contributions settled`        | _(parent process, n/a)_  | Electron child stdout/stderr |
| `Frontend <Contribution>.<phase>`          | Puppeteer page console   | Electron child stdout/stderr |
| `Backend <Contribution>.<phase>`           | _(parent process, n/a)_  | Electron child stdout/stderr |

Theia's frontend logger forwards all console output to the backend over RPC, so the Electron child process's stdout/stderr contains both frontend and backend Stopwatch lines.
The browser script cannot observe backend logs because the backend runs as a separate process whose stdout is not captured.
