## Jobs not starting

CI jobs are dispatched by a script which responds to both the `ci-build-me`
label and comments by MinaProtocol organization members containing exactly
`!ci-build-me`. If your CI job has not started after adding the `ci-build-me`
label, please comment on the pull request with `!ci-build-me` to attempt to
re-trigger the script.
If no CI jobs started, check that your membership to O(1) Labs/mina organisation
is public. If your membership is private, the jobs will not started and
`!ci-build-me` won't have an impact.

If CI jobs are not running after applying both the `ci-build-me` label and
comment, you may be able to find and fix the error in the script. The script
lives in `frontend/ci-build-me/src/index.js`, and instructions for deploying
the new version are in the readme at `frontend/ci-build-me/README.md`. You
should still follow normal procedure: submit a pull request and await approval
for the changes before attempting to deploy the fixed script.

## Integration test failures

If your CI error is related to a timeout logged by one of the integration test
runners, this is a known issue and re-running the test in the Buildkite 
interface will usually succeed.

If an issue arises, please post an update in both `development` on the Mina
Protocol discord and `engineering-internal` on the O(1) Labs discord with the
details and links to the failures.

## CI environment mismatch

The CI runs its jobs in multiple Docker images. The images that it is using are
specified in `buildkite/src/Constants/ContainerImages.dhall`: the CI uses all
Debian images prefixed by `minaToolchainBullseye`.

These images are generated by the CI itself, in particular based on the content of the
`dockerfiles` directory and the `opam.export` file (which describes versions of
OCaml packages). If you PR modifies how the images are generated (for example by
changing a package version in `opam.export`), then the CI will not automatically
use these new images and will potentially fail.

In this case, you should:
1. Let the CI run once with the old images. Jobs may fail, but ensure that jobs
   of the form `Docker: toolchain-*` succeed. This is the step that creates new
   Docker images from your PR.
2. Look for the newly generated images on
   [Google Cloud](https://console.cloud.google.com/gcr/images/o1labs-192920/global/mina-toolchain)
   where they have been automatically uploaded. Look for your branch name and
   commit hash in the second column. You should find several images suffixed
   with `-bullseye` or `-focal` (Debian versions).
3. For each such image, retrieve its full name and hash by hovering its link and
   clicking the `Copy full image name` tooltip that appears (or retrieve it on
   the image's page).
4. Edit `buildkite/src/Constants/ContainerImages.dhall` and replace the name of
   each `minaToolchain-*` image by the ones you retrieved.
5. Commit the change, the CI should now be using the new images.

## Contact

If you have a bugfix for failing CI, or are seeing a CI failure across
multiple PRs, the best people to contact are:
* @bkase (bkase#2492 on discord) (Europe - misc.)
* @lk86 (linuskrom#2287 on discord) (US West Coast)
* @OxO1 (awilson#6424 on discord) (US West Coast)
* @mrmr1993 (matthew#4797 on discord) (UK)
