CI tests take too long
bug
It's been impressive to see our test suites expanded over the years, but the current situation is a little painful.
CI takes 20+ minutes, the vast majority of which results from running the suite of cram tests. (These are parallalised over the python/biopython matrix, but for each combination the tests run sequentially.)
We often have a situation where we run CI on the PR ([e.g.](https://github.com/nextstrain/augur/actions/runs/25768898851)), then wait for run CI on the master branch post-merge ([e.g.](https://github.com/nextstrain/augur/actions/runs/25770332275)), then run it all again in order to release a new version ([e.g.](https://github.com/nextstrain/augur/actions/runs/25771313633)). **This takes over an hour!**
On my machine sequential cram tests take ~11 minutes. Timing each reveals some clear bottlenecks:
```console
$ for t in tests/functional/**/**/*.t; do start=$SECONDS; cram --shell=/opt/homebrew/bin/bash -q "$t" >/dev/null 2>&1; ec=$?; echo "$((SECONDS - start))s exit=$ec $t"; done | sort -rn
79s exit=0 tests/functional/merge/cram/merge-metadata.t
29s exit=0 tests/functional/tree/cram/iqtree-more-threads.t
19s exit=0 tests/functional/subsample/cram/proximal-subsampling.t
16s exit=0 tests/functional/measurements_export.t # not part of the above glob
14s exit=0 tests/functional/curate/cram/metadata-input.t
11s exit=0 tests/functional/export_v2/cram/metadata-columns.t
11s exit=0 tests/functional/curate/cram/titlecase.t
10s exit=0 tests/functional/tree/cram/iqtree-override-args.t
10s exit=0 tests/functional/subsample/cram/proximal-subsampling-errors.t
10s exit=0 tests/functional/merge/cram/merge-metadata-and-sequences.t
```
At the very least we should see if we can make these 10s+ tests run faster.
We should also see if we can parallalise them. In theory this should be possible, but we have a range of test-setup code and so some of the tests may not correctly utilise the temp directory setup typical of cram. I _think_ each job in the pytest-cram matrix runs on a 4 vCPU box itself.
Finally, I'm not keen on adding more complexity to `ci.yaml` but running the entire test suite 3 times when the underlying code is identical is a little silly. Are there any simple solutions to this?
关闭于 2026-05-18 5 条评论