ITADN

Reduce containers overhead

#267Openkim-fehl 创建于 2026-03-14
bug
K
kim-fehlcommented
### Description of the bug Currently, there are more than 20 container images in the pipeline totaling ~60 GB of disk usage I remember that for nf-core modules the recommended principle is "one tool – one container", but here we mostly have local modules and I think some redundancy (caused by incremental development) can be reduced. IMAGE | ID | DISK USAGE | CONTENT SIZE -- | -- | -- | -- SEQERA: anndata2ri_bioconductor-singlecellexperiment_anndata_r-seurat:5fae42aabf7a1c5f | 291a48658716 | 3.38GB | 846MB SEQERA: anndata:0.10.9--1eab54e300e1e584 | 3471efcc8b48 | 936MB | 231MB SEQERA: anndata_pyyaml:82c6914e861435f7 | 7946ce8a97db | 1.06GB | 261MB SEQERA: anndata_upsetplot:784e0f450da10178 | 766c2dff4b54 | 1.18GB | 306MB SEQERA: bbknn_pyyaml_scanpy:4cf2984722da607f | 453b5e1f8972 | 1.6GB | 382MB SEQERA: bioconductor-celldex_bioconductor-hdf5array_bioconductor-singlecellexperiment_r-yaml:13bf33457e3e7490 | fdb9aa052292 | 2.33GB | 634MB SEQERA: celltypist_scanpy:44b604b24dd4cf33 | bfe009b0a96c | 1.78GB | 431MB SEQERA: harmonypy_pyyaml_scanpy:f6cc57196369fb1e | 0c62b23a31d6 | 1.63GB | 392MB SEQERA: leidenalg_python-igraph_pyyaml_scanpy:4936fa196b5f4340 | 8644a451da2a | 1.66GB | 401MB SEQERA: liana_pyyaml:776fdd7103df146d | 131e6bd9dccb | 2.24GB | 507MB SEQERA: multiqc:1.33--ee7739d47738383b | abd5751768f8 | 2.01GB | 432MB SEQERA: pandas:2.2.3--9b034ee33172d809 | 50da2ef5f060 | 765MB | 190MB SEQERA: python-igraph_pyyaml_scanpy:cc0304f4731f72f9 | 8f65ff8a2191 | 1.66GB | 401MB SEQERA: python_pyyaml_scanpy:b5509a698e9aae25 | e0dac9eda4d7 | 1.85GB | 461MB SEQERA: python_pyyaml_scanpy_scikit-image:750e7b74b6d036e4 | e2816307a73f | 2.04GB | 509MB SEQERA: pyyaml_scanpy:3c9e9f631f45553d | 7ed2839670f9 | 1.63GB | 392MB SEQERA: pyyaml_scanpy:a3a797e09552fddc | 228c2994c5f4 | 1.86GB | 466MB SEQERA: scanpy_upsetplot:1ce883f3ff369ca8 | a91e0a660553 | 1.67GB | 414MB SEQERA: scvi-tools:1.3.3--df115aabdccb7d6b | 551e3b44c383 | 4.66GB | 1.08GB SEQERA: scvi-tools:1.4.1--47f5b0e6b70fd131 | 0ac460cb48b1 | 3.47GB | 797MB nicotru/celda:1d48a68e9d534b2b | 3a4f38d26238 | 2.95GB | 759MB nicotru/scds:7788dbeb87bc7eec | e6aac618e327 | 2.48GB | 651MB nicotru/seurat:b3b12d17271014d9 | 22f891364efc | 3.35GB | 853MB nicotru/soupx:f6297681695fbfcf | 222d79287a15 | 2.82GB | 700MB saditya88/singler:0.0.1 | cb267ab7d826 | 9.13GB | 2.64GB (This issue has been brought to my attention as I rent the server and also pay for disk space 😃) I asked Codex to analyze repo structure and find some ways to optimize container usage, not touching nf-core/modules and accounting for python version pinning you [mentioned](https://github.com/nf-core/scdownstream/pull/266#discussion_r2931786801). Here's the output: > ## Implementation Plan > - Use nf-core module containers as the canonical baseline for overlapping local tool families. > - Align local `SCVITOOLS_SCVI` and `SCVITOOLS_SCANVI` to the same `scvi-tools=1.3.3` container/env family already used by vendored `SCVITOOLS_SOLO` and `SCVITOOLS_SCAR`. > - Collapse the local generic `scanpy` `1.11.5` / `1.11.2` split onto one pinned local baseline that is compatible with the nf-core `scrublet` stack. > - Standardize local `scanpy` modules that only need core scanpy functionality on `python=3.12.11`, `scanpy=1.11.2`, `pyyaml=6.0.2`. > - Apply the same base version to additive local scanpy envs (`neighbors`, `paga`, `leiden`, `harmony`, `bbknn`) while keeping their extra packages. > - Collapse the local upsetplot fork. > - Change `ADATA_UPSETGENES` to use `anndata` directly instead of `scanpy` for reading `.h5ad`. > - Move `ADATA_UPSETGENES` and `DOUBLET_REMOVAL` onto one shared pinned local env: `python=3.12.11`, `anndata=0.12.7`, `upsetplot=0.9.0`. > - Replace `docker.io/saditya88/singler:0.0.1` with a local Dockerfile built from a minimal R/Wave-compatible base containing the actual R dependencies used by `singleR.R`, including `bioconductor-hdf5array` and `anndataR`. > > ## Test Plan > - Run nf-tests for affected local modules and subworkflows: > - `integrate` > - `quality_control` > - `doublet_detection` > - `celltype_assignment` > - affected local modules under `scanpy`, `scvitools`, `adata/upsetgenes`, `doublet_detection/doublet_removal`, and `celltypes/singler` > - Run pipeline tests with `-profile test,docker` and `-profile test_full,docker`. Does this plan make sense? You also mentioned that private docker hub images can be replaced with Seqera ones, but Codex thought it's too much for one conservative pass :) ### Command used and terminal output ```console ``` ### Relevant files _No response_ ### System information _No response_
1 条评论