# nf-core/scdownstream: Output

## Introduction

This document describes the output produced by the pipeline.
Most of the plots are taken from the MultiQC report, which summarises results at the end of the pipeline.

The directories listed below will be created in the results directory after the pipeline has finished.
All paths are relative to the top-level results directory.

## Pipeline overview

The pipeline is built using [Nextflow](https://www.nextflow.io/) and processes data using the following steps:

1. Per-sample preprocessing
   1. Convert all RDS files to H5AD format
   2. Create filtered matrix (if not provided)
   3. Present QC for raw counts ([`MultiQC`](http://multiqc.info/))
   4. Remove ambient RNA
      - [DecontX](https://bioconductor.org/packages/release/bioc/html/decontX.html)
      - [SoupX](https://cran.r-project.org/web/packages/SoupX/readme/README.html)
      - [CellBender](https://cellbender.readthedocs.io/en/latest/)
      - [scAR](https://docs.scvi-tools.org/en/stable/user_guide/models/scar.html)
   5. Apply user-defined QC filters (can be defined per sample in the samplesheet)
   6. Doublet detection (Majority vote possible)
      - [SOLO](https://docs.scvi-tools.org/en/stable/user_guide/models/solo.html)
      - [Scrublet](https://scanpy.readthedocs.io/en/stable/api/generated/scanpy.pp.scrublet.html)
      - [DoubletDetection](https://doubletdetection.readthedocs.io/en/v2.5.2/doubletdetection.doubletdetection.html)
      - [scDblFinder](https://bioconductor.org/packages/release/bioc/html/scDblFinder.html)
   7. Cell cycle scoring ([Tirosh et al. 2015](https://doi.org/10.1038/nature14590))
2. Sample aggregation
   1. Merge into a single H5AD file
   2. Present QC for merged counts ([`MultiQC`](http://multiqc.info/))
   3. Integration
      - [scVI](https://docs.scvi-tools.org/en/stable/user_guide/models/scvi.html)
      - [scANVI](https://docs.scvi-tools.org/en/stable/user_guide/models/scanvi.html)
      - [Symphony](https://github.com/immunogenomics/symphony) / Harmony (via [symphonypy](https://pypi.org/project/symphonypy/))
      - [BBKNN](https://github.com/Teichlab/bbknn)
      - [Combat](https://scanpy.readthedocs.io/en/latest/api/generated/scanpy.pp.combat.html)
      - [Seurat](https://satijalab.org/seurat/articles/integration_introduction)
      - [PCA](https://scanpy.readthedocs.io/en/stable/generated/scanpy.pp.pca.html)
      - [scimilarity](https://github.com/Genentech/scimilarity)
      - [Expimap](https://docs.scarches.org/en/latest/api/models.html#scarches.models.EXPIMAP)
3. Cell type annotation
   - [CellTypist](https://www.celltypist.org/)
   - [SingleR](https://www.bioconductor.org/packages/release/bioc/html/SingleR.html)
   - [CyteType](https://github.com/NygenAnalytics/cytetype)
4. Clustering and dimensionality reduction
   1. [Leiden clustering](https://scanpy.readthedocs.io/en/stable/generated/scanpy.tl.leiden.html)
   2. [UMAP](https://scanpy.readthedocs.io/en/stable/generated/scanpy.tl.umap.html)
5. Create [Quarto](https://quarto.org/) and [`MultiQC`](http://multiqc.info/) reports

### Per-sample preprocessing

<details markdown="1">
<summary>Output files</summary>

- `01_load_h5ad/` (when `--save_intermediates` is enabled)
  - Converted H5AD files from RDS, 10x H5, or CSV inputs.
- `02_quality_control/`
  - `converted/`: legacy alias in docs; see `01_load_h5ad/` for format conversion outputs when intermediates are saved.
  - `unify/`: Gene symbol unification and metadata standardisation (also `unify/hugounifier/` when HUGO-unifier runs).
  - `empty_droplet_removal/`: Results of empty droplet removal. Only if no `filtered` matrix is provided in the samplesheet.
  - `qc_raw/`: QC plots for the raw input data.
  - `ambient_rna_removal/`: Results of ambient RNA removal.
  - `custom_thresholds/`: Results of applying user-defined QC thresholds.
  - `doublet_detection/`: Directories related to doublet detection.
    - `input_rds/`: RDS version of the H5AD file that is used as input to the doublet detection tools.
    - `(doubletdetection|scdblfinder|scrublet|solo)/`: Results of doublet detection.
      Each directory contains a filtered `h5ad`/`rds` and a `csv`/`pkl` file
      with the doublet annotations.
    - `${sample_id}.h5ad`: The H5AD without doublets.
  - `qc_filtered/`: QC plots for the preprocessed data.
  - `cell_cycle/`: Cell cycle scoring results.
    - `${sample_id}_cellcycle.pkl`: `S_score`, `G2M_score`, and `phase` columns
      for each cell. Merged into the final H5AD via `FINALIZE_QC_ANNDATAS`.
    - `${sample_id}_cellcycle.h5ad`: Intermediate H5AD with cell cycle scores added, available for inspection.
  - `sizes/`: Cell counts at each QC stage (MultiQC table input).
  - `finalized/`: Per-sample QC AnnData objects when `--save_intermediates` is enabled.

</details>

The output of each step is stored under numbered stage directories in the results folder.
Per-sample QC outputs live under `02_quality_control/`.

### Sample aggregation

<details markdown="1">
<summary>Output files</summary>

- `03_combine/`
  - `merge/`
    - `merged_inner.h5ad`: The merged H5AD file with only the intersection of the genes. Will be used for integration.
    - `merged_outer.h5ad`: The merged H5AD file with all genes. Will be used as
      base for the final H5AD file.
    - `merged_sample_genes.png`: UpSet plot showing the overlap of genes between samples.
  - `integrate/`
    - `input_hvg`
      - `*.h5ad`: The H5AD file that is used as input to the integration tools.
      - `*.rds`: RDS version of the H5AD file.
    - `${tool}`
      - `*.h5ad/*.rds`: The integrated H5AD or RDS file.
      - `X_${tool}.pkl`: Low-dimensional representation of the integrated data.
      - `symphony_reference.h5ad` (Symphony only): Compact Symphony reference AnnData for query mapping, published from de novo Symphony runs.

</details>

The `03_combine` directory contains the results of the sample aggregation step.
The `merge` directory contains the merged H5AD files, which are used as input to the integration tools.
The `integrate` directory contains the results of the integration step.
The integrated H5AD files are stored in subdirectories named after the integration tool used.

### Cell type annotation

<details markdown="1">
<summary>Output files</summary>

- `04_celltypes/`
  - `celltypist/`
    - `*.h5ad`: The H5AD file with cell type annotations.
    - `*.pkl`: The cell type annotations in a pickle file.
  - `singleR`
    - `*.h5ad`: The H5AD file with cell type annotations.
    - `*.csv`: The cell type annotations in a CSV file.
    - `*.pdf`: The cell type annotation plots in PDF format.
  - `celldexreferenceprocessing`
    - `dir`: A directory containing `h5` and `rds` files for the reference(s)

</details>

The `04_celltypes` directory contains the results of the per-sample cell type annotation step. Annotations from `celltypist` and `singleR` are merged back into the final per-sample AnnData object via the `FINALIZE_QC_ANNDATAS` step.

### CyteType

<details markdown="1">
<summary>Output files</summary>

- `08_cytetype/`
  - `*.h5ad`: The H5AD file with CyteType annotations merged into `adata.obs` (only if `--save_intermediates` is enabled).
  - `*.pkl`: The CyteType-added obs columns in a pickle file (always emitted; merged into the final merged cohort H5AD by `FINALIZE`).

</details>

CyteType runs after integration, clustering, and global differential expression. Annotations are merged into the merged cohort AnnData object via the `FINALIZE` step.

### Clustering and dimensionality reduction

<details markdown="1">
<summary>Output files</summary>

- `05_cluster_dimred/`
  - `${integration}/`
    - `neighbors/`
      - `*.h5ad`: The H5AD file with the neighbourhood graph.
    - `leiden/`
      - ${resolution}/`
        - `*.h5ad`: The H5AD file with the Leiden clustering.
        - `*.pkl`: The Leiden clustering in a pickle file.
    - `umap/`
      - `*.h5ad`: The H5AD file with the UMAP coordinates.
      - `*.pkl`: The UMAP coordinates in a pickle file.
    - `tsne/` (when `--tsne` is enabled)
      - `*.h5ad`: The H5AD file with the t-SNE coordinates.
      - `*.pkl`: The t-SNE coordinates in a pickle file.

</details>

The `05_cluster_dimred` directory contains the results of the clustering and dimensionality reduction step.
The results are stored in subdirectories named after the integration tool used.

### Per-group analyses

<details markdown="1">
<summary>Output files</summary>

- `06_per_group/`
  - `{integration}/{subset}/{grouping}/`
    - Context path: `{integration}` (or `merged`), `{subset}` (or `global`), then either `leiden/{resolution}` or the grouping column (usually `label`).
    - Example Leiden context: `scvi/global/leiden/1.0/`
    - Example label context: `merged/global/label/`
    - `paga/`
      - `*.png`: PAGA graph plot (always published).
      - `*.h5ad` / `*.pkl` / `*_connectivities.npy`: Intermediate PAGA artefacts (when `--save_intermediates`).
    - `liana/`
      - `*_dotplot.png`: Dotplot of top ligand-receptor interactions (always published).
      - `*_circle.png`: Circle plot of interaction counts between cell groups (always published).
      - `*_tileplot.png`: Tileplot of the top ligand-receptor interactions (always published).
      - `*.h5ad` / `*.pkl`: Rank-aggregate AnnData and results table (when `--save_intermediates`).
      - `by_sample/` (when `--cell2cell` and `--save_intermediates`)
        - `*.csv.gz`: Long-format LIANA results per context.
        - `*_contexts.tsv`: Context metadata with optional condition labels.
    - `cell2cell/` (when `--cell2cell true`)
      - `*_tensor_factors.png`: Factor overview across contexts, LR pairs, senders and receivers.
      - `*_loadings_lr_clustermap.png`: Ligand-receptor loadings heatmap across factors.
      - `*_loadings_contexts_clustermap.png`: Context loadings heatmap across factors.
      - `*_context_boxplots.png`: Context loadings by condition (only when each context maps to one condition).
      - `*_pathway_enrichment_dotplot.png`: PROGENy pathway enrichment across factors (requires network access to Omnipath).
      - `*_pathway_enrichment.csv`: Enrichment scores and p-values (when `--save_intermediates`).
      - `*_factor_*_loadings_product.png`: Sender-receiver loadings-product heatmaps.
      - `*_loadings_*.csv`: Factor loading tables (when `--save_intermediates`).
      - `*_tensor.pkl`: Serialised Tensor-cell2cell object (when `--save_intermediates`).
    - `differential_expression/`
      - `scanpy/{method}/{global|filtered}/`
        - `*_markers.csv`: Filtered marker-gene tables from Scanpy `rank_genes_groups`.
        - `*.png`: Marker summary, dot plots, and multi-panel volcano plots.
        - `*_volcano.png`: Multi-panel volcano figure (one panel per group, or a single `{A} vs {B}` panel when there are exactly two groups).
        - `*.pkl`: Serialised `rank_genes_groups` results merged into the final AnnData `uns` slot (when `--save_intermediates`).
      - `pseudobulk/aggregation/`
        - `*.h5ad`: Pseudobulk count matrices per cell-type stratum.
        - `*_samples.tsv`: Pseudobulk sample metadata (donor, cell type, condition, cell and count totals).
      - `pseudobulk/pydeseq2/`
        - `*_results.csv`: PyDESeq2 result tables per cell-type stratum.
        - `*_volcano.png`: Volcano plots for each PyDESeq2 contrast.
      - `pseudobulk/edgepython/`
        - `*_results.csv`: edgePython pseudobulk result tables per cell-type stratum.
        - `*_volcano.png`: Volcano plots for each edgePython contrast.
      - `single_cell/edgepython/`
        - `*_results.csv`: edgePython single-cell DE result tables.
        - `*_volcano.png`: Volcano plots for each edgepython_sc contrast.

</details>

Per-group analyses are organised by integration context first, then by analysis type. Resuming into an existing outdir may leave directories from the previous flat `{integration}-{subset}-{resolution}` layout; use a clean outdir for a tidy tree.

LIANA rank-aggregate writes a dotplot and tileplot of top interactions by `magnitude_rank`, and a circle plot of interactions with `specificity_rank <= 0.05`. Tensor-cell2cell is opt-in (`--cell2cell true`) and consumes by-sample LIANA results. All differential expression is requested through the `de` analysis token and [`de_methods`](https://nf-co.re/scdownstream/parameters#de_methods). Pseudobulk aggregation runs automatically when `pydeseq2` or `edgepython` are selected, or when [`pseudobulk`](https://nf-co.re/scdownstream/parameters#pseudobulk) is enabled. Summary plots are also embedded in MultiQC under the corresponding integration section.

### Finalize

<details markdown="1">
<summary>Output files</summary>

- `09_finalized/`
  - `merged.h5ad`: The final H5AD file with all results.
  - `merged.rds`: RDS version of the final H5AD file (only when `--tords` is enabled).
  - `merged_metadata.csv`: Metadata of the final H5AD file.

</details>

The `09_finalized` directory contains the final results of the pipeline.
The final H5AD file contains all results from the pipeline and is stored in the `merged.h5ad` file.
The metadata of the final H5AD file is stored in the `merged_metadata.csv` file.

### MultiQC

<details markdown="1">
<summary>Output files</summary>

- `11_multiqc/`
  - `multiqc_report.html`: a stand-alone HTML file that can be viewed in your web browser.
  - `multiqc_data/`: directory containing parsed statistics from the different tools used in the pipeline.
  - `multiqc_plots/`: directory containing static images from the report in various formats.

</details>

[MultiQC](http://multiqc.info) is a visualization tool that generates a single HTML report summarising all samples in your project.
Most of the pipeline QC results are visualised in the report and further statistics are available in the report data directory.
The report includes preprocessing QC plots, cell-cycle score summaries, doublet prediction overlap or score distributions, filtering retention tables, integration and clustering summaries, marker-gene plots, and pseudobulk aggregation summaries when those steps run.

Results generated by MultiQC collate pipeline QC from supported tools e.g. FastQC.
The pipeline has special steps which also allow the software versions to be reported in the MultiQC output for future traceability.
For more information about how to use MultiQC reports, see <http://multiqc.info>.

### Quarto

<details markdown="1">
<summary>Output files</summary>

- `10_reports/`
  - `qc-report.html`: a stand-alone HTML file that can be viewed in your web browser.
  - `qc-report.qmd`: a Quarto document containing all code used to render the report.
  - `_extensions/`: directory containing the Quarto extension used to render the report, which includes the nf-core visual identity.
  - `params.yml`: YAML file containing the parameters used when rendering the report.
  - `artifacts/`
    - `concatenated.h5ad`: the final H5AD file all the results when running with
      `--qc_only`, which does not include _e.g._ integration and clustering

</details>

[Quarto](https://quarto.org/) is an open source scientific and technical publishing system that generates a single HTML report that summarises all the samples in your project, including many plots and figures.
Many of the pipeline results are visualised in the report, granting an overview of all of the data and analytical steps performed by the pipeline.

### Pipeline information

<details markdown="1">
<summary>Output files</summary>

- `pipeline_info/`
  - Reports generated by Nextflow: `execution_report.html`, `execution_timeline.html`, `execution_trace.txt` and `pipeline_dag.dot`/`pipeline_dag.svg`.
  - Reports generated by the pipeline: `pipeline_report.html`, `pipeline_report.txt` and `software_versions.yml`. The `pipeline_report*` files will only be present if the `--email` / `--email_on_fail` parameter's are used when running the pipeline.
  - Reformatted samplesheet files used as input to the pipeline: `samplesheet.valid.csv`.
  - Parameters used by the pipeline run: `params.json`.

</details>

[Nextflow](https://www.nextflow.io/docs/latest/tracing.html) provides excellent functionality for generating various reports relevant to the running and execution of the pipeline. This will allow you to troubleshoot errors with the running of the pipeline, and also provide you with other information such as launch commands, run times and resource usage.
