---
title: "Census Datasets example"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Census Datasets example}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
options(width = 88, max.print = 256)
```

<!--
THIS VIGNETTE IS BASED ON:
https://github.com/chanzuckerberg/cellxgene-census/blob/main/api/python/notebooks/api_demo/census_datasets.ipynb
-->

This tutorial demonstrates basic use of the `census_datasets` data frame that contains metadata of the Census source datasets. This metadata can be joined to the cell metadata data frame (`obs`) via the column dataset_id.

**Contents**

1. Fetching the datasets table.
2. Fetching the expression data from a single dataset.
3. Downloading the original source H5AD file of a dataset.

## Fetching the datasets table

Each Census contains a top-level data frame itemizing the datasets contained therein. You can read this `SOMADataFrame` into an [Arrow Table](https://arrow.apache.org/docs/r/reference/Table.html):

```{r,message=FALSE,warning=FALSE}
library("cellxgene.census")
census <- open_soma()
census_datasets <- census$get("census_info")$get("datasets")$read()$concat()
print(census_datasets)
```

and then an R data frame:

```{r}
census_datasets <- as.data.frame(census_datasets)
print(census_datasets[, c(
  "dataset_id",
  "dataset_title",
  "dataset_total_cell_count"
)])
```

The sum of cell counts across all datasets should match the number of cells across all SOMA experiments (human, mouse).

```{r}
census_data <- census$get("census_data")
all_experiments <- lapply(census_data$to_list(), function(x) census_data$get(x$name))
print(all_experiments)
```

```{r}
experiments_total_cells <- sum(sapply(all_experiments, function(x) {
  nrow(x$obs$read(column_names = c("soma_joinid"))$concat())
}))

print(paste("Found", experiments_total_cells, "cells in all experiments."))
print(paste(
  "Found", sum(as.vector(census_datasets$dataset_total_cell_count)),
  "cells in all datasets."
))
```

## Fetching the expression data from a single dataset

Let's pick one dataset to slice out of the census, and turn into a Seurat in-memory object. (This requires the `Seurat` package to have been installed beforehand.)

```{r}
census_datasets[census_datasets$dataset_id == "0bd1a1de-3aee-40e0-b2ec-86c7a30c7149", ]
```

Create a query on the mouse experiment, "RNA" measurement, for the `dataset_id`.

```{r}
library("tiledbsoma")
obs_query <- SOMAAxisQuery$new(
  value_filter = "dataset_id == '0bd1a1de-3aee-40e0-b2ec-86c7a30c7149'"
)
expt_query <- census_data$get("mus_musculus")$axis_query(
  measurement_name = "RNA",
  obs_query = obs_query
)
dataset_seurat <- expt_query$to_seurat(c(counts = "raw"))
print(dataset_seurat)
```

## Downloading the original source H5AD file of a dataset

You can use the `cellxgene.census::get_source_h5ad_uri()` API to fetch a URI pointing to the H5AD associated with this `dataset_id`. This is the same H5AD you can download from [CZ CELLxGENE Discover](https://github.com/chanzuckerberg/cellxgene-census/blob/main/api/python/notebooks/api_demo/census_datasets.ipynb), and may contain additional data-submitter provided information which was not included in the Census.

To do this you can fetch the location in the cloud or directly download to your system.

```{r}
# Option 1: Direct download
download_source_h5ad(
  dataset_id = "0bd1a1de-3aee-40e0-b2ec-86c7a30c7149",
  file = "/tmp/Tabula_Muris_Senis-bone_marrow.h5ad",
  overwrite = TRUE
)
```

```{r}
# Option 2: Get location and download via preferred method
get_source_h5ad_uri("0bd1a1de-3aee-40e0-b2ec-86c7a30c7149")
```

The local H5AD file can be used in R using [SeuratDisk's anndata converter](https://mojaveazure.github.io/seurat-disk/articles/convert-anndata.html).

### Close the census

After use, the census object should be closed to release memory and other resources.

```{r}
census$close()
```

This also closes all SOMA objects accessed via the top-level `census`. Closing can be automated using `on.exit(census$close(), add = TRUE)` immediately after `census <- open_soma()`.
