---
title: "Querying and fetching the single-cell data and cell/gene metadata"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Querying and fetching the single-cell data and cell/gene metadata}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
options(width = 88, max.print = 256)
```

<!--
THIS VIGNETTE IS BASED ON:
https://github.com/chanzuckerberg/cellxgene-census/blob/main/api/python/notebooks/api_demo/census_query_extract.ipynb
-->

This tutorial showcases the easiest ways to query the expression data and cell/gene metadata from the Census, and load them into R data frames, [Seurat](https://satijalab.org/seurat/) assays, and [SingleCellExperiment](https://bioconductor.org/packages/release/bioc/html/SingleCellExperiment.html) objects.

**Contents**

1. Opening the census.
2. Querying cell metadata (obs).
3. Querying gene metadata (var).
4. Querying expression data as `Seurat`.
5. Querying expression data as `SingleCellExperiment`.

## Opening the census

The `cellxgene.census` R package contains a convenient API to open any version of the Census (by default, the newest stable version).

```{r,message=FALSE,warning=FALSE}
library("cellxgene.census")
census <- open_soma()
```

You can learn more about the `cellxgene.census` methods by accessing their corresponding documentation, for example `?cellxgene.census::open_soma`.

## Querying cell metadata (obs)

The human gene metadata of the Census, for RNA assays, is located at `census$get("census_data")$get("homo_sapiens")$obs`. This is a `SOMADataFrame` and as such it can be materialized as an R data frame (tibble) using `as.data.frame(obs$read()$concat())`.

The mouse cell metadata is at `census$get("census_data")$get("mus_musculus").obs`.

For slicing the cell metadata there are two relevant arguments that can be passed through `read():`

- `column_names` — character vector indicating what metadata columns to fetch.
- `value_filter` — R expression with selection conditions to fetch rows.
  - Expressions are one or more comparisons
  - Comparisons are one of `<column> <op> <value>` or `<column> <op> <column>`
  - Expressions can combine comparisons using && or ||
  - op is one of < | > | <= | >= | == | != or %in%

To learn what metadata columns are available for fetching and filtering we can directly look at the keys of the cell metadata.

```{r}
census$get("census_data")$get("homo_sapiens")$obs$colnames()
```

`soma_joinid` is a special `SOMADataFrame` column that is used for join operations. The definition for all other columns can be found at the [Census schema](https://chanzuckerberg.github.io/cellxgene-census/cellxgene_census_docsite_schema.html).

All of these can be used to fetch specific columns or specific rows matching a condition. For the latter we need to know the values we are looking for *a priori*.

For example let's see what are the possible values available for `sex`. To this we can load all cell metadata but fetching only for the column `sex`.

```{r}
unique(as.data.frame(census$get("census_data")$get("homo_sapiens")$obs$read(column_names = "sex")$concat()))
```

As you can see there are only three different values for sex, that is `"male"`, `"female"` and `"unknown"`.

With this information we can fetch all cell metatadata for a specific sex value, for example `"unknown"`.

```{r}
as.data.frame(census$get("census_data")$get("homo_sapiens")$obs$read(value_filter = "sex == 'unknown'")$concat())
```

You can use both `column_names` and `value_filter` to perform specific queries. For example let's fetch the `disease` column for the `cell_type` `"B cell"` in the `tissue_general` `"lung"`.

```{r}
cell_metadata_b_cell <- census$get("census_data")$get("homo_sapiens")$obs$read(
  value_filter = "cell_type == 'B cell' & tissue_general == 'lung'",
  column_names = "disease"
)

cell_metadata_b_cell <- as.data.frame(cell_metadata_b_cell$concat())

table(cell_metadata_b_cell)
```

## Querying gene metadata (var)

The human gene metadata of the Census is located at `census$get("census_data")$get("homo_sapiens")$ms$get("RNA")$var`. Similarly to the cell metadata, it is a `SOMADataFrame` and thus we can also use its method `read()`.

The mouse gene metadata is at `census$get("census_data")$get("mus_musculus")$ms$get("RNA")$var`.

Let's take a look at the metadata available for column selection and row filtering.

```{r}
census$get("census_data")$get("homo_sapiens")$ms$get("RNA")$var$colnames()
```

With the exception of soma_joinid these columns are defined in the [Census schema](https://cellxgene-census.readthedocs.io/en/latest/cellxgene_census_docsite_schema.html). Similarly to the cell metadata, we can use the same operations to learn and fetch gene metadata.

For example, to get the `feature_name` and `feature_length` of the genes `"ENSG00000161798"` and `"ENSG00000188229"` we can do the following.

```{r}
var_df <- census$get("census_data")$get("homo_sapiens")$ms$get("RNA")$var$read(
  value_filter = "feature_id %in% c('ENSG00000161798', 'ENSG00000188229')",
  column_names = c("feature_name", "feature_length")
)

as.data.frame(var_df$concat())
```

## Querying expression data as `Seurat`

A convenient way to query and fetch expression data is to use the `get_seurat` method of the `cellxgene.census` API. This is a method that combines the column selection and value filtering we described above to obtain slices of the expression data based on metadata queries.

The method will return a [`Seurat` object](https://satijalab.org/seurat/), it takes as an input a census object, the string for an organism, and for both cell and gene metadata we can specify filters and column selection as described above but with the following arguments:

- `obs_column_names` — character vector indicating the columns to select for cell metadata.
- `obs_value_filter` — expression with selection conditions to fetch cells meeting a criteria.
- `var_column_names` — character vector indicating the columns to select for gene metadata.
- `var_value_filter` — expression with selection conditions to fetch genes meeting a criteria.

For example if we want to fetch the expression data for:

- Genes `"ENSG00000161798"` and `"ENSG00000188229"`.
- All `"B cells"` of `"lung"` with `"COVID-19"`.
- With all gene metadata and adding `sex` cell metadata.

```{r, warning = FALSE, message = FALSE}
library("Seurat")

seurat_obj <- get_seurat(
  census, "Homo sapiens",
  obs_column_names = c("cell_type", "tissue_general", "disease", "sex"),
  var_value_filter = "feature_id %in% c('ENSG00000161798', 'ENSG00000188229')",
  obs_value_filter = "cell_type == 'B cell' & tissue_general == 'lung' & disease == 'COVID-19'"
)
seurat_obj
```

```{r}
head(seurat_obj[[]])
```


```{r}
head(seurat_obj$RNA[[]])
```

For a full description refer to `?cellxgene.census::get_seurat`.

## Querying expression data as `SingleCellExperiment`

Similarly to the previous section, there is a `get_single_cell_experiment` method in the `cellxgene.census` API. It behaves exactly the same as `get_seurat` but it returns a [`SingleCellExperiment` object](https://bioconductor.org/packages/release/bioc/html/SingleCellExperiment.html).

For example, to repeat the same query we can simply do the following.

```{r, warning = FALSE, message = FALSE}
library("SingleCellExperiment")

sce_obj <- get_single_cell_experiment(
  census, "Homo sapiens",
  obs_column_names = c("cell_type", "tissue_general", "disease", "sex"),
  var_value_filter = "feature_id %in% c('ENSG00000161798', 'ENSG00000188229')",
  obs_value_filter = "cell_type == 'B cell' & tissue_general == 'lung' & disease == 'COVID-19'"
)
sce_obj
```


```{r}
head(colData(sce_obj))
```


```{r}
head(rowData(sce_obj))
```

For a full description refer to `?cellxgene.census::get_single_cell_experiment`.

### Close the census

After use, the census object should be closed to release memory and other resources.

```{r}
census$close()
```

This also closes all SOMA objects accessed via the top-level `census`. Closing can be automated using `on.exit(census$close(), add = TRUE)` immediately after `census <- open_soma()`.
