# sample

> Randomly draw rows (with optional seed) from a CSV using ten different sampling methods - [reservoir](https://en.wikipedia.org/wiki/Reservoir_sampling) (default), [indexed](https://en.wikipedia.org/wiki/Random_access), [bernoulli](https://en.wikipedia.org/wiki/Bernoulli_sampling), [systematic](https://en.wikipedia.org/wiki/Systematic_sampling), [stratified](https://en.wikipedia.org/wiki/Stratified_sampling), [weighted](https://doi.org/10.1016/j.ipl.2005.11.003), [varopt](https://doi.org/10.1016/j.ipl.2005.11.003), [mergeable-reservoir](https://dl.acm.org/doi/10.1145/3147.3165), [cluster](https://en.wikipedia.org/wiki/Cluster_sampling) & timeseries sampling. The `--varopt` & `--mergeable-reservoir` modes support mergeable sketch I/O (`--sketch-out`/`--sketch-in`) so sharded inputs can be sampled and combined without re-reading the corpus. Supports sampling from CSVs on remote URLs. Uses the stats cache to skip unnecessary scanning and inform its sampling strategies.

**[Table of Contents](TableOfContents.md)** | **Source: [src/cmd/sample.rs](https://github.com/dathere/qsv/blob/master/src/cmd/sample.rs)** | [📇](TableOfContents.md#legend "uses an index when available.")[🏎️](TableOfContents.md#legend "multithreaded and/or faster when an index (📇) is available.")[🌐](TableOfContents.md#legend "has web-aware options.")[🎲](TableOfContents.md#legend "randomly generated or randomized output with a --seed option for reproducibility.")[🪄](TableOfContents.md#legend "\"automagical\" commands that uses stats and/or frequency tables to work \"smarter\" & \"faster\".")

<a name="nav"></a>
[Description](#description) | [Examples](#examples) | [Usage](#usage) | [Arguments](#arguments) | [Sample Options](#sample-options) | [Sampling Methods Options](#sampling-methods-options) | [Time-Series Sampling Options](#time-series-sampling-options) | [Sketch Options](#sketch-options) | [Remote File Options](#remote-file-options) | [Common Options](#common-options)

<a name="description"></a>

## Description [↩](#nav)

Randomly samples CSV data.

It supports ten sampling methods:  
* RESERVOIR: the default sampling method when NO INDEX is present and no sampling method
is specified. Visits every CSV record exactly once, using MEMORY PROPORTIONAL to the
sample size (k) - O(k).
<https://en.wikipedia.org/wiki/Reservoir_sampling>

* INDEXED: the default sampling method when an INDEX is present and no sampling method
is specified. Uses random I/O to sample efficiently, as it only visits records selected
by random indexing, using MEMORY PROPORTIONAL to the sample size (k) - O(k).
<https://en.wikipedia.org/wiki/Random_access>

* BERNOULLI: the sampling method when the --bernoulli option is specified.
Each record has an independent probability p of being selected, where p is
specified by the <sample-size> argument. For example, if p=0.1, then each record
has a 10% chance of being selected, regardless of the other records. The final
sample size is random and follows a binomial distribution. Uses CONSTANT MEMORY - O(1).
When sampling from a remote URL, processes the file in chunks without downloading it
entirely, making it especially efficient for sampling large remote files.
<https://en.wikipedia.org/wiki/Bernoulli_sampling>

* SYSTEMATIC: the sampling method when the --systematic option is specified.
Selects every nth record from the input, where n is the integer part of <sample-size>
and the fraction part is the percentage of the population to sample.
For example, if <sample-size> is 10.5, it will select every 10th record and 50% of the
population. If <sample-size> is a whole number (no fractional part), it will select
every nth record for the whole population. Uses CONSTANT memory - O(1). The starting
point can be specified as "random" or "first". Useful for time series data or when you
want evenly spaced samples.
<https://en.wikipedia.org/wiki/Systematic_sampling>

* STRATIFIED: the sampling method when the --stratified option is specified.
Stratifies the population by the specified column and then samples from each stratum.
Particularly useful when a population has distinct subgroups (strata) that are
heterogeneous within but homogeneous between in terms of the variable of interest.
For example, if you want to sample 1,000 records from a population of 100,000 across the US,
you can stratify the population by US state and then sample 20 records from each stratum.
This will ensure that you have a representative sample from each of the 50 states.
The sample size must be a whole number. Uses MEMORY PROPORTIONAL to the
number of strata (s) and samples per stratum (k) as specified by <sample-size> - O(s*k).
<https://en.wikipedia.org/wiki/Stratified_sampling>

* WEIGHTED: the sampling method when the --weighted option is specified.
Samples records with probabilities proportional to values in a specified weight column.
Records with higher weights are more likely to be selected. For example, if you have
sales data and want to sample transactions weighted by revenue, high-value transactions
will have a higher chance of being included. Non-numeric weights are treated as zero.
The weights are automatically normalized using the maximum weight in the dataset.
Specify the desired sample size with <sample-size>. Uses MEMORY PROPORTIONAL to the
sample size (k) - O(k).
"Weighted random sampling with a reservoir" <https://doi.org/10.1016/j.ipl.2005.11.003>

* VAROPT: the sampling method when the --varopt option is specified.
Variance-bounded weighted reservoir sampling using the A-ExpJ keying scheme of
Efraimidis and Spirakis (2006). For each record, computes a key u^(1/w) and
retains the <sample-size> items with the largest keys. Unlike the --weighted
method, it does NOT require a stats cache, runs in a single pass, and supports
merge across partitions through the --sketch-out and --sketch-in options.
Suitable for heavy-tailed weight distributions where bounded-variance
estimators are needed. Uses MEMORY PROPORTIONAL to the sample size (k) - O(k).
This is a native Rust implementation written from the original paper; the
analogous VarOpt sketches in the Apache DataSketches library use the same
family of algorithms but are NOT used here.
Algorithm: "Weighted random sampling with a reservoir"
doi 10.1016/j.ipl.2005.11.003

* MERGEABLE-RESERVOIR: the sampling method when the --mergeable-reservoir flag is set.
Uniform reservoir sample using Vitter's Algorithm R. Same statistical
distribution as the default RESERVOIR method, but the sampler state is
mergeable: a sketch written by one run can be combined with sketches from
other runs via the --sketch-out and --sketch-in options, producing a uniform
sample of the combined stream WITHOUT re-reading the input files. Useful
for sharded or incremental sampling pipelines. Uses MEMORY PROPORTIONAL to
the sample size (k) - O(k). Native Rust implementation; the analogous
ReservoirItemsSketch in the Apache DataSketches library implements the same
algorithm but is NOT used here.
See en.wikipedia.org/wiki/Reservoir_sampling

* CLUSTER: the sampling method when the --cluster option is specified.
Samples entire groups of records together based on a cluster identifier column.
The number of clusters is specified by the <sample-size> argument.
Useful when records are naturally grouped (e.g., by household, neighborhood, etc.).
For example, if you have records grouped by neighborhood and specify a sample size of 10,
it will randomly select 10 neighborhoods and include ALL records from those neighborhoods
in the output. This ensures that natural groupings in the data are preserved.
Uses MEMORY PROPORTIONAL to the number of clusters (c) - O(c).
<https://en.wikipedia.org/wiki/Cluster_sampling>

* TIMESERIES: the sampling method when the --timeseries option is specified.
Samples records based on time intervals from a time-series dataset. Groups records by
time windows (e.g., hourly, daily, weekly) and selects one record per interval.
Supports adaptive sampling (e.g., prefer business hours or weekends) and aggregation
(e.g., mean, sum, min, max) within each interval. The starting point can be "first"
(earliest), "last" (most recent), or "random". Particularly useful for time-series data
where simple row-based sampling would always return the same records due to sorting.
Uses MEMORY PROPORTIONAL to the number of records - O(n).

Supports sampling from CSVs on remote URLs. Note that the entire file is downloaded first
to a temporary file before sampling begins for all sampling methods except Bernoulli, which
streams the file as it samples it, stopping when the desired sample size is reached or the
end of the file is reached.

Sampling from stdin is also supported for all sampling methods, copying stdin to a in-memory
buffer first before sampling begins.

If a stats cache is available, it will be used to do extra checks on systematic,
weighted and cluster sampling, and to speed up sampling in general.

This command is intended to provide a means to sample from a CSV data set that
is too big to fit into memory (for example, for use with commands like
'qsv stats' with the '--everything' option).


<a name="examples"></a>

## Examples [↩](#nav)

> Take a sample of 1000 records from data.csv using RESERVOIR or INDEXED sampling
> depending on whether an INDEX is present.

```console
qsv sample 1000 data.csv
```

> Take a sample of approximately 10% of the records from data.csv using RESERVOIR
> or INDEXED sampling depending on whether an INDEX is present.

```console
qsv sample 0.1 data.csv
```

> Take a sample using BERNOULLI sampling where each record has a 5% chance of being selected

```console
qsv sample --bernoulli 0.05 data.csv
```

> Take a sample using SYSTEMATIC sampling where every 10th record is selected
> and approximately 50% of the population is sampled, starting from a random point.

```console
qsv sample --systematic random 10.5 data.csv
```

> Take a sample using STRATIFIED sampling where 20 records are sampled from each
> stratum defined by the 'State' column.

```console
qsv sample --stratified State 20 data.csv
```

> Take a sample using WEIGHTED sampling where records are sampled with probabilities
> proportional to the 'Revenue' column, for a total sample size of 1000 records.

```console
qsv sample --weighted Revenue 1000 data.csv
```

> Take a sample using CLUSTER sampling where 10 clusters defined by the
> 'Neighborhood' column are randomly selected and all records from those clusters
> are included in the sample.

```console
qsv sample --cluster Neighborhood 10 data.csv
```

> Take a sample using VAROPT (A-ExpJ weighted reservoir) sampling, weighted by
> the 'Revenue' column, for a sample size of 1000 records. Unlike --weighted,
> this does NOT require a stats cache.

```console
qsv sample --varopt Revenue 1000 data.csv
```

> Sample two shards and merge their sketches into a single uniform sample
> without re-reading the inputs.

```console
qsv sample --mergeable-reservoir --sketch-out a.sk 1000 shard_a.csv
```

```console
qsv sample --mergeable-reservoir --sketch-out b.sk 1000 shard_b.csv
```

```console
qsv sample --sketch-in a.sk,b.sk 1000 -o merged.csv
```

For more examples, see [tests](https://github.com/dathere/qsv/blob/master/tests/test_sample.rs).

See also <https://github.com/dathere/qsv/wiki/Selection-and-Inspection#sample>

<a name="usage"></a>

## Usage [↩](#nav)

```console
qsv sample [options] <sample-size> [<input>]
qsv sample --help
```

<a name="arguments"></a>

## Arguments [↩](#nav)

| &nbsp;&nbsp;&nbsp;Argument&nbsp;&nbsp;&nbsp;&nbsp; | Description |
|----------|-------------|
| &nbsp;`<input>`&nbsp; | The CSV file to sample. This can be a local file, stdin, or a URL (http and https schemes supported). |
| &nbsp;`<sample-size>`&nbsp; | When using INDEXED, RESERVOIR or WEIGHTED sampling, the sample size. Can either be a whole number or a value between value between 0 and 1. If a fraction, specifies the sample size as a percentage of the population. (e.g. 0.15 - 15 percent of the CSV) When using BERNOULLI sampling, the probability of selecting each record (between 0 and 1). When using SYSTEMATIC sampling, the integer part is the interval between records to sample & the fractional part is the percentage of the population to sample. When there is no fractional part, it will select every nth record for the entire population. When using STRATIFIED sampling, the stratum sample size. When using CLUSTER sampling, the number of clusters. When using TIMESERIES sampling, the interval number (treated as hours by default, e.g., 1 = 1 hour). Use --ts-interval for custom intervals like "1d" (daily), "1w" (weekly), "1m" (monthly), "1y" (yearly), etc. |

<a name="sample-options"></a>

## Sample Options [↩](#nav)

| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Option&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | Type | Description | Default |
|--------|------|-------------|--------|
| &nbsp;`‑‑seed`&nbsp; | integer | Random Number Generator (RNG) seed. |  |
| &nbsp;`‑‑rng`&nbsp; | string | The Random Number Generator (RNG) algorithm to use. | `standard` |

<a name="sampling-methods-options"></a>

## Sampling Methods Options [↩](#nav)

| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Option&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | Type | Description | Default |
|--------|------|-------------|--------|
| &nbsp;`‑‑bernoulli`&nbsp; | flag | Use Bernoulli sampling instead of indexed or reservoir sampling. When this flag is set, <sample-size> must be between 0 and 1 and represents the probability of selecting each record. |  |
| &nbsp;`‑‑systematic`&nbsp; | string | Use systematic sampling (every nth record as specified by <sample-size>). If <arg> is "random", the starting point is randomly chosen between 0 & n. If <arg> is "first", the starting point is the first record. The sample size must be a whole number. Uses CONSTANT memory - O(1). |  |
| &nbsp;`‑‑stratified`&nbsp; | string | Use stratified sampling. The strata column is specified by <col>. Can be either a column name or 0-based column index. The sample size must be a whole number. Uses MEMORY PROPORTIONAL to the number of strata (s) and samples per stratum (k) - O(s*k). |  |
| &nbsp;`‑‑weighted`&nbsp; | string | Use weighted sampling. The weight column is specified by <col>. Can be either a column name or 0-based column index. The column will be parsed as a number. Records with non-number weights will be skipped. Uses MEMORY PROPORTIONAL to the sample size (k) - O(k). |  |
| &nbsp;`‑‑varopt`&nbsp; | string | Use VAROPT weighted reservoir sampling (A-ExpJ keying). The weight column is specified by <col> (column name or 0-based index). Variance-bounded, single-pass, no stats-cache required, mergeable via --sketch-out / --sketch-in. Records with non-positive or non-numeric weights are silently skipped. Uses MEMORY PROPORTIONAL to the sample size (k) - O(k). |  |
| &nbsp;`‑‑mergeable‑reservoir`&nbsp; | flag | Use a mergeable Algorithm-R reservoir sampler. Distribution is identical to the default RESERVOIR method, but the resulting sketch is mergeable via --sketch-out / --sketch-in. Cannot be combined with another sampling-method flag. |  |
| &nbsp;`‑‑cluster`&nbsp; | string | Use cluster sampling. The cluster column is specified by <col>. Can be either a column name or 0-based column index. Uses MEMORY PROPORTIONAL to the number of clusters (c) - O(c). |  |
| &nbsp;`‑‑timeseries`&nbsp; | string | Use time-series sampling. The time column is specified by <col>. Can be either a column name or 0-based column index. Sorts records by the specified time column and then groups by time intervals and selects one record per interval. Supports various date formats (19 formats recognized by qsv-dateparser). Uses MEMORY PROPORTIONAL to the number of records - O(n). |  |

<a name="time-series-sampling-options"></a>

## Time-Series Sampling Options [↩](#nav)

| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Option&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | Type | Description | Default |
|--------|------|-------------|--------|
| &nbsp;`‑‑ts‑interval`&nbsp; | string | Time interval for grouping records. Format: <number><unit> where unit is h (hour), d (day), w (week), m (month), y (year). Examples: "1h", "1d", "1w", "2d" (every 2 days). If not specified, <sample-size> is treated as hours. |  |
| &nbsp;`‑‑ts‑start`&nbsp; | string | Starting point for time-series sampling. Options: "first" (earliest timestamp, default), "last" (most recent timestamp), "random" (random starting point). | `first` |
| &nbsp;`‑‑ts‑adaptive`&nbsp; | string | Adaptive sampling mode for time-series data. Options: "business-hours" (prefer 9am-5pm Mon-Fri), "weekends" (prefer weekends), "business-days" (prefer weekdays), "both" (combine business-hours and weekends). |  |
| &nbsp;`‑‑ts‑aggregate`&nbsp; | string | Aggregation function to apply within each time interval. Options: "first", "last", "mean", "sum", "count", "min", "max", "median". When specified, aggregates all records in each interval instead of selecting a single record. |  |
| &nbsp;`‑‑ts‑input‑tz`&nbsp; | string | Timezone for parsing input timestamps. Can be an IANA timezone name or "local" for the local timezone. | `UTC` |
| &nbsp;`‑‑ts‑prefer‑dmy`&nbsp; | flag | Prefer to parse dates in dmy format. Otherwise, use mdy format. |  |

<a name="sketch-options"></a>

## Sketch Options [↩](#nav)

| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Option&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | Type | Description | Default |
|--------|------|-------------|--------|
| &nbsp;`‑‑sketch‑out`&nbsp; | string | After sampling, also write a binary sketch describing the internal sampler state to <file>. The blob can later be merged into another run via --sketch-in. Only valid with --varopt or --mergeable-reservoir. The format is qsv-specific and is not interoperable with serialized sketches from other tools. |  |
| &nbsp;`‑‑sketch‑in`&nbsp; | string | Comma-separated list of sketch files produced by --sketch-out. CSV input is NOT read; the listed sketches (which must all be of the same sampler kind) are merged and the resulting sample is emitted as CSV. <sample-size> may be used to cap the merged sample below the sketches' own k. |  |

<a name="remote-file-options"></a>

## Remote File Options [↩](#nav)

| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Option&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | Type | Description | Default |
|--------|------|-------------|--------|
| &nbsp;`‑‑user‑agent`&nbsp; | string | Specify custom user agent to use when the input is a URL. It supports the following variables - $QSV_VERSION, $QSV_TARGET, $QSV_BIN_NAME, $QSV_KIND and $QSV_COMMAND. Try to follow the syntax here - <https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent> |  |
| &nbsp;`‑‑timeout`&nbsp; | integer | Inactivity timeout for downloading URLs in seconds. Aborts only if no data is received from the server for this long. If 0, no timeout. | `30` |
| &nbsp;`‑‑max‑size`&nbsp; | integer | Maximum size of the file to download in MB before sampling. Will download the entire file if not specified. If the CSV is partially downloaded, the sample will be taken only from the downloaded portion. |  |
| &nbsp;`‑‑force`&nbsp; | flag | Do not use stats cache, even if its available. |  |

<a name="common-options"></a>

## Common Options [↩](#nav)

| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Option&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | Type | Description | Default |
|--------|------|-------------|--------|
| &nbsp;`‑h,`<br>`‑‑help`&nbsp; | flag | Display this message |  |
| &nbsp;`‑o,`<br>`‑‑output`&nbsp; | string | Write output to <file> instead of stdout. |  |
| &nbsp;`‑n,`<br>`‑‑no‑headers`&nbsp; | flag | When set, the first row will be considered as part of the population to sample from. (When not set, the first row is the header row and will always appear in the output.) |  |
| &nbsp;`‑d,`<br>`‑‑delimiter`&nbsp; | string | The field delimiter for reading/writing CSV data. Must be a single character. (default: ,) |  |

---
**Source:** [`src/cmd/sample.rs`](https://github.com/dathere/qsv/blob/master/src/cmd/sample.rs)
| **[Table of Contents](TableOfContents.md)** | **[README](../../README.md)**
