# qsv Command Help

> Auto-generated from qsv command USAGE text. See [README](../../README.md) for full documentation.

| Command | Description |
| --- | --- |
| [apply](apply.md)<br>[📇](#legend "uses an index when available.")[🧠](#legend "expensive operations are memoized with available inter-session Redis/Disk caching for fetch commands.")[🤖](#legend "command uses Natural Language Processing or Generative AI.")[🚀](#legend "multithreaded even without an index.")[🔣](#legend "requires UTF-8 encoded input.")[👆](#legend "has powerful column selector support. See `select` for syntax.")[⛩️](#legend "uses MiniJinja template engine.") | Apply series of string, date, math & currency transformations to given CSV column/s. It also has some basic [NLP](https://en.wikipedia.org/wiki/Natural_language_processing) functions ([similarity](https://crates.io/crates/strsim), [sentiment analysis](https://crates.io/crates/vader_sentiment), [profanity](https://docs.rs/censor/latest/censor/), [eudex](https://github.com/ticki/eudex#eudex-a-blazingly-fast-phonetic-reductionhashing-algorithm), [language](https://crates.io/crates/whatlang) & [name gender](https://github.com/Raduc4/gender_guesser?tab=readme-ov-file#gender-guesser)) detection. Its `summarize` subcommand condenses a column or group of columns using an OpenAI API-compatible LLM (local or commercial) with customizable, [MiniJinja](https://docs.rs/minijinja/latest/minijinja/)-templated per-record prompts. |
| [applydp](applydp.md)<br>[📇](#legend "uses an index when available.")[🚀](#legend "multithreaded even without an index.")[🔣](#legend "requires UTF-8 encoded input.")[👆](#legend "has powerful column selector support. See `select` for syntax.") [![CKAN](../images/ckan.png)](#legend "has CKAN-aware integration options.") | applydp is a slimmed-down version of `apply` with only [Datapusher+](https://github.com/dathere/datapusher-plus) relevant subcommands/operations (`qsvdp` binary variant only). |
| [behead](behead.md) | Drop headers from a CSV. |
| [blake3](blake3.md)<br>[🚀](#legend "multithreaded even without an index.") | Compute or check [BLAKE3](https://github.com/BLAKE3-team/BLAKE3/?tab=readme-ov-file#blake3) hashes of files. |
| [cat](cat.md)<br>[🗄️](#legend "Extended input support.") | Concatenate CSV files by row or by column. |
| [clean](clean.md) | Remove qsv-generated cache files (`.idx` index, `stats` & `frequency` caches) to reduce clutter & simplify data packaging. With `--stale`, only removes caches whose source changed or is gone. Opt-in flags also clean `schema`, `validate` & `moarstats` outputs. |
| [clipboard](clipboard.md)<br>[🖥️](#legend "part of the User Interface (UI) feature group.") | Provide input from the clipboard or save output to the clipboard. |
| [color](color.md)<br>[🤯](#legend "loads entire CSV into memory, though `dedup`, `stats` & `transpose` have \"streaming\" modes as well.")[🐻‍❄️](#legend "command powered/accelerated by  vectorized query engine.")[🖥️](#legend "part of the User Interface (UI) feature group.") | Outputs tabular data as a pretty, colorized table that always fits into the terminal. Apart from CSV and its dialects, Arrow, Avro/IPC, Parquet, JSON array & JSONL formats are supported with the "polars" feature. |
| [count](count.md)<br>[📇](#legend "uses an index when available.")[🐻‍❄️](#legend "command powered/accelerated by  vectorized query engine.")[🏎️](#legend "multithreaded and/or faster when an index (📇) is available.") | Count the rows and optionally compile record width statistics of a CSV file. (11.87 seconds for a 15gb, 28m row NYC 311 dataset without an index. Instantaneous with an index.) If the `polars` feature is enabled, uses Polars' multithreaded, mem-mapped CSV reader for fast counts even without an index |
| [datefmt](datefmt.md)<br>[📇](#legend "uses an index when available.")[🚀](#legend "multithreaded even without an index.")[👆](#legend "has powerful column selector support. See `select` for syntax.") | Formats recognized date fields ([19 formats recognized](https://docs.rs/qsv-dateparser/latest/qsv_dateparser/#accepted-date-formats)) to a specified date format using [strftime date format specifiers](https://docs.rs/chrono/latest/chrono/format/strftime/). |
| [dedup](dedup.md)<br>[🤯](#legend "loads entire CSV into memory, though `dedup`, `stats` & `transpose` have \"streaming\" modes as well.")[🚀](#legend "multithreaded even without an index.")[👆](#legend "has powerful column selector support. See `select` for syntax.") | Remove duplicate rows (See also `extdedup`, `extsort`, `sort` & `sortcheck` commands). |
| [denull](denull.md) | Detect null sentinels — literal text like `NULL` or `N/A` standing in for a missing value, which makes `stats` type a numeric column as String (its `nullcount` stays 0, no quartiles are computed) and silently degrades `viz`, `schema` & `describegpt` downstream. Reports by default; `--apply` blanks the sentinels in the columns it confirmed, per column. Scans once with bounded memory. Numeric sentinels (`-999`) are deliberately NOT detected — they parse as valid numbers and no scan can tell them from real data. |
| [describegpt](describegpt.md)<br>[📇](#legend "uses an index when available.")[🗃️](#legend "Limited Extended input support.")[🤖](#legend "command uses Natural Language Processing or Generative AI.")[🌐](#legend "has web-aware options.")[🪄](#legend "\"automagical\" commands that uses stats and/or frequency tables to work \"smarter\" & \"faster\".")[📚](#legend "has lookup table support, enabling runtime \"lookups\" against local or remote reference CSVs.")[⛩️](#legend "uses MiniJinja template engine.") [![CKAN](../images/ckan.png)](#legend "has CKAN-aware integration options.") | Infer a ["neuro-symbolic"](https://en.wikipedia.org/wiki/Neuro-symbolic_AI) Data Dictionary, Description & Tags or ask questions about a CSV with a [configurable, MiniJinja prompt file](../../resources/describegpt_defaults.toml), using any [OpenAI API](https://platform.openai.com/docs/introduction)-compatible cloud/local LLM. (e.g. [Markdown](../describegpt/nyc311-describegpt.md), [JSON](../describegpt/nyc311-describegpt.json), [TOON](../describegpt/nyc311-describegpt.toon), [JSON Schema](../describegpt/nyc311-describegpt-schema.json), [Semantic Markdown](../describegpt/nyc311-describegpt-semanticmd.md), [OKF](../describegpt/nyc311-describegpt-okf.md), [Everything](../describegpt/nyc311-describegpt-everything.md), [Content Type inferencing](../describegpt/property-alleghenney.md), [Spanish](../describegpt/nyc311-describegpt-spanish.md), [Mandarin](../describegpt/nyc311-describegpt-mandarin.md), [Controlled Tags](../describegpt/nyc311-describegpt-tagvocab.md); [--prompt "What are the top 10 complaint types by community board & borough by year?"](../describegpt/nyc311-describegpt-prompt.md) - [deterministic, hallucination-free SQL RAG result](../describegpt/nyc311-describegpt-prompt.csv); [iterative, session-based SQL RAG refinement](../describegpt/allegheny_discussion3.md) - [refined SQL RAG result](../describegpt/mostexpensive6.csv)) |
| [diff](diff.md)<br>[🚀](#legend "multithreaded even without an index.") | Find the difference between two CSVs with ludicrous speed! e.g. _compare two CSVs with 1M rows x 9 columns in under 600ms!_ |
| [edit](edit.md) | Replace the value of a cell specified by its row and column. |
| [enum](enum.md)<br>[👆](#legend "has powerful column selector support. See `select` for syntax.") | Add a new column enumerating rows by adding a column of incremental or uuid identifiers. Can also be used to copy a column or fill a new column with a constant value. |
| [excel](excel.md)<br>[🚀](#legend "multithreaded even without an index.") | Exports a specified Excel/ODS sheet to a CSV file. |
| [exclude](exclude.md)<br>[👆](#legend "has powerful column selector support. See `select` for syntax.") | Removes a set of CSV data from another set based on the specified columns. |
| [explode](explode.md)<br>[🔣](#legend "requires UTF-8 encoded input.")[👆](#legend "has powerful column selector support. See `select` for syntax.") | Explode rows into multiple ones by splitting a column value based on the given separator. The inverse of `implode`. |
| [extdedup](extdedup.md)<br>[👆](#legend "has powerful column selector support. See `select` for syntax.") | Remove duplicate rows from an arbitrarily large CSV/text file using a memory-mapped, [on-disk hash table](https://crates.io/crates/odht). Unlike the `dedup` command, this command does not load the entire file into memory nor does it sort the deduped file. |
| [extsort](extsort.md)<br>[📇](#legend "uses an index when available.")[🚀](#legend "multithreaded even without an index.")[👆](#legend "has powerful column selector support. See `select` for syntax.") | Sort an arbitrarily large CSV/text file using a multithreaded [external merge sort](https://en.wikipedia.org/wiki/External_sorting) algorithm. |
| [fetch](fetch.md)<br>[📇](#legend "uses an index when available.")[🧠](#legend "expensive operations are memoized with available inter-session Redis/Disk caching for fetch commands.")[🌐](#legend "has web-aware options.") | Send/Fetch data to/from web services for every row using **HTTP Get**. Comes with [HTTP/2](https://http2-explained.haxx.se/en/part1) [adaptive flow control](https://medium.com/coderscorner/http-2-flow-control-77e54f7fd518), [jaq](https://github.com/01mf02/jaq?tab=readme-ov-file#jaq) JSON query language support, dynamic throttling ([RateLimit](https://www.ietf.org/archive/id/draft-ietf-httpapi-ratelimit-headers-06.html)) & caching with available persistent caching using [Redis](https://redis.io/) or a disk-cache. |
| [fetchpost](fetchpost.md)<br>[📇](#legend "uses an index when available.")[🧠](#legend "expensive operations are memoized with available inter-session Redis/Disk caching for fetch commands.")[🌐](#legend "has web-aware options.")[⛩️](#legend "uses MiniJinja template engine.") | Similar to `fetch`, but uses **HTTP Post** ([HTTP GET vs POST methods](https://www.geeksforgeeks.org/difference-between-http-get-and-post-methods/)). Supports HTML form (application/x-www-form-urlencoded), JSON (application/json) and custom content types - with the ability to render payloads using CSV data using the [MiniJinja](https://docs.rs/minijinja/latest/minijinja/) template engine. |
| [fill](fill.md)<br>[👆](#legend "has powerful column selector support. See `select` for syntax.") | Fill empty values. |
| [fixedwidth](fixedwidth.md) | Convert fixed-width text (fields at fixed byte-column positions, no delimiters) to CSV. |
| [fixlengths](fixlengths.md) | Force a CSV to have same-length records by either padding or truncating them. |
| [flatten](flatten.md) | A flattened view of CSV records. Useful for viewing one record at a time. e.g. `qsv slice -i 5 data.csv \| qsv flatten`. |
| [fmt](fmt.md) | Reformat a CSV with different delimiters, record terminators or quoting rules. (Supports ASCII delimited data.) |
| [foreach](foreach.md) | Execute a shell command once per record in a given CSV file. |
| [frequency](frequency.md)<br>[📇](#legend "uses an index when available.")[😣](#legend "uses additional memory proportional to the cardinality of the columns in the CSV.")[🏎️](#legend "multithreaded and/or faster when an index (📇) is available.")[👆](#legend "has powerful column selector support. See `select` for syntax.")[🪄](#legend "\"automagical\" commands that uses stats and/or frequency tables to work \"smarter\" & \"faster\".")[![Luau](../images/luau.png)](#legend "uses Luau 0.728 as an embedded scripting DSL.") | Build [frequency distribution tables](https://en.wikipedia.org/wiki/Frequency_(statistics)) of each column. Uses multithreading to go faster if an index is present (Examples: [CSV](../../scripts/nyc311-1m.freqs.csv) [JSON](../../scripts/nyc311-1m.freqs.json) [TOON](../../scripts/nyc311-1m.freqs.toon)). |
| [get](get.md)<br>[📇](#legend "uses an index when available.")[🧠](#legend "expensive operations are memoized with available inter-session Redis/Disk caching for fetch commands.")[🌐](#legend "has web-aware options.") [![CKAN](../images/ckan.png)](#legend "has CKAN-aware integration options.") | Get tabular data from local files, URLs (http/https & `dathere://`) & [CKAN](https://ckan.org) (`ckan://`) into a managed, queryable disk cache - with conditional revalidation (ETag/Last-Modified), transparent [zstd](https://github.com/facebook/zstd) compression, [BLAKE3](https://github.com/BLAKE3-team/BLAKE3) hashing & automatic indexing. Cached resources are reusable by ANY qsv command via the `dc:` prefix (e.g. `qsv stats dc:data.csv`), with stale entries auto-refreshed. Efficiently seeds `luau` lookup tables, `validate` dynamicEnum reference data & speeds up Datapusher+ harvesting. |
| [geocode](geocode.md)<br>[📇](#legend "uses an index when available.")[🧠](#legend "expensive operations are memoized with available inter-session Redis/Disk caching for fetch commands.")[🚀](#legend "multithreaded even without an index.")[🌐](#legend "has web-aware options.")[🔣](#legend "requires UTF-8 encoded input.")[👆](#legend "has powerful column selector support. See `select` for syntax.")[🌎](#legend "has geospatial capabilities.") | Geocodes a location against an updatable local copy of the [Geonames](https://www.geonames.org/) cities & the [Maxmind GeoLite2](https://www.maxmind.com/en/geolite-free-ip-geolocation-data) databases — with caching and multi-threading, this offline path geocodes up to 360,000 records/sec! Can also geocode online (forward & reverse) via the [OpenCage](https://opencagedata.com) geocoder. |
| [geoconvert](geoconvert.md)<br>[🌎](#legend "has geospatial capabilities.") | Convert between various spatial formats and CSV/SVG including GeoJSON, SHP, and more. |
| [headers](headers.md)<br>[🗄️](#legend "Extended input support.") | Show the headers of a CSV. Or show the intersection of all headers between many CSV files. |
| [implode](implode.md)<br>[😣](#legend "uses additional memory proportional to the cardinality of the columns in the CSV.")[👆](#legend "has powerful column selector support. See `select` for syntax.") | Implode rows by grouping on key column(s) and joining a value column with a given separator. The inverse of `explode`. |
| [index](index.md) | Create an index for a CSV. This is very quick (even the 15gb, 28m row NYC 311 dataset takes all of 14 seconds to index) & provides constant time indexing/random access into the CSV. With an index, `count`, `sample` & `slice` work instantaneously; random access mode is enabled in `luau`; and multithreading is enabled for the `frequency`, `split`, `stats` & `schema` commands. |
| [input](input.md) | Read CSV data with special commenting, quoting, trimming, line-skipping & non-UTF8 encoding handling rules. Typically used to "normalize" a CSV for further processing with other qsv commands. |
| [join](join.md)<br>[📇](#legend "uses an index when available.")[😣](#legend "uses additional memory proportional to the cardinality of the columns in the CSV.")[👆](#legend "has powerful column selector support. See `select` for syntax.") | Inner, outer, right, cross, anti & semi joins. Automatically creates a simple, in-memory hash index to make it fast. |
| [joinp](joinp.md)<br>[🐻‍❄️](#legend "command powered/accelerated by  vectorized query engine.")[🚀](#legend "multithreaded even without an index.")[🪄](#legend "\"automagical\" commands that uses stats and/or frequency tables to work \"smarter\" & \"faster\".") | Inner, outer, right, cross, anti, semi, non-equi & asof joins using the [Pola.rs](https://www.pola.rs) engine. Unlike the `join` command, `joinp` can process files larger than RAM, is multithreaded, has join key validation, a maintain row order option, pre and post-join filtering, join keys unicode normalization, supports "special" [non-equi joins](https://docs.pola.rs/user-guide/transformations/joins/#non-equi-joins) and [asof joins](https://docs.pola.rs/user-guide/transformations/joins/#asof-join) (which is [particularly useful for time series data](https://github.com/dathere/qsv/blob/30cc920d0812a854fcbfedc5db81788a0600c92b/tests/test_joinp.rs#L509-L983)) & its output columns can be coalesced. |
| [json](json.md)<br>[👆](#legend "has powerful column selector support. See `select` for syntax.") | Convert JSON array to CSV. |
| [jsonl](jsonl.md)<br>[🚀](#legend "multithreaded even without an index.")[🔣](#legend "requires UTF-8 encoded input.") | Convert newline-delimited JSON ([JSONL](https://jsonlines.org/)/[NDJSON](http://ndjson.org/)) to CSV. See `tojsonl` command to convert CSV to JSONL. |
| [lens](lens.md)<br>[🗃️](#legend "Limited Extended input support.")[🐻‍❄️](#legend "command powered/accelerated by  vectorized query engine.")[🖥️](#legend "part of the User Interface (UI) feature group.") | Interactively view, search & filter tabular data files using the [csvlens](https://github.com/YS-L/csvlens#csvlens) engine. Apart from CSV and its dialects, Arrow, Avro/IPC, Parquet, JSON array & JSONL formats are supported with the "polars" feature. |
| [luau](luau.md)<br>[📇](#legend "uses an index when available.")[🌐](#legend "has web-aware options.")[🔣](#legend "requires UTF-8 encoded input.")[📚](#legend "has lookup table support, enabling runtime \"lookups\" against local or remote reference CSVs.") [![CKAN](../images/ckan.png)](#legend "has CKAN-aware integration options.") [![Luau](../images/luau.png)](#legend "uses Luau 0.728 as an embedded scripting DSL.") | Create multiple new computed columns, filter rows, compute aggregations and build complex data pipelines by executing a [Luau](https://luau-lang.org) [0.728](https://github.com/Roblox/luau/releases/tag/0.728) expression/script for every row of a CSV file ([sequential mode](https://github.com/dathere/qsv/blob/bb72c4ef369d192d85d8b7cc6e972c1b7df77635/tests/test_luau.rs#L254-L298)), or using [random access](https://www.webopedia.com/definitions/random-access/) with an index ([random access mode](https://github.com/dathere/qsv/blob/bb72c4ef369d192d85d8b7cc6e972c1b7df77635/tests/test_luau.rs#L367-L415)). Can process a single Luau expression or [full-fledged data-wrangling scripts using lookup tables](https://github.com/dathere/qsv-lookup-tables#example) with discrete BEGIN, MAIN and END sections. It is not just another qsv command, it is qsv's [Domain-specific Language](https://en.wikipedia.org/wiki/Domain-specific_language) (DSL) with [numerous qsv-specific helper functions](https://github.com/dathere/qsv/blob/master/src/cmd/luau.rs#L1473-L2755) to build production data pipelines. |
| [moarstats](moarstats.md)<br>[📇](#legend "uses an index when available.")[🏎️](#legend "multithreaded and/or faster when an index (📇) is available.") | Add up to an additional 56 statistical measures, including extended outlier, robust & bivariate statistics to an existing stats CSV file. ([example](../moarstats/NYC_311_SR_2010-2020-sample-1M.stats.csv)). |
| [partition](partition.md)<br>[👆](#legend "has powerful column selector support. See `select` for syntax.") | Partition a CSV based on a column value. |
| [pivotp](pivotp.md)<br>[🐻‍❄️](#legend "command powered/accelerated by  vectorized query engine.")[🚀](#legend "multithreaded even without an index.")[🪄](#legend "\"automagical\" commands that uses stats and/or frequency tables to work \"smarter\" & \"faster\".") | Pivot CSV data. Features "smart" aggregation auto-selection based on data type & stats. |
| [pragmastat](pragmastat.md)<br>[📇](#legend "uses an index when available.")[🤯](#legend "loads entire CSV into memory, though `dedup`, `stats` & `transpose` have \"streaming\" modes as well.")[🎲](#legend "randomly generated or randomized output with a --seed option for reproducibility.")[🪄](#legend "\"automagical\" commands that uses stats and/or frequency tables to work \"smarter\" & \"faster\".") | Compute pragmatic statistics using the [Pragmastat](https://pragmastat.dev/) library. Uses the stats cache to auto-filter non-numeric columns and support Date/DateTime columns. |
| [pro](pro.md) | Interact with the [qsv pro](https://qsvpro.dathere.com) API. |
| [profile](profile.md)<br>[📇](#legend "uses an index when available.")[🧠](#legend "expensive operations are memoized with available inter-session Redis/Disk caching for fetch commands.")[🤖](#legend "command uses Natural Language Processing or Generative AI.")[📚](#legend "has lookup table support, enabling runtime \"lookups\" against local or remote reference CSVs.")[⛩️](#legend "uses MiniJinja template engine.") [![CKAN](../images/ckan.png)](#legend "has CKAN-aware integration options.") | Extract, derive & infer metadata from a CSV (local path or URL) - using the statistical profile of a dataset, mapped and driven by a configurable metadata [scheming](https://github.com/ckan/ckanext-scheming) YAML spec ([DCAT-US v3](https://resources.data.gov/resources/dcat-us3/), [DCAT-AP v3](https://semiceu.github.io/DCAT-AP/releases/3.0.0/) and [Croissant 1.1](https://docs.mlcommons.org/croissant/docs/croissant-spec-1.1.html) bundled; [Geoconnex](https://docs.geoconnex.us/reference/overview) when built with the `geoconnex` feature), with optional CKAN/DCAT metadata discovery for URL inputs. This enables [FAIRification](https://www.go-fair.org/fair-principles/fairification-process/) at scale. |
| [prompt](prompt.md)<br>[🐻‍❄️](#legend "command powered/accelerated by  vectorized query engine.")[🖥️](#legend "part of the User Interface (UI) feature group.") | Open a file dialog to either pick a file as input or save output to a file. |
| [pseudo](pseudo.md)<br>[🔣](#legend "requires UTF-8 encoded input.")[👆](#legend "has powerful column selector support. See `select` for syntax.") | [Pseudonymise](https://en.wikipedia.org/wiki/Pseudonymization) the value of the given column by replacing them with an incremental identifier. |
| [py](py.md)<br>[📇](#legend "uses an index when available.")[🔣](#legend "requires UTF-8 encoded input.") | Create a new computed column or filter rows by evaluating a Python expression on every row of a CSV file. Python's [f-strings](https://www.freecodecamp.org/news/python-f-strings-tutorial-how-to-use-f-strings-for-string-formatting/) is particularly useful for extended formatting, [with the ability to evaluate Python expressions as well](https://github.com/dathere/qsv/blob/4cd00dca88addf0d287247fa27d40563b6d46985/src/cmd/python.rs#L23-L31). [Requires Python 3.11 or greater](https://github.com/dathere/qsv/blob/master/docs/INTERPRETERS.md#building-qsv-with-python-feature). |
| [rename](rename.md) | Rename the columns of a CSV efficiently. |
| [replace](replace.md)<br>[📇](#legend "uses an index when available.")[🏎️](#legend "multithreaded and/or faster when an index (📇) is available.")[👆](#legend "has powerful column selector support. See `select` for syntax.") | Replace CSV data using a regex. Applies the regex to each field individually. |
| [reverse](reverse.md)<br>[📇](#legend "uses an index when available.")[🤯](#legend "loads entire CSV into memory, though `dedup`, `stats` & `transpose` have \"streaming\" modes as well.") | Reverse order of rows in a CSV. Unlike the `sort --reverse` command, it preserves the order of rows with the same key. If an index is present, it works with constant memory. Otherwise, it will load all the data into memory. |
| [safenames](safenames.md)<br>[![CKAN](../images/ckan.png)](#legend "has CKAN-aware integration options.") | Modify headers of a CSV to only have ["safe" names](../../src/cmd/safenames.rs#L5-L14) - guaranteed "database-ready"/"CKAN-ready" names. |
| [sample](sample.md)<br>[📇](#legend "uses an index when available.")[🏎️](#legend "multithreaded and/or faster when an index (📇) is available.")[🌐](#legend "has web-aware options.")[🎲](#legend "randomly generated or randomized output with a --seed option for reproducibility.")[🪄](#legend "\"automagical\" commands that uses stats and/or frequency tables to work \"smarter\" & \"faster\".") | Randomly draw rows (with optional seed) from a CSV using ten different sampling methods - [reservoir](https://en.wikipedia.org/wiki/Reservoir_sampling) (default), [indexed](https://en.wikipedia.org/wiki/Random_access), [bernoulli](https://en.wikipedia.org/wiki/Bernoulli_sampling), [systematic](https://en.wikipedia.org/wiki/Systematic_sampling), [stratified](https://en.wikipedia.org/wiki/Stratified_sampling), [weighted](https://doi.org/10.1016/j.ipl.2005.11.003), [varopt](https://doi.org/10.1016/j.ipl.2005.11.003), [mergeable-reservoir](https://dl.acm.org/doi/10.1145/3147.3165), [cluster](https://en.wikipedia.org/wiki/Cluster_sampling) & timeseries sampling. The `--varopt` & `--mergeable-reservoir` modes support mergeable sketch I/O (`--sketch-out`/`--sketch-in`) so sharded inputs can be sampled and combined without re-reading the corpus. Supports sampling from CSVs on remote URLs. Uses the stats cache to skip unnecessary scanning and inform its sampling strategies. |
| [schema](schema.md)<br>[📇](#legend "uses an index when available.")[😣](#legend "uses additional memory proportional to the cardinality of the columns in the CSV.")[🐻‍❄️](#legend "command powered/accelerated by  vectorized query engine.")[🏎️](#legend "multithreaded and/or faster when an index (📇) is available.")[👆](#legend "has powerful column selector support. See `select` for syntax.")[🪄](#legend "\"automagical\" commands that uses stats and/or frequency tables to work \"smarter\" & \"faster\".") | Infer either a [JSON Schema Validation Draft 2020-12](https://json-schema.org/draft/2020-12/json-schema-validation) ([Example](https://github.com/dathere/qsv/blob/master/resources/test/311_Service_Requests_from_2010_to_Present-2022-03-04.csv.schema.json)) or [Polars Schema](https://docs.pola.rs/user-guide/lazy/schemas/) ([Example](https://github.com/dathere/qsv/blob/master/resources/test/NYC_311_SR_2010-2020-sample-1M.pschema.json)) from CSV data. In JSON Schema Validation mode, it produces a `.schema.json` file replete with inferred data type & domain/range validation rules derived from [`stats`](../../README.md#stats_deeplink). Uses multithreading to go faster if an index is present. See [`validate`](../../README.md#validate_deeplink) command to use the generated JSON Schema to validate if similar CSVs comply with the schema. With the `--polars` option, it produces a `.pschema.json` file that all polars commands (`sqlp`, `joinp` & `pivotp`) use to determine the data type of each column & to optimize performance. Both schemas are editable and can be fine-tuned. For JSON Schema, to refine the inferred validation rules. For Polars Schema, to change the inferred Polars data types. |
| [scoresql](scoresql.md)<br>[🐻‍❄️](#legend "command powered/accelerated by  vectorized query engine.")[🪄](#legend "\"automagical\" commands that uses stats and/or frequency tables to work \"smarter\" & \"faster\".") | Analyze a SQL query against CSV file caches (stats, moarstats, frequency) to produce a performance score with actionable optimization suggestions BEFORE running the query. Supports Polars (default) and DuckDB modes. |
| [search](search.md)<br>[📇](#legend "uses an index when available.")[🏎️](#legend "multithreaded and/or faster when an index (📇) is available.")[👆](#legend "has powerful column selector support. See `select` for syntax.") | Run a regex over a CSV. Applies the regex to selected fields & shows only matching rows. |
| [searchset](searchset.md)<br>[📇](#legend "uses an index when available.")[🏎️](#legend "multithreaded and/or faster when an index (📇) is available.")[👆](#legend "has powerful column selector support. See `select` for syntax.") | _Run multiple regexes over a CSV in a single pass._ Applies the regexes to each field individually & shows only matching rows. |
| [select](select.md)<br>[👆](#legend "has powerful column selector support. See `select` for syntax.") | Select, re-order, reverse, duplicate or drop columns. |
| [slice](slice.md)<br>[📇](#legend "uses an index when available.")[🗃️](#legend "Limited Extended input support.")[🏎️](#legend "multithreaded and/or faster when an index (📇) is available.") | Slice rows from any part of a CSV. When an index is present, this only has to parse the rows in the slice (instead of all rows leading up to the start of the slice). |
| [snappy](snappy.md)<br>[🚀](#legend "multithreaded even without an index.")[🌐](#legend "has web-aware options.") | Does streaming compression/decompression of the input using Google's [Snappy](https://github.com/google/snappy/blob/main/docs/README.md) framing format ([more info](../../README.md#automatic-compressiondecompression)). |
| [sniff](sniff.md)<br>[📇](#legend "uses an index when available.")[🤖](#legend "command uses Natural Language Processing or Generative AI.")[🌐](#legend "has web-aware options.") [![CKAN](../images/ckan.png)](#legend "has CKAN-aware integration options.") | Quickly sniff & infer CSV metadata (delimiter, header row, preamble rows, quote character, flexible, is_utf8, average record length, number of records, content length & estimated number of records if sniffing a CSV on a URL, number of fields, field names & data types). It is also a general mime type detector. |
| [sort](sort.md)<br>[🤯](#legend "loads entire CSV into memory, though `dedup`, `stats` & `transpose` have \"streaming\" modes as well.")[🚀](#legend "multithreaded even without an index.")[👆](#legend "has powerful column selector support. See `select` for syntax.")[🎲](#legend "randomly generated or randomized output with a --seed option for reproducibility.") | Sorts CSV data in [lexicographical](https://en.wikipedia.org/wiki/Lexicographic_order), [natural](https://en.wikipedia.org/wiki/Natural_sort_order), numerical, reverse, unique or random (with optional seed) order (Also see `extsort` & `sortcheck` commands). |
| [sortcheck](sortcheck.md)<br>[👆](#legend "has powerful column selector support. See `select` for syntax.") | Check if a CSV is sorted. With the --json options, also retrieve record count, sort breaks & duplicate count. |
| [split](split.md)<br>[📇](#legend "uses an index when available.")[🏎️](#legend "multithreaded and/or faster when an index (📇) is available.") | Split one CSV file into many CSV files. It can split by number of rows, number of chunks or file size. Uses multithreading to go faster if an index is present when splitting by rows or chunks. |
| [sqlp](sqlp.md)<br>[📇](#legend "uses an index when available.")[🗄️](#legend "Extended input support.")[🐻‍❄️](#legend "command powered/accelerated by  vectorized query engine.")[🚀](#legend "multithreaded even without an index.")[🪄](#legend "\"automagical\" commands that uses stats and/or frequency tables to work \"smarter\" & \"faster\".") | Run [Polars](https://pola.rs) SQL (a PostgreSQL dialect) queries against several CSVs, Parquet, JSONL and Arrow files - converting queries to blazing-fast Polars [LazyFrame](https://docs.pola.rs/user-guide/lazy/) expressions, processing larger than memory CSV files. Query results can be saved in CSV, JSON, JSONL, Parquet, Apache Arrow IPC and Apache Avro formats. |
| [stats](stats.md)<br>[📇](#legend "uses an index when available.")[🤯](#legend "loads entire CSV into memory, though `dedup`, `stats` & `transpose` have \"streaming\" modes as well.")[🏎️](#legend "multithreaded and/or faster when an index (📇) is available.")[👆](#legend "has powerful column selector support. See `select` for syntax.")[🪄](#legend "\"automagical\" commands that uses stats and/or frequency tables to work \"smarter\" & \"faster\".") | Compute up to 48 [summary statistics](../STATS_DEFINITIONS.md) & make GUARANTEED data type inferences (Null, String, Float, Integer, Date, DateTime, Boolean) for each column in a CSV ([Example](https://github.com/dathere/qsv/blob/master/scripts/NYC_311_SR_2010-2020-sample-1M.stats.csv)). Uses multithreading to go faster if an index is present. With an index, can compile "streaming" stats on a 1M row sample of NYC's 311 data in [less than 0.25 seconds vs 2.24 seconds without one](https://qsv.dathere.com/benchmarks). |
| [synthesize](synthesize.md)<br>[📇](#legend "uses an index when available.")[🎲](#legend "randomly generated or randomized output with a --seed option for reproducibility.")[🤖](#legend "command uses Natural Language Processing or Generative AI.") | Generate a synthetic CSV that is statistically faithful to a source CSV. Runs `stats` + `frequency` on the source so synthesized columns reproduce its per-column attributes — frequency-weighted sampling for categorical columns, quartile-bucketed numeric/date generation, null-ratio preservation. With a Data Dictionary from `describegpt --dictionary --infer-content-type`, semantic Content Types pick realistic [fake-rs](https://github.com/cksac/fake-rs) fakers (names, emails, addresses, UUIDs, etc.) for non-enumerable columns. A dictionary `relationships` array preserves inter-column structure within each row — `joint` (functional dependencies like city/state/zip), `ordered` (monotonic chains like created_date ≤ closed_date) and `correlated` (numeric correlation via a Gaussian copula). Fully reproducible with `--seed`. |
| [table](table.md)<br>[🤯](#legend "loads entire CSV into memory, though `dedup`, `stats` & `transpose` have \"streaming\" modes as well.") | Align output of a CSV using [elastic tabstops](https://github.com/BurntSushi/tabwriter) for viewing; or to create an "aligned TSV" file or Fixed Width Format file. To interactively view a CSV, use the `lens` command. |
| [template](template.md)<br>[📇](#legend "uses an index when available.")[🚀](#legend "multithreaded even without an index.")[🔣](#legend "requires UTF-8 encoded input.")[📚](#legend "has lookup table support, enabling runtime \"lookups\" against local or remote reference CSVs.")[⛩️](#legend "uses MiniJinja template engine.") [![CKAN](../images/ckan.png)](#legend "has CKAN-aware integration options.") | Renders a template using CSV data with the [MiniJinja](https://docs.rs/minijinja/latest/minijinja/) template engine ([Example](https://github.com/dathere/qsv/blob/4645ec07b5befe3b0c0e49bf0f547315d0d7514b/src/cmd/template.rs#L18-L44)). |
| [to](to.md)<br>[🗄️](#legend "Extended input support.")[🐻‍❄️](#legend "command powered/accelerated by  vectorized query engine.")[🚀](#legend "multithreaded even without an index.") | Convert CSV files to [Parquet](https://parquet.apache.org), [PostgreSQL](https://www.postgresql.org), [SQLite](https://www.sqlite.org/index.html), Excel (XLSX), [LibreOffice Calc](https://www.libreoffice.org/discover/calc/) (ODS) and [Data Package](https://datahub.io/docs/data-packages/tabular). |
| [tojsonl](tojsonl.md)<br>[📇](#legend "uses an index when available.")[😣](#legend "uses additional memory proportional to the cardinality of the columns in the CSV.")[🗃️](#legend "Limited Extended input support.")[🚀](#legend "multithreaded even without an index.")[🔣](#legend "requires UTF-8 encoded input.")[🪄](#legend "\"automagical\" commands that uses stats and/or frequency tables to work \"smarter\" & \"faster\".") | Smartly converts CSV to a newline-delimited JSON ([JSONL](https://jsonlines.org/)/[NDJSON](http://ndjson.org/)). By scanning the CSV first, it "smartly" infers the appropriate JSON data type for each column. See `jsonl` command to convert JSONL to CSV. |
| [transpose](transpose.md)<br>[🤯](#legend "loads entire CSV into memory, though `dedup`, `stats` & `transpose` have \"streaming\" modes as well.")[👆](#legend "has powerful column selector support. See `select` for syntax.") | Transpose rows/columns of a CSV. |
| [validate](validate.md)<br>[📇](#legend "uses an index when available.")[🗄️](#legend "Extended input support.")[🚀](#legend "multithreaded even without an index.")[🌐](#legend "has web-aware options.")[📚](#legend "has lookup table support, enabling runtime \"lookups\" against local or remote reference CSVs.") [![CKAN](../images/ckan.png)](#legend "has CKAN-aware integration options.") | Validate CSV data [_blazingly-fast_](https://github.com/Stranger6667/jsonschema-rs?tab=readme-ov-file#performance "using jsonschema-rs - the fastest JSON Schema validator for Rust") using [JSON Schema Validation (Draft 2020-12)](https://json-schema.org/draft/2020-12/json-schema-validation.html) (e.g. _up to 780,031 rows/second_[^1] using [NYC's 311 schema](https://github.com/dathere/qsv/blob/master/resources/test/311_Service_Requests_from_2010_to_Present-2022-03-04.csv.schema.json) generated by the [`schema`](../../README.md#schema_deeplink) command) & put invalid records into a separate file along with a detailed validation error report. Supports several custom JSON Schema formats & keywords: * `currency` custom format with [ISO-4217](https://en.wikipedia.org/wiki/ISO_4217) validation * `dynamicEnum` custom keyword that supports enum validation against a CSV on the filesystem or a URL (http/https/ckan & dathere URL schemes supported) * `uniqueCombinedWith` custom keyword to validate uniqueness across multiple columns for composite key validation. If no JSON schema file is provided, validates if a CSV conforms to the [RFC 4180 standard](../../README.md#rfc-4180-csv-standard) and is UTF-8 encoded. |
| [viz](viz.md)<br>[🪄](#legend "\"automagical\" commands that uses stats and/or frequency tables to work \"smarter\" & \"faster\".")[📇](#legend "uses an index when available.")[🏎️](#legend "multithreaded and/or faster when an index (📇) is available.")[👆](#legend "has powerful column selector support. See `select` for syntax.")[🤖](#legend "command uses Natural Language Processing or Generative AI.")[🌐](#legend "has web-aware options.")[🌎](#legend "has geospatial capabilities.") | Generate interactive charts & maps from CSV data using [plotly](https://plotly.com). `viz smart` creates a [Data Schematic](../DATA_SCHEMATIC.md) — a *"[neuro-symbolic](https://en.wikipedia.org/wiki/Neuro-symbolic_AI)"* interactive rendering of a dataset's schema & statistics — picking appropriate visualizations using the dataset's statistics, frequency distributions, data dictionary & optional LLM metadata inferencing/classification, with automatic geocoding enrichment. Outputs self-contained, interactive HTML or static PNG/SVG/PDF/JPEG/WebP with the `viz_static` feature. ([Gallery](https://dathere.github.io/qsv/gallery.html)) |

---

### Legend

✨: enabled by a [feature flag](../../README.md#feature-flags).  
📇: uses an index when available.  
🤯: loads entire CSV into memory, though `dedup`, `stats` & `transpose` have "streaming" modes as well.  
😣: uses additional memory proportional to the cardinality of the columns in the CSV.  
🧠: expensive operations are memoized with available inter-session Redis/Disk caching for fetch commands.  
🗄️: [Extended input support](../../README.md#extended-input-support).  
🗃️: [Limited Extended input support](../../README.md#limited-extended-input-support).  
🐻‍❄️: command powered/accelerated by [![polars 0.55.2](https://img.shields.io/badge/polars-0.55.2-blue?logo=polars)](https://github.com/pola-rs/polars/releases/tag/rs-0.55.2) vectorized query engine.  
🤖: command uses Natural Language Processing or Generative AI.  
🏎️: multithreaded and/or faster when an index (📇) is available.  
🚀: multithreaded even without an index.  
![CKAN](../images/ckan.png) : has [CKAN](https://ckan.org)-aware integration options.  
🌐: has web-aware options.  
🔣: requires UTF-8 encoded input.  
👆: has powerful column selector support. See [`select`](https://github.com/dathere/qsv/blob/master/src/cmd/select.rs#L2) for syntax.  
🪄: "automagical" commands that uses stats and/or frequency tables to work "smarter" & "faster".  
📚: has lookup table support, enabling runtime "lookups" against local or remote reference CSVs.  
🌎: has geospatial capabilities.  
⛩️: uses [MiniJinja](https://docs.rs/minijinja/latest/minijinja/) template engine.  
![Luau](../images/luau.png) : uses [Luau](https://luau.org/) [0.728](https://github.com/Roblox/luau/releases/tag/0.728) as an embedded scripting [DSL](https://en.wikipedia.org/wiki/Domain-specific_language).  
🎲: randomly generated or randomized output with a --seed option for reproducibility.  
🖥️: part of the User Interface (UI) feature group.  

---
**[README](../../README.md)**
