## qsv: Blazing-fast Data-Wrangling toolkit

[![Linux build status](https://github.com/dathere/qsv/actions/workflows/rust.yml/badge.svg)](https://github.com/dathere/qsv/actions/workflows/rust.yml)
[![Windows build status](https://github.com/dathere/qsv/actions/workflows/rust-windows.yml/badge.svg)](https://github.com/dathere/qsv/actions/workflows/rust-windows.yml)
[![macOS build status](https://github.com/dathere/qsv/actions/workflows/rust-macos.yml/badge.svg)](https://github.com/dathere/qsv/actions/workflows/rust-macos.yml)
[![Security audit](https://github.com/dathere/qsv/actions/workflows/security-audit.yml/badge.svg)](https://github.com/dathere/qsv/actions/workflows/security-audit.yml)
[![Crates.io](https://img.shields.io/crates/v/qsv.svg?logo=crates.io)](https://crates.io/crates/qsv)
[![Discussions](https://img.shields.io/github/discussions/dathere/qsv)](https://github.com/dathere/qsv/discussions)
[![Minimum supported Rust version](https://img.shields.io/badge/Rust-1.97-red?logo=rust)](#minimum-supported-rust-version)
[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fjqnatividad%2Fqsv.svg?type=shield)](https://app.fossa.com/projects/git%2Bgithub.com%2Fjqnatividad%2Fqsv?ref=badge_shield) [![DOI](https://zenodo.org/badge/320463703.svg)](https://doi.org/10.5281/zenodo.17851335)
[![Wiki](https://img.shields.io/badge/Wiki-Available-blue)](https://github.com/dathere/qsv/wiki)

<div align="center">

 &nbsp;          |  Table of Contents
:--------------------------|:-------------------------
![qsv logo](docs/images/qsv_logo-gemini-indy-robothorse-small.png "Nano Banana Prompt: Can you make the horse robotic? Also, add an \"MCP\" label on the robotic horse. Keep the same pose and dimensions.")<br/>[_Hi-ho "Quicksilver" away!_](https://www.youtube.com/watch?v=p9lf76xOA5k)<br/><sub><sup>[original logo details](https://github.com/dathere/qsv/discussions/295) * [Base AI-reimagined logo](docs/images/qsv_logo-gemini-indy-robothorse-small.png) * [Event logo archive](docs/images/event-logos/)</sup></sub><br/>|qsv is a data-wrangling toolkit for querying, slicing,<br>sorting, analyzing, filtering, enriching, transforming,<br>validating, joining, formatting, converting, chatting,<br>[FAIR](https://www.go-fair.org/fair-principles/)ifying & documenting tabular data (CSV, Excel, [etc](#file-formats)).<br>Commands are simple, composable & ___["blazing fast"](https://github.com/dathere/qsv/discussions/1348)___.<br><br>* [Commands](#available-commands)<br>* Installation: [CLI](#installation-options) • [MCP Server](.claude/skills/docs/guides/START_HERE.md) • [Cowork Plugin](.claude/skills/docs/guides/START_HERE.md#step-2-optional-install-the-cowork-plugin)<br> * [Whirlwind Tour](docs/whirlwind_tour.md#a-whirlwind-tour) / [Notebooks](contrib/notebooks/) / [Lessons & Exercises](https://100.dathere.com)<br>* [FAQ](https://github.com/dathere/qsv/discussions/categories/faq)<br>* [Performance Tuning](docs/PERFORMANCE.md#performance-tuning)<br>* 👉 [Benchmarks](https://qsv.dathere.com/benchmarks) 🚀 &nbsp;•&nbsp; [Interactive Benchmarks](https://github.com/dathere/qsv/wiki/Benchmarks) 📈<br>* 📊 [Visualizations Gallery](https://dathere.github.io/qsv/gallery.html) 🗺️<br>* 🗃️ [Neurosymbolic](https://dathere.github.io/qsv/smart_allegheny_dogs.html) [Visual](https://dathere.github.io/qsv/smart_boston_311_2025.html) [Data](https://dathere.github.io/qsv/smart_nyc311.html) [Schematics](https://dathere.github.io/qsv/pitt311data.html) 📈<br>* [Environment Variables](docs/ENVIRONMENT_VARIABLES.md)<br>* [Feature Flags](#feature-flags)<br>* [Goals/Non-goals](#goals--non-goals)<br>* [Testing](#testing)<br>* [NYC&nbsp;SOD&nbsp;2022](https://docs.google.com/presentation/d/e/2PACX-1vQ12ndZL--gkz0HLQRaxqsNOwzddkv1iUKB3sq661yA77OPlAsmHJHpjaqt9s9QEf73VqMfb0cv4jHU/pub?start=false&loop=false&delayms=3000)/[csv,conf,v8](https://docs.google.com/presentation/d/10T_3MyIqS5UsKxJaOY7Ktrd-GfhJelQImlE_qYmtuis/edit#slide=id.g2e0f1e7aa0e_0_62)/[PyConUS&nbsp;2025](https://docs.google.com/presentation/d/e/2PACX-1vRKFnU0Hm8oDrtCYbxcf96kHVsPcoLU05jPVNYaAs09D05gPMWDJ96q_4_zgUvadGro4deohisy-XtY/pub?start=false&loop=false&delayms=3000)/<br>&nbsp;&nbsp;&nbsp;[csv,conf,v9](https://docs.google.com/presentation/d/1j-S0q5gqR8agsqIPBVXabGEntMlc4FDTwb4r-v8-9tA/edit?usp=sharing)/[NYC&nbsp;SOD&nbsp;2026](https://docs.google.com/presentation/d/e/2PACX-1vTobPFucA1QO6u8dF3CyHjOctoom1DBmgQF558I_gx5e8cWPr0HLvJISvoaZyCMwLZZdDHlhK2cil0o/pub?start=false&loop=false&delayms=5000)<br>* **_"Have we achieved ACI?"_** series - [1](https://dathere.com/2026/01/the-peoples-api-is-finally-here/) • [2](https://dathere.github.io/NYC-Snow-Analysis-2010-2026-Claude-Cowork/)  • [3](https://dathere.github.io/peoples-api-demos/NYC-Housing-Policy-SOD2026/) <br>* [Sponsor](#sponsor)
</div>
<div align="center">

## Try it out at [qsv.dathere.com](https://qsv.dathere.com)! <!-- markdownlint-disable-line -->

</div>

| <a name="available-commands">Command | Description |
| --- | --- |
| [apply](docs/help/apply.md)✨<br>📇🧠🤖🚀🔣👆⛩️ | Apply series of string, date, math & currency transformations to given CSV column/s. It also has some basic [NLP](https://en.wikipedia.org/wiki/Natural_language_processing) functions ([similarity](https://crates.io/crates/strsim), [sentiment analysis](https://crates.io/crates/vader_sentiment), [profanity](https://docs.rs/censor/latest/censor/), [eudex](https://github.com/ticki/eudex#eudex-a-blazingly-fast-phonetic-reductionhashing-algorithm), [language](https://crates.io/crates/whatlang) & [name gender](https://github.com/Raduc4/gender_guesser?tab=readme-ov-file#gender-guesser)) detection. Its `summarize` subcommand condenses a column or group of columns using an OpenAI API-compatible LLM (local or commercial) with customizable, [MiniJinja](https://docs.rs/minijinja/latest/minijinja/)-templated per-record prompts.  |
| [applydp](docs/help/applydp.md)✨<br>📇🚀🔣👆 ![CKAN](docs/images/ckan.png) | <a name="applydp_deeplink"></a>applydp is a slimmed-down version of `apply` with only [Datapusher+](https://github.com/dathere/datapusher-plus) relevant subcommands/operations (`qsvdp` binary variant only). |
| [behead](docs/help/behead.md) | Drop headers from a CSV. |
| [blake3](docs/help/blake3.md)<br>🚀 | Compute or check [BLAKE3](https://github.com/BLAKE3-team/BLAKE3/?tab=readme-ov-file#blake3) hashes of files. |
| [cat](docs/help/cat.md)<br>🗄️ | Concatenate CSV files by row or by column. |
| [clean](docs/help/clean.md) | Remove qsv-generated cache files (`.idx` index, `stats` & `frequency` caches) to reduce clutter & simplify data packaging. With `--stale`, only removes caches whose source changed or is gone. Opt-in flags also clean `schema`, `validate` & `moarstats` outputs. |
| [clipboard](docs/help/clipboard.md)✨<br>🖥️ | Provide input from the clipboard or save output to the clipboard. |
| [color](docs/help/color.md)✨<br>🤯🐻‍❄️🖥️ | Outputs tabular data as a pretty, colorized table that always fits into the terminal. Apart from CSV and its dialects, Arrow, Avro/IPC, Parquet, JSON array & JSONL formats are supported with the "polars" feature. |
| [count](docs/help/count.md)<br>📇🐻‍❄️🏎️ | Count the rows and optionally compile record width statistics of a CSV file. (11.87 seconds for a 15gb, 28m row NYC 311 dataset without an index. Instantaneous with an index.) If the `polars` feature is enabled, uses Polars' multithreaded, mem-mapped CSV reader for fast counts even without an index |
| [datefmt](docs/help/datefmt.md)<br>📇🚀👆 | Formats recognized date fields ([19 formats recognized](https://docs.rs/qsv-dateparser/latest/qsv_dateparser/#accepted-date-formats)) to a specified date format using [strftime date format specifiers](https://docs.rs/chrono/latest/chrono/format/strftime/). |
| [dedup](docs/help/dedup.md)<br>🤯🚀👆 | Remove duplicate rows (See also `extdedup`, `extsort`, `sort` & `sortcheck` commands). |
| [denull](docs/help/denull.md) | Detect null sentinels — literal text like `NULL` or `N/A` standing in for a missing value, which makes `stats` type a numeric column as String (its `nullcount` stays 0, no quartiles are computed) and silently degrades `viz`, `schema` & `describegpt` downstream. Reports by default; `--apply` blanks the sentinels in the columns it confirmed, per column. Scans once with bounded memory. Numeric sentinels (`-999`) are deliberately NOT detected — they parse as valid numbers and no scan can tell them from real data. |
| [describegpt](docs/help/describegpt.md)<br>📇🗃️🤖🌐🪄📚⛩️ ![CKAN](docs/images/ckan.png) | <a name="describegpt_deeplink"></a>Infer a ["neuro-symbolic"](https://en.wikipedia.org/wiki/Neuro-symbolic_AI) Data Dictionary, Description & Tags or ask questions about a CSV with a [configurable, MiniJinja prompt file](resources/describegpt_defaults.toml), using any [OpenAI API](https://platform.openai.com/docs/introduction)-compatible cloud/local LLM. (e.g. [Markdown](docs/describegpt/nyc311-describegpt.md), [JSON](docs/describegpt/nyc311-describegpt.json), [TOON](docs/describegpt/nyc311-describegpt.toon), [JSON Schema](docs/describegpt/nyc311-describegpt-schema.json), [Semantic Markdown](docs/describegpt/nyc311-describegpt-semanticmd.md), [OKF](docs/describegpt/nyc311-describegpt-okf.md), [Everything](docs/describegpt/nyc311-describegpt-everything.md), [Content Type inferencing](docs/describegpt/property-alleghenney.md), [Spanish](docs/describegpt/nyc311-describegpt-spanish.md), [Mandarin](docs/describegpt/nyc311-describegpt-mandarin.md), [Controlled Tags](docs/describegpt/nyc311-describegpt-tagvocab.md);<br>[--prompt "What are the top 10 complaint types by community board & borough by year?"](docs/describegpt/nyc311-describegpt-prompt.md) - [deterministic, hallucination-free SQL RAG result](docs/describegpt/nyc311-describegpt-prompt.csv); [iterative, session-based SQL RAG refinement](docs/describegpt/allegheny_discussion3.md) - [refined SQL RAG result](docs/describegpt/mostexpensive6.csv)) |
| [diff](docs/help/diff.md)<br>🚀 | Find the difference between two CSVs with ludicrous speed!<br/>e.g. _compare two CSVs with 1M rows x 9 columns in under 600ms!_ |
| [edit](docs/help/edit.md) | Replace the value of a cell specified by its row and column. |
| [enum](docs/help/enum.md)<br>👆 | Add a new column enumerating rows by adding a column of incremental or uuid identifiers. Can also be used to copy a column or fill a new column with a constant value.  |
| [excel](docs/help/excel.md)<br>🚀 | Exports a specified Excel/ODS sheet to a CSV file. |
| [exclude](docs/help/exclude.md)<br>👆 | Removes a set of CSV data from another set based on the specified columns.  |
| [explode](docs/help/explode.md)<br>🔣👆 | Explode rows into multiple ones by splitting a column value based on the given separator. The inverse of `implode`. |
| [extdedup](docs/help/extdedup.md)<br>👆 | Remove duplicate rows from an arbitrarily large CSV/text file using a memory-mapped, [on-disk hash table](https://crates.io/crates/odht). Unlike the `dedup` command, this command does not load the entire file into memory nor does it sort the deduped file. |
| [extsort](docs/help/extsort.md)<br>📇🚀👆 | Sort an arbitrarily large CSV/text file using a multithreaded [external merge sort](https://en.wikipedia.org/wiki/External_sorting) algorithm. |
| [fetch](docs/help/fetch.md)✨<br>📇🧠🌐 | Send/Fetch data to/from web services for every row using **HTTP Get**. Comes with [HTTP/2](https://http2-explained.haxx.se/en/part1) [adaptive flow control](https://medium.com/coderscorner/http-2-flow-control-77e54f7fd518), [jaq](https://github.com/01mf02/jaq?tab=readme-ov-file#jaq) JSON query language support, dynamic throttling ([RateLimit](https://www.ietf.org/archive/id/draft-ietf-httpapi-ratelimit-headers-06.html)) & caching with available persistent caching using [Redis](https://redis.io/) or a disk-cache. |
| [fetchpost](docs/help/fetchpost.md)✨<br>📇🧠🌐⛩️ | Similar to `fetch`, but uses **HTTP Post** ([HTTP GET vs POST methods](https://www.geeksforgeeks.org/difference-between-http-get-and-post-methods/)). Supports HTML form (application/x-www-form-urlencoded), JSON (application/json) and custom content types - with the ability to render payloads using CSV data using the [MiniJinja](https://docs.rs/minijinja/latest/minijinja/) template engine. |
| [fill](docs/help/fill.md)<br>👆 | Fill empty values.  |
| [fixedwidth](docs/help/fixedwidth.md) | Convert fixed-width text (fields at fixed byte-column positions, no delimiters) to CSV. |
| [fixlengths](docs/help/fixlengths.md) | Force a CSV to have same-length records by either padding or truncating them. |
| [flatten](docs/help/flatten.md) | A flattened view of CSV records. Useful for viewing one record at a time.<br />e.g. `qsv slice -i 5 data.csv \| qsv flatten`. |
| [fmt](docs/help/fmt.md) | Reformat a CSV with different delimiters, record terminators or quoting rules. (Supports ASCII delimited data.)  |
| [foreach](docs/help/foreach.md)✨ | Execute a shell command once per record in a given CSV file. |
| [frequency](docs/help/frequency.md)<br>📇😣🏎️👆🪄![Luau](docs/images/luau.png) | Build [frequency distribution tables](https://en.wikipedia.org/wiki/Frequency_(statistics)) of each column. Uses multithreading to go faster if an index is present (Examples: [CSV](scripts/nyc311-1m.freqs.csv) [JSON](scripts/nyc311-1m.freqs.json) [TOON](scripts/nyc311-1m.freqs.toon)). |
| [get](docs/help/get.md)✨<br>📇🧠🌐 ![CKAN](docs/images/ckan.png) | <a name="get_deeplink"></a>Get tabular data from local files, URLs (http/https & `dathere://`) & [CKAN](https://ckan.org) (`ckan://`) into a managed, queryable disk cache - with conditional revalidation (ETag/Last-Modified), transparent [zstd](https://github.com/facebook/zstd) compression, [BLAKE3](https://github.com/BLAKE3-team/BLAKE3) hashing & automatic indexing. Cached resources are reusable by ANY qsv command via the `dc:` prefix (e.g. `qsv stats dc:data.csv`), with stale entries auto-refreshed. Efficiently seeds `luau` lookup tables, `validate` dynamicEnum reference data & speeds up Datapusher+ harvesting. |
| [geocode](docs/help/geocode.md)✨<br>📇🧠🚀🌐🔣👆🌎 | Geocodes a location against an updatable local copy of the [Geonames](https://www.geonames.org/) cities & the [Maxmind GeoLite2](https://www.maxmind.com/en/geolite-free-ip-geolocation-data) databases — with caching and multi-threading, this offline path geocodes up to 360,000 records/sec! Can also geocode online (forward & reverse) via the [OpenCage](https://opencagedata.com) geocoder. |
| [geoconvert](docs/help/geoconvert.md)✨<br>🌎 | Convert between various spatial formats and CSV/SVG including GeoJSON, SHP, and more. |
| [headers](docs/help/headers.md)<br>🗄️ | Show the headers of a CSV. Or show the intersection of all headers between many CSV files. |
| [implode](docs/help/implode.md)<br>😣👆 | Implode rows by grouping on key column(s) and joining a value column with a given separator. The inverse of `explode`. |
| [index](docs/help/index.md) | Create an index (📇) for a CSV. This is very quick (even the 15gb, 28m row NYC 311 dataset takes all of 14 seconds to index) & provides constant time indexing/random access into the CSV. With an index, `count`, `sample` & `slice` work instantaneously; random access mode is enabled in `luau`; and multithreading (🏎️) is enabled for the `frequency`, `split`, `stats` & `schema` commands. |
| [input](docs/help/input.md) | Read CSV data with special commenting, quoting, trimming, line-skipping & non-UTF8 encoding handling rules. Typically used to "normalize" a CSV for further processing with other qsv commands. |
| [join](docs/help/join.md)<br>📇😣👆 | Inner, outer, right, cross, anti & semi joins. Automatically creates a simple, in-memory hash index to make it fast.  |
| [joinp](docs/help/joinp.md)✨<br>🐻‍❄️🚀🪄 | Inner, outer, right, cross, anti, semi, non-equi & asof joins using the [Pola.rs](https://www.pola.rs) engine. Unlike the `join` command, `joinp` can process files larger than RAM, is multithreaded, has join key validation, a maintain row order option, pre and post-join filtering, join keys unicode normalization, supports "special" [non-equi joins](https://docs.pola.rs/user-guide/transformations/joins/#non-equi-joins) and [asof joins](https://docs.pola.rs/user-guide/transformations/joins/#asof-join) (which is [particularly useful for time series data](https://github.com/dathere/qsv/blob/30cc920d0812a854fcbfedc5db81788a0600c92b/tests/test_joinp.rs#L509-L983)) & its output columns can be coalesced. |
| [json](docs/help/json.md)<br>👆 | Convert JSON array to CSV.
| [jsonl](docs/help/jsonl.md)<br>🚀🔣 | Convert newline-delimited JSON ([JSONL](https://jsonlines.org/)/[NDJSON](http://ndjson.org/)) to CSV. See `tojsonl` command to convert CSV to JSONL.
| [lens](docs/help/lens.md)✨<br>🗃️🐻‍❄️🖥️ | Interactively view, search & filter tabular data files using the [csvlens](https://github.com/YS-L/csvlens#csvlens) engine. Apart from CSV and its dialects, Arrow, Avro/IPC, Parquet, JSON array & JSONL formats are supported with the "polars" feature. |
| [luau](docs/help/luau.md)✨<br>📇🌐🔣📚 ![CKAN](docs/images/ckan.png) ![Luau](docs/images/luau.png) | <a name="luau_deeplink"></a>Create multiple new computed columns, filter rows, compute aggregations and build complex data pipelines by executing a [Luau](https://luau-lang.org) [0.728](https://github.com/Roblox/luau/releases/tag/0.728) expression/script for every row of a CSV file ([sequential mode](https://github.com/dathere/qsv/blob/bb72c4ef369d192d85d8b7cc6e972c1b7df77635/tests/test_luau.rs#L254-L298)), or using [random access](https://www.webopedia.com/definitions/random-access/) with an index ([random access mode](https://github.com/dathere/qsv/blob/bb72c4ef369d192d85d8b7cc6e972c1b7df77635/tests/test_luau.rs#L367-L415)).<br>Can process a single Luau expression or [full-fledged data-wrangling scripts using lookup tables](https://github.com/dathere/qsv-lookup-tables#example) with discrete BEGIN, MAIN and END sections.<br> It is not just another qsv command, it is qsv's [Domain-specific Language](https://en.wikipedia.org/wiki/Domain-specific_language) (DSL) with [numerous qsv-specific helper functions](https://github.com/dathere/qsv/blob/master/src/cmd/luau.rs#L1473-L2755) to build production data pipelines. |
| [moarstats](docs/help/moarstats.md)<br>📇🏎️ | Add up to an additional 56 statistical measures, including extended outlier, robust & bivariate statistics to an existing stats CSV file. ([example](docs/moarstats/NYC_311_SR_2010-2020-sample-1M.stats.csv)).|
| [partition](docs/help/partition.md)<br>👆 | Partition a CSV based on a column value. |
| [pivotp](docs/help/pivotp.md)✨<br>🐻‍❄️🚀🪄 | Pivot CSV data. Features "smart" aggregation auto-selection based on data type & stats. |
| [pragmastat](docs/help/pragmastat.md)<br>📇🤯🎲🪄 | Compute pragmatic statistics using the [Pragmastat](https://pragmastat.dev/) library. Uses the stats cache to auto-filter non-numeric columns and support Date/DateTime columns. |
| [pro](docs/help/pro.md) | Interact with the [qsv pro](https://qsvpro.dathere.com) API. |
| [profile](docs/help/profile.md)✨<br>📇🧠🤖📚⛩️ ![CKAN](docs/images/ckan.png) | Extract, derive & infer metadata from a CSV (local path or URL) - using the statistical profile of a dataset, mapped and driven by a configurable metadata [scheming](https://github.com/ckan/ckanext-scheming) YAML spec ([DCAT-US v3](https://resources.data.gov/resources/dcat-us3/), [DCAT-AP v3](https://semiceu.github.io/DCAT-AP/releases/3.0.0/) and [Croissant 1.1](https://docs.mlcommons.org/croissant/docs/croissant-spec-1.1.html) bundled; [Geoconnex](https://docs.geoconnex.us/reference/overview) when built with the `geoconnex` feature), with optional CKAN/DCAT metadata discovery for URL inputs. This enables [FAIRification](https://www.go-fair.org/fair-principles/fairification-process/) at scale. |
| [prompt](docs/help/prompt.md)✨<br>🐻‍❄️🖥️ | Open a file dialog to either pick a file as input or save output to a file. |
| [pseudo](docs/help/pseudo.md)<br>🔣👆 | [Pseudonymise](https://en.wikipedia.org/wiki/Pseudonymization) the value of the given column by replacing them with an incremental identifier.  |
| [py](docs/help/py.md)✨<br>📇🔣 | Create a new computed column or filter rows by evaluating a Python expression on every row of a CSV file. Python's [f-strings](https://www.freecodecamp.org/news/python-f-strings-tutorial-how-to-use-f-strings-for-string-formatting/) is particularly useful for extended formatting, [with the ability to evaluate Python expressions as well](https://github.com/dathere/qsv/blob/4cd00dca88addf0d287247fa27d40563b6d46985/src/cmd/python.rs#L23-L31). [Requires Python 3.11 or greater](https://github.com/dathere/qsv/blob/master/docs/INTERPRETERS.md#building-qsv-with-python-feature). |
| [rename](docs/help/rename.md) |  Rename the columns of a CSV efficiently. |
| [replace](docs/help/replace.md)<br>📇🏎️👆 | Replace CSV data using a regex. Applies the regex to each field individually. |
| [reverse](docs/help/reverse.md)<br>📇🤯 | Reverse order of rows in a CSV. Unlike the `sort --reverse` command, it preserves the order of rows with the same key. If an index is present, it works with constant memory. Otherwise, it will load all the data into memory. |
| [safenames](docs/help/safenames.md)<br>![CKAN](docs/images/ckan.png) | <a name="safenames_deeplink"></a>Modify headers of a CSV to only have ["safe" names](/src/cmd/safenames.rs#L5-L14) - guaranteed "database-ready"/"CKAN-ready" names.  |
| [sample](docs/help/sample.md)<br>📇🏎️🌐🎲🪄 | Randomly draw rows (with optional seed) from a CSV using ten different sampling methods - [reservoir](https://en.wikipedia.org/wiki/Reservoir_sampling) (default), [indexed](https://en.wikipedia.org/wiki/Random_access), [bernoulli](https://en.wikipedia.org/wiki/Bernoulli_sampling), [systematic](https://en.wikipedia.org/wiki/Systematic_sampling), [stratified](https://en.wikipedia.org/wiki/Stratified_sampling), [weighted](https://doi.org/10.1016/j.ipl.2005.11.003), [varopt](https://doi.org/10.1016/j.ipl.2005.11.003), [mergeable-reservoir](https://dl.acm.org/doi/10.1145/3147.3165), [cluster](https://en.wikipedia.org/wiki/Cluster_sampling) & timeseries sampling. The `--varopt` & `--mergeable-reservoir` modes support mergeable sketch I/O (`--sketch-out`/`--sketch-in`) so sharded inputs can be sampled and combined without re-reading the corpus. Supports sampling from CSVs on remote URLs. Uses the stats cache to skip unnecessary scanning and inform its sampling strategies. |
| [schema](docs/help/schema.md)<br>📇😣🐻‍❄️🏎️👆🪄 | <a name="schema_deeplink"></a>Infer either a [JSON Schema Validation Draft 2020-12](https://json-schema.org/draft/2020-12/json-schema-validation) ([Example](https://github.com/dathere/qsv/blob/master/resources/test/311_Service_Requests_from_2010_to_Present-2022-03-04.csv.schema.json)) or [Polars Schema](https://docs.pola.rs/user-guide/lazy/schemas/) ([Example](https://github.com/dathere/qsv/blob/master/resources/test/NYC_311_SR_2010-2020-sample-1M.pschema.json)) from CSV data.<br>In JSON Schema Validation mode, it produces a `.schema.json` file replete with inferred data type & domain/range validation rules derived from [`stats`](#stats_deeplink). Uses multithreading to go faster if an index is present. See [`validate`](#validate_deeplink) command to use the generated JSON Schema to validate if similar CSVs comply with the schema.<br>With the `--polars` option, it produces a `.pschema.json` file that all polars commands (`sqlp`, `joinp` & `pivotp`) use to determine the data type of each column & to optimize performance.<br>Both schemas are editable and can be fine-tuned. For JSON Schema, to refine the inferred validation rules. For Polars Schema, to change the inferred Polars data types. |
| [scoresql](docs/help/scoresql.md)✨<br>🐻‍❄️🪄 | Analyze a SQL query against CSV file caches (stats, moarstats, frequency) to produce a performance score with actionable optimization suggestions BEFORE running the query. Supports Polars (default) and DuckDB modes. |
| [search](docs/help/search.md)<br>📇🏎️👆 | Run a regex over a CSV. Applies the regex to selected fields & shows only matching rows.  |
| [searchset](docs/help/searchset.md)<br>📇🏎️👆 | _Run multiple regexes over a CSV in a single pass._ Applies the regexes to each field individually & shows only matching rows.  |
| [select](docs/help/select.md)<br>👆 | Select, re-order, reverse, duplicate or drop columns.  |
| [slice](docs/help/slice.md)<br>📇🗃️🏎️ | Slice rows from any part of a CSV. When an index is present, this only has to parse the rows in the slice (instead of all rows leading up to the start of the slice).  |
| [snappy](docs/help/snappy.md)<br>🚀🌐 | <a name="snappy_deeplink"></a>Does streaming compression/decompression of the input using Google's [Snappy](https://github.com/google/snappy/blob/main/docs/README.md) framing format ([more info](#automatic-compressiondecompression)). |
| [sniff](docs/help/sniff.md)<br>📇🤖🌐 ![CKAN](docs/images/ckan.png) | Quickly sniff & infer CSV metadata (delimiter, header row, preamble rows, quote character, flexible, is_utf8, average record length, number of records, content length & estimated number of records if sniffing a CSV on a URL, number of fields, field names & data types). It is also a general mime type detector. |
| [sort](docs/help/sort.md)<br>🤯🚀👆🎲 | Sorts CSV data in [lexicographical](https://en.wikipedia.org/wiki/Lexicographic_order), [natural](https://en.wikipedia.org/wiki/Natural_sort_order), numerical, reverse, unique or random (with optional seed) order (Also see `extsort` & `sortcheck` commands).  |
| [sortcheck](docs/help/sortcheck.md)<br>👆 | Check if a CSV is sorted. With the --json options, also retrieve record count, sort breaks & duplicate count. |
| [split](docs/help/split.md)<br>📇🏎️ | Split one CSV file into many CSV files. It can split by number of rows, number of chunks or file size. Uses multithreading to go faster if an index is present when splitting by rows or chunks. |
| [sqlp](docs/help/sqlp.md)✨<br>📇🗄️🐻‍❄️🚀🪄 | <a name="sqlp_deeplink"></a>Run [Polars](https://pola.rs) SQL (a PostgreSQL dialect) queries against several CSVs, Parquet, JSONL and Arrow files - converting queries to blazing-fast Polars [LazyFrame](https://docs.pola.rs/user-guide/lazy/) expressions, processing larger than memory CSV files. Query results can be saved in CSV, JSON, JSONL, Parquet, Apache Arrow IPC and Apache Avro formats. |
| [stats](docs/help/stats.md)<br>📇🤯🏎️👆🪄 | <a name="stats_deeplink"></a>Compute up to 48 [summary statistics](docs/STATS_DEFINITIONS.md) & make GUARANTEED data type inferences (Null, String, Float, Integer, Date, DateTime, Boolean) for each column in a CSV ([Example](https://github.com/dathere/qsv/blob/master/scripts/NYC_311_SR_2010-2020-sample-1M.stats.csv)).<br>Uses multithreading to go faster if an index is present. With an index, can compile "streaming" stats on a 1M row sample of NYC's 311 data in [less than 0.25 seconds vs 2.24 seconds without one](https://qsv.dathere.com/benchmarks). |
| [synthesize](docs/help/synthesize.md)✨<br>📇🎲🤖 | <a name="synthesize_deeplink"></a>Generate a synthetic CSV that is statistically faithful to a source CSV. Runs `stats` + `frequency` on the source so synthesized columns reproduce its per-column attributes — frequency-weighted sampling for categorical columns, quartile-bucketed numeric/date generation, null-ratio preservation. With a Data Dictionary from `describegpt --dictionary --infer-content-type`, semantic Content Types pick realistic [fake-rs](https://github.com/cksac/fake-rs) fakers (names, emails, addresses, UUIDs, etc.) for non-enumerable columns. A dictionary `relationships` array preserves inter-column structure within each row — `joint` (functional dependencies like city/state/zip), `ordered` (monotonic chains like created_date ≤ closed_date) and `correlated` (numeric correlation via a Gaussian copula). Fully reproducible with `--seed`. |
| [table](docs/help/table.md)<br>🤯 | Align output of a CSV using [elastic tabstops](https://github.com/BurntSushi/tabwriter) for viewing; or to create an "aligned TSV" file or Fixed Width Format file. To interactively view a CSV, use the `lens` command. |
| [template](docs/help/template.md)<br>📇🚀🔣📚⛩️ ![CKAN](docs/images/ckan.png) | Renders a template using CSV data with the [MiniJinja](https://docs.rs/minijinja/latest/minijinja/) template engine ([Example](https://github.com/dathere/qsv/blob/4645ec07b5befe3b0c0e49bf0f547315d0d7514b/src/cmd/template.rs#L18-L44)). |
| [to](docs/help/to.md)✨<br>🗄️🐻‍❄️🚀 | Convert CSV files to [Parquet](https://parquet.apache.org), [PostgreSQL](https://www.postgresql.org), [SQLite](https://www.sqlite.org/index.html), Excel (XLSX), [LibreOffice Calc](https://www.libreoffice.org/discover/calc/) (ODS) and [Data Package](https://datahub.io/docs/data-packages/tabular). |
| [tojsonl](docs/help/tojsonl.md)<br>📇😣🗃️🚀🔣🪄 | Smartly converts CSV to a newline-delimited JSON ([JSONL](https://jsonlines.org/)/[NDJSON](http://ndjson.org/)). By scanning the CSV first, it "smartly" infers the appropriate JSON data type for each column. See `jsonl` command to convert JSONL to CSV. |
| [transpose](docs/help/transpose.md)<br>🤯👆 | Transpose rows/columns of a CSV.  |
| [validate](docs/help/validate.md)<br>📇🗄️🚀🌐📚 ![CKAN](docs/images/ckan.png) | <a name="validate_deeplink"></a>Validate CSV data [_blazingly-fast_](https://github.com/Stranger6667/jsonschema-rs?tab=readme-ov-file#performance "using jsonschema-rs - the fastest JSON Schema validator for Rust") using [JSON Schema Validation (Draft 2020-12)](https://json-schema.org/draft/2020-12/json-schema-validation.html) (e.g. _up to 780,031 rows/second_[^1] using [NYC's 311 schema](https://github.com/dathere/qsv/blob/master/resources/test/311_Service_Requests_from_2010_to_Present-2022-03-04.csv.schema.json) generated by the [`schema`](#schema_deeplink) command) & put invalid records into a separate file along with a detailed validation error report.<br><br>Supports several custom JSON Schema formats & keywords:<br> * `currency` custom format with [ISO-4217](https://en.wikipedia.org/wiki/ISO_4217) validation<br> * `dynamicEnum` custom keyword that supports enum validation against a CSV on the filesystem or a URL (http/https/ckan & dathere URL schemes supported)<br>* `uniqueCombinedWith` custom keyword to validate uniqueness across multiple columns for composite key validation.<br><br>If no JSON schema file is provided, validates if a CSV conforms to the [RFC 4180 standard](#rfc-4180-csv-standard) and is UTF-8 encoded. |
| [viz](docs/help/viz.md)✨<br>🪄📇🏎️👆🤖🌐🌎 | <a name="viz_deeplink"></a>Generate interactive charts & maps from CSV data using [plotly](https://plotly.com). `viz smart` creates a [Data Schematic](docs/DATA_SCHEMATIC.md) — a *"[neuro-symbolic](https://en.wikipedia.org/wiki/Neuro-symbolic_AI)"* interactive rendering of a dataset's schema & statistics — picking appropriate visualizations using the dataset's statistics, frequency distributions, data dictionary & optional LLM metadata inferencing/classification, with automatic geocoding enrichment. Outputs self-contained, interactive HTML or static PNG/SVG/PDF/JPEG/WebP with the `viz_static` feature. ([Gallery](https://dathere.github.io/qsv/gallery.html)) |

<div style="text-align: right"><sub><sup>Performance metrics compiled on an M2 Pro 12-core Mac Mini with 32gb RAM</sup></sub></div>

<a name="legend_deeplink">✨</a>: enabled by a [feature flag](#feature-flags).  
📇: uses an index when available.  
🤯: loads entire CSV into memory, though `dedup`, `stats` & `transpose` have "streaming" modes as well.  
😣: uses additional memory proportional to the cardinality of the columns in the CSV.  
🧠: expensive operations are memoized with available inter-session Redis/Disk caching for fetch commands.  
🗄️: [Extended input support](#extended-input-support).  
🗃️: [Limited Extended input support](#limited-extended-input-support).  
🐻‍❄️: command powered/accelerated by [![polars 0.55.2](https://img.shields.io/badge/polars-0.55.2-blue?logo=polars)](https://github.com/pola-rs/polars/releases/tag/rs-0.55.2) vectorized query engine.  
🤖: command uses Natural Language Processing or Generative AI.  
🏎️: multithreaded and/or faster when an index (📇) is available.  
🚀: multithreaded even without an index.  
![CKAN](docs/images/ckan.png) : has [CKAN](https://ckan.org)-aware integration options.  
🌐: has web-aware options.  
🔣: requires UTF-8 encoded input.  
👆: has powerful column selector support. See [`select`](https://github.com/dathere/qsv/blob/master/src/cmd/select.rs#L2) for syntax.  
🪄: "automagical" commands that uses stats and/or frequency tables to work "smarter" & "faster".  
📚: has lookup table support, enabling runtime "lookups" against local or remote reference CSVs.  
🌎: has geospatial capabilities.  
⛩️: uses [MiniJinja](https://docs.rs/minijinja/latest/minijinja/) template engine.  
![Luau](docs/images/luau.png) : uses [Luau](https://luau.org/) [0.728](https://github.com/Roblox/luau/releases/tag/0.728) as an embedded scripting [DSL](https://en.wikipedia.org/wiki/Domain-specific_language).  
🎲: randomly generated or randomized output with a --seed option for reproducibility.  
🖥️: part of the User Interface (UI) feature group.  

[^1]: see [`validate_index` benchmark](https://qsv.dathere.com/benchmarks)

## Installation Options

> [!TIP]
> To install the qsv MCP Server and/or the qsv Claude Cowork plugin, see the [Getting Started guide](.claude/skills/docs/guides/START_HERE.md).

### Option 0: qsv pro

If you prefer to explore your data using a graphical interface instead of the command-line, feel free to try out **[qsv pro](https://qsvpro.dathere.com)**. Leveraging qsv, qsv pro can help you quickly analyze spreadsheet data by just dropping a file, along with many other interactive features. Learn more at [qsvpro.dathere.com](https://qsvpro.dathere.com) or download qsv pro directly by clicking one of the badges below.

<div style="display: flex; gap: 1rem;">
<a target="_blank" href="https://qsv.dathere.com/download/qsv-pro/windows"><img alt="qsv pro Windows download badge" src="https://github.com/user-attachments/assets/64cad7a8-8aa6-4c66-aa1e-15594a5efe6d" width="200" /></a>
<a target="_blank" href="https://qsv.dathere.com/download/qsv-pro/macos"><img alt="qsv pro macOS download badge" src="https://github.com/user-attachments/assets/85e64182-8625-4411-b924-8ab060c6a4d4" width="200" /></a>
<a target="_blank" href="https://qsv.dathere.com/download/qsv-pro/linux-deb"><img alt="qsv pro Linux (deb) download badge" src="https://github.com/user-attachments/assets/aeac617a-f699-4d71-888e-aa9ceaca37b6" width="200" /></a>
<a target="_blank" href="https://qsv.dathere.com/download/qsv-pro/linux-rpm"><img alt="qsv pro Linux (rpm) download badge" src="https://github.com/user-attachments/assets/b995871f-d255-490f-9bba-c3239b8bf11e" width="200" /></a>
<a target="_blank" href="https://qsv.dathere.com/download/qsv-pro/linux-appimage"><img alt="qsv pro Linux (AppImage) download badge" src="https://github.com/user-attachments/assets/2fe50993-2c26-4ec2-a9e6-9c36e6396e66" width="200" /></a>
</div>

### Option 1: Download Prebuilt Binaries

Full-featured prebuilt [binary variants](#variants) of the latest qsv version for Linux, macOS & Windows are available [for download](https://github.com/dathere/qsv/releases/latest), including binaries compiled with [Rust Nightly](https://stackoverflow.com/questions/70745970/rust-nightly-vs-beta-version) ([more info](https://github.com/dathere/qsv/blob/master/docs/PERFORMANCE.md#nightly-release-builds)). You may click a badge below based on your platform to download a ZIP with pre-built binaries.

<div style="display: flex; gap: 1rem;">
<a target="_blank" href="https://qsv.dathere.com/download/linux-x86_64-gnu"><img alt="qsv Linux x86_64 GNU download badge" src="https://github.com/user-attachments/assets/f71812ec-30bc-4bfe-a8a3-a639d6a6c5aa" width="200" /></a>
<a target="_blank" href="https://qsv.dathere.com/download/linux-aarch64-gnu"><img alt="qsv Linux AArch64 GNU download badge" src="https://github.com/user-attachments/assets/80dccb2a-c0e5-4f85-a4e2-282d7e84d03f" width="200" /></a>
<a target="_blank" href="https://qsv.dathere.com/download/linux-x86_64-musl"><img alt="qsv Linux x86_64 MUSL download badge" src="https://github.com/user-attachments/assets/aa2ee7ec-c183-4426-87e6-c2e9c136b69f" width="200" /></a>
<a target="_blank" href="https://qsv.dathere.com/download/linux-powerpc64le-gnu"><img alt="qsv linux-powerpc64le-gnu download badge" src="https://github.com/user-attachments/assets/e5c81cd3-7a2d-4107-a124-a6f3be3cdd40" width="200" /></a>
<a target="_blank" href="https://qsv.dathere.com/download/macos-silicon"><img alt="qsv macOS download badge" src="https://github.com/user-attachments/assets/6b9534ae-ea09-44f9-b34a-259a3d9bcfa1" width="200" /></a>
<a target="_blank" href="https://qsv.dathere.com/download/windows-msvc"><img alt="qsv Windows MSVC download badge" src="https://github.com/user-attachments/assets/d4418a34-541a-4a9a-8f29-da25d9ff3d7f" width="200" /></a>
<a target="_blank" href="https://qsv.dathere.com/download/windows-aarch64-msvc"><img alt="qsv Windows AArch64 MSVC download badge" src="https://github.com/user-attachments/assets/2754df2b-9125-4c6d-aaf4-951cefe69a7f" width="200" /></a>
<a target="_blank" href="https://qsv.dathere.com/download/windows-gnu"><img alt="qsv Windows GNU download badge" src="https://github.com/user-attachments/assets/771660f3-e898-4ee2-acb5-e5f268434752" width="200" /></a>
</div>

Prebuilt binaries for Apple Silicon, Windows for ARM, [IBM Power Servers (PowerPC64 LE Linux)](https://www.ibm.com/products/power) and [IBM Z mainframes (s390x)](https://www.ibm.com/products/z) have CPU optimizations enabled ([`target-cpu=native`](https://rust-lang.github.io/packed_simd/perf-guide/target-feature/rustflags.html#target-cpu)). The macOS Apple Silicon, Linux x86_64 (GNU), Linux ARM64 (GNU) and Windows x86_64 (MSVC) prebuilts are also compiled with [Profile Guided Optimization (PGO)](https://doc.rust-lang.org/rustc/profile-guided-optimization.html#what-is-profile-guided-optimization) for even more performance gains.

We do not enable CPU optimizations on prebuilt binaries on x86_64 platforms as there are too many CPU variants which often lead to Illegal Instruction (SIGILL) faults. If you still get SIGILL faults, "portable" binaries (all CPU optimizations disabled) are also included in the release zip archives (qsv with a "p for portable" suffix - e.g. `qsvp`, `qsvplite` `qsvpdp`).

For Windows, an MSI "Easy installer" for the x86_64 MSVC `qsvp` binary is also available. After downloading and installing the Easy installer, launch the Easy installer and click "Install qsv" to download the latest `qsvp` pre-built binary to a folder that is added to your `PATH`. Afterwards qsv should be installed and you may launch a new terminal to use qsv.

<a download href="https://github.com/dathere/qsv-easy-windows-installer/releases/download/v1.1.1/qsv-easy-installer_1.1.1_x64_en-US.msi"><img alt="qsv Windows Easy Installer download badge" src="https://github.com/user-attachments/assets/ca24398b-0aaf-40be-abe0-c79a2b2da520" width="200" /></a>

For macOS, ["ad-hoc" signatures](https://users.rust-lang.org/t/distributing-cli-apps-on-macos/70223) are used to sign our binaries, so you will need to [set appropriate Gatekeeper security settings](https://support.apple.com/en-us/HT202491) or run the following command to remove the quarantine attribute from qsv before you run it for the first time:

```bash
# replace qsv with qsvmcp, qsvlite, qsvdp, qsvpy* if you installed those binary variants
sudo xattr -d com.apple.quarantine qsv
```

An additional benefit of using the prebuilt binaries is that they have the `self_update` feature enabled, allowing you to quickly update qsv to the latest version with a simple `qsv --update`. For further security, the `self_update` feature only fetches [releases from this GitHub repo](https://github.com/dathere/qsv/releases) and automatically verifies the signature of the downloaded zip archive before installing the update.

> [!NOTE]
> The `luau` feature is not available in `musl` prebuilt binaries[^3].

#### Manually verifying the Integrity of the Prebuilt Binaries Zip Archives
All prebuilt binaries zip archives are signed with [zipsign](https://github.com/Kijewski/zipsign#zipsign) with the following public key [qsv-zipsign-public.key](https://github.com/dathere/qsv/raw/master/src/qsv-zipsign-public.key). To verify the integrity of the downloaded zip archives:

```bash
# if you don't have zipsign installed yet
cargo install zipsign

# verify the integrity of the downloaded prebuilt binary zip archive
# after downloading the zip archive and the qsv-zipsign-public.key file.
# replace <PREBUILT-BINARY-ARCHIVE.zip> with the name of the downloaded zip archive
# e.g. zipsign verify zip qsv-0.118.0-aarch64-apple-darwin.zip qsv-zipsign-public.key
zipsign verify zip <PREBUILT-BINARY-ARCHIVE.zip> qsv-zipsign-public.key
```

### Option 2: Package Managers & Distributions

qsv is also distributed by several package managers and distros.

[![Packaging status](https://repology.org/badge/vertical-allrepos/qsv.svg)](https://repology.org/project/qsv/versions)

Here are the relevant commands for installing qsv using the various package managers and distros:
```bash
# Arch Linux Extra Repository (https://archlinux.org/packages/extra/x86_64/qsv/)
pacman -S qsv

# Homebrew on macOS/Linux (https://formulae.brew.sh/formula/qsv#default)
brew install qsv

# MacPorts on macOS (https://ports.macports.org/port/qsv/)
sudo port install qsv

# Mise on Linux/macOS/Windows (https://mise.jdx.dev)
mise use -g qsv@latest

# Nixpkgs on Linux/macOS (https://search.nixos.org/packages?channel=unstable&show=qsv&from=0&size=50&sort=relevance&type=packages&query=qsv)
nix-shell -p qsv

# Scoop on Windows (https://scoop.sh/#/apps?q=qsv)
scoop install qsv

# Void Linux (https://voidlinux.org/packages/?arch=x86_64&q=qsv)
sudo xbps-install qsv

# Conda-forge (https://anaconda.org/conda-forge/qsv)
conda install conda-forge::qsv
```

Note that qsv provided by these package managers/distros enable different features (Homebrew, for instance, enables the `apply`, `fetch`, `foreach`, `geocode`, `lens`, `luau` and `to` features. However, it does automatically install shell completion for `bash`, `fish` and `zsh` shells).

To find out what features are enabled in a package/distro's qsv, run `qsv --version` ([more info](https://github.com/dathere/qsv/blob/master/docs/PERFORMANCE.md#version-details)).

In the true spirit of open source, these packages are maintained by volunteers who wanted to make qsv easier to install in various environments. They are much appreciated, and we loosely collaborate with the package maintainers through GitHub, but know that these packages are maintained by third-parties.

#### Debian package
datHere also maintains a Debian package targeting the latest Ubuntu LTS on x86_64 architecture to make it easier to install qsv with DataPusher+.

To install qsv on Ubuntu/Debian:

```bash
wget -O - https://dathere.github.io/qsv-deb-releases/qsv-deb.gpg | sudo gpg --dearmor -o /usr/share/keyrings/qsv-deb.gpg
echo "deb [signed-by=/usr/share/keyrings/qsv-deb.gpg] https://dathere.github.io/qsv-deb-releases ./" | sudo tee /etc/apt/sources.list.d/qsv.list
sudo apt update
sudo apt install qsv
```

### Option 3: Compile from Source

If you have [Rust installed](https://www.rust-lang.org/tools/install), you can compile from source[^2]:

```bash
git clone https://github.com/dathere/qsv.git
cd qsv
cargo build --release --locked --bin qsv --features all_features
```

The compiled binary will end up in `./target/release/`.

To compile different [variants](#variants) and enable optional [features](#feature-flags):

```bash
# to compile qsv with all features enabled
cargo build --release --locked --bin qsv --features feature_capable,apply,fetch,foreach,geocode,geoconnex,get,get_cloud,luau,mcp,magika,polars,profile,self_update,synthesize,to,ui,viz_static
# shorthand
cargo build --release --locked --bin qsv -F all_features
# enable all CPU optimizations for the current CPU (warning: creates non-portable binary)
CARGO_BUILD_RUSTFLAGS='-C target-cpu=native' cargo build --release --locked --bin qsv -F all_features

# or build qsv with only the fetch and foreach features enabled
cargo build --release --locked --bin qsv -F feature_capable,fetch,foreach

# for qsvmcp - MCP server optimized variant
cargo build --release --locked --bin qsvmcp -F qsvmcp

# for qsvlite
cargo build --release --locked --bin qsvlite -F lite

# for qsvdp
cargo build --release --locked --bin qsvdp -F datapusher_plus
```

[^2]: Of course, you'll also need a linker & a C compiler. Linux users should generally install GCC or Clang, according to their distribution’s documentation.
For example, if you use Ubuntu, you can install the `build-essential` package. On macOS, you can get a C compiler by running `$ xcode-select --install`.
For Windows, this means installing [Visual Studio 2022](https://visualstudio.microsoft.com/downloads/). When prompted for workloads, include "Desktop Development with C++",
the Windows 10 or 11 SDK & the English language pack, along with any other language packs your require.

> [!NOTE]
> To build with Rust nightly, see [Nightly Release Builds](docs/PERFORMANCE.md#nightly-release-builds).
The `feature_capable`, `qsvmcp`, `lite` and `datapusher_plus` are MUTUALLY EXCLUSIVE features. See [Special Build Features](docs/FEATURES.md#special-features-for-building-qsv-binary-variants) for more info.

### Variants

There are five binary variants of qsv:

* `qsv` - [feature](#feature-flags)-capable(✨), with the [prebuilt binaries](https://github.com/dathere/qsv/releases/latest) enabling all applicable features except Python [^3]
* `qsvpy` - same as `qsv` but with the Python feature enabled. Three subvariants are available - qsvpy311, qsvpy312 & qsvpy313 - which are compiled with the latest patch version of Python 3.11, 3.12 & 3.13 respectively. We need to have a binary for each Python version as Python is dynamically linked ([more info](docs/INTERPRETERS.md#building-qsv-with-python-feature)).
* `qsvmcp` - optimized for [MCP (Model Context Protocol)](https://modelcontextprotocol.io/) server use with geocode, get, get_cloud, mcp, polars, profile, self_update, synthesize, to, and viz features enabled. Shares `src/main.rs` with `qsv`.
* `qsvlite` - all features disabled (~16% of the size of `qsv`). If you are migrating from [xsv](https://github.com/BurntSushi/xsv) and want the same experience and feature set, this is the variant for you.
* `qsvdp` - optimized for use with [DataPusher+](https://github.com/dathere/datapusher-plus) with only DataPusher+ relevant commands; [`applydp`](#applydp_deeplink), a slimmed-down version of the `apply` feature; the `--progressbar` option disabled; and the self-update only checking for new releases, requiring an explicit `--update` (~16% of the size of `qsv`).

> [!NOTE]
> There are "portable" subvariants of qsv available with the "p" suffix - `qsvp`, `qsvplite` and `qsvpdp`. These subvariants are compiled without any CPU features enabled. Use these subvariants if you have an old CPU architecture or getting "Illegal instruction (SIGILL)" errors when running the regular qsv binaries.

[^3]: The `luau`feature is NOT enabled by default on the prebuilt binaries for musl platforms. This is because we cross-compile using GitHub Action Runners using Ubuntu 20.04 LTS with the [musl libc](https://musl.libc.org/) toolchain. However, Ubuntu is a glibc-based, not a musl-based distro. We get around this by [cross-compiling](https://blog.logrocket.com/guide-cross-compilation-rust/).   
Unfortunately, this prevents us from cross-compiling binaries with the `luau` feature enabled as doing so requires statically linking the host OS libc library. If you need the `luau` feature on `musl`, you will need to compile from source on your own musl-based Linux Distro (e.g. Alpine, Void, [etc.](https://wiki.musl-libc.org/projects-using-musl)).  

### Shell Completion
qsv has extensive, extendable [shell completion](https://en.wikipedia.org/wiki/Command-line_completion) support. It currently supports the following shells: `bash`, `zsh`, `powershell`, `fish`, `nushell`, `fig` & `elvish`. You may download a shell completions script for your shell by clicking one of the badges below:

<div style="display: flex; gap: 1rem;">
<a download href="https://qsv.dathere.com/download/bash-shell"><img alt="qsv Bash shell completions download badge" src="https://github.com/user-attachments/assets/3e35edf8-27d0-485d-84b1-36d8f1ef7075" width="140" /></a>
<a download target="_blank" href="https://qsv.dathere.com/download/powershell-shell"><img alt="qsv PowerShell shell completions download badge" src="https://github.com/user-attachments/assets/738358c8-c925-4778-b9c1-ef4ae1f2c52d" width="140" /></a>
<a download target="_blank" href="https://qsv.dathere.com/download/zsh-shell"><img alt="qsv zsh shell completions download badge" src="https://github.com/user-attachments/assets/a633e577-143c-47d3-bb5a-c4ca102bc007" width="140" /></a>
<a download target="_blank" href="https://qsv.dathere.com/download/fish-shell"><img alt="qsv fish shell completions download badge" src="https://github.com/user-attachments/assets/cd24def7-bb75-4842-aa74-f92b5bc659ed" width="140" /></a>
<a download target="_blank" href="https://qsv.dathere.com/download/nushell-shell"><img alt="qsv nushell shell completions download badge" src="https://github.com/user-attachments/assets/fa76664e-8e3b-4663-bf16-ad489a42bdef" width="140" /></a>
<a download target="_blank" href="https://qsv.dathere.com/download/fig-shell" width="160" /><img alt="qsv fig shell completions download badge" src="https://github.com/user-attachments/assets/7151e8ac-903b-4fa4-b63b-aceaaa0e2a9e" width="140" /></a>
<a download target="_blank" href="https://qsv.dathere.com/download/elvish-shell"><img alt="qsv elvish shell completions download badge" src="https://github.com/user-attachments/assets/9d2aa23e-73a7-4f77-8ab2-f168bf8ca216" width="140" /></a>
</div>

To customize shell completions, see the [Shell Completion](contrib/completions/README.md) documentation. If you're using Bash, you can also follow the step-by-step tutorial at [100.dathere.com](https://100.dathere.com/exercises-setup.html#optional-set-up-qsv-completions) to learn how to enable the Bash shell completions.

## Regular Expression Syntax

The `--select` option and several commands (`apply`, `applydp`, `datefmt`, `exclude`, `fetchpost`, `replace`, `schema`, `search`, `searchset`, `select`, `sqlp`, `stats` & `validate`) allow the user to specify regular expressions. We use the [`regex`](https://docs.rs/regex) crate to parse, compile and execute these expressions. [^4]

[^4]: This is the same regex engine used by [`ripgrep`](https://github.com/BurntSushi/ripgrep#ripgrep-rg) - the [blazingly fast grep replacement](https://blog.burntsushi.net/ripgrep/) that powers Visual Studio's [magical](https://lab.cccb.org/en/arthur-c-clarke-any-sufficiently-advanced-technology-is-indistinguishable-from-magic/) ["Find in Files"](https://github.com/microsoft/vscode-ripgrep) feature.

Its syntax can be found [here](https://docs.rs/regex/latest/regex/#syntax) and *"is similar to other regex engines, but it lacks several features that are not known how to implement efficiently. This includes, but is not limited to, look-around and backreferences. In exchange, all regex searches in this crate have worst case O(m * n) time complexity, where m is proportional to the size of the regex and n is proportional to the size of the string being searched."*

If you want to test your regular expressions, [regex101](https://regex101.com) supports the syntax used by the `regex` crate. Just select the "Rust" flavor.

> [!CAUTION]
> JSON SCHEMA VALIDATION REGEX: The `schema` command, when inferring a JSON Schema Validation file, will derive a regex expression for the selected columns when the `--pattern-columns` option is used. Though the derived regex is guaranteed to work, it may not be the most efficient.<br/>Before using the generated JSON Schema file in production with the `validate` command, it is recommended that users inspect and optimize the derived regex as required.<br/>While doing so, note that the `validate` command in JSON Schema Validation mode, can also support "fancy" regex expressions with look-around and backreferences using the `--fancy-regex` option.

## File formats

qsv recognizes UTF-8/ASCII encoded, CSV (`.csv`), SSV (`.ssv`) and TSV files (`.tsv` & `.tab`). CSV files are assumed to have "," (comma) as a delimiter, SSV files have ";" (semicolon) as a delimiter
and TSV files, "\t" (tab) as a delimiter. The delimiter is a single ascii character that can be set either by the `--delimiter` command-line option or
with the `QSV_DEFAULT_DELIMITER` environment variable or automatically detected when `QSV_SNIFF_DELIMITER` is set.

When using the `--output` option, qsv will UTF-8 encode the file & automatically change the delimiter used in the generated file based on the file extension - i.e. comma for `.csv`, semicolon for `.ssv`, tab for `.tsv` & `.tab` files.

JSON files are recognized & converted to CSV with the [`json`](/src/cmd/json.rs#L2) command.
[JSONL](https://jsonlines.org/)/[NDJSON](http://ndjson.org/) files are also recognized & converted to/from CSV with the [`jsonl`](/src/cmd/jsonl.rs#L2) and [`tojsonl`](/src/cmd/tojsonl.rs#L2) commands respectively.

The `fetch` & `fetchpost` commands also produces JSONL files when its invoked without the `--new-column` option & TSV files with the `--report` option.

The `excel`, `safenames`, `sniff`, `sortcheck` & `validate` commands produce JSON files with their JSON options following the [JSON API 1.1 specification](https://jsonapi.org/format/), so it can return detailed machine-friendly metadata that can be used by other systems.

The `schema` command produces a [JSON Schema Validation (Draft 2020-12)](https://json-schema.org/draft/2020-12/json-schema-validation.html) file with the ".schema.json" file extension, which can be used with the `validate` command to validate other CSV files with an identical schema.

The `describegpt` and `frequency` commands also both produce [TOON](https://toonformat.dev) files. TOON is a compact, human-readable encoding of the JSON data model for LLM prompts.

The `excel` command recognizes Excel & Open Document Spreadsheet(ODS) files (`.xls`, `.xlsx`, `.xlsm`, `.xlsb` & `.ods` files).

Speaking of Excel, if you're having trouble opening qsv-generated CSV files in Excel, set the QSV_OUTPUT_BOM environment variable to add a [Byte Order Mark](https://en.wikipedia.org/wiki/Byte_order_mark) to the beginning of the generated CSV file. This is a workaround for [Excel's UTF-8 encoding detection bug](https://stackoverflow.com/questions/155097/microsoft-excel-mangles-diacritics-in-csv-files).

The `to` command converts CSVs to Parquet, Excel `.xlsx`, LibreOffice/OpenOffice Calc `.ods` & [Data Package](https://datahub.io/docs/data-packages/tabular) formats, and populates [PostgreSQL](https://www.postgresql.org) and [SQLite](https://www.sqlite.org/index.html) databases.

The `sqlp` command returns query results in CSV, JSON, JSONL, [Parquet](https://parquet.apache.org), [Apache Arrow IPC](https://arrow.apache.org/docs/format/Columnar.html#ipc-file-format) & [Apache AVRO](https://avro.apache.org) formats. Polars SQL also supports reading external files directly in various formats with its `read_csv`, `read_ndjson`, `read_parquet` & `read_ipc` [table functions](https://github.com/pola-rs/polars/blob/91a423fea2dc067837db65c3608e3cbc1112a6fc/crates/polars-sql/src/table_functions.rs#L18-L43).

The `sniff` command can also detect the mime type of any file with the `--no-infer` or `--just-mime` options, may it be local or remote (http and https schemes supported).
It can detect more than 130 file formats, including MS Office/Open Document files, JSON, XML, PDF, PNG, JPEG and specialized geospatial formats like GPX, GML, KML, TML, TMX, TSX, TTML.
Click [here](https://docs.rs/file-format/latest/file_format/#reader-features) for a complete list.

> [!TIP]
> When the `polars` feature is enabled, qsv can also natively read `.parquet`, `.ipc`, `.arrow`, `.json` & `.jsonl` files.

### Extended Input Support

The `cat`, `headers`, `sqlp`, `to` & `validate` commands have extended input support (🗄️). If the input is `-` or empty, the command will try to use stdin as input. If it's not, it will check if its a directory, and if so, add all the files in the directory as input files.

If its a file, it will first check if it has an `.infile-list` extension. If it does, it will load the text file and parse each line as an input file path. This is a much faster and convenient way to process a large number of input files, without having to pass them all as separate command-line arguments. Further, the file paths can be anywhere in the file system, even on separate volumes. If an input file path is not fully qualified, it will be treated as relative to the current working directory. Empty lines and lines starting with `#` are ignored. Invalid file paths will be logged as warnings and skipped.

For both directory and `.infile-list` input, snappy compressed files with a `.sz` or `.zip` extension will be automatically decompressed.

Finally, if its just a regular file, it will be treated as a regular input file.

#### Limited Extended Input Support
The `describegpt`, `lens`, `slice` & `tojsonl` commands have limited extended input support (🗃️). They are different in that they only process one file. If provided an `.infile-list` or a compressed `.sz` or `.zip` file, they will only process the first file.

> **Note on `.zip` inputs.** qsv treats a `.zip` archive as a container of **delimited-text** files (CSV/TSV/TAB/SSV). The first such entry (in archive order) is used; commands with full Extended Input Support use _all_ of them. Directory and system entries (`__MACOSX`, `.DS_Store`, …) are skipped, and path-traversal ("zip-slip") entries are rejected. Nesting a **special binary format** (Parquet, Avro, or Arrow) inside a `.zip` is **not a recommended workflow** — those formats are already compressed, so zipping them serves no purpose — and it is **not handled uniformly**: most commands select only the first CSV/TSV/TAB/SSV entry (and error if a `.zip` contains none), while the commands with full Extended Input Support (`cat`, `headers`, `sqlp`, `to` & `validate`) _do_ extract non-tabular supported entries. For predictable results, provide `.parquet`/`.avro`/`.arrow` files _directly_ instead (qsv reads them natively). See [#3988](https://github.com/dathere/qsv/issues/3988).

### Automatic Compression/Decompression

qsv supports _automatic compression/decompression_ using the [Snappy frame format](https://github.com/google/snappy/blob/main/framing_format.txt). Snappy was chosen instead of more popular compression formats like gzip because it was designed for [high-performance streaming compression & decompression](https://github.com/google/snappy/tree/main/docs#readme) (up to 2.58 gb/sec compression, 0.89 gb/sec decompression).

For all commands except the `index`, `extdedup` & `extsort` commands, if the input file has an ".sz" extension, qsv will _automatically_ do streaming decompression as it reads it. Further, if the input file has an extended CSV/TSV ".sz" extension (e.g nyc311.csv.sz/nyc311.tsv.sz/nyc311.tab.sz), qsv will also use the file extension to determine the delimiter to use.   

Similarly, if the `--output` file has an ".sz" extension, qsv will _automatically_ do streaming compression as it writes it.
If the output file has an extended CSV/TSV ".sz" extension, qsv will also use the file extension to determine the delimiter to use.  

Note however that compressed files cannot be indexed, so index-accelerated commands (`frequency`, `schema`, `split`, `stats`, `tojsonl`) will not be multithreaded. Random access is also disabled without an index, so `slice` will not be instantaneous and `luau`'s random-access mode will not be available.

There is also a dedicated [`snappy`](/src/cmd/snappy.rs#L2) command with four subcommands for direct snappy file operations — a multithreaded `compress` subcommand (5-6x faster than the built-in, single-threaded auto-compression); a `decompress` subcommand with detailed compression metadata; a `check` subcommand to quickly inspect if a file has a Snappy header; and a `validate` subcommand to confirm if a Snappy file is valid.

The `snappy` command can be used to compress/decompress ANY file, not just CSV/TSV files.

Using the `snappy` command, we can compress NYC's 311 data (15gb, 28m rows) to 4.95 gb in _5.77 seconds_ with the multithreaded `compress` subcommand - _2.58 gb/sec_ with a 0.33 (3.01:1) compression ratio.  With `snappy decompress`, we can roundtrip decompress the same file in _16.71 seconds_ - _0.89 gb/sec_.

Compare that to [zip 3.0](https://infozip.sourceforge.net/Zip.html), which compressed the same file to 2.9 gb in _248.3 seconds on the same machine - 43x slower at 0.06 gb/sec_ with a 0.19 (5.17:1) compression ratio - for just an additional 14% (2.45 gb) of saved space. zip also took 4.3x longer to roundtrip decompress the same file in _72 seconds_ - _0.20 gb/sec_.

> [!TIP]
> qsv has additional compression support beyond Snappy:
>
> The `sqlp` command can:
> - Automatically decompress gzip, zstd and zlib compressed input files
> - Automatically compress output files when using Arrow, Avro and Parquet formats (via `--format` and `--compression` options)
>
> When the `polars` feature is enabled, qsv can automatically decompress these compressed file formats:
> - CSV: `.csv.gz`, `.csv.zst`, `.csv.zlib`
> - TSV/TAB: `.tsv.gz`, `.tsv.zst`, `.tsv.zlib`; `.tab.gz`, `.tab.zst`, `.tab.zlib`  
> - SSV: `.ssv.gz`, `.ssv.zst`, `.ssv.zlib`
>
> `.zip` archives are auto-decompressed in **all** builds (including qsvlite — zip needs no polars): the first CSV/TSV/TAB/SSV entry in archive order is used, with the delimiter taken from that entry's extension. A `.zip` with no CSV/TSV/TAB/SSV entry errors rather than parsing an arbitrary file as CSV.
>
> This automatic decompression applies both to commands that read through qsv's standard CSV reader (e.g. `count`, `stats`, `frequency`) and to reference data loaded as `luau`, `validate` & `describegpt` lookup tables.
>
> Compressed **remote/downloaded** sources are also auto-decompressed *before* caching, so the cached copy is plain CSV (correctly delimited, auto-indexed): remote `luau`/`validate`/`describegpt` lookup tables (http/https, `dathere://`, `ckan://`) and `get`/`dc:` cached resources. `.zip`/`.sz` work in every build that has those commands; `.gz`/`.zlib`/`.zst` need the `flate2`/`zstd` codecs (bundled in the standard `qsv` & `qsvmcp` builds), otherwise an actionable error names the missing build feature.
>
> Commands with Extended and Limited Extended Input support handle `.zip` differently: they expand a `.zip`'s contents as input files (all entries for Extended Input commands like `cat`/`sqlp`/`to`; only the first entry for Limited Extended Input commands like `slice`/`tojsonl`).

## RFC 4180 CSV Standard

qsv follows the [RFC 4180](https://datatracker.ietf.org/doc/html/rfc4180) CSV standard. However, in real life, CSV formats vary significantly & qsv is actually not strictly compliant with the specification so it can process "real-world" CSV files.
qsv leverages the awesome [Rust CSV](https://docs.rs/csv/latest/csv/) crate to read/write CSV files.

Click [here](https://docs.rs/csv-core/latest/csv_core/struct.Reader.html#rfc-4180) to find out more about how qsv conforms to the standard using this crate.

When dealing with "atypical" CSV files, you can use the `input` & `fmt` commands to normalize them to be RFC 4180-compliant.

## UTF-8 Encoding

qsv requires UTF-8 encoded input (of which ASCII is a subset).

Should you need to re-encode CSV/TSV files, you can use the `input` command to "lossy save" to UTF-8 - replacing invalid UTF-8 sequences with `�` ([U+FFFD REPLACEMENT CHARACTER](https://doc.rust-lang.org/std/char/constant.REPLACEMENT_CHARACTER.html)).

Alternatively, if you want to truly transcode to UTF-8, there are several utilities like [`iconv`](https://en.wikipedia.org/wiki/Iconv) that you can use to do so on [Linux/macOS](https://stackoverflow.com/questions/805418/how-can-i-find-encoding-of-a-file-via-a-script-on-linux) & [Windows](https://superuser.com/questions/1163753/converting-text-file-to-utf-8-on-windows-command-prompt).

### Windows Powershell and Windows Excel Usage Note

Unlike other modern operating systems, Microsoft Windows' [default encoding](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_character_encoding?view=powershell-7.4) [is UTF16-LE](https://stackoverflow.com/questions/66072117/why-does-windows-use-utf-16le). This will cause problems when redirecting qsv's output to a CSV file in Powershell & trying to open it with Excel - everything will be in the first column, as the UTF16-LE encoded CSV file will not be properly recognized by Excel.

```
# the following command will produce a UTF16-LE encoded CSV file on Windows
qsv stats wcp.csv > wcpstats.csv
```

Which is weird, since you'd think [Microsoft's own Excel would properly recognize UTF16-LE encoded CSV files](https://answers.microsoft.com/en-us/msoffice/forum/all/opening-csv-file-with-utf16-encoding-in-excel-2010/ed522cb9-e88d-4b82-b88e-a2d4bd99f874?auth=1). Regardless, to create a properly UTF-8 encoded file on Windows, use the `--output` option instead:

```
# so instead of redirecting stdout to a file on Windows
qsv stats wcp.csv > wcpstats.csv

# do this instead, so it will be properly UTF-8 encoded
qsv stats wcp.csv --output wcpstats.csv
```

Alternatively, qsv can add a [Byte Order Mark](https://en.wikipedia.org/wiki/Byte_order_mark) (BOM) to the beginning of a CSV to indicate it's UTF-8 encoded. You can do this by setting the `QSV_OUTPUT_BOM` environment variable to `1`.

This will allow Excel on Windows to properly recognize the CSV file as UTF-8 encoded.

Note that this is not a problem with Excel on macOS, as macOS (like most other *nixes) uses UTF-8 as its default encoding.

Nor is it a problem with qsv output files produced on other operating systems, as Excel on Windows can properly recognize UTF-8 encoded CSV files.

## Interpreters
For complex data-wrangling tasks, you can use Luau and Python scripts.

Luau is recommended over Python for complex data-wrangling tasks as it is faster, more memory-efficient, has no external dependencies and has several data-wrangling helper functions as qsv's DSL.

See [Luau vs Python](docs/INTERPRETERS.md) for more info.

Another "interpreter" included with qsv is [MiniJinja](https://docs.rs/minijinja/latest/minijinja/), which is used in the `template` and `fetchpost` commands.

## Memory Management
qsv supports three memory allocators - jemalloc (default), mimalloc and the standard allocator.<br>See [Memory Allocator](docs/PERFORMANCE.md#memory-allocator) for more info.

It also has Out-of-Memory prevention, with two modes - NORMAL (default) & CONSERVATIVE.<br>See [Out-of-Memory Prevention](docs/PERFORMANCE.md#out-of-memory-oom-prevention) for more info.

## Environment Variables & dotenv file support

qsv supports an extensive list of environment variables and supports `.env` files to set them.

For details, see [Environment Variables](docs/ENVIRONMENT_VARIABLES.md) and the [`dotenv.template`](dotenv.template) file.
## Feature Flags

qsv has several [feature flags](https://doc.rust-lang.org/cargo/reference/features.html) that can be used to enable/disable optional features.

See [Features](docs/FEATURES.md) for more info.

## Minimum Supported Rust Version

qsv's MSRV policy is to require the latest stable [Rust version](https://github.com/rust-lang/rust/blob/master/RELEASES.md) that is [supported by Homebrew](https://formulae.brew.sh/formula/rust#default), currently [![HomeBrew](https://img.shields.io/homebrew/v/rust?logo=homebrew)](https://formulae.brew.sh/formula/rust). 
qsv itself may upgrade its MSRV, but a new qsv release will only be made once Homebrew supports the latest Rust stable.

## Goals / Non-Goals

QuickSilver's goals, in priority order, are to be:
* **As Fast as Possible** - To do so, it has frequent releases, an aggressive MSRV policy, takes advantage of CPU features, employs [various caching strategies](docs/PERFORMANCE.md#caching), uses [HTTP/2](https://www.cloudflare.com/learning/performance/http2-vs-http1.1/#:~:text=Multiplexing%3A%20HTTP%2F1.1%20loads%20resources,resource%20blocks%20any%20other%20resource.), and is multithreaded when possible and it makes sense. It also uses the latest dependencies when possible, and will use Cargo [`patch`](https://doc.rust-lang.org/cargo/reference/overriding-dependencies.html#the-patch-section) to get unreleased fixes/features from its dependencies. See [Performance](docs/PERFORMANCE.md) for more info.
* **Able to Process Very Large Files** - Most qsv commands are streaming, using constant memory, and can process arbitrarily large CSV files. For those commands that require loading the entire CSV into memory (denoted by 🤯), qsv has Out-of-Memory prevention, batch processing strategies and "ext"ernal commands that use the disk to process larger than memory files. See [Memory Management](docs/PERFORMANCE.md#memory-management) for more info.
* **A Complete Data-Wrangling Toolkit** - qsv aims to be a comprehensive data-wrangling toolkit that you can use for quick analysis and investigations, but is also robust enough for production data pipelines. Its many commands are targeted towards common data-wrangling tasks and can be combined/composed into complex data-wrangling scripts with its Luau-based DSL.  
Luau will also serve as the backbone of a whole library of **qsv recipes** - reusable scripts for common tasks (e.g. street-level geocoding, removing PII, data enrichment, etc.) that prompt for easily modifiable parameters.   
* **Composable/Interoperable** - qsv is designed to be composable, with a focus on interoperability with other common CLI tools like 'awk', 'xargs', 'ripgrep', 'sed', etc., and with well known ETL/ELT tools like Airbyte, Airflow, Pentaho Kettle, etc. Its commands can be combined with other tools via pipes, and it supports other common file formats like JSON/JSONL, Parquet, Arrow IPC, Avro, Excel, ODS, PostgreSQL, SQLite, etc. See [File Formats](#file-formats) for more info.
* **As Portable as Possible** - qsv is designed to be portable, with installers on several platforms with an integrated self-update mechanism. In preference order, it supports Linux, macOS and Windows. See [Installation Options](#installation-options) for more info.
* **As Easy to Use as Possible** - qsv is designed to be easy to use. As easy-to-use that is,
 as command line interfaces go :shrug:. Its commands have numerous options but have sensible defaults. The usage text is written for a data analyst audience, not developers; and there are numerous examples in the usage text, with the tests doubling as examples as well. With [qsv pro](https://qsvpro.dathere.com), it has much expanded functionality while being easier to use with its Graphical User Interface.
* **As Secure as Possible** - qsv is designed to be secure. It has no external runtime dependencies, is [written](https://aws.amazon.com/blogs/opensource/why-aws-loves-rust-and-how-wed-like-to-help/) [in](https://msrc.microsoft.com/blog/2019/07/why-rust-for-safe-systems-programming/) [Rust](https://opensource.googleblog.com/2023/06/rust-fact-vs-fiction-5-insights-from-googles-rust-journey-2022.html), and its codebase is automatically audited for security vulnerabilities with automated [DevSkim](https://github.com/microsoft/DevSkim#devskim), ["cargo audit"](https://rustsec.org) and [Codacy](https://app.codacy.com/gh/dathere/qsv/dashboard) Github Actions workflows.  
It uses the latest stable Rust version, with an aggressive MSRV policy and the latest version of all its dependencies.
It has an extensive test suite with ~5,100 tests, including several [property tests](https://medium.com/criteo-engineering/introduction-to-property-based-testing-f5236229d237) which [randomly generate](https://github.com/BurntSushi/quickcheck#quickcheck) parameters for oft-used commands.   
Its prebuilt binary archives are [zipsigned](https://github.com/Kijewski/zipsign#zipsign), so you can [verify their integrity](#manually-verifying-the-integrity-of-the-prebuilt-binaries-zip-archives). Its self-update mechanism automatically verifies the integrity of the prebuilt binaries archive before applying an update.
See [Security](SECURITY.md) for more info.
* **As Easy to Contribute to as Possible** - qsv is designed to be easy to contribute to, with a focus on maintainability. It's modular architecture allows the easy addition of self-contained commands gated by feature flags, the source code is heavily commented, the usage text is embedded, and there are helper functions that make it easy to create additional commands and supporting tests. See [Features](docs/FEATURES.md) and [Contributing](CONTRIBUTING.md) for more info.

QuickSilver's non-goals are to be:
* **As Small as Possible** - qsv is designed to be small, but not at the expense of performance, features, composability, portability, usability, security or maintainability. However, we do have a `qsvlite` variant that is ~16% of the size of `qsv` and a `qsvdp` variant that is ~16% of the size of `qsv`. Those variants, however, have reduced functionality.
Further, several commands are gated behind feature flags, so you can compile qsv with only the features you need.
* **Multi-lingual** - qsv's _usage text_ and _messages_ are English-only. There are no plans to support other languages. This does not mean it can only process English input files.  
It can process well-formed CSVs in _any_ language so long as its UTF-8 encoded. Further, it supports alternate delimiters/separators other than comma; the `apply whatlang` operation detects 87 languages; and its `apply thousands, currency and eudex` operations supports different languages and country conventions for number, currency and date parsing/formatting.  
Finally, though the default Geonames index of the `geocode` command is English-only, the index can be rebuilt with the `geocode index-update` subcommand with the `--languages` option to return place names in multiple languages ([with support for 254 languages](http://download.geonames.org/export/dump/alternatenames/)).

## Testing
qsv has 5,126 tests: 4,400 for the qsv binaries (3,553 integration in the [tests](https://github.com/dathere/qsv/tree/master/tests) directory + 847 unit across `src/` modules), plus 726 for the MCP server & Cowork plugin (in [`.claude/skills/tests`](https://github.com/dathere/qsv/tree/master/.claude/skills/tests)).
Most commands have their own test suite in a separate file with the convention `test_<COMMAND>.rs`.
Apart from preventing regressions, the tests also serve as good illustrative examples, and are often linked from the usage text of each corresponding command.

To test each binary variant:

```bash
# to test qsv
cargo test --features all_features

# to test qsvlite
cargo test --features lite
# to test all tests with "stats" in the name with qsvlite
cargo test stats --features lite

# to test qsvmcp
cargo test --features qsvmcp

# to test qsvdp
cargo test --features datapusher_plus

# to test a specific command
# here we test only stats and use the
# t alias for test and the -F shortcut for --features
cargo t stats -F all_features

# to test a specific command with a specific feature
# here we test only luau command with the luau feature
cargo t luau -F feature_capable,luau

# to test the count command with multiple features
# we use "test_count" as we don't want to run other tests
# that have "count" in the testname - e.g. test_geocode_countryinfo
cargo t test_count -F feature_capable,luau,polars

# to test using an alternate allocator
# other than the default jemalloc allocator
cargo t --no-default-features -F all_features,mimalloc
```

## License

[MIT](LICENSE-MIT)


[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fjqnatividad%2Fqsv.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2Fjqnatividad%2Fqsv?ref=badge_large)

## Third-Party Software

qsv vendors and redistributes several open source components — among them
[plotly.js](https://github.com/plotly/plotly.js) and (bundled inside it)
[MapLibre GL JS](https://github.com/maplibre/maplibre-gl-js), which power `qsv viz`;
[DataTables](https://datatables.net/), which powers the `viz smart` data viewer;
[LuaDate](https://github.com/Tieske/date), used by `qsv luau`; and vendored DCAT-US, DCAT-AP and
geoconnex schemas used by `qsv profile`.

[**THIRD_PARTY_NOTICES.md**](THIRD_PARTY_NOTICES.md) records each component's upstream, version or
pinned commit, and license, together with the copyright notices and license texts those licenses
require us to carry. Sample datasets have their own credits in
[`examples/viz/README.md`](examples/viz/README.md) — note that `world_cities.csv` is GeoNames-derived
and CC BY 4.0, which **requires attribution** if you reuse it.

If you think something here is attributed incorrectly or not at all, please
[open an issue](https://github.com/dathere/qsv/issues) — we would rather over-attribute than
under-attribute.

## Origins

qsv is a fork of the popular [xsv](https://github.com/BurntSushi/xsv) utility. Building on this solid foundation, it was forked in Sept 2021 and has since evolved to a general purpose data wrangling toolkit, adding numerous commands and features.
See [FAQ](https://github.com/dathere/qsv/discussions/287) for more details.

## AI Policy
qsv has a very fast development pace with frequent releases. To keep up with this pace, we use the latest AI tools to help with various tasks, including code generation, documentation, testing, bug fixing, code review, etc.
To ensure the quality of the work, we maintain more than 4,000 tests; use multi-agent, multi-model, adversarial code reviews with [roborev](https://roborev.io) and GitHub Copilot; heavily comment the code to help Agents, Users and Developers understand it; and use tools like [Serena](https://oraios.github.io/serena) and [Context7](https://context7.com) to give Agents semantic context.

We accredit the Agent/Model used in commit messages and pull request descriptions. Contributors must do the same — see the [AI Policy in CONTRIBUTING.md](CONTRIBUTING.md#ai-policy) for the full contributor guidelines (attribution, review, licensing).

## Sponsor

<div align="center">

|qsv was made possible by|
:-------------------------:|
|[![datHere Logo](docs/images/datHere-logo-tagline.png)](https://datHere.com)<br>|
|Standards-based, best-of-breed, open source solutions<br>to make your **Data Useful, Usable & Used.**   |

</div>

## Naming Collision

This project is unrelated to [Intel's Quick Sync Video](https://www.intel.com/content/www/us/en/architecture-and-technology/quick-sync-video/quick-sync-video-general.html).
