## [20.0.0] - 2026-05-03 🧹 The "Spring Cleaning" Release 🌱 Using Claude Code, [roborev](https://www.roborev.io/), [Serena](https://oraios.github.io/serena), [Context7](https://context7.com) and GitHub Copilot orchestrated using an adversarial review workflow — we systematically audited every command for correctness, safety, and performance. Over the past four weeks, we did the first end-to-end pass over the qsv codebase. The result is the largest correctness-and-safety sweep in qsv's history: ALL commands were touched by review-driven cleanups, with dozens of latent bugs, panic paths, and performance cliffs swept out, while adding more than 250 new tests across the board. This is a major version bump because that sweep also surfaced four user-visible behaviors that were demonstrably wrong and could not be fixed without breaking compatibility: - `safenames` verify-mode now correctly counts duplicate-suffix renames as unsafe (previously under-reported). - `enum --hash` is now collision-resistant across multi-column inputs (previously ["ab","c"] and ["a","bc"] hashed identically). - `excel --metadata csv` column ordering now actually matches its header row (previously the type, visible, and headers columns held each other's values). - `util::safe_header_names` now enforces its 60-char cap in bytes end-to-end (previously chars-based, allowing UTF-8 names up to 240 bytes — past Postgres' NAMEDATALEN). Plus a few smaller but breaking corrections: `headers --intersect` is renamed to `--union` (the flag never computed an intersection), `luau qsv_loadcsv headersvare` now 1-indexed per Lua convention, and MSRV is bumped to Rust 1.95. Beyond the cleanup, this release adds one new top-level command: - NEW `implode` command: the inverse of `explode`. Groups rows by key column(s) and joins a value column into a single delimited string per group — useful for collapsing normalized output back into compact form. And a notable performance win: - `frequency`: parallel tree-reduce of partial frequency tables delivers a ~1.3x speedup on multi-core machines. Smaller per-command perf wins also landed in `fill` (+22%), `datefmt` (+9%), `cat`, `dedup`, `replace`, `search/searchset`, and `transpose`. Detailed MCP Server and Cowork Plugin changes are documented in the MCP Server/Cowork Plugin CHANGELOG. > [!IMPORTANT] > This is a major release with breaking changes. Pipelines that consume `qsv excel --metadata csv` by column position, store `qsv enum --hash` digests across versions, parse `qsv safenames` verify-mode output, or invoke `qsv headers --intersect` will need updates. See the Changed and Removed sections below for migration notes. --- ### Added - `implode`: new command — inverse of `explode` [#3733](https://github.com/dathere/qsv/pull/3733) (closes [#917](https://github.com/dathere/qsv/issues/917)) - `generators`: mark required options in help markdown and MCP skills [#3734](https://github.com/dathere/qsv/pull/3734) - `sortcheck`: add `--numeric` and `--natural` flags; allocation-free streaming loop [#3756](https://github.com/dathere/qsv/pull/3756) - `exclude`: add stdin support and memcheck [#3749](https://github.com/dathere/qsv/pull/3749) ### Changed - **BREAKING** `excel`: `--metadata csv` column ordering for `type`, `visible`, and `headers` is corrected. Previously the CSV header row declared `type, visible, headers` but the data rows pushed values in the order `headers, typ, visible`, so under each named column the wrong values appeared (the `type` column held the headers list, `visible` held the type, and `headers` held the visibility). The CSV output now matches the `--metadata json` (`SheetMetadata` struct) field order: `index, sheet_name, type, visible, headers, column_count, …`. Pipelines that consumed `qsv excel --metadata csv` and indexed by column position must shift those three columns; consumers that indexed by header name see corrected values automatically. - **BREAKING** `enum`: `--hash` digest values change. The hashed input now carries a `u64` length prefix per field (to fix the multi-column collision bug above), so every `--hash` digest differs from earlier qsv versions — single-column hashes change identity values too, and stored hashes from earlier qsv versions will not match. Same input still hashes deterministically across rows and runs in ≥ this version. - **BREAKING** `luau`: `qsv_loadcsv` now returns the headers table 1-indexed (per Lua convention). Scripts that accessed `headers[0]` or iterated `for i = 0, #headers - 1` must shift to `headers[1]` and `for i = 1, #headers` (or `ipairs(headers)`). Previously `headers[1]` returned the *second* header. - **BREAKING** `headers`: rename `--intersect` to `--union`. The flag has always computed a deduplicated union of headers across inputs, not a true set intersection — the name was a long-standing misnomer. `--intersect` is removed entirely (no alias) given the surrounding breaking-change window. Migration: replace `qsv headers --intersect …` with `qsv headers --union …`; output is unchanged. - **BREAKING** `safenames`: verify-mode (`--mode v / V / j / J`) outputs change. (1) Verify counts now include header positions that would be renamed by the duplicate-suffix pass — inputs containing duplicate column names will report higher unsafe counts than earlier qsv versions; the count now matches what `--mode a` would actually rewrite. (2) `--mode V / j / J` displays unsafe-header strings with leading/trailing whitespace and surrounding `"` already trimmed (matching what the safe-rename pass actually evaluates), and `duplicate_headers` is now sorted alphabetically rather than appearing in undefined HashMap iteration order. Pipelines that parsed verbose/JSON output and depended on the old ordering or untrimmed strings must update. - **BREAKING** `util::safe_header_names`: the 60-length cap is now enforced in **bytes** on the *final* name, including any duplicate-disambiguation suffix. Previously the truncation was chars-based (`take(60).sum()`) and only applied to the base, so non-ASCII headers could produce up to ~240-byte names and duplicate-disambiguated headers added `_<n>` *after* truncation, pushing past Postgres' `NAMEDATALEN` (63 bytes). Now the rewrite path lowercases and prepends the leading-`_` prefix *before* truncating, then snaps to a UTF-8 char boundary at ≤60 bytes. ASCII-only inputs see the same output as before for non-suffixed cases. Long ASCII headers that previously generated 61–63-char suffixed variants will be 1–2 chars shorter at the boundary. Headers containing multibyte UTF-8 (CJK, accented chars, emoji) that previously produced names >60 bytes will now be aggressively trimmed to fit. Affects every caller (`safenames`, `applydp`, `apply`, `fetch`, `python`); stored mappings keyed on the old over-long forms will not match. - **BREAKING** MSRV bumped to Rust 1.95 - `describegpt`: split `process_phase_output` into per-branch helpers (dictionary context-only, full dictionary, JSON, TSV, TOON, Markdown). No behavior change — same output, smaller functions. - `luau`: `qsv_coalesce` now stringifies non-string values (numbers and booleans render via `to_string`; nil / arrays / objects are skipped). Previously, numbers and booleans were silently treated as missing values via `as_str().unwrap_or_default()`. Scripts relying on `qsv_coalesce(some_bool, fallback)` to skip booleans will now return `"true"`/`"false"` for the boolean. - `describegpt`: per-phase helper split, widened cache key, ~21% LOC reduction [#3720](https://github.com/dathere/qsv/pull/3720) [#3721](https://github.com/dathere/qsv/pull/3721) [#3722](https://github.com/dathere/qsv/pull/3722) - `frequency`: parallel tree-reduce of partial FTables (~1.3x speedup) [#3728](https://github.com/dathere/qsv/pull/3728) - `moarstats`: collapse duplicated outlier bivariate scan; safety/perf cleanup, unit tests [#3718](https://github.com/dathere/qsv/pull/3718) [#3719](https://github.com/dathere/qsv/pull/3719) - `validate`: use `cold_hint` (stabilized in Rust 1.95) [#3717](https://github.com/dathere/qsv/pull/3717); correctness, perf cleanup [#3743](https://github.com/dathere/qsv/pull/3743) [#3779](https://github.com/dathere/qsv/pull/3779) - `frequency`: correctness, perf, refactor cleanup [#3745](https://github.com/dathere/qsv/pull/3745) - `apply`: review-driven cleanup, perf [#3741](https://github.com/dathere/qsv/pull/3741) - `template`: subdir bug fix, lookup perf, render-error visibility, helper extraction [#3740](https://github.com/dathere/qsv/pull/3740) - `dedup`: allocation-free ignore-case [#3754](https://github.com/dathere/qsv/pull/3754) - `datefmt`: ~9% perf [#3753](https://github.com/dathere/qsv/pull/3753) - `fill`: ~22% faster hot path [#3762](https://github.com/dathere/qsv/pull/3762) - `replace`: streaming parallel write; dead match-flag tracking [#3777](https://github.com/dathere/qsv/pull/3777) - `search`/`searchset`: parallel memory streaming; `--quick` fixes; USAGE alignment [#3776](https://github.com/dathere/qsv/pull/3776) - `cat`: rowskey speedup [#3750](https://github.com/dathere/qsv/pull/3750) - `transpose`: correctness, perf cleanup, polish [#3781](https://github.com/dathere/qsv/pull/3781) - `cleanup`: rename `fail_oom_clierror`; surface `geocode` update-check error [#3806](https://github.com/dathere/qsv/pull/3806) - applied select clippy lints ### Fixed - `excel`: review-driven cleanup of `src/cmd/excel.rs` — fix four correctness bugs. (1) Negative `--sheet` indices that overshot the sheet count silently selected a wrong sheet because the `abs_diff` clamp "bounced" past zero (e.g. `--sheet -4` on a 3-sheet workbook returned the 2nd sheet); now errors with `usage error: negative sheet index N is out of range for K sheets`. (2) `get_requested_range` lower-cased the sheet name into the caller's `sheet` variable and never restored it, so `--range 'Sheet2!A1:B2' --error-format formula|both` failed because calamine's `worksheet_formula` is case-sensitive, and the success message printed the wrong case; now the canonical name from the workbook's sheet list is preserved. (3) The float-to-i64 conversion guard used `*float_val > i64::MAX as f64`, but `i64::MAX as f64` rounds *up* past `i64::MAX`, so a value of `2^63` slipped through, saturating-cast to `i64::MAX`, and was emitted as `9223372036854775807` (off by one). Replaced with `is_finite() && val >= i64::MIN as f64 && val < i64::MAX as f64 && fract() == 0.0`, with a NaN/Inf fallback to `Display` (the previous code would have hit `zmij::format_finite` UB on non-finite floats). (4) `--range` against an empty sheet reported "larger than sheet" because the bounds check used a `(0, 0)` fallback that swallowed the empty-sheet case; now reports "sheet is empty" distinctly. Also: hoisted `to_lowercase()` allocations out of the table-name and named-range search loops, fixed two USAGE typos (`3nd` → `3rd`, stray paren in the negative-index example), removed a stray blank line in `NamesMetadata`'s derive, and dropped the now-unused `cmp` import. - `enum`: review-driven cleanup of `src/cmd/enumerate.rs` — fix silent multi-column hash collision and lossy UTF-8 hashing. Previously `--hash` over multiple columns concatenated the UTF-8 strings of each field into a single buffer before hashing, so distinct rows like `["ab","c"]` and `["a","bc"]` flattened to `"abc"` and hashed identically; non-UTF-8 fields were silently replaced with the empty string via `unwrap_or_default()`, masking different invalid byte sequences. Switch to streaming `Xxh3` over raw bytes with a per-field length prefix, eliminating both bugs and removing per-row `String` allocation. Also fix a degenerate-selection bug where a `--hash` selector resolving to *only* the existing `hash` column (e.g. `--hash hash` on `name,hash` headers) silently re-included the `hash` column — the post-filter parse/selection round-trip turned the empty selection into "all columns"; now constructed via `Selection::from_indices` and rejected with a clear error. Also: collapse three redundant `select()` calls in hash setup, avoid per-row `ByteRecord` reallocation when an existing `hash` column is dropped, fix docstring mode count (was "four", now "six") and duplicated `3.` numbering, and add `enumerate_hash_no_concat_collision` regression test. - `util`: review-driven hardening of `src/util.rs` — defend `process_input` against zip-slip, fix panics in `optimal_batch_size`, `safe_header_names` (whitespace-only headers), `qsv_check_for_update` (empty release list), and `format_systemtime` (pre-1970 timestamps); drop UB-prone `unsafe { get_unchecked }` from `write_json` / `csv_to_jsonl` (this also fixes a long-standing trailing-comma bug when records were shorter than headers); log warnings (including the input path) when row counting hits a CSV read error instead of silently undercounting. **Behavior change:** `download_file` now propagates `timeout_secs` validation errors instead of silently coercing a bad timeout to 30s — passing `--timeout 0` or `--timeout` > 3600 to a downloading command will now error. `get_envvar_flag` now trims whitespace, accepts `on`/`off`, and warns at most once per unrecognized (key, value) pair. - `describegpt`: widen cache key to cover all template-affecting flags (`--tag-vocab`, `--num-tags`, `--enum-threshold`, `--sample-size`, `--fewshot-examples`, the `QSV_DUCKDB_PATH` toggle, and the generated Data Dictionary). Previously, changing any of these between runs silently returned stale cached output. First run after upgrade will re-invoke the LLM once per phase as prior cache entries no longer match. - `luau`: fix stale per-column globals when a row has fewer fields than headers (in both sequential and random-access modes); fix `_LASTROW`/`_ROWCOUNT` math under `--no-headers` in random-access mode (was off-by-one); fix `_LASTROW` underflow on empty input; fix `qsv_loadcsv` swallowing CSV parse errors; bound `qsv_lag` / `qsv_diff` history with `VecDeque` to prevent unbounded memory growth on large CSVs; reject `lag`/`periods <= 0` with clear errors; short-circuit MAIN on empty random-access input so END can still run. - `headers`: review-driven cleanup of `src/cmd/headers.rs`. The docopt USAGE for `--intersect` claimed it computed a set intersection, but the implementation (and the test fixtures) have always emitted a deduplicated *union* across inputs — see the BREAKING `--union` rename in *Changed*. Tightened the multi-input path: now actually sets `flag_just_names` when more than one input is given (previously the docopt promised this but the code only suppressed the index prefix), removing a redundant `num_inputs == 1` guard and the wasted `TabWriter` wrap on that path. `--trim` now operates on raw bytes (no `String::from_utf8_lossy` round-trip, so non-UTF-8 header bytes pass through untouched) and additionally strips leading/trailing tab characters in addition to spaces and double-quotes. - `safenames`: review-driven cleanup of `src/cmd/safenames.rs` — fix two correctness bugs in verify modes (`--mode v / V / j / J`). (1) Headers that the rename pass changed only via the duplicate-suffix step (e.g. `col1, col1` → `col1, col1_2`) were counted as *safe* because the verify loop checked membership against the rewritten list rather than comparing positionally; the count now agrees with always-mode's `changed_count` for the same input. See BREAKING note in *Changed* for output impact. (2) The verify loop iterated the *original* (untrimmed, quote-included) headers but compared against the quote-and-space-trimmed list that was actually passed to `safe_header_names`, so a literally-quoted header like `"col"` was wrongly flagged unsafe; now compares trimmed-to-trimmed. Also: replaced two `simd_json::to_string{,_pretty}(...).unwrap()` calls with `?` propagation (a serialize failure now surfaces a CLI error instead of panicking); collapsed the verify-mode body into a single pass (replaces O(N²) `Vec::contains` lookups + four `String` allocations per header with a `HashSet` dedup and `entry().or_insert(0)` count update); sorted `duplicate_headers` for deterministic output (previously `foldhash::HashMap` iteration order leaked into stderr/JSON, forcing tests to accept two orderings); write `safe_headers` directly in the always/conditional path instead of clearing a `StringRecord` and re-pushing each field; rewrote the `--mode` docopt block to make case sensitivity explicit (the parser keys off only the first character, with case-sensitive `v`/`V` and `j`/`J` distinctions previously buried) and dropped a misleading "verify does not count quoted identifiers as unsafe" note that contradicted the actual implementation and the in-tree example output. Tightened the `Some(reserved_names_vec).as_ref()` call to `Some(&reserved_names_vec)`. - `util`: `safe_header_names` now enforces the length limit in **bytes** (≤60, snapped to a UTF-8 char boundary) end-to-end, including any disambiguation suffix. Previously the truncation was a hybrid: `is_safe_name` rejected names >60 *bytes*, but the rewrite path used `chars().map(char::len_utf8).take(60).sum()` (chars-based, up to 240 bytes for 4-byte UTF-8) and the duplicate-suffix step appended `_2`/`_3` *after* truncation, so a non-ASCII or duplicate-disambiguated header could land at 62–240+ bytes — past the documented bound and Postgres' default `NAMEDATALEN` of 63 bytes. The function now lowercases and prepends the leading-`_` prefix *before* truncation (case-folding can change byte length, prefixing adds bytes) and snaps the truncation to a char boundary via `str::floor_char_boundary`. Affects every caller (`safenames`, `applydp`, `apply`, `fetch`, `python`); see BREAKING note in *Changed*. - `apply`: malformed CSV, init bugs, doc typos, perf [#3741](https://github.com/dathere/qsv/pull/3741) - `blake3`: check-mode interop fixes, parser polish [#3782](https://github.com/dathere/qsv/pull/3782) - `cat`: `--no-headers` fix [#3750](https://github.com/dathere/qsv/pull/3750) - `clipboard`: surface error details, use `?` for clipboard ops [#3783](https://github.com/dathere/qsv/pull/3783) - `color`: drop dead clones, mark in-memory [#3785](https://github.com/dathere/qsv/pull/3785) - `config`: HumanCount overflow, env-var unwrap panics, sniff UTF-8 panic [#3770](https://github.com/dathere/qsv/pull/3770) - `count`: quoting fix, stdin temp-file leak [#3751](https://github.com/dathere/qsv/pull/3751) - `datefmt`: strict tz/flag validation [#3753](https://github.com/dathere/qsv/pull/3753) - `dedup`: edge-case fixes [#3754](https://github.com/dathere/qsv/pull/3754) - `diff`: surface builder error, dedupe index/colname parsing [#3784](https://github.com/dathere/qsv/pull/3784) - `edit`: `--in-place`, bounds checks, silent no-ops [#3786](https://github.com/dathere/qsv/pull/3786) - `explode`: validate separator and column selection [#3787](https://github.com/dathere/qsv/pull/3787) - `extdedup`: key-collision and dupes-writer issues [#3759](https://github.com/dathere/qsv/pull/3759) - `extsort`: CRLF off-by-one fix (line→record) [#3790](https://github.com/dathere/qsv/pull/3790); error propagation, temp-file handle release [#3789](https://github.com/dathere/qsv/pull/3789) - `fetch`/`fetchpost`: cache, panic, safety fixes [#3747](https://github.com/dathere/qsv/pull/3747) - `fixlengths`: `--remove-empty` crash on flexible rows; widen insert arithmetic [#3764](https://github.com/dathere/qsv/pull/3764) - `fmt`: `--no-final-newline` bugs, tempfile leak [#3767](https://github.com/dathere/qsv/pull/3767) - `foreach`: dry-run truncation, panic, multi-column drop, child-failure propagation [#3757](https://github.com/dathere/qsv/pull/3757) - `geocode`: remove latent panics, fix FIPS JSON shape, perf [#3739](https://github.com/dathere/qsv/pull/3739) - `geoconvert`: lat/lon swap, tempfile leak, UTF-8 panic [#3768](https://github.com/dathere/qsv/pull/3768) - `input`: panic, validation, clarity fixes [#3791](https://github.com/dathere/qsv/pull/3791) - `join`: unify key transform; fix silent `--keys-output` drop [#3769](https://github.com/dathere/qsv/pull/3769) - `joinp`: correctness, validation, schema-handling [#3731](https://github.com/dathere/qsv/pull/3731) - `json`: preserve BigInt precision, surface `jaq` runtime errors [#3794](https://github.com/dathere/qsv/pull/3794); clarify `--jaq` numeric precision in USAGE; defer BigInt `to_string` allocation [#3795](https://github.com/dathere/qsv/pull/3795) - `jsonl`: honor `--ignore-errors` for header inference; fix line-number reporting [#3796](https://github.com/dathere/qsv/pull/3796) - `lens`: fix `--streaming-stdin`; reject invalid `--wrap-mode` [#3797](https://github.com/dathere/qsv/pull/3797) - `lookup`: harden cache, download errors, CKAN auth handling [#3803](https://github.com/dathere/qsv/pull/3803) - `luau`: correctness and consistency [#3742](https://github.com/dathere/qsv/pull/3742) - `moarstats`: Atkinson re-population bug; harden test coverage [#3799](https://github.com/dathere/qsv/pull/3799) - `partition`: UTF-8 panic, O(N) collision check [#3771](https://github.com/dathere/qsv/pull/3771) - `pivotp`: correctness, clarity, cleanup [#3732](https://github.com/dathere/qsv/pull/3732) - `pragmastat`: Windows backup path; suppress meaningless date ratios [#3805](https://github.com/dathere/qsv/pull/3805) - `prompt`: stream file I/O; avoid unnecessary clone [#3798](https://github.com/dathere/qsv/pull/3798) - `pseudo`: reject `--increment 0`; preserve last-valid-counter row on overflow [#3792](https://github.com/dathere/qsv/pull/3792) - `py`: hoist Python module setup; jagged-row panic [#3758](https://github.com/dathere/qsv/pull/3758) - `reverse`: avoid u64 underflow on indexed reverse [#3808](https://github.com/dathere/qsv/pull/3808) - `sample`: streaming bernoulli header bug; dead retry loop; cluster pre-alloc [#3774](https://github.com/dathere/qsv/pull/3774); add integration tests for streaming Bernoulli URL path [#3775](https://github.com/dathere/qsv/pull/3775) - `schema`: correctness, panic-safety [#3746](https://github.com/dathere/qsv/pull/3746) - `scoresql`: USING panic, string-literal handling, filter heuristic + tests [#3810](https://github.com/dathere/qsv/pull/3810) - `select`: review-driven cleanup, fix `/` panic, quoted-name CSV-escape, empty-name silent fall-through [#3772](https://github.com/dathere/qsv/pull/3772); `--sort` round-trip (quotes, non-UTF-8, dup names) [#3773](https://github.com/dathere/qsv/pull/3773) - `slice`: panic and underflow fixes [#3748](https://github.com/dathere/qsv/pull/3748) - `snappy`: preserve validate error; guard decompress ratio on stdin [#3809](https://github.com/dathere/qsv/pull/3809) - `sort`: `--numeric --natural --unique` consistency [#3755](https://github.com/dathere/qsv/pull/3755) - `split`: correctness fixes, error propagation, tests [#3780](https://github.com/dathere/qsv/pull/3780); Windows `--filter` quoted-arg corruption fix via `raw_arg` [#3788](https://github.com/dathere/qsv/pull/3788) - `sqlp`: word-boundary alias replacement [#3730](https://github.com/dathere/qsv/pull/3730) - `stats`: cache & boolean-pattern fixes [#3744](https://github.com/dathere/qsv/pull/3744); close cache short-circuit gaps for select/round/typesonly/infer-boolean [#3800](https://github.com/dathere/qsv/pull/3800) - `tojsonl`: guard non-finite Number; hoist header escaping; drop unused `unused_assignments` allows [#3807](https://github.com/dathere/qsv/pull/3807) ### Removed - **BREAKING** `headers`: removed `--intersect` flag (use `--union` instead) [#3763](https://github.com/dathere/qsv/pull/3763) ### Dependencies - Bump polars to 0.53.0 (multiple bumps; latest tracks py-1.40.1) - Bump `mlua` to 0.12.0-rc.1; Luau from 709 to 716 - Bump `zip` from 7 to 8 - Switch csv crate to qsv-tuned fork (replaces ryu with zmij) - Bump `jsonschema` from 0.46.1 to 0.46.4 [#3723](https://github.com/dathere/qsv/pull/3723) [#3778](https://github.com/dathere/qsv/pull/3778) [#3804](https://github.com/dathere/qsv/pull/3804) - Bump `mimalloc` from 0.1.49 to 0.1.50 [#3729](https://github.com/dathere/qsv/pull/3729) - Bump `rayon` from 1.11.0 to 1.12.0 [#3710](https://github.com/dathere/qsv/pull/3710) - Bump `tokio` from 1.51.1 to 1.52.0 [#3712](https://github.com/dathere/qsv/pull/3712) - Bump `libc` from 0.2.184 to 0.2.186 [#3709](https://github.com/dathere/qsv/pull/3709) [#3736](https://github.com/dathere/qsv/pull/3736) - Bump `reqwest` from 0.13.2 to 0.13.3 [#3766](https://github.com/dathere/qsv/pull/3766) - Bump `blake3` from 1.8.4 to 1.8.5 [#3738](https://github.com/dathere/qsv/pull/3738) - Bump `magika` from 1.0.1 to 1.1.0 [#3737](https://github.com/dathere/qsv/pull/3737) - Bump `robinraju/release-downloader` from 1.12 to 1.13 [#3726](https://github.com/dathere/qsv/pull/3726) - Bump `qsv-stats` from 0.49.0 to 0.50.0 [#3727](https://github.com/dathere/qsv/pull/3727) - Bump `qsv_docopt` from 1.9.0 to 1.10.0 [#3724](https://github.com/dathere/qsv/pull/3724) - Update `self_update` to latest upstream (qsv PR merged) - Update `geosuggest` to 0.8.3 - Update `csvs_convert` - Removed `kiddo` patch fork now that 0.5.3 is released with our PR merged **Full Changelog**: https://github.com/dathere/qsv/compare/19.1.0...20.0.0

## [12.0.0] - 2025-12-24 🎄 Stuff your virtual stocking and jingle your data bells - qsv 12.0.0 slides down the chimney packed fuller than Santa’s sleigh! Unwrap delightful surprises like the shiny new `moarstats` command, gift-wrapped weighted statistics, and AI-powered FAIR metadata inferencing now speaking in multiple languages (no elf translation required). As the star on top, meet TOON - the [brand new LLM-optimized, token-efficient format](https://openapi.com/blog/what-the-toon-format-is-token-oriented-object-notation) - ready to sleigh your AI projects all through 2026. Ho-ho-hold my data, this update’s a festive feast! ## 🌟 Major Features ### NEW: moarstats Command A powerful new command for "[moar](https://www.dictionary.com/culture/slang/moar)" advanced statistical analysis, providing statistics beyond what the `stats` command offers: - **Comprehensive Statistics**: Over 50+ advanced statistical measures including: - Detailed outlier analysis (count, sum, average) - Winsorized and trimmed means (5%, 10%, 20%, 25%) - Multiple dispersion measures (IQR to range ratio, quartile coefficient of dispersion) - Distribution statistics (skewness, multiple kurtosis measures) - **Advanced Option** (`--advanced`): Access computationally intensive statistics: - Gini coefficient for inequality measurement - Excess Kurtosis to measure "tailedness" of the distribution - Shannon Entropy for data diversity analysis - **Available on all binary variants** for universal access ### Enhanced describegpt Command Major enhancements to AI-powered data description capabilities: - **⛩️ Minijinja Template Engine Integration**: - Custom prompt templating with full Minijinja and Minijinja-contrib filters - More powerful and flexible prompt customization - **Multilingual Support**: - `--language` option for generating descriptions in any language/dialect - Automatic language detection in prompts - SQL comments also generated in requested language - beyond language/dialect, this option can also be used to describe a dataset using a persona (e.g. Yoda, Spock, Valley Girl, Christopher Walken, Silly Santa after taking a Data Science Course, etc.) - **Advanced Features**: - `--addl-columns` option with detailed attribution and system metadata - `--export-prompt <file>` to save the default prompts to the specified file. This file can then be tailored and used with the `--prompt-file <file>` option. - Iterative, session-based SQL RAG with `--prompt` option - Sampling in prompt mode for better SQL generation - Lookup table and CKAN support for controlled vocabularies - Convenience values for `--addl-cols-list` (i.e., "everything", "everything!", "moar", "moar!") ### Weighted Statistics Support Comprehensive weighted statistics implementation across multiple commands: - **stats Command** (`--weight <column>`): - Weighted mean, standard deviation, variance - Weighted MAD (Median Absolute Deviation) and percentiles - Weighted modes and antimodes - Weighted harmonic and geometric means - All weighted calculations handle non-finite values gracefully - **frequency Command** (`--weight <column>`): - Weighted frequency distributions - Proper handling of weighted "Other" and "ALL UNIQUE" category - Non-finite weights automatically skipped ### Token Object Oriented Notation ([TOON](https://toonformat.dev)) Format Support - A compact, human-readable encoding of the JSON data model for LLM prompts - **Commands Supporting TOON**: - `describegpt --format TOON` - `frequency --toon` - **Benefits**: More readable than JSON, easier to parse than CSV for hierarchical data and more token-efficient, terse format targeted for LLMs ### stats Command Enhancements - **Percentile Improvements**: - `--percentile-list` special values: "deciles" and "quintiles" - Percentile labels now include prefix before value (e.g., "p50: 42.5") - Validation of percentile-list on startup - **New Columns**: Added `n_counts` for more detailed count information - **Performance Optimizations**: - Optimized Stats struct layout - Eliminated redundant, unnecessary sorting - Removed redundant filtering for weighted stats functions - Microoptimizations throughout ### transpose Command - **New `--long` Option**: Transform data from wide to long format - Column selection support using select syntax - Streaming implementation per GitHub Copilot review suggestions ### diff Command - upgraded csv-diff from 0.1.1 to faster 0.1.2, improving performance in optimal cases by up to 25% 🚀 ### lens Command - Aligned `--no-streaming-stdin` behavior with csvlens upstream ## 📊 Output Format Changes ### schema Command - Updated `$schema` from Draft 7 to **JSON Schema Draft 2020-12** ## ⚡ Performance Improvements ### suite-wide - replaced already fast ryu float to string conversion crate crate with even faster zmij crate (https://vitaut.net/posts/2025/faster-dtoa/) ### stats Command - Optimized Stats struct memory layout - Eliminated redundant sorting operations - Removed unnecessary clone operations - Better handling of real-world data (assumes no infinity values) ### frequency Command - Microoptimizations for faster frequency computation - Optimized top_n/bottom_n retrieval ## 🐛 Bug Fixes ### frequency Command - Fixed behavior when compiling weighted frequencies with `ALL_UNIQUE` - Fixed issue where "Other (0),0,0,0" could appear in output - Proper handling of non-finite weights (automatically skipped) ## 🏗️ Infrastructure & Quality ### Testing - Test suite expanded from 2,060 to **2,380 tests** - Comprehensive test coverage for all new features - Weighted statistics thoroughly tested - Advanced moarstats options validated ### Code Quality - Extensive GitHub Copilot review integration - Multiple refactoring passes for code clarity - Clippy suggestions incorporated throughout - Better error handling and edge case management ### FAIR Principles - Added **CITATION.cff** (by rzmk) for academic citation - Added **Zenodo DOI badge** for dataset citation - Enhanced FAIRification of qsv as a research tool ## 📚 Documentation Improvements ### Statistical Documentation - Comprehensive documentation for statistics produced by stats command (by @kulnor) WIP - Enhanced usage text for stats, frequency, and moarstats - Better examples throughout documentation ### Command Documentation - Updated describegpt with multilingual examples - Added controlled tag vocabulary examples - Enhanced TOON format documentation - Better SQL RAG workflow documentation --- ## Migration Notes ### Breaking Changes 1. **schema command**: `$schema` output changed from Draft 7 to Draft 2020-12 - Most schemas should be compatible - Validation tools must support JSON Schema Draft 2020-12 2. **stats command**: Output now includes percentile label prefixes - Example: "p50: 10" of the 50th percentile value instead of just the value "10" - May affect parsing scripts that expect raw numbers --- ## Added * feat: `describegpt` add `--add-cols` and `--addl-cols-list <list>` options https://github.com/dathere/qsv/pull/3179 * feat: `describegpt` add `--language` option https://github.com/dathere/qsv/pull/3184 * feat: `describegpt` use minijinja engine for prompt processing https://github.com/dathere/qsv/pull/3188 * feat: `describegpt` add language autodetection in `--prompt` (chat) mode https://github.com/dathere/qsv/pull/3193 * feat: `describegpt` sampling in prompt mode for better SQL generation… https://github.com/dathere/qsv/pull/3198 * feat: `describegpt` add --prompt sessions for iterative SQL RAG refinement https://github.com/dathere/qsv/pull/3200 * feat: `describegpt` add TOON format support https://github.com/dathere/qsv/pull/3205 * feat: `frequency` add TOON format https://github.com/dathere/qsv/pull/3206 * feat: `frequency` add weighted frequencies https://github.com/dathere/qsv/pull/3218 * feat: add new `moarstats` command https://github.com/dathere/qsv/pull/3207 * feat: `moarstats` add even moar! Now with detailed outliers info! https://github.com/dathere/qsv/pull/3208 * feat: `moarstats` - add configurable Winsorized and Trimmed means https://github.com/dathere/qsv/pull/3209 * build(deps): bump ryu from 1.0.20 to 1.0.21 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3210 * chore: `moarstats` remove redundant Bowley's Skewness Coefficient https://github.com/dathere/qsv/pull/3212 * feat: `moarstats` add kurtosis & gini stats behind `--advanced` option https://github.com/dathere/qsv/pull/3217 * feat: `moarstats` moar, moar, moar stats! https://github.com/dathere/qsv/pull/3220 * feat: `moarstats` add shannon entropy to advanced statistics https://github.com/dathere/qsv/pull/3227 * feat: `stats` `--percentile-list` special values "deciles" and "quintiles" https://github.com/dathere/qsv/pull/3176 * docs: added qsv stats descriptions document by @kulnor in https://github.com/dathere/qsv/pull/3172 * feat: add CITATION.cff by @rzmk in https://github.com/dathere/qsv/pull/3182 * feat: `stats` add percentile label prefixes in front of percentile values https://github.com/dathere/qsv/pull/3183 * feat: `stats` add weighted statistics https://github.com/dathere/qsv/pull/3213 * feat: `transpose` add `--long` option https://github.com/dathere/qsv/pull/3194 * feat: `transpose` add `--long` column selection https://github.com/dathere/qsv/pull/3197 ## Changed * feat: `schema` change `$schema` from `https://json-schema.org/draft-07/schema` to `https://json-schema.org/draft/2020-12/schema` https://github.com/dathere/qsv/pull/3203 * deps: bump blake3 to latest upstream * deps: bump csvlens to 0.15.0 * deps: bump geozero to 0.15.0 * deps: indexmap - enable serde feature * deps: bump redis to 1 * deps: cached use upstream fork with redis updated to 1 * deps: jsonschema use latest upstream * deps: polars use latest upstream * deps: replaced ryu with faster zmij binary to decimal floating point library * build(deps): bump actions/upload-artifact from 5 to 6 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3189 * build(deps): bump csv-diff from 0.1.1 to 0.1.2 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3228 * build(deps): bump governor from 0.10.2 to 0.10.4 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3196 * build(deps): bump itoa from 1.0.15 to 1.0.16 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3214 * build(deps): bump minijinja from 2.13.0 to 2.14.0 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3185 * build(deps): bump minijinja-contrib from 2.13.0 to 2.14.0 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3186 * build(deps): bump qsv-stats from 0.43.0 to 0.44.0 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3215 * build(deps): bump qsv-stats from 0.44.0 to 0.45.0 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3216 * build(deps): bump reqwest from 0.12.24 to 0.12.25 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3177 * build(deps): bump reqwest from 0.12.25 to 0.12.26 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3191 * build(deps): bump reqwest from 0.12.26 to 0.12.27 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3221 * build(deps): bump reqwest from 0.12.27 to 0.12.28 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3226 * build(deps): bump serde_json from 1.0.145 to 1.0.146 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3219 * build(deps): bump serde_json from 1.0.146 to 1.0.147 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3229 * build(deps): bump tempfile from 3.23.0 to 3.24.0 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3230 * build(deps): bump toml from 0.9.8 to 0.9.9+spec-1.0.0 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3199 * bumped several indirect dependencies * applied select clippy & Codacy suggestions * bumped MSRV to 1.92 ## Fixed: * fix: `frequency` fix ALL_UNIQUE weighted behavior https://github.com/dathere/qsv/pull/3224 * fix: `frequency` fix "Other (0),0,0,0" should never happen https://github.com/dathere/qsv/pull/3225 ## Removed: * deps: blake3 removed unnecessary conditional compilation directive **Full Changelog**: https://github.com/dathere/qsv/compare/11.0.2...12.0.0

## [11.0.1] - 2025-12-08 qsv 11.0.1 brings significant enhancements to larger-than-memory data processing, AI-powered metadata inferencing, JSON Schema inferencing & validation, and data viewing capabilities, along with important bug fixes and performance improvements. All in preparation for at-scale, secure, interactive, "zero-copy" "Data Steward-in-the-Loop" FAIRification on the desktop in qsv pro. ## 🌟 Major Features ### `stats` & `frequency` Command Enhancements - **Larger than Memory Files**: `stats` & `frequency` can now handle arbitrarily large files, even when "advanced" statistics are enabled with its new dynamic parallel chunk sizing algorithm! - **N Counts**: Added "n_counts" (`n_negative`, `n_zero` and `n_positive`) columns to `stats` output for more detailed count information for numeric fields. ### `describegpt` Command Enhancements The `describegpt` command has received substantial improvements for AI-powered metadata inferencing: - **Format Option**: Replaced `--json` flag with `--format` option for more flexible output formatting - Supports multiple output formats - Markdown (default), TSV and JSON - Removed `--jsonl` option for cleaner API - **Controlled Tag Vocabulary**: New tag vocabulary system for consistent categorization - `--tag-vocab` option to specify controlled vocabulary - Lookup support for tag vocabularies - retrieve a tag vocabulary from a local or remote CSV using `http://`, `https://`, `dathere://` and `ckan://` URL schemes. - **Enhanced Boolean Inference**: `--infer-boolean` is now enabled by default for better data type detection - **Performance Metrics**: Added elapsed time tracking to monitor processing duration - **Improved Prompt Templates**: Updated default description prompt with PII/PHI alerts and better attribution metadata ### `schema` & `validate` Command Improvements Enhanced JSON Schema inference and validation capabilities: - **Strict Formats**: New `--strict-formats` option for stricter JSON Schema format validation, enforcing JSON Schema format constraints for email, hostname and IP address (IPV4 and IPV6) formats. - **Output Option**: New `--output` option for specifying schema output destination - Polars schema now uses consistent naming conventions across commands - Updated `joinp`, `pivotp`, and `sqlp` commands to use new `.pschema.json` naming convention - **Configurable Email Validation**: `validate` has numerous options to tweak email validation - taking advantage of `schema`'s email format constraint inferencing. ### `sample` Command time-series sampling A new `--timeseries` sampling method with grouping (hourly, daily, weekly), adaptive sampling (prefer business hours or weekends) with various aggregation (mean, sum, min, max) within each interval with configurable starting points (first, last or random). ### `lens` Command Features Enhanced CSV viewing capabilities with csvlens integration: - **Auto-Reload**: New `--auto-reload` option to automatically reload file when it changes - Useful for monitoring live data files - **Streaming stdin**: New `--streaming-stdin` option for real-time data viewing - Supports viewing data as it's being piped in - **Row Marking**: Updated csvlens dependency with row marking feature ### Breaking Changes - `describegpt`: `--json` flag replaced with `--format` option - `describegpt`: `--jsonl` option removed - `schema`, `joinp`, `pivotp`, `sqlp`: Updated Polars schema naming conventions (existing workflows should work but output format may differ slightly) --- ## Added * Created [Event Logo Archive](https://github.com/dathere/qsv/tree/master/docs/images/event-logos) with AI-generated seasonal/version logos * `describegpt`: add controlled vocabulary support for tags https://github.com/dathere/qsv/pull/3122 * `describegpt`: add elapsed time https://github.com/dathere/qsv/pull/3168 * `describegpt`: add lookup support https://github.com/dathere/qsv/pull/3170 * `excel`: add `--cell` option https://github.com/dathere/qsv/pull/3133 * `frequency`: add dynamic parallel chunk sizing https://github.com/dathere/qsv/pull/3135 * `lens`: add `--auto-reload` option https://github.com/dathere/qsv/pull/3128 * `lens`: add `--streaming-stdin` option https://github.com/dathere/qsv/pull/3171 * `sample`: add timeseries sampling options https://github.com/dathere/qsv/pull/3130 * `schema`: infer addl JSON Schema predefined formats - email, ipv4, ipv6, hostname https://github.com/dathere/qsv/pull/3125 * `schema`: add `--output` option and standardize Polars Schema file name https://github.com/dathere/qsv/pull/3126 * `stats`: dynamic parallel chunk sizing with indexed files https://github.com/dathere/qsv/pull/3134 * `stats`: add n_negative, n_zero, n_positive count columns https://github.com/dathere/qsv/pull/3157 * `validate:` add email validation options https://github.com/dathere/qsv/pull/3148 * `tests`: add tests for https://100.dathere.com/lessons/4 by @rzmk in https://github.com/dathere/qsv/pull/3151 * Added Claude AI guidance for contributors * Enhanced `--version` output with more comprehensive system metadata ## Changed * refactor: `describegpt` improve tags inferencing with Tag Vocabulary https://github.com/dathere/qsv/pull/3139 * feat: `describegpt` - major refactor https://github.com/dathere/qsv/pull/3143 * feat: `describegpt` improved Polars SQL processing https://github.com/dathere/qsv/pull/3147 * feat: `describegpt` replace `--json` option with `--format` option supporting 3 formats - markdown, json and TSV; remove `--jsonl` option https://github.com/dathere/qsv/pull/3167 * refactor: `frequency` & `stats` - parallel chunk sizing - allow forcing of cpu based chunking https://github.com/dathere/qsv/pull/3138 * Align partition stdin handling with split/stats pattern by @Copilot in https://github.com/dathere/qsv/pull/3162 * deps: use latest polars upstream with new SQL fixes and features (https://github.com/pola-rs/polars/commit/e1be17f2ccb9dee0d570c6126b54c0e44ae7131d) * build(deps): bump actions/setup-python from 6.0.0 to 6.1.0 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3120 * build(deps): bump actix-web from 4.12.0 to 4.12.1 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3127 * build(deps): bump flate2 from 1.1.5 to 1.1.7 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3159 * build(deps): bump jsonschema from 0.37.1 to 0.37.2 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3129 * build(deps): bump jsonschema from 0.37.2 to 0.37.3 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3131 * build(deps): bump jsonschema from 0.37.3 to 0.37.4 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3140 * build(deps): bump log from 0.4.28 to 0.4.29 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3150 * build(deps): bump minijinja from 2.12.0 to 2.13.0 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3142 * build(deps): bump minijinja-contrib from 2.12.0 to 2.13.0 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3141 * build(deps): bump pyo3 from 0.27.1 to 0.27.2 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3137 * build(deps): bump qsv-stats from 0.40.0 to 0.41.0 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3136 * build(deps): bump qsv-stats from 0.41.0 to 0.42.0 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3156 * build(deps): bump qsv-stats from 0.42.0 to 0.43.0 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3169 * build(deps): bump rfd from 0.15.4 to 0.16.0 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3121 * build(deps): bump uuid from 1.18.1 to 1.19.0 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3146 * Improved qsvpy build process for Apple Silicon * Updated GitHub Actions workflows for better reliability * bumped several indirect dependencies * applied select clippy & Codacy suggestions * Improved dependency version management * Better feature flag handling ## Fixed * fix: `apply` panic on empty selection https://github.com/dathere/qsv/pull/3165 * fix: more robust snappy and file extension detection https://github.com/dathere/qsv/pull/3166 * fix: `partition` add proper stdin handling regression introduced when `--limit` option was added https://github.com/dathere/qsv/pull/3161 * Fix broken layout of environment variable documentation by @tmtmtmtm in https://github.com/dathere/qsv/pull/3163 ## Removed * `describegpt`: remove `--jsonl` option https://github.com/dathere/qsv/pull/3167 * chore: remove jemalloc support https://github.com/dathere/qsv/pull/3153 ## New Contributors * @Copilot made their first contribution in https://github.com/dathere/qsv/pull/3162 **Full Changelog**: https://github.com/dathere/qsv/compare/10.0.0...11.0.1

## [11.0.0] - 2025-12-07 qsv 11.0.0 brings significant enhancements to larger-than-memory data processing, AI-powered metadata inferencing, schema validation, and data viewing capabilities, along with important bug fixes and performance improvements. All in preparation for at-scale, interactive, "Data Steward-in-the-Loop" FAIRification in qsv pro. ## 🌟 Major Features ### `stats` & `frequency` Command Enhancements - **Larger than Memory Files**: `stats` & `frequency` can now handle arbitrarily large files, even when "advanced" statistics are enabled with its new dynamic parallel chunk sizing algorithm! - **N Counts**: Added `n_counts` (`n_negative`, `n_zero` and `n_positive`) columns to `stats` output for more detailed count information for numeric fields ### `describegpt` Command Enhancements The `describegpt` command has received substantial improvements for AI-powered metadata inferencing: - **Format Option**: Replaced `--json` flag with `--format` option for more flexible output formatting - Supports multiple output formats - Markdown (default), TSV and JSON - Removed `--jsonl` option for cleaner API - **Controlled Tag Vocabulary**: New tag vocabulary system for consistent categorization - `--tag-vocab` option to specify controlled vocabulary - Lookup support for tag vocabularies - retrieve a tag vocabulary from a local or remote CSV using `http://`, `https://`, `dathere://` and `ckan://` URL schemes. - **Enhanced Boolean Inference**: `--infer-boolean` is now enabled by default for better data type detection - **Performance Metrics**: Added elapsed time tracking to monitor processing duration - **Improved Prompts**: Updated default description prompt with PII/PHI alerts and better attribution metadata ### `schema` & `validate` Command Improvements Enhanced schema inference and validation capabilities: - **Strict Formats**: New `--strict-formats` option for stricter JSON Schema format validation, enforcing JSON Schema format constraints for email, hostname and IP address (IPV4 and IPV6) formats. - **Output Option**: New `--output` option for specifying schema output destination - Polars schema now uses consistent naming conventions across commands - Updated `joinp`, `pivotp`, and `sqlp` commands to use new `.pschema.json` naming convention - **Configurable Email Validation**: `validate` has numerous options to tweak email validation - taking advantage of `schema`'s email format constraint inferencing. ### `sample` Command time-series sampling A new `--timeseries` sampling method with grouping (hourly, daily, weekly), adaptive sampling (prefer business hours or weekends) with various aggregation (mean, sum, min, max) within each interval with configurable starting points (first, last or random). ### `lens` Command Features Enhanced CSV viewing capabilities with csvlens integration: - **Auto-Reload**: New `--auto-reload` option to automatically reload file when it changes - Useful for monitoring live data files - **Streaming stdin**: New `--streaming-stdin` option for real-time data viewing - Supports viewing data as it's being piped in - **Row Marking**: Updated csvlens dependency with row marking feature ### Breaking Changes - `describegpt`: `--json` flag replaced with `--format` option - `describegpt`: `--jsonl` option removed - `schema`, `joinp`, `pivotp`, `sqlp`: Updated Polars schema naming conventions (existing workflows should work but output format may differ slightly) --- ## Added * Created [Event Logo Archive](https://github.com/dathere/qsv/tree/master/docs/images/event-logos) with AI-generated seasonal/version logos * `describegpt`: add controlled vocabulary support for tags https://github.com/dathere/qsv/pull/3122 * `describegpt`: add elapsed time https://github.com/dathere/qsv/pull/3168 * `describegpt`: add lookup support https://github.com/dathere/qsv/pull/3170 * `excel`: add `--cell` option https://github.com/dathere/qsv/pull/3133 * `frequency`: add dynamic parallel chunk sizing https://github.com/dathere/qsv/pull/3135 * `lens`: add `--auto-reload` option https://github.com/dathere/qsv/pull/3128 * `lens`: add `--streaming-stdin` option https://github.com/dathere/qsv/pull/3171 * `sample`: add timeseries sampling options https://github.com/dathere/qsv/pull/3130 * `schema`: infer addl JSON Schema predefined formats - email, ipv4, ipv6, hostname https://github.com/dathere/qsv/pull/3125 * `schema`: add `--output` option and standardize Polars Schema file name https://github.com/dathere/qsv/pull/3126 * `stats`: dynamic parallel chunk sizing with indexed files https://github.com/dathere/qsv/pull/3134 * `stats`: add n_negative, n_zero, n_positive count columns https://github.com/dathere/qsv/pull/3157 * `validate:` add email validation options https://github.com/dathere/qsv/pull/3148 * `tests`: add tests for https://100.dathere.com/lessons/4 by @rzmk in https://github.com/dathere/qsv/pull/3151 * Added Claude AI guidance for contributors * Enhanced `--version` output with more comprehensive system metadata ## Changed * refactor: `describegpt` improve tags inferencing with Tag Vocabulary https://github.com/dathere/qsv/pull/3139 * feat: `describegpt` - major refactor https://github.com/dathere/qsv/pull/3143 * feat: `describegpt` improved Polars SQL processing https://github.com/dathere/qsv/pull/3147 * feat: `describegpt` replace `--json` option with `--format` option supporting 3 formats - markdown, json and TSV; remove `--jsonl` option https://github.com/dathere/qsv/pull/3167 * refactor: `frequency` & `stats` - parallel chunk sizing - allow forcing of cpu based chunking https://github.com/dathere/qsv/pull/3138 * Align partition stdin handling with split/stats pattern by @Copilot in https://github.com/dathere/qsv/pull/3162 * deps: use latest polars upstream with new SQL fixes and features (https://github.com/pola-rs/polars/commit/e1be17f2ccb9dee0d570c6126b54c0e44ae7131d) * deps: latest self_update upstream * build(deps): bump actions/setup-python from 6.0.0 to 6.1.0 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3120 * build(deps): bump actix-web from 4.12.0 to 4.12.1 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3127 * build(deps): bump flate2 from 1.1.5 to 1.1.7 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3159 * build(deps): bump jsonschema from 0.37.1 to 0.37.2 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3129 * build(deps): bump jsonschema from 0.37.2 to 0.37.3 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3131 * build(deps): bump jsonschema from 0.37.3 to 0.37.4 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3140 * build(deps): bump log from 0.4.28 to 0.4.29 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3150 * build(deps): bump minijinja from 2.12.0 to 2.13.0 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3142 * build(deps): bump minijinja-contrib from 2.12.0 to 2.13.0 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3141 * build(deps): bump pyo3 from 0.27.1 to 0.27.2 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3137 * build(deps): bump qsv-stats from 0.40.0 to 0.41.0 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3136 * build(deps): bump qsv-stats from 0.41.0 to 0.42.0 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3156 * build(deps): bump qsv-stats from 0.42.0 to 0.43.0 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3169 * build(deps): bump rfd from 0.15.4 to 0.16.0 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3121 * build(deps): bump uuid from 1.18.1 to 1.19.0 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3146 * Improved qsvpy build process for Apple Silicon * Updated GitHub Actions workflows for better reliability * bumped several indirect dependencies * applied select clippy & Codacy suggestions * Improved dependency version management * Better feature flag handling ## Fixed * fix: `apply` panic on empty selection https://github.com/dathere/qsv/pull/3165 * fix: more robust snappy and file extension detection https://github.com/dathere/qsv/pull/3166 * fix: `partition` add proper stdin handling regression introduced when `--limit` option was added https://github.com/dathere/qsv/pull/3161 * Fix broken layout of environment variable documentation by @tmtmtmtm in https://github.com/dathere/qsv/pull/3163 ## Removed * `describegpt`: remove `--jsonl` option https://github.com/dathere/qsv/pull/3167 * chore: remove jemalloc support https://github.com/dathere/qsv/pull/3153 ## New Contributors * @Copilot made their first contribution in https://github.com/dathere/qsv/pull/3162 **Full Changelog**: https://github.com/dathere/qsv/compare/10.0.0...11.0.0

## [9.1.0] - 2025-11-03 --- ## Added * `frequency`: add `--pretty-json` option https://github.com/dathere/qsv/commit/c67fd061a0cd101b0e04aaab79087c04324b0e46 * `frequency`: add `--rank-strategy` option https://github.com/dathere/qsv/pull/3075 * `frequency`: add `-null-text` option https://github.com/dathere/qsv/pull/3082 ## Changed * `describegpt`: explicitly use `frequency`'s dense rank strategy https://github.com/dathere/qsv/commit/dc3f270000fde3321ae0ad239010471db5ca3cad * `describegpt`: allow `--prompt` to be loaded from a text file https://github.com/dathere/qsv/commit/b11a10c306f0065f1852b23b935c5b04b0e69238 * `describegpt`: use much faster BLAKE3 hash for cache key * `frequency`: change default rank-strategy from min (AKA "1224" ranking) to dense (AKA "1223" ranking) * `lens`: bumped csvlens from 0.13.0 to [0.14.0](https://github.com/YS-L/csvlens/releases/tag/v0.14.0) * `lens`: automatically set to monochrome mode when using `--find` option https://github.com/dathere/qsv/commit/85398690b0ebbc9dea227d13f528c7703451de8b * `luau`: bumped embedded Luau from 0.694 to 0.697 https://github.com/dathere/qsv/commit/3e68e2991757aba2b0597d722b1108fdc8009628 * `stats`: fingerprint hash now uses much-faster, parallelizable BLAKE3 instead of SHA256 * `table`: document that it also creates "aligned TSVs" and Fixed Width Format files https://github.com/dathere/qsv/commit/aaa84b0b22c8cf60361554ddee5213b1d6f8ca49 * tests: change default Python to 3.13 * docs: documented that Extended Input Support (🗄️) does `.zip` auto-decompression * docs: documented Limited Extended Input Support (🗃️) * use latest [qsv-tuned csv crate with performance optimizations](https://github.com/dathere/qsv/blob/aaa84b0b22c8cf60361554ddee5213b1d6f8ca49/Cargo.toml#L304C1-L313C82) * build(deps): bump flate2 from 1.1.4 to 1.1.5 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3071 * build(deps): bump human-panic from 2.0.3 to 2.0.4 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3077 * deps: bump Polars from 0.51.0 at py-1.35.0-beta.1 to 0.52.0 https://github.com/dathere/qsv/commit/618edf0214a5ceb6df38cb61aafbc9e16ab35613 * deps: use latest Polars at time of 9.0.0 release https://github.com/dathere/qsv/commit/c9e934c64af06f71c3a75ec891f895746f2123f9 * build(deps): bump qsv-stats from 0.39.1 to 0.40.0 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3078 * build(deps): bump actions/upload-artifact from 4 to 5 by @dependabot[bot] in https://github.com/dathere/qsv/pull/3074 * applied several clippy lint suggestions * bumped several indirect dependencies * align nightly to 2025-10-24, the same nightly as Polars * bumped MSRV to Rust 1.91 ## Fixed * `describegpt`: add SQL escaping to eliminate SQL injection attack vector; add `.csv` extension to `--sql-output` when Polars SQL query runs successfully https://github.com/dathere/qsv/commit/ad52a35f3c900d4591d30091d10b0a0874c3c254 * `frequency`: fix `--select` option always returning `<ALL_UNIQUE>` https://github.com/dathere/qsv/pull/3082 * publish: fixed some publishing workflows ## Removed * Removed SHA256 and replaced with mush faster, parallelizable BLAKE3 hash https://github.com/dathere/qsv/pull/3072 and https://github.com/dathere/qsv/pull/3080 * publish: removed `maximize-build-space` step in workflows as it was not working as advertised * tests: removed `target-cpu=native` RUSTFLAG in CI tests to avoid intermittent SIGILL (Illegal Instruction) faults **Full Changelog**: https://github.com/dathere/qsv/compare/8.1.1...9.1.0

# 🇮🇹 csv,conf,v9 edition 🍝 &nbsp; | &nbsp; :----|:---- |<img width="410" height="410" alt="csvconfv9-flavor-small" src="https://github.com/user-attachments/assets/8c747193-6f72-4cb4-80f8-2d33740e2512" />|Just in time for [csv,conf,v9](https://csvconf.com/), we're Bologna-bound and will be talking all things qsv, CSV, open data, metadata standards, AI, [POSE](https://civicdataecosystem.org) and [CKAN](https://ckan.org)!<br><br>For this feature release, we polished `describegpt` a bit more for the occassion...<br><br>**[_Towards the "People's API!"! Verso l'API del Popolo!_](https://www.linkedin.com/posts/joelnatividad_towards-the-peoples-api-activity-7369788691717865472-VLGk)**<br>(Answering People/Policymaker Interface)| --- ### 🚀 Enhanced `describegpt` Command * **Configurable Frequency Limits**: Make frequency distribution limit configurable for better control over data analysis * **[Few-shot Learning](https://en.wikipedia.org/wiki/Prompt_engineering#Text-to-text)**: Add `--fewshot-examples` option to improve LLM response quality with contextual examples * **Advanced SQL Generation**: Fine-tuned SQL generation guidance for better date handling and query optimization * **Conditional SQL Results**: Implement conditional `--sql-results` format for more efficient "SQL RAG" processing - i.e. if the generated SQL query executes successfully - the results are saved to the specified file with a `.csv` extension. If a "SQL hallucination" fails, the file is saved with a `.sql` extension instead for the user to tweak and edit. * **TogetherAI Support**: Add support for TogetherAI models endpoint, expanding LLM provider options * **Enhanced Error Handling**: Improved SQL parsing error handling and more informative error messages * **Disk Cache by Default**: The disk cache is now enabled by default for better performance * **TOML Configuration**: Migrate from JSON to more readable TOML format for more easily modifiable prompt files. (see https://github.com/dathere/qsv/blob/master/resources/describegpt_defaults.toml) * **Better Local LLM Support**: `--api-key` can now be set to NONE for local LLM configurations that may not necessarily run on `localhost` (e.g. a shared Local LLM service running on the local network) ### `partition` Command Enhancements * **New `--limit` Option**: Implement `--limit` option to set the maximum number of open files * **Streaming to Enhanced Batching Logic**: Convert from streaming to a simplified, two-pass batched approach designed to partition on columns with high cardinality for very large datasets --- ## Added * `describegpt`: add configurable frequency limit https://github.com/dathere/qsv/pull/2950 * `describegpt`: migrate prompt file from JSON to more easier to edit TOML format https://github.com/dathere/qsv/pull/2954 * `describegpt`: refactor default prompt file; add `--fewshot-examples` option https://github.com/dathere/qsv/pull/2955 * `describegpt`: add TogetherAI support for models endpoint https://github.com/dathere/qsv/pull/2965 * `partition`: add `--limit` option https://github.com/dathere/qsv/pull/2960 * added Windows ARM64 prebuilt binaries ## Changed * `describegpt`: enable disk cache by default https://github.com/dathere/qsv/pull/2951 * `describegpt`: Polars SQL generation tweaks https://github.com/dathere/qsv/pull/2958 * `python`: replace deprecated `with_gil` with `attach` https://github.com/dathere/qsv/pull/2949. This sets the stage for ["free-threaded" Python 3.14](https://docs.python.org/3.14/whatsnew/3.14.html#whatsnew314-pep779) support when its released in October 2025. Buh-bye GIL! * deps: bump embedded Luau from 0.688 to 0.690 https://github.com/dathere/qsv/pull/2967 * deps: bump Polars to 0.50.0 at py-1.33.0 tag * build(deps): bump actions/setup-python from 5.6.0 to 6.0.0 by @dependabot[bot] in https://github.com/dathere/qsv/pull/2962 * build(deps): bump actions/stale from 9 to 10 by @dependabot[bot] in https://github.com/dathere/qsv/pull/2963 * build(deps): bump log from 0.4.27 to 0.4.28 by @dependabot[bot] in https://github.com/dathere/qsv/pull/2961 * build(deps): bump mlua from 0.11.2 to 0.11.3 by @dependabot[bot] in https://github.com/dathere/qsv/pull/2948 * build(deps): bump pyo3 from 0.25.1 to 0.26.0 by @dependabot[bot] in https://github.com/dathere/qsv/pull/2946 * build(deps): bump uuid from 1.18.0 to 1.18.1 by @dependabot[bot] in https://github.com/dathere/qsv/pull/2956 * build(deps): bump zip from 4.5.0 to 4.6.0 by @dependabot[bot] in https://github.com/dathere/qsv/pull/2952 * applied select clippy lints * updated indirect dependencies **Full Changelog**: https://github.com/dathere/qsv/compare/7.0.1...7.1.0

# 🇮🇹 csv,conf,v9 edition 🍝 &nbsp; | &nbsp; :----|:---- |<img width="410" height="410" alt="csvconfv9-flavor-small" src="https://github.com/user-attachments/assets/8c747193-6f72-4cb4-80f8-2d33740e2512" />|Just in time for [csv,conf,v9](https://csvconf.com/), we're Bologna-bound and will be talking all things qsv, CSV, open data, metadata standards, AI, [POSE](https://civicdataecosystem.org) and [CKAN](https://ckan.org)!<br><br>For this feature release, we polished `describegpt` a bit more for the occassion...<br><br>**[_Towards the "People's API!"! Verso l'API del Popolo!_](https://www.linkedin.com/posts/joelnatividad_towards-the-peoples-api-activity-7369788691717865472-VLGk)**<br>(Answering People/Policymaker Interface)| --- ### 🚀 Enhanced `describegpt` Command * **Configurable Frequency Limits**: Make frequency distribution limit configurable for better control over data analysis * **[Few-shot Learning](https://en.wikipedia.org/wiki/Prompt_engineering#Text-to-text)**: Add `--fewshot-examples` option to improve LLM response quality with contextual examples * **Advanced SQL Generation**: Fine-tuned SQL generation guidance for better date handling and query optimization * **Conditional SQL Results**: Implement conditional `--sql-results` format for more efficient "SQL RAG" processing - i.e. if the generated SQL query executes successfully - the results are saved to the specified file with a `.csv` extension. If a "SQL hallucination" fails, the file is saved with a `.sql` extension instead for the user to tweak and edit. * **TogetherAI Support**: Add support for TogetherAI models endpoint, expanding LLM provider options * **Enhanced Error Handling**: Improved SQL parsing error handling and more informative error messages * **Disk Cache by Default**: The disk cache is now enabled by default for better performance * **TOML Configuration**: Migrate from JSON to more readable TOML format for more easily modifiable prompt files. (see https://github.com/dathere/qsv/blob/master/resources/describegpt_defaults.toml) * **Better Local LLM Support**: `--api-key` can now be set to NONE for local LLM configurations that may not necessarily run on `localhost` (e.g. a shared Local LLM service running on the local network) ### `partition` Command Enhancements * **New `--limit` Option**: Implement `--limit` option to set the maximum number of open files * **Streaming to Enhanced Batching Logic**: Convert from streaming to a simplified, two-pass batched approach designed to partition on columns with high cardinality for very large datasets --- ## Added * `describegpt`: add configurable frequency limit https://github.com/dathere/qsv/pull/2950 * `describegpt`: migrate prompt file from JSON to more easier to edit TOML format https://github.com/dathere/qsv/pull/2954 * `describegpt`: refactor default prompt file; add `--fewshot-examples` option https://github.com/dathere/qsv/pull/2955 * `describegpt`: add TogetherAI support for models endpoint https://github.com/dathere/qsv/pull/2965 * `partition`: add `--limit` option https://github.com/dathere/qsv/pull/2960 * added Windows ARM64 prebuilt binaries ## Changed * `describegpt`: enable disk cache by default https://github.com/dathere/qsv/pull/2951 * `describegpt`: Polars SQL generation tweaks https://github.com/dathere/qsv/pull/2958 * `python`: replace deprecated `with_gil` with `attach` https://github.com/dathere/qsv/pull/2949. This sets the stage for ["free-threaded" Python 3.14](https://docs.python.org/3.14/whatsnew/3.14.html#whatsnew314-pep779) support when its released in October 2025. Buh-bye GIL! * deps: bump embedded Luau from 0.688 to 0.690 https://github.com/dathere/qsv/pull/2967 * deps: bump Polars to 0.50.0 at py-1.33.0 tag * build(deps): bump actions/setup-python from 5.6.0 to 6.0.0 by @dependabot[bot] in https://github.com/dathere/qsv/pull/2962 * build(deps): bump actions/stale from 9 to 10 by @dependabot[bot] in https://github.com/dathere/qsv/pull/2963 * build(deps): bump log from 0.4.27 to 0.4.28 by @dependabot[bot] in https://github.com/dathere/qsv/pull/2961 * build(deps): bump mlua from 0.11.2 to 0.11.3 by @dependabot[bot] in https://github.com/dathere/qsv/pull/2948 * build(deps): bump pyo3 from 0.25.1 to 0.26.0 by @dependabot[bot] in https://github.com/dathere/qsv/pull/2946 * build(deps): bump uuid from 1.18.0 to 1.18.1 by @dependabot[bot] in https://github.com/dathere/qsv/pull/2956 * build(deps): bump zip from 4.5.0 to 4.6.0 by @dependabot[bot] in https://github.com/dathere/qsv/pull/2952 * applied select clippy lints * updated indirect dependencies **Full Changelog**: https://github.com/dathere/qsv/compare/7.0.1...7.1.0

## [7.0.1] - 2025-08-28 A patch release with some minor bug fixes, benchmark tweaks and build system improvements. ## Added * publish: add dedicated powerpc64le-unknown-linux-gnu publishing workflow (WIP) ## Changed * docs: `describegpt` expanded error message about LLM URL or API key * deps: remove planus pinned dependency ## Fixed * fix: `geocode` `--batch 0` causes panic when polars feature is enabled * publish: remove luau feature from x86_64-pc-windows builds that was causing builds to fail * publish: remove powerpc64le from main publish workflow * benchmarks: updated to v6.8.0 with fixes to luau and clustered sample benchmarks **Full Changelog**: https://github.com/dathere/qsv/compare/7.0.0...7.0.1