# describegpt

> Infer a ["neuro-symbolic"](https://en.wikipedia.org/wiki/Neuro-symbolic_AI) Data Dictionary, Description & Tags or ask questions about a CSV with a [configurable, MiniJinja prompt file](../../resources/describegpt_defaults.toml), using any [OpenAI API](https://platform.openai.com/docs/introduction)-compatible cloud/local LLM. (e.g. [Markdown](../describegpt/nyc311-describegpt.md), [JSON](../describegpt/nyc311-describegpt.json), [TOON](../describegpt/nyc311-describegpt.toon), [JSON Schema](../describegpt/nyc311-describegpt-schema.json), [Semantic Markdown](../describegpt/nyc311-describegpt-semanticmd.md), [OKF](../describegpt/nyc311-describegpt-okf.md), [Everything](../describegpt/nyc311-describegpt-everything.md), [Content Type inferencing](../describegpt/property-alleghenney.md), [Spanish](../describegpt/nyc311-describegpt-spanish.md), [Mandarin](../describegpt/nyc311-describegpt-mandarin.md), [Controlled Tags](../describegpt/nyc311-describegpt-tagvocab.md); [--prompt "What are the top 10 complaint types by community board & borough by year?"](../describegpt/nyc311-describegpt-prompt.md) - [deterministic, hallucination-free SQL RAG result](../describegpt/nyc311-describegpt-prompt.csv); [iterative, session-based SQL RAG refinement](../describegpt/allegheny_discussion3.md) - [refined SQL RAG result](../describegpt/mostexpensive6.csv))

**[Table of Contents](TableOfContents.md)** | **Source: [src/cmd/describegpt.rs](https://github.com/dathere/qsv/blob/master/src/cmd/describegpt.rs)** | [📇](TableOfContents.md#legend "uses an index when available.")[🗃️](TableOfContents.md#legend "Limited Extended input support.")[🤖](TableOfContents.md#legend "command uses Natural Language Processing or Generative AI.")[🌐](TableOfContents.md#legend "has web-aware options.")[🪄](TableOfContents.md#legend "\"automagical\" commands that uses stats and/or frequency tables to work \"smarter\" & \"faster\".")[📚](TableOfContents.md#legend "has lookup table support, enabling runtime \"lookups\" against local or remote reference CSVs.")[⛩️](TableOfContents.md#legend "uses MiniJinja template engine.") [![CKAN](../images/ckan.png)](TableOfContents.md#legend "has CKAN-aware integration options.")

<a name="nav"></a>
[Description](#description) | [Examples](#examples) | [Usage](#usage) | [Data Analysis/Inferencing Options](#data-analysis/inferencing-options) | [Dictionary Options](#dictionary-options) | [Tag Options](#tag-options) | [Stats/Frequency Options](#stats/frequency-options) | [Custom Prompt Options](#custom-prompt-options) | [LLM API Options](#llm-api-options) | [Caching Options](#caching-options) | [MCP Sampling Options](#mcp-sampling-options) | [Common Options](#common-options)

<a name="description"></a>

## Description [↩](#nav)

Create a "neuro-symbolic" Data Dictionary and/or infer Description & Tags about a Dataset
using an OpenAI API-compatible Large Language Model (LLM).

It does this by compiling Summary Statistics & a Frequency Distribution of the Dataset,
and then prompting the LLM with detailed, configurable, MiniJinja-templated prompts with
these extended statistical context.

The Data Dictionary is "neuro-symbolic" as it uses a hybrid approach. It's primarily populated
deterministically using Summary Statistics & Frequency Distribution, and only the human-friendly
Label and Description (plus Content Type when --infer-content-type is set) are populated by the
"neural network" LLM using the same statistical context.

CHAT MODE:  
You can also use the --prompt option to ask a natural language question about the Dataset.

If the question can be answered by solely using the Dataset's Summary Statistics and
Frequency Distribution data, the LLM will return the answer directly.

CHAT SQL RETRIEVAL-AUGMENTED GENERATION (RAG) SUB-MODE:  
If the question cannot be answered using the Dataset's Summary Statistics & Frequency Distribution,
it will first create a Data Dictionary and a small random sample (default: 100 rows) of the Dataset
and provide it to the LLM as additional context to help it generate a SQL query that DETERMINISTICALLY
answers the natural language question.

Two SQL dialects are currently supported - DuckDB (highly recommended) & Polars. If the
QSV_DUCKDB_PATH environment variable is set to the absolute path of the DuckDB binary,
DuckDB will be used to answer the question. Otherwise, if the "polars" feature is enabled,
Polars SQL will be used.

If neither DuckDB nor Polars is available, the SQL query will be returned in a Markdown code block,
along with the reasoning behind the query.

Even in "SQL RAG" mode, though the SQL query is guaranteed to be deterministic, the query itself
may not be correct. In the event of a SQL query execution failure, run the same --prompt with
the --fresh option to request the LLM to generate a new SQL query.

When using DuckDB, all loaded DuckDB extensions will be sent as additional context to the LLM to let
it know what functions (even UDFs!) it can use in the SQL queries it generates. If you want a
specific function or technique to be used in the SQL query, mention it in the prompt.

SUPPORTED MODELS & LLM PROVIDERS:  
OpenAI's open-weights gpt-oss model (both 20b and 120b variants) was used during development &
is recommended for most use cases.
It was also tested with OpenAI, TogetherAI, OpenRouter and Google Gemini cloud providers.
For Gemini, use the base URL "<https://generativelanguage.googleapis.com/v1beta/openai">.
Local LLMs tested include Ollama, Jan and LM Studio.

> [!NOTE]
> LLMs are prone to inaccurate information being produced. Verify output results before using them.

CACHING:  
As LLM inferencing takes time and can be expensive, describegpt caches the LLM inferencing results
in a either a disk cache (default) or a Redis cache. It does so by calculating the BLAKE3 hash of the
input file and using it as the primary cache key along with the prompt type, model and every flag that
influences the rendered prompt (including prompt-file, context-file, language, tag-vocab, num-tags,
enum-threshold, infer-content-type, sample-size, fewshot-examples, the QSV_DUCKDB_PATH toggle and the
generated Data Dictionary), so changing any of them produces a fresh LLM call rather than stale cached output.

The default disk cache is stored in the ~/.qsv-cache/describegpt directory with a default TTL of 28 days
and cache hits NOT refreshing an existing cached value's TTL.
Adjust the QSV_DISKCACHE_TTL_SECS & QSV_DISKCACHE_TTL_REFRESH env vars to change disk cache settings.
A QSV_DISKCACHE_TTL_SECS of 0 disables time-based expiration (entries are cached indefinitely).

Alternatively a Redis cache can be used instead of the disk cache. This is especially useful if you want
to share the cache across the network with other users or computers.
The Redis cache is stored in database 3 by default with a TTL of 28 days and cache hits NOT refreshing
an existing cached value's TTL. Adjust the QSV_DG_REDIS_CONNSTR, QSV_REDIS_MAX_POOL_SIZE,
QSV_REDIS_TTL_SECS & QSV_REDIS_TTL_REFRESH env vars to change Redis cache settings.
A QSV_REDIS_TTL_SECS of 0 disables expiration (entries are cached indefinitely).


<a name="examples"></a>

## Examples [↩](#nav)

> Generate a Data Dictionary, Description & Tags of data.csv using the bundled defaults
> (a local LLM on <http://localhost:1234/v1> running the google/gemma-4-26b-a4b model)

```console
qsv describegpt data.csv --all
```

> Same, but against OpenAI's cloud API instead of a local LLM
> (replace <API_KEY> with your OpenAI API key)

```console
qsv describegpt data.csv --base-url https://api.openai.com/v1 --api-key <API_KEY> --model openai/gpt-oss-20b --all
```

> Generate a Data Dictionary of data.csv using the DeepSeek R1:14b model on a local Ollama instance

```console
qsv describegpt data.csv -u http://localhost:11434/v1 --model deepseek-r1:14b --dictionary
```

> Generate a Data Dictionary that also infers a semantic Content Type for each field
> (e.g. email, city, latitude) so the dictionary can later drive synthetic data generation

```console
qsv describegpt data.csv --dictionary --infer-content-type
```

> Ask questions about the sample NYC 311 dataset using LM Studio with the default gemma-4-26b-a4b model.
> Questions that can be answered using the Summary Statistics & Frequency Distribution of the dataset.

```console
qsv describegpt NYC_311.csv --prompt "What is the most common complaint?"
```

> Ask detailed natural language questions that require SQL queries and auto-invoke SQL RAG mode
> Generate a DuckDB SQL query to answer the question

```console
QSV_DUCKDB_PATH=/path/to/duckdb \
qsv describegpt NYC_311.csv -p "What's the breakdown of complaint types by borough descending order?"
```

> Prompt requires a natural language query. Convert query to SQL using the LLM and save results to
> a file with the --sql-results option.  If generated SQL query runs successfully,
> the file is "results.csv". Otherwise, it is "results.sql".

```console
qsv describegpt NYC_311.csv -p "Aggregate complaint types by community board" --sql-results results
```

> Cache Dictionary, Description & Tags inference results using the Redis cache instead of the disk cache

```console
qsv describegpt data.csv --all --redis-cache
```

> Get fresh Description & Tags inference results from the LLM and refresh disk cache entries for both

```console
qsv describegpt data.csv --description --tags --fresh
```

> Get fresh inference results from the LLM and refresh the Redis cache entries for all three

```console
qsv describegpt data.csv --all --redis-cache --fresh
```

> Forget a cached response for data.csv's data dictionary if it exists and then exit

```console
qsv describegpt data.csv --dictionary --forget
```

> Flush/Remove ALL cached entries in the disk cache

```console
qsv describegpt --flush-cache
```

> Flush/Remove ALL cached entries in the Redis cache

```console
qsv describegpt --redis-cache --flush-cache
```

> Generate Data Dictionary but exclude ID columns from frequency analysis to reduce overhead

```console
qsv describegpt data.csv --dictionary --freq-options "--select '!id,!uuid' --limit 20"
```

> Generate Data Dictionary, Description & Tags but reduce frequency context
> by showing only top 5 values per field

```console
qsv describegpt data.csv --all --freq-options "--limit 5"
```

> Generate Description using weighted frequencies with ascending sort

```console
qsv describegpt data.csv --description --freq-options "--limit 50 --asc --weight count_column"
```

> Generate a Data Dictionary, Description & Tags using a previously compiled stats CSV file and
> frequency CSV file instead of running the stats and frequency commands

```console
qsv describegpt data.csv --all --stats-options "file:my_stats.csv" --freq-options "file:my_freq.csv"
```

For more examples, see [tests](https://github.com/dathere/qsv/blob/master/tests/test_describegpt.rs).

For more detailed info on how describegpt works and how to prepare a prompt file,
see <https://github.com/dathere/qsv/blob/master/docs/Describegpt.md> and
<https://github.com/dathere/qsv/wiki/AI-and-Documentation#describegpt>

<a name="usage"></a>

## Usage [↩](#nav)

```console
qsv describegpt [options] [<input>]
qsv describegpt --prepare-context [options] [<input>]
qsv describegpt --process-response [options]
qsv describegpt (--redis-cache) (--flush-cache)
qsv describegpt --help
```

<a name="data-analysis/inferencing-options"></a>

## Data Analysis/Inferencing Options [↩](#nav)

| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Option&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | Type | Description | Default |
|--------|------|-------------|--------|
| &nbsp;`‑‑dictionary`&nbsp; | flag | Create a Data Dictionary using a hybrid "neuro-symbolic" pipeline - i.e. the Dictionary is populated deterministically using Summary Statistics and Frequency Distribution data, and only the human-friendly Label and Description (and Content Type when --infer-content-type is set) are populated by the LLM using the same statistical context. The dataset's content language is also auto-detected locally (using whatlang, not the LLM) from sampled values of its text columns, and emitted as detected_language, detected_language_code (ISO 639-3) and detected_language_confidence in the JSON dictionary (top-level) and JSON Schema ("x-qsv" object) outputs when detection confidence >= 80% (tunable with a float value in the --language option); the fields are omitted when detection is not confident. When detection is confident, the LLM is also asked to generate its output (Labels, Descriptions, Tags and the dataset Description) in the detected language, unless an explicit language is given with the --language option. |  |
| &nbsp;`‑‑description`&nbsp; | flag | Infer a general Description of the dataset based on detailed statistical context. An Attribution signature is embedded in the Description. |  |
| &nbsp;`‑‑tags`&nbsp; | flag | Infer Tags that categorize the dataset based on detailed statistical context. Useful for grouping datasets and filtering. |  |
| &nbsp;`‑A,`<br>`‑‑all`&nbsp; | flag | Shortcut for --dictionary --description --tags. |  |

<a name="dictionary-options"></a>

## Dictionary Options [↩](#nav)

| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Option&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | Type | Description | Default |
|--------|------|-------------|--------|
| &nbsp;`‑‑num‑examples`&nbsp; | integer | The number of Example values to include in the dictionary. | `5` |
| &nbsp;`‑‑truncate‑str`&nbsp; | integer | The maximum length of an Example value in the dictionary. An ellipsis is appended to the truncated value. If zero, no truncation is performed. Bounds the prompt sent to the LLM, and applies to the csv/json/semantic-md dictionary output. The jsonschema output is machine-facing, so it always carries the full, untruncated Example values. | `25` |
| &nbsp;`‑‑infer‑content‑type`&nbsp; | flag | Also have the LLM classify each field's semantic "Content Type", mapped to a curated, documented vocabulary (e.g. email, city, category, name, credit card, etc.) see <https://github.com/dathere/qsv/blob/master/src/cmd/synthesize/faker_map.rs>. Adds a "Content Type" column/field to the Data Dictionary output. Fields where cardinality equals the row count (i.e. every row has a distinct non-null value - primary keys, surrogate keys, sequence numbers) are deterministically classified as "unique_id", overriding any token the LLM returned for that field. For Date/DateTime fields, the LLM also infers the column's strftime date format (e.g. "date:%m/%d/%Y"); the Markdown, JSON & JSON Schema dictionaries then render Min/Max AND Examples in that inferred format so they match how the dates actually appear in the data, instead of qsv's normalized form. (TSV output keeps Min/Max & Examples in qsv's raw normalized form.) For a CONTINUOUS numeric measure on a canonical scale (a percentage, a 0-1 ratio/probability, a bounded index), the LLM also proposes an "x-qsv.gauge_range" [min, max]; qsv keeps it only when the field is a numeric measure AND the observed data lies within it, so that a "viz smart" dictionary-driven Data Schematic draws that KPI tile as a GAUGE. (A KPI "vs target" delta uses "x-qsv.target", which is a GOAL you hand-author - never inferred.) The LLM also infers dataset-level INTER-COLUMN RELATIONSHIPS, emitted as a "relationships" array (top-level in the JSON dictionary, and in the dataset-level "x-qsv" object of the JSON Schema one). Each entry has a "kind" of "joint", "ordered", "correlated" or "pipeline", plus a "members" list of the field names involved. A "pipeline" is a process whose stages narrow monotonically - planned to committed to spent, impressions to clicks to conversions. Its stages may be separate COLUMNS (listed in "members", upstream first - the opposite direction from "ordered", which ascends) or the VALUES of one category column (via "stage_column", an ordered "stages" list, and an optional "value_column" to sum). "qsv viz smart --dictionary" reads these to draw its pipeline funnel panel; edit them in the saved schema to correct or add one. |  |
| &nbsp;`‑‑infer‑null‑values`&nbsp; | flag | Also have the LLM propose each field's null sentinels - literal values that stand in for "missing" (e.g. NULL, N/A, -999, 9999-12-31). Emitted into the JSON Schema dictionary's per-property "x-qsv" object, split into two lists that carry different warranties.<ul><li>"null_values" - proposed by the LLM AND independently confirmed by qsv to occur as a literal value in that String column. Every observed casing is listed.</li><li>"null_candidates" - proposed by the LLM but NOT confirmable by any scan; numeric/date placeholders that parse as valid values of the column's own type (-999 is a legal integer), or tokens never seen in the data. Each carries "confirm_required": true.</li></ul> A proposal that is IMPOSSIBLE is dropped rather than reported: a non-numeric token in a numeric column (no cell could hold it, or the column would not have typed as numeric), and an echo of the null placeholder the Frequency Distribution displays for cells qsv already counts as empty. A confirmed "null_values" entry means only that the literal is PRESENT in the column; the judgment that it MEANS "missing" remains the LLM's. This complements "qsv denull", which reports a narrower set - only the columns that would promote to a numeric type once their sentinels are blanked. A purely categorical column (status = ok/pending/NULL) is outside denull's remit. Sentinels are REPORTED, never applied - qsv does not modify your data on an LLM's say-so. Only "qsv denull --apply" masks values, and only in the columns it independently confirmed. Only rendered by "--format JSONSchema"; the other formats ignore them. |  |
| &nbsp;`‑‑two‑pass`&nbsp; | flag | Run a second LLM call that takes the full first-pass Data Dictionary as JSON context and refines each field's Label, Description and (when --infer-content-type is set) Content Type using cross-field awareness. The LLM can then relate fields that belong together (e.g. street_no + street_name + city + state + zip describing a single mailing address; first_name + last_name naming a single person; lat + lng forming a coordinate pair). The refined dictionary becomes the emitted output and is also what downstream Description, Tags and Prompt inference phases see as dictionary context. Roughly doubles dictionary LLM cost and latency, so opt-in. Most useful when combined with --infer-content-type. Allowed with the --dictionary, --all and --prompt inference flags. Mutually exclusive with --prepare-context and --process-response (MCP sampling is single-turn per inference phase). |  |
| &nbsp;`‑‑addl‑cols`&nbsp; | flag | Add additional columns to the dictionary from the Summary Statistics. |  |
| &nbsp;`‑‑addl‑cols‑list`&nbsp; | string | A comma-separated list of additional stats columns to add to the dictionary. The columns must be present in the Summary Statistics. If the columns are not present in the Summary Statistics or already in the dictionary, they will be ignored. These values are case-insensitive and automatically set the --addl-cols option to true. "everything" can be used to add all 45 "available" statistics columns. You can adjust the available columns with --stats-options. "everything!" automatically sets --stats-options to compute "all" 51 supported stats. The 6 addl cols are the mode/s & antimode/s stats with each having counts & occurrences. "moar" gets you even moar stats, with detailed outliers info. "moar!" gets you even moar with --advanced stats (Kurtosis, Gini Coefficient & Shannon Entropy) | `sort_order, sortiness, mean, median, mad, stddev, variance, cv` |

<a name="tag-options"></a>

## Tag Options [↩](#nav)

| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Option&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | Type | Description | Default |
|--------|------|-------------|--------|
| &nbsp;`‑‑num‑tags`&nbsp; | integer | The maximum number of tags to infer when the --tags option is used. Maximum allowed value is 50. | `10` |
| &nbsp;`‑‑tag‑vocab`&nbsp; | string | The CSV file containing the tag vocabulary to use for inferring tags. If no tag vocabulary file is provided, the model will use free-form tags. Supports local files, remote URLs (http/https), CKAN resources (ckan://), and dathere:// scheme. Remote resources are cached locally. The CSV file must have two columns with headers: first column is the tag, second column is the description. Note that qsvlite only supports local files. |  |
| &nbsp;`‑‑cache‑dir`&nbsp; | string | The directory to use for caching downloaded tag vocabulary resources. If the directory does not exist, qsv will attempt to create it. If the QSV_CACHE_DIR envvar is set, it will be used instead. | `~/.qsv-cache` |
| &nbsp;`‑‑ckan‑api`&nbsp; | string | The URL of the CKAN API to use for downloading tag vocabulary resources with the "ckan://" scheme. If the QSV_CKAN_API envvar is set, it will be used instead. | `https://data.dathere.com/api/3/action` |
| &nbsp;`‑‑ckan‑token`&nbsp; | string | The CKAN API token to use. Only required if downloading private resources. If the QSV_CKAN_TOKEN envvar is set, it will be used instead. |  |

<a name="stats/frequency-options"></a>

## Stats/Frequency Options [↩](#nav)

| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Option&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | Type | Description | Default |
|--------|------|-------------|--------|
| &nbsp;`‑‑stats‑options`&nbsp; | string | Options for the stats command used to generate summary statistics. If it starts with "file:" prefix, the statistics are read from the specified CSV file instead of running the stats command. e.g. "file:my_custom_stats.csv" | `--infer-dates --infer-boolean --mad --quartiles --percentiles --force --stats-jsonl` |
| &nbsp;`‑‑freq‑options`&nbsp; | string | Options for the frequency command used to generate frequency distributions. You can use this to exclude certain variable types from frequency analysis (e.g., --select '!id,!uuid'), limit results differently per use case, or control output format. If --limit is specified here, it takes precedence over --enum-threshold. If it starts with "file:" prefix, the frequency data is read from the specified CSV file instead of running the frequency command. e.g. "file:my_custom_frequency.csv" A "file:"-backed CSV is assumed to use frequency's default "(NULL)" null text; a custom --null-text in a file-supplied CSV is not recognized when validating inferred date/datetime formats. | `--rank-strategy dense` |
| &nbsp;`‑‑enum‑threshold`&nbsp; | integer | The threshold for compiling Enumerations with the frequency command before bucketing other unique values into the "Other" category. This is a convenience shortcut for --freq-options --limit <n>. If --freq-options contains --limit, this flag is ignored. | `10` |

<a name="custom-prompt-options"></a>

## Custom Prompt Options [↩](#nav)

| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Option&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | Type | Description | Default |
|--------|------|-------------|--------|
| &nbsp;`‑p,`<br>`‑‑prompt`&nbsp; | string | Custom prompt to answer questions about the dataset. The prompt will be answered based on the dataset's Summary Statistics, Frequency data & Data Dictionary. If the prompt CANNOT be answered by looking at these metadata, a SQL query will be generated to answer the question. If the "polars" or the "QSV_DUCKDB_PATH" environment variable is set & the `--sql-results` option is used, the SQL query will be automatically executed and its results returned. Otherwise, the SQL query will be returned along with the reasoning behind it. If it starts with "file:" prefix, the prompt is read from the file specified. e.g. "file:my_long_prompt.txt" |  |
| &nbsp;`‑‑sql‑results`&nbsp; | string | The file to save the SQL query results to. Only valid if the --prompt option is used & the "polars" or the "QSV_DUCKDB_PATH" environment variable is set. If the SQL query executes successfully, the results will be saved with a ".csv" extension. Otherwise, it will be saved with a ".sql" extension so the user can inspect why it failed and modify it. |  |
| &nbsp;`‑‑prompt‑file`&nbsp; | string | The configurable TOML file containing prompts to use for inferencing. If no file is provided, default prompts will be used. The prompt file uses the MiniJinja template engine (<https://docs.rs/minijinja>) See <https://github.com/dathere/qsv/blob/master/resources/describegpt_defaults.toml> |  |
| &nbsp;`‑‑context‑file`&nbsp; | string | Path to a file with additional context about the dataset - e.g. variable/code labels, provenance & domain notes - injected into the prompts as the {{ context }} MiniJinja variable. The file TYPE is sniffed from its contents (not its extension). Supported types: plain text, Markdown, CSV, Excel/ODS spreadsheets (extracted to CSV), and PDF or image files (JPEG/PNG/WebP/GIF) sent to the LLM as a multimodal attachment (needs a multimodal model & endpoint; max ~32 MB). Word/PowerPoint (docx/pptx) are NOT supported - convert to PDF or text. By default qsv injects this context into the USER message; custom prompt file templates may reference {{ context }} anywhere to place it instead. If the option is unset or the file is empty, {{ context }} renders as an empty string and the prompts are unaffected. The file's contents are part of the cache key, so editing it produces a fresh LLM call. |  |
| &nbsp;`‑‑markdown‑template`&nbsp; | string | TOML file with MiniJinja templates for Markdown output. The TOML contains four wrapper templates - one per inference kind: dictionary_md_template, description_md_template, tags_md_template and custom_prompt_md_template - plus a dictionary_md_body_template that drives the per-field dictionary table that fills the dictionary wrapper's {{ llm_response }}. All template fields are optional; any omitted field falls back to the embedded default, so a minimal TOML can override just the templates you want to change. Custom MiniJinja filters (pipe_escape, br_replace, human_count, dict_cell, humanize_examples) and template variables are documented inline in the default TOML referenced below. If no file is provided, built-in defaults are used (matching legacy output). See <https://github.com/dathere/qsv/blob/master/resources/describegpt_md_defaults.toml> |  |
| &nbsp;`‑‑sample‑size`&nbsp; | integer | The number of rows to randomly sample from the input file for the sample data. Uses the INDEXED sampling method with the qsv sample command. | `100` |
| &nbsp;`‑‑fewshot‑examples`&nbsp; | flag | By default, few-shot examples are NOT included in the LLM prompt when generating SQL queries. When this option is set, few-shot examples in the default prompt file are included. Though this will increase the quality of the generated SQL, it comes at a cost - increased LLM API call cost in terms of tokens and execution time. See <https://en.wikipedia.org/wiki/Prompt_engineering> for more info. |  |
| &nbsp;`‑‑session`&nbsp; | string | Enable stateful session mode for iterative SQL RAG refinement. The session name is the file path of the markdown file where session messages will be stored. When used with --prompt, subsequent queries in the same session will refine the baseline SQL query. SQL query results (10-row sample) and errors are automatically included in subsequent messages for context. |  |
| &nbsp;`‑‑session‑len`&nbsp; | integer | Maximum number of recent messages to keep in session context before summarizing older messages. Only used when --session is specified. | `10` |
| &nbsp;`‑‑no‑score‑sql`&nbsp; | flag | Disable scoresql validation of generated SQL queries before execution. By default, when --prompt generates a SQL query and --sql-results is set, the query is scored and iteratively improved if below threshold. |  |
| &nbsp;`‑‑score‑threshold`&nbsp; | integer | Minimum scoresql score for a SQL query to be accepted. Typical range is 0-100; values >100 will always trigger retries and the below-threshold warning. | `50` |
| &nbsp;`‑‑score‑max‑retries`&nbsp; | integer | Max LLM re-prompts to improve a low-scoring SQL query. | `3` |

<a name="llm-api-options"></a>

## LLM API Options [↩](#nav)

| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Option&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | Type | Description | Default |
|--------|------|-------------|--------|
| &nbsp;`‑u,`<br>`‑‑base‑url`&nbsp; | string | The LLM API URL. Supports APIs & local LLMs compatible with the OpenAI API specification. Some common base URLs: OpenAI: <https://api.openai.com/v1> Gemini: <https://generativelanguage.googleapis.com/v1beta/openai> TogetherAI: <https://api.together.ai/v1> |  |
| &nbsp;`‑m,`<br>`‑‑model`&nbsp; | string | The model to use for inferencing. This model must be compatible with OpenAI API spec. Works with both cloud LLM providers and local LLMs. Tested open weights models include OpenAI's gpt-oss-20b and gpt-oss-120b; Google's Gemma family of open models; and Mistral's Magistral reasoning models. Precedence: explicit CLI flag > QSV_LLM_MODEL env var > prompt file model > built-in default (google/gemma-4-26b-a4b). No docopt default — same rationale as --base-url above. |  |
| &nbsp;`‑‑language`&nbsp; | string | The output language/dialect/tone to use for the response. (e.g., "Spanish", "French", "Hindi", "Mandarin", "Italian", "Castilian", "Franglais", "Taglish", "Pig Latin", "Valley Girl", "Pirate", "Shakespearean English", "Chavacano", "Gen Z", "Yoda", etc.) If set to a string, that language/dialect is ALWAYS used for the response, overriding auto-detection. If set to a float (0.0 to 1.0), it specifies the auto-detection confidence threshold. If not set, auto-detection runs with an 80% confidence threshold. |  |
| &nbsp;`‑‑addl‑props`&nbsp; | string | Additional model properties to pass to the LLM chat/completion API. Various models support different properties beyond the standard ones. For instance, gpt-oss-20b supports the "reasoning_effort" property. e.g. to set the "reasoning_effort" property to "high" & "temperature" to 0.5, use '{"reasoning_effort": "high", "temperature": 0.5}' |  |
| &nbsp;`‑k,`<br>`‑‑api‑key`&nbsp; | string | The API key to use. If set, takes precedence over the QSV_LLM_APIKEY envvar. Required when the base URL is not localhost. Set to NONE to suppress sending the API key. |  |
| &nbsp;`‑t,`<br>`‑‑max‑tokens`&nbsp; | integer | Limits the number of generated tokens in the output. Set to 0 to disable token limits. If the --base-url is localhost, indicating a local LLM, the default is automatically set to 0. | `10000` |
| &nbsp;`‑‑timeout`&nbsp; | integer | Timeout for completions in seconds. If 0, no timeout is used. Defaults to 300 if not set. If the --base-url is localhost, indicating a local LLM, the timeout is automatically disabled unless you set --timeout (or the QSV_TIMEOUT envvar) explicitly. |  |
| &nbsp;`‑‑user‑agent`&nbsp; | string | Specify custom user agent. It supports the following variables - $QSV_VERSION, $QSV_TARGET, $QSV_BIN_NAME, $QSV_KIND and $QSV_COMMAND. Try to follow the syntax here - <https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent> |  |
| &nbsp;`‑‑export‑prompt`&nbsp; | string | Export the default prompts to the specified file that can be used with the --prompt-file option. The file will be saved with a .toml extension. If the file already exists, it will be overwritten. It will exit after exporting the prompts. |  |

<a name="caching-options"></a>

## Caching Options [↩](#nav)

| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Option&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | Type | Description | Default |
|--------|------|-------------|--------|
| &nbsp;`‑‑no‑cache`&nbsp; | flag | Disable default disk cache. |  |
| &nbsp;`‑‑disk‑cache‑dir`&nbsp; | string | The directory to store the disk cache. Note that if the directory does not exist, it will be created. If the directory exists, it will be used as is, and will not be flushed. This option allows you to maintain several disk caches for different describegpt jobs (e.g. one for a data portal, another for internal data exchange). | `~/.qsv-cache/describegpt` |
| &nbsp;`‑‑redis‑cache`&nbsp; | flag | Use Redis instead of the default disk cache to cache LLM completions. It connects to "redis://127.0.0.1:6379/3" by default, with a connection pool size of 20, with a TTL of 28 days, and cache hits NOT refreshing an existing cached value's TTL. This option automatically disables the disk cache. |  |
| &nbsp;`‑‑fresh`&nbsp; | flag | Send a fresh request to the LLM API, refreshing a cached response if it exists. When a --prompt SQL query fails, you can also use this option to request the LLM to generate a new SQL query. |  |
| &nbsp;`‑‑forget`&nbsp; | flag | Remove a cached response if it exists and then exit. |  |
| &nbsp;`‑‑flush‑cache`&nbsp; | flag | Flush the current cache entries on startup. WARNING: This operation is irreversible. |  |

<a name="mcp-sampling-options"></a>

## MCP Sampling Options [↩](#nav)

| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Option&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | Type | Description | Default |
|--------|------|-------------|--------|
| &nbsp;`‑‑prepare‑context`&nbsp; | flag | Output the prompt context as JSON to stdout without calling the LLM. JSON includes system/user prompts, cache state, and analysis results for each inference phase. Useful for inspecting prompts or piping to custom LLM integrations. Used by the MCP server for sampling mode. |  |
| &nbsp;`‑‑process‑response`&nbsp; | flag | Process LLM responses provided as JSON via stdin. Takes the output format from --prepare-context with LLM responses filled in, and produces the final output (dictionary, description, tags, or prompt results). Used by the MCP server for sampling mode. |  |

<a name="common-options"></a>

## Common Options [↩](#nav)

| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Option&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | Type | Description | Default |
|--------|------|-------------|--------|
| &nbsp;`‑h,`<br>`‑‑help`&nbsp; | flag | Display this message |  |
| &nbsp;`‑‑format`&nbsp; | string | Output format: Markdown, TSV, JSON, TOON, JSONSchema, SemanticMd, or OKF. TOON is a compact, human-readable encoding of the JSON data model for LLM prompts. See <https://toonformat.dev/> for more info. JSONSchema emits the Data Dictionary as a JSON Schema (draft 2020-12) document, enriched with LLM-inferred Label, Description and Content Type (the latter only when the infer-content-type flag is set). qsv- and LLM- specific metadata not modeled by the JSON Schema spec (cardinality, null_count, weighted example counts, content_type, null_values, null_candidates, addl stats columns) is preserved via a single x-qsv annotation object per property; unknown keywords are ignored by validators per the 2020-12 spec. The JSONSchema format requires the dictionary inference phase (the dictionary or all flag). The description inference, when also run, becomes the schema's top-level description; tags, when also run, are embedded at x-qsv.tags. The prompt inference is not supported. SemanticMd emits the Data Dictionary as a Semantic Markdown document (<https://semanticmd.org/>) - human-readable markdown with light, agent-parseable conventions that a companion converter turns into JSON. It enriches each column with a catalog-wide Concept ID (for cross-dataset join discovery), an analytical Role (dimension/measure/identifier/timestamp), join keys & cardinality, data-quality flags, and a richer per-column statistics block; it also emits a dataset grain and a temporal/spatial envelope. To populate Concept/Role/grain, SemanticMd implies --infer-content-type. Like JSONSchema, it requires the dictionary inference phase (the dictionary or all flag). The description inference, when also run, becomes the '# Dataset' description; tags, when also run, are embedded in the document frontmatter. The prompt inference is not supported. OKF emits the Data Dictionary as an Open Knowledge Format document (<https://github.com/GoogleCloudPlatform/knowledge-catalog/tree/main/okf>) - a leaner, vendor-neutral plain-markdown-plus-YAML-frontmatter format. It emits frontmatter (type/title/description/resource/timestamp/tags) and a Schema section with a Column/Type/Content Type/Description/Enumeration table. The type frontmatter key is set via the okf-type flag (default "CSV Table"); ds-source maps to resource and ds-updated maps to timestamp. Like SemanticMd, OKF implies infer-content-type, requires the dictionary inference phase (the dictionary or all flag), folds the description inference into the body prose and a single-sentence frontmatter description, and embeds tags in the frontmatter. The prompt inference is not supported. | `Markdown` |
| &nbsp;`‑‑allow‑extra‑cols`&nbsp; | flag | When the format is JSONSchema, emit additionalProperties as true at the schema root (default is false, strict). Only meaningful with the JSONSchema format; ignored otherwise. |  |
| &nbsp;`‑‑strict‑dates`&nbsp; | flag | When the format is JSONSchema, emit format date or date-time for columns that stats infers as Date or DateTime. Off by default because qsv's --infer-dates is permissive (accepts strings like "June 27, 1968") and JSON Schema's date formats require RFC 3339, so the validate roundtrip would otherwise fail. Set this only when your source columns are guaranteed to be RFC 3339 full-date / date-time. Mirrors the same flag on the schema command. |  |
| &nbsp;`‑‑ds‑source`&nbsp; | string | For the SemanticMd & OKF formats only: the dataset source/provenance recorded in the document frontmatter (e.g. a source URL or publisher). For OKF this populates the `resource` key. Optional; the frontmatter key is omitted when unset. Ignored by other formats. |  |
| &nbsp;`‑‑ds‑updated`&nbsp; | string | For the SemanticMd & OKF formats only: the dataset's last-updated date recorded in the document frontmatter. For OKF this populates the `timestamp` key. Optional. Ignored by other formats. |  |
| &nbsp;`‑‑ds‑license`&nbsp; | string | For the SemanticMd format only: the dataset license recorded in the document frontmatter. Optional. Ignored by other formats. |  |
| &nbsp;`‑‑okf‑type`&nbsp; | string | For the OKF format only: the value of the required `type` frontmatter key (e.g. "CSV Table", "BigQuery Table"). Optional; defaults to "CSV Table". Ignored by other formats. |  |
| &nbsp;`‑o,`<br>`‑‑output`&nbsp; | string | Write output to <file> instead of stdout. If --format is set to TSV, separate files will be created for each prompt type with the pattern {filestem}.{kind}.tsv (e.g., output.dictionary.tsv, output.tags.tsv). |  |
| &nbsp;`‑q,`<br>`‑‑quiet`&nbsp; | flag | Do not print status messages to stderr. |  |

---
**Source:** [`src/cmd/describegpt.rs`](https://github.com/dathere/qsv/blob/master/src/cmd/describegpt.rs)
| **[Table of Contents](TableOfContents.md)** | **[README](../../README.md)**
