# `describegpt` command

`describegpt` allows users to infer extended metadata about a CSV dataset using large language models, in particular GPT chat completion models from OpenAI's API, Ollama, or an API compatible with the OpenAI API specification such as Jan.

`describegpt` uses `qsv stats` and `qsv frequency` in the background to provide context to the model. It does not send the entire dataset to the model. This "zero-copy" approach is not only ideal for reducing costs (as detailed metadata, even for very large dataset is "tiny" in comparison) and increasing performance, it also protects the privacy of sensitive data.

Note that this command uses LLMs for inferencing and is therefore prone to inaccurate information being produced. Verify output results before using them.

## Basic Inference Options

`describegpt` provides several options to infer different types of metadata about your dataset:

- `--dictionary` - Creates a Data Dictionary using a hybrid "neuro-symbolic" pipeline. The Data Dictionary is deterministically populated using Summary Statistics and Frequency Distribution data, and only the human-friendly Label and Description (and the Content Type, when `--infer-content-type` is set) are populated by the LLM using the same statistical context.

- `--description` - Infers a general Description of the dataset based on detailed statistical context.

- `--tags` - Infers Tags that categorize the dataset based on detailed statistical context. Useful for grouping datasets and filtering.

- `-A, --all` - Shortcut for `--dictionary --description --tags`. Generates all three types of metadata in a single run.

## Dictionary Options

When using the `--dictionary` option, the following options control what the Data Dictionary
includes: `--num-examples` (default 5), `--truncate-str` (default 25), `--infer-content-type`,
`--infer-null-values`, `--two-pass`, `--addl-cols` and `--addl-cols-list`.

### `--infer-content-type`

By default, the Data Dictionary's LLM pass only infers a human-friendly **Label** and **Description** for each field. With `--infer-content-type`, the LLM additionally infers a semantic **Content Type** for each field — a single token describing *what the field actually contains* — and a "Content Type" column/field is added to the Data Dictionary output (Markdown, JSON, TSV, and TOON).

The Content Type is chosen from a fixed, curated vocabulary so the value stays stable and machine-mappable:

`first_name`, `last_name`, `full_name`, `username`, `password`, `email`, `phone`, `street_address`, `building_number`, `secondary_address`, `city`, `state`, `state_abbr`, `zip_code`, `country`, `country_code`, `latitude`, `longitude`, `time_zone`, `company_name`, `job_title`, `uuid`, `credit_card`, `currency_code`, `isbn`, `ip_address`, `mac_address`, `url`, `user_agent`, `file_name`, `file_path`, `mime_type`, `color_hex`, `time`, `category`, `lorem_word`, `lorem_sentence`, `lorem_paragraph`, `free_text`, `street_name`, `industry`, `profession`, `unique_id`, `ipv6_address`, `license_plate`, `date`, `datetime`, `duration`, `unknown`

Primitive types (`integer`, `decimal`, `boolean`) are intentionally **not** in the vocabulary — they are already covered by the Data Dictionary's deterministic `Type` column. `date` and `datetime` *are* in the vocabulary, and may carry an optional LLM-inferred strftime suffix (e.g. `date:%Y-%m-%d`). Plain numeric or boolean fields with no specific semantic meaning are classified as `unknown`, and any value the LLM returns outside the vocabulary is coerced to `unknown`.

When `--infer-content-type` is not set, the Data Dictionary output is unchanged.

```bash
qsv describegpt data.csv --dictionary --infer-content-type
```

The curated vocabulary is designed to map cleanly onto fake-data generators, so a Data Dictionary produced with `--infer-content-type` can later drive synthetic test-data generation.

## Tag Options

When using the `--tags` option, you can control how tags are inferred:

### `--num-tags <n>`

The maximum number of tags to infer when the `--tags` option is used. The value must be between 1 and 50. The default is 10.

### `--tag-vocab <file>`

The CSV file containing the tag vocabulary to use for inferring tags. If no tag vocabulary file is provided, the model will use free-form tags.

The CSV file must have two columns with headers:
- First column: tag name
- Second column: tag description

Example CSV format:

```csv
tag,description
alphabetical_data,Data containing letters or alphabetical characters
numerical_data,Data containing numbers or numerical values
test_data,Sample or test data used for demonstration
```

The tag vocabulary supports multiple sources:
- **Local files**: Provide a path to a local CSV file
- **Remote URLs**: HTTP/HTTPS URLs pointing to CSV files (e.g., `https://example.com/tags.csv`)
- **CKAN resources**: Use the `ckan://` scheme to reference CKAN dataset resources (e.g., `ckan://dataset-id/resource-id`)
- **dathere:// scheme**: Reference GitHub raw content using the `dathere://` scheme

Remote resources (HTTP/HTTPS, CKAN, and dathere://) are automatically cached locally to avoid repeated downloads. The cache TTL is 1 hour by default.

**Note**: `qsvlite` only supports local files. Remote URLs, CKAN resources, and dathere:// schemes are not available in `qsvlite`.

### `--cache-dir <dir>`

The directory to use for caching downloaded tag vocabulary resources. If the directory does not exist, qsv will attempt to create it. The default is `~/.qsv-cache`.

If the `QSV_CACHE_DIR` environment variable is set, it will be used instead of this option.

### `--ckan-api <url>`

The URL of the CKAN API to use for downloading tag vocabulary resources with the `ckan://` scheme. The default is `https://data.dathere.com/api/3/action`.

If the `QSV_CKAN_API` environment variable is set, it will be used instead of this option.

### `--ckan-token <token>`

The CKAN API token to use. Only required if downloading private CKAN resources. If the `QSV_CKAN_TOKEN` environment variable is set, it will be used instead of this option.

### Examples

Using tag vocabulary with a local CSV file:

```bash
qsv describegpt data.csv --tags --tag-vocab tags.csv
```

Using tag vocabulary with a remote URL:

```bash
qsv describegpt data.csv --tags --tag-vocab https://example.com/tags.csv
```

Using tag vocabulary with CKAN resources:

```bash
qsv describegpt data.csv --tags --tag-vocab ckan://dataset-id/resource-id --ckan-api https://data.example.com/api/3/action
```

Using tag vocabulary with a custom cache directory:

```bash
qsv describegpt data.csv --tags --tag-vocab https://example.com/tags.csv --cache-dir /tmp/qsv-cache
```

Limiting the number of tags inferred:

```bash
qsv describegpt data.csv --tags --tag-vocab tags.csv --num-tags 5
```

## QSV_LLM_APIKEY

When working with a cloud-based LLM, `describegpt` requires an API key. You can set this key using the `QSV_LLM_APIKEY` environment variable. Check [/docs/ENVIRONMENT_VARIABLES.md](ENVIRONMENT_VARIABLES.md) for more info.

When working with a Local LLM (i.e. if `--base-url` or `QSV_LLM_BASE_URL` contains "localhost"), `describegpt` will NOT require an API key.

## `--api-key <key>`

You can also specify your API key directly in your CLI using the `--api-key` option.

Note that the precedence is: CLI flag > environment variable > error. If you pass `--api-key`, it overrides
the `QSV_LLM_APIKEY` environment variable. Pass `--api-key NONE` to suppress the API key entirely.

## `--format <format>`

Use the `--format` option to set the output format. Supported values are `Markdown` (the default), `TSV`,
`JSON`, `TOON`, `JSONSchema`, `SemanticMd`, and `OKF`. `OKF` emits the Data Dictionary as an
Open Knowledge Format document; its frontmatter `type` key is set with `--okf-type`
(default `"CSV Table"`).

- `Markdown` - human-readable Markdown (default).
- `TSV` - tab-separated values.
- `JSON` - JSON output, useful for piping to other commands.
- `TOON` - a compact, human-readable encoding of the JSON data model, optimized for LLM prompts.
- `JSONSchema` - emits the Data Dictionary as a JSON Schema (draft 2020-12). Requires the dictionary
  inference phase. See `--allow-extra-cols` and `--strict-dates` for JSONSchema-specific behavior.
- `SemanticMd` - emits the Data Dictionary as a Semantic Markdown document. Implies `--infer-content-type`
  and also requires the dictionary inference phase. See `--ds-source`, `--ds-updated`, and `--ds-license`.

Note that `--format` only controls the structure of the command's output. It does **not** instruct the LLM
to generate a particular content shape - that is driven by your prompts.

## `--max-tokens <value>`

`--max-tokens` is an option that allows you to specify the maximum number of tokens in the completion **output**. This is limited by the maximum number of tokens allowed by the model including the input tokens.

Input tokens may include the output of `qsv stats` and `qsv frequency` from your dataset, which can be large based on your dataset's size. Therefore we use `gpt-oss-20b` as the default model for `describegpt` as it has a maximum token limit of 131,072.

It is highly recommended to set the `--max-tokens` option to set the maximum number of tokens in the completion output. Your output may be truncated if you set this value too low or you may receive errors depending on your options. The default is set to `10000` as a safety measure.

When running a Local LLM (detected if the `base_url` contains localhost), the max token limit is automatically disabled. Your completions are only limited by the LLM model you're using.

## `--prompt-file`

With `describegpt` you can use a prompt file to add your own custom prompts and as an alternative to specifying certain options through the CLI. You can use the `--prompt-file` option to specify a prompt file to use.

If you do not specify a prompt file, default prompts will be used.

| Field                    | Description                                                                                 |
| --------------------     | ------------------------------------------------------------------------------------------- |
| `name`                   | The name of your prompt file.                                                               |
| `description`            | A description of your prompt file.                                                          |
| `author`                 | Your name.                                                                                  |
| `version`                | The version of your prompt file.                                                            |
| `tokens`                 | The maximum number of tokens in the completion output.                                      |
| `format`                 | Output format: `Markdown`, `TSV`, `JSON`, `TOON`, `JSONSchema`, `SemanticMd`, or `OKF` (default `markdown`). The `--format` CLI flag takes precedence over this field. |
| `system_prompt`          | Overall guidance prompt to the LLM.                                                         |
| `dictionary_prompt`      | The prompt for the `--dictionary` option.                                                   |
| `description_prompt`     | The prompt for the `--description` option.                                                  |
| `tags_prompt`            | The prompt for the `--tags` option.                                                         |
| `base_url`               | The URL of the LLM API. When it contains "localhost", automatically sets `tokens` to 0.     |
| `model`                  | The LLM model to use.                                                                       |
| `timeout`                | The timeout in seconds to use when waiting for LLM prompt completions.                      |
| `custom_prompt_guidance` | The guidance used to generate SQL queries in SQL RAG mode.                                  | 
| `duckdb_sql_guidance`    | DuckDB-specific SQL generation guidelines.                                                  |
| `polars_sql_guidance`    | Polars-specific SQL generation guidelines.                                                  |
| `dd_fewshot_examples`    | DuckDB "few-shot" examples. See https://www.promptingguide.ai/techniques/fewshot            |
| `p_fewshot_examples`     | Polars "few-shot" examples.                                                                 |
| `language`               | The natural language the LLM should respond in.                                             |
| `prompt`                 | The base prompt prepended to the per-option prompts.                                        |
| `dictionary_refine_prompt` | The prompt for the second pass of `--two-pass` Data Dictionary inference.                 |

All fields except those with a serde default must be present in your prompt file. If you do not want to use a certain prompt, you can set it to an empty string.

> [!IMPORTANT]
> `language` and `prompt` have no serde default — omitting either causes a TOML deserialization
> error. `dictionary_refine_prompt` does have a default, so omitting it is harmless.
> See `resources/describegpt_defaults.toml` for a complete, working reference file.

Within your prompts, you can use the following MiniJinja variables (use `{{ variable }}` syntax).
These are replaced in the prompt sent to the LLM with their respective values at run-time:

-   `{{ stats }}` - summary stats generated by qsv stats (CSV format)
-   `{{ frequency }}` - frequency distribution generated by qsv frequency (CSV format)
-   `{{ dictionary }}` - the Data Dictionary that was inferred by the LLM using Stats & Frequency (JSON format)
-   `{{ json_add }}` - inserts ` (in JSON format)` when JSON output is requested. Note the leading space.
-   `{{ input_table_name }}` - replaced by the name of the input file
-   `{{ generated_by_signature }}` - replaced with model name and current timestamp
-   `{{ duckdb_version }}` - DuckDB version (up to the minor version)
-   `{{ context }}` - additional, user-supplied context provided via `--context-file`. The file type is
    sniffed from its contents (not its extension). Supported types: plain text, Markdown, CSV, Excel/ODS
    spreadsheets (extracted to CSV), and PDF or image files (JPEG/PNG/WebP/GIF) — the latter are sent to
    the LLM as a multimodal attachment (needs a multimodal model & endpoint; ~32 MB max), and here the
    variable renders a short marker. Word/PowerPoint (docx/pptx) are not supported — convert to PDF or
    text first. By default qsv injects this context into the **user** message; if a custom template
    references `{{ context }}` in any prompt, that explicit placement is honored instead. Empty when not
    set, so guard usage with `{% if context %}`.

See the header of `resources/describegpt_defaults.toml` for the full list of available template variables.


See `resources/describegpt_defaults.toml` for the default values.

## Running LLMs locally with Ollama

Since the release of Ollama v0.2.0, Ollama provides the necessary OpenAI compatible endpoints to work with describegpt. You may find the Ollama OpenAI compatibility documentation here: https://github.com/ollama/ollama/blob/main/docs/openai.md.

An example command for getting an inferred description is as follows:

```bash
qsv describegpt <filepath> --base-url http://localhost:11434/v1 --model <model> --max-tokens <number> --description
```

Remove the arrow brackets `<>` and replace `filepath` with your file's path, `<model>` with the model you want to use, and `number` with the max tokens you want to set based on your model's context size.

## SQL Query Generation and Execution ("SQL RAG" mode)

When using the `--prompt` option, `describegpt` can automatically generate and execute SQL queries to answer questions that cannot be answered using just the summary statistics and frequency distribution data. This is called "SQL RAG" mode.

### Using SQL with Polars (default)

By default, when the `polars` feature is enabled, `describegpt` uses qsv's `sqlp` command to execute SQL queries. You can specify the `--sql-results` option to save the query results to a CSV file:

```bash
qsv describegpt data.csv --prompt "What's the breakdown of complaint types by borough?" --sql-results results.csv
```

### Using SQL with DuckDB

You can also use DuckDB to execute SQL queries by setting the `QSV_DUCKDB_PATH` environment variable to the fully qualified path of the DuckDB binary. When set and non-empty, DuckDB will be used instead of Polars:

```bash
export QSV_DUCKDB_PATH=/usr/local/bin/duckdb
qsv describegpt data.csv --prompt "What's the breakdown of complaint types by borough?" --sql-results results.csv
```

When DuckDB is used, the SQL query generation guidelines are automatically modified to use DuckDB generation guidelines.
See `resources/describegpt_defaults.toml` for the default guidelines for DuckDB and Polars.



