describegpt: --dictionary infer is unstable run-to-run; same data + model yields different role assignments
documentation
## Problem
Re-inferring a dictionary for the **same dataset with the same model** produces materially
different semantics run to run — not just reworded labels, but different `role` assignments, which
change which panels `viz smart` draws.
All runs below used `google/gemma-4-26b-a4b` via LM Studio with `--fresh` (so describegpt's
completion cache is bypassed) and the same flags:
```
qsv describegpt <input> --dictionary --description --infer-content-type --two-pass \
--format jsonschema --fresh
```
## Observations
**`examples/viz/world_cities.csv`** — three runs, three outcomes:
| run | result |
|---|---|
| A | correct roles; `metro_population_m: sum`, `elevation_m: mean`, `avg_annual_temp_c: mean` — but all content in Italian (see #4406) |
| B (`--language English`) | English labels, but **every one of the 8 columns** came back `role: dimension`, including the three numeric measures. Zero aggregation annotations. |
**`examples/viz/seismic_events.csv`** — a fresh run demoted `magnitude`, `depth_km` and
`felt_reports` from `measure` to `dimension` and set `concept: unknown` on all of them, where the
previous inference had them as measures. It also dropped the `gauge_range: [0, 10]` a previous
inference had emitted for `magnitude` — the Richter scale is a canonical domain, and the committed
figure draws that KPI as a gauge on it.
`magnitude` is `Float` with cardinality 45 and `depth_km` is `Float` with cardinality 320, so these
are not borderline calls.
## Impact
`role` drives panel selection, so this is not cosmetic — a numeric measure demoted to `dimension`
loses its box/correlation/KPI treatment and may chart as a frequency bar instead. Two consequences:
1. Anyone regenerating a `--dictionary infer` dashboard can get a structurally different dashboard
from identical inputs.
2. `verify_aggregation` correctly drops `x-qsv.aggregation` on a non-measure, so a bad `role` roll
silently takes the aggregation annotations with it.
Worth noting the *aggregation* values were reliable whenever the role was right — on
`sales_sample.csv` the model annotated all six numeric measures correctly, `unit_price: mean` and
`shipping_cost: sum` included. The instability is concentrated in `role`/`concept`, not in the new
`aggregation` vocabulary.
## Mitigation already in place
PR #4404 tracks the four `--dictionary infer` gallery sidecars, so the committed gallery is now
immune to this: `QSV_VIZ_REGEN_LLM=1` reuses the committed dictionaries instead of re-rolling.
That protects the gallery but not users running `--dictionary infer` on their own data.
## Possible directions
Not proposing a fix, just what seems worth weighing:
- A **sanity check** at verification time: a `Float`/`Integer` column with high cardinality tagged
`role: dimension` is suspect, and could at minimum warn. This is the same propose-then-verify
shape `verify_gauge_range` / `verify_currency` / `verify_aggregation` already use — the stats
needed are already in hand.
- `--addl-props '{"temperature": 0}'` already lets a user pin sampling; it may be worth
**defaulting** dictionary inference to a low temperature, since this task wants determinism
rather than variety, or at least documenting that.
- Documenting that an inferred sidecar is a draft to be reviewed and committed — which is already
the design intent ("edit it to fine-tune, reused on later runs"), just not stated as a
reproducibility practice.
Found while working on #4401 (PR #4404).
1 条评论