ITADN

describegpt: --dictionary infer is unstable run-to-run; same data + model yields different role assignments

#4407Openjqnatividad 创建于 15 天前
documentation
J
jqnatividadcommented
## Problem Re-inferring a dictionary for the **same dataset with the same model** produces materially different semantics run to run — not just reworded labels, but different `role` assignments, which change which panels `viz smart` draws. All runs below used `google/gemma-4-26b-a4b` via LM Studio with `--fresh` (so describegpt's completion cache is bypassed) and the same flags: ``` qsv describegpt <input> --dictionary --description --infer-content-type --two-pass \ --format jsonschema --fresh ``` ## Observations **`examples/viz/world_cities.csv`** — three runs, three outcomes: | run | result | |---|---| | A | correct roles; `metro_population_m: sum`, `elevation_m: mean`, `avg_annual_temp_c: mean` — but all content in Italian (see #4406) | | B (`--language English`) | English labels, but **every one of the 8 columns** came back `role: dimension`, including the three numeric measures. Zero aggregation annotations. | **`examples/viz/seismic_events.csv`** — a fresh run demoted `magnitude`, `depth_km` and `felt_reports` from `measure` to `dimension` and set `concept: unknown` on all of them, where the previous inference had them as measures. It also dropped the `gauge_range: [0, 10]` a previous inference had emitted for `magnitude` — the Richter scale is a canonical domain, and the committed figure draws that KPI as a gauge on it. `magnitude` is `Float` with cardinality 45 and `depth_km` is `Float` with cardinality 320, so these are not borderline calls. ## Impact `role` drives panel selection, so this is not cosmetic — a numeric measure demoted to `dimension` loses its box/correlation/KPI treatment and may chart as a frequency bar instead. Two consequences: 1. Anyone regenerating a `--dictionary infer` dashboard can get a structurally different dashboard from identical inputs. 2. `verify_aggregation` correctly drops `x-qsv.aggregation` on a non-measure, so a bad `role` roll silently takes the aggregation annotations with it. Worth noting the *aggregation* values were reliable whenever the role was right — on `sales_sample.csv` the model annotated all six numeric measures correctly, `unit_price: mean` and `shipping_cost: sum` included. The instability is concentrated in `role`/`concept`, not in the new `aggregation` vocabulary. ## Mitigation already in place PR #4404 tracks the four `--dictionary infer` gallery sidecars, so the committed gallery is now immune to this: `QSV_VIZ_REGEN_LLM=1` reuses the committed dictionaries instead of re-rolling. That protects the gallery but not users running `--dictionary infer` on their own data. ## Possible directions Not proposing a fix, just what seems worth weighing: - A **sanity check** at verification time: a `Float`/`Integer` column with high cardinality tagged `role: dimension` is suspect, and could at minimum warn. This is the same propose-then-verify shape `verify_gauge_range` / `verify_currency` / `verify_aggregation` already use — the stats needed are already in hand. - `--addl-props '{"temperature": 0}'` already lets a user pin sampling; it may be worth **defaulting** dictionary inference to a low temperature, since this task wants determinism rather than variety, or at least documenting that. - Documenting that an inferred sidecar is a draft to be reviewed and committed — which is already the design intent ("edit it to fine-tune, reused on later runs"), just not stated as a reproducibility practice. Found while working on #4401 (PR #4404).
1 条评论