# Roadmap: DuckDB Semantic Views

## Milestones

- ✅ **v0.1.0 MVP** -- Phases 1-7 (shipped 2026-02-28)
- ✅ **v0.2.0 Native DDL + Time Dimensions** -- Phases 8-14 (shipped 2026-03-03)
- ✅ **v0.3.0 Zero-Copy Query Pipeline** -- (shipped 2026-03-03)
- ✅ **v0.4.0 Simplified Dimensions** -- (shipped 2026-03-03)
- ✅ **v0.5.0 Parser Extension Spike** -- Phases 15-18 (shipped 2026-03-08)
- ✅ **v0.5.1 DDL Polish** -- Phases 19-23 (shipped 2026-03-09)
- ✅ **v0.5.2 SQL DDL & PK/FK Relationships** -- Phases 24-28 (shipped 2026-03-13)
- ✅ **v0.5.3 Advanced Semantic Features** -- Phases 29-32 (shipped 2026-03-15)
- ✅ **v0.5.4 Snowflake-Parity & Registry Publishing** -- Phases 33-36 (shipped 2026-03-27)
- :construction: **v0.5.5 SHOW/DESCRIBE Alignment & Refactoring** -- Phases 37-42 (in progress)

## Phases

<details>
<summary>v0.1.0 MVP (Phases 1-7) -- SHIPPED 2026-02-28</summary>

- [x] Phase 1: Scaffold (3/3 plans) -- completed 2026-02-24
- [x] Phase 2: Storage and DDL (4/4 plans) -- completed 2026-02-24
- [x] Phase 3: Expansion Engine (3/3 plans) -- completed 2026-02-25
- [x] Phase 4: Query Interface (3/3 plans) -- completed 2026-02-25
- [x] Phase 5: Hardening and Docs (2/2 plans) -- completed 2026-02-26
- [x] Phase 6: Tech Debt Code Cleanup (1/1 plan) -- completed 2026-02-26
- [x] Phase 7: Verification & Formal Closure (2/2 plans) -- completed 2026-02-27

Full details: [milestones/v1.0-ROADMAP.md](milestones/v1.0-ROADMAP.md)

</details>

<details>
<summary>v0.2.0 Native DDL + Time Dimensions (Phases 8-14) -- SHIPPED 2026-03-03</summary>

- [x] Phase 8: C++ Shim Infrastructure (2/2 plans) -- completed 2026-03-01
- [x] Phase 9: Time Dimensions (2/2 plans) -- completed 2026-03-01
- [x] Phase 10: pragma_query_t Catalog Persistence (3/3 plans) -- completed 2026-03-01
- [x] Phase 11: Scalar Function DDL (4/4 plans) -- completed 2026-03-02
- [x] Phase 11.1: Snowflake-aligned STRUCT/LIST DDL Syntax (5/5 plans) -- completed 2026-03-02
- [x] Phase 12: EXPLAIN + Typed Output (4/4 plans) -- completed 2026-03-02
- [x] Phase 13: Type-mapping + PBTs (2/2 plans) -- completed 2026-03-02
- [x] Phase 14: DuckLake Integration Test + CI (3/3 plans) -- completed 2026-03-02

Full details: [milestones/v0.2-ROADMAP.md](milestones/v0.2-ROADMAP.md)

</details>

<details>
<summary>v0.3.0 Zero-Copy Query Pipeline -- SHIPPED 2026-03-03</summary>

Replaced binary-read dispatch with zero-copy vector references (`duckdb_vector_reference_vector`).
Eliminated ~600 LOC of per-type read/write code. Type mismatches handled at SQL generation time
via `build_execution_sql` cast wrapper. Streaming chunk-by-chunk instead of collect-all-then-write.

</details>

<details>
<summary>v0.4.0 Simplified Dimensions -- SHIPPED 2026-03-03</summary>

Breaking change: removed `time_dimensions` DDL parameter and `granularities` query parameter.
Time truncation expressed via dimension `expr` directly (e.g., `date_trunc('month', created_at)`).
DDL simplified from 6 to 4 named params; query function from 3 to 2 named params.

</details>

<details>
<summary>v0.5.0 Parser Extension Spike (Phases 15-18) -- SHIPPED 2026-03-08</summary>

- [x] Phase 15: Entry Point POC (2/2 plans) -- completed 2026-03-07
- [x] Phase 16: Parser Hook Registration (1/1 plan) -- completed 2026-03-07
- [x] Phase 17: DDL Execution (1/1 plan) -- completed 2026-03-07
- [x] Phase 17.1: Python vtab crash investigation (2/2 plans) -- completed 2026-03-08
- [x] Phase 18: Verification and Integration (2/2 plans) -- completed 2026-03-08

Full details: [milestones/v0.5-ROADMAP.md](milestones/v0.5-ROADMAP.md)

</details>

<details>
<summary>v0.5.1 DDL Polish (Phases 19-23) -- SHIPPED 2026-03-09</summary>

- [x] Phase 19: Parser Hook Validation Spike (1/1 plan) -- completed 2026-03-09
- [x] Phase 20: Extended DDL Statements (2/2 plans) -- completed 2026-03-09
- [x] Phase 21: Error Location Reporting (3/3 plans) -- completed 2026-03-09
- [x] Phase 22: Documentation (1/1 plan) -- completed 2026-03-09
- [x] Phase 23: Parser Proptests and Caret Integration Tests (2/2 plans) -- completed 2026-03-09

Full details: [milestones/v0.5.1-ROADMAP.md](milestones/v0.5.1-ROADMAP.md)

</details>

<details>
<summary>v0.5.2 SQL DDL & PK/FK Relationships (Phases 24-28) -- SHIPPED 2026-03-13</summary>

- ~~Phase 24: PK/FK Model~~ -- CANCELLED (absorbed into Phase 25-01)
- [x] Phase 25: SQL Body Parser (4/4 plans) -- completed 2026-03-12
- [x] Phase 25.1: Parser Robustness & Security (2/2 plans) -- completed 2026-03-13
- [x] Phase 26: PK/FK Join Resolution (2/2 plans) -- completed 2026-03-13
- [x] Phase 27: Alias-Based Query Expansion (3/3 plans) -- completed 2026-03-13
- [x] Phase 28: Integration Testing & Docs (3/3 plans) -- completed 2026-03-13

Full details: [milestones/v0.5.2-ROADMAP.md](milestones/v0.5.2-ROADMAP.md)

</details>

<details>
<summary>v0.5.3 Advanced Semantic Features (Phases 29-32) -- SHIPPED 2026-03-15</summary>

- [x] Phase 29: FACTS Clause & Hierarchies (2/2 plans) -- completed 2026-03-14
- [x] Phase 30: Derived Metrics (2/2 plans) -- completed 2026-03-14
- [x] Phase 31: Fan Trap Detection (2/2 plans) -- completed 2026-03-14
- [x] Phase 32: Role-Playing Dimensions & USING RELATIONSHIPS (2/2 plans) -- completed 2026-03-14

Full details: [milestones/v0.5.3-ROADMAP.md](milestones/v0.5.3-ROADMAP.md)

</details>

<details>
<summary>v0.5.4 Snowflake-Parity & Registry Publishing (Phases 33-36) -- SHIPPED 2026-03-27</summary>

- [x] Phase 33: UNIQUE Constraints & Cardinality Inference (2/2 plans) -- completed 2026-03-15
- [x] Phase 34: DuckDB 1.5 Upgrade & LTS Branch (2/2 plans) -- completed 2026-03-16
- [x] Phase 34.1: DDL Surface Parity (3/3 plans) -- completed 2026-03-22
- [x] Phase 34.1.1: SHOW Command Filtering (1/1 plan) -- completed 2026-03-22
- [x] Phase 35: Documentation Site (1/1 plan) -- completed 2026-03-27
- [x] Phase 36: Registry Publishing & Maintainer Docs (2/2 plans) -- completed 2026-03-27

Full details: [milestones/v0.5.4-ROADMAP.md](milestones/v0.5.4-ROADMAP.md)

</details>

### v0.5.5 SHOW/DESCRIBE Alignment & Refactoring (In Progress)

**Milestone Goal:** Align all 6 SHOW/DESCRIBE output formats with Snowflake column schemas, and refactor the two largest modules (expand.rs, graph.rs) into clean module directories -- preparing for future MetricFlow-style library extraction.

- [x] **Phase 37: Extract Shared Utilities** - Break circular dependencies by extracting util.rs and errors.rs as leaf modules (completed 2026-04-01)
- [x] **Phase 38: Module Directory Splits** - Split expand.rs and graph.rs into module directories with unchanged public APIs (completed 2026-04-01)
- [x] **Phase 39: Metadata Storage** - Add created_on, database_name, schema_name, and fact output_type to the semantic view model (completed 2026-04-02)
- [x] **Phase 40: SHOW Command Alignment** - Align all SHOW SEMANTIC commands to Snowflake column schemas (completed 2026-04-02)
- [x] **Phase 41: DESCRIBE Rewrite** - Complete rewrite of DESCRIBE SEMANTIC VIEW to Snowflake property-per-row format (completed 2026-04-02)

## Phase Details

### Phase 37: Extract Shared Utilities
**Goal**: Circular dependencies between expand/graph and parse/body_parser are broken by extracting shared functions and types into leaf modules
**Depends on**: Nothing (first phase of v0.5.5)
**Requirements**: REF-03, REF-04
**Success Criteria** (what must be TRUE):
  1. `suggest_closest` and `replace_word_boundary` are importable from `crate::util` by both expand and graph modules
  2. `ParseError` is importable from `crate::errors` by both parse and body_parser modules
  3. All 482+ existing tests pass with zero behavior changes after extraction
**Plans**: 1 plan
Plans:
- [ ] 37-01-PLAN.md -- Extract util.rs and errors.rs leaf modules, update all import paths

### Phase 38: Module Directory Splits
**Goal**: The two largest source files (expand.rs at 4,299 lines and graph.rs at 2,333 lines) are decomposed into module directories with single-responsibility submodules, while preserving the exact public API surface
**Depends on**: Phase 37
**Requirements**: REF-01, REF-02, REF-05
**Success Criteria** (what must be TRUE):
  1. `src/expand/` exists as a module directory with submodules (validation, resolution, facts, fan_trap, role_playing, join_resolver, sql_gen) and `mod.rs` re-exports the full prior public API
  2. `src/graph/` exists as a module directory with submodules (relationship, facts, derived_metrics, using, toposort) and `mod.rs` re-exports the full prior public API
  3. No file named `expand.rs` or `graph.rs` exists in `src/` (old files removed to avoid Rust ambiguity errors)
  4. `just test-all` passes -- all Rust tests, sqllogictests, and DuckLake CI tests green with zero behavior changes
**Plans**: 2 plans
Plans:
- [ ] 38-01-PLAN.md -- Split expand.rs into expand/ module directory (7 submodules + test_helpers)
- [ ] 38-02-PLAN.md -- Split graph.rs into graph/ module directory (5 submodules + test_helpers)

### Phase 39: Metadata Storage
**Goal**: Semantic view definitions carry timestamp and context metadata needed by downstream SHOW/DESCRIBE commands, with full backward compatibility for pre-v0.5.5 stored views
**Depends on**: Phase 38
**Requirements**: META-01, META-02, META-03, META-04, META-05
**Success Criteria** (what must be TRUE):
  1. Newly created semantic views store a `created_on` ISO 8601 timestamp captured at define time via DuckDB `now()`
  2. Newly created semantic views store `database_name` and `schema_name` from the DuckDB connection context at define time
  3. Facts in the model carry an `output_type` field for downstream data_type display
  4. Pre-v0.5.5 stored JSON (without new fields) deserializes without error -- missing fields default to None via `serde(default)`
  5. `just test-all` passes with new metadata fields exercised in unit tests
**Plans**: 1 plan
Plans:
- [x] 39-01-PLAN.md -- Add metadata fields to model, capture at define time, backward compat tests

### Phase 40: SHOW Command Alignment
**Goal**: All SHOW SEMANTIC commands output Snowflake-aligned column schemas -- dropping `expr`, renaming `source_table` to `table_name`, and adding metadata columns
**Depends on**: Phase 39
**Requirements**: SHOW-01, SHOW-02, SHOW-03, SHOW-04, SHOW-05, SHOW-06, SHOW-07, SHOW-08
**Success Criteria** (what must be TRUE):
  1. `SHOW SEMANTIC VIEWS` returns 5 columns: created_on, name, kind, database_name, schema_name
  2. `SHOW SEMANTIC DIMENSIONS`, `SHOW SEMANTIC METRICS`, and `SHOW SEMANTIC FACTS` each return 6 columns: database_name, schema_name, semantic_view_name, table_name, name, data_type
  3. `SHOW SEMANTIC DIMENSIONS FOR METRIC` returns 4 columns: table_name, name, data_type, required (BOOLEAN, constant FALSE)
  4. No SHOW command exposes an `expr` column; all use `table_name` (not `source_table`)
  5. LIKE, STARTS WITH, and LIMIT filtering continues to work on all SHOW commands
**Plans**: 2 plans
Plans:
- [x] 40-01-PLAN.md -- Update list.rs, show_dims.rs, show_metrics.rs, show_facts.rs to Snowflake schemas + update phase34_1_show_commands.test
- [x] 40-02-PLAN.md -- Update show_dims_for_metric.rs with BOOLEAN required column + update remaining sqllogictest files

### Phase 41: DESCRIBE Rewrite
**Goal**: DESCRIBE SEMANTIC VIEW outputs a Snowflake-aligned property-per-row format where each row describes one property of one object, replacing the current single-row JSON-blob format
**Depends on**: Phase 39
**Requirements**: DESC-01, DESC-02, DESC-03, DESC-04, DESC-05, DESC-06, DESC-07
**Success Criteria** (what must be TRUE):
  1. `DESCRIBE SEMANTIC VIEW` returns rows with 5 columns: object_kind, object_name, parent_entity, property, property_value
  2. TABLE objects emit properties: BASE_TABLE_NAME, PRIMARY_KEY, BASE_TABLE_DATABASE_NAME, BASE_TABLE_SCHEMA_NAME
  3. RELATIONSHIP, DIMENSION, FACT, METRIC, and DERIVED_METRIC objects each emit their documented property sets (TABLE, EXPRESSION, DATA_TYPE, FOREIGN_KEY, REF_KEY, REF_TABLE as applicable)
  4. A semantic view with tables, relationships, dimensions, facts, metrics, and derived metrics produces the correct multi-row property output for all object kinds
  5. sqllogictest coverage validates the complete property-per-row schema across all object kinds
**Plans**: 2 plans
Plans:
- [x] 41-01-PLAN.md -- Rewrite describe.rs to property-per-row format + comprehensive phase41_describe.test
- [x] 41-02-PLAN.md -- Update all existing test files with DESCRIBE assertions + error.rs help message

### Phase 42: Refactor, Tidy-ups, and Test Reorganisation
**Goal**: Eight code review findings addressed -- persistence correctness hardened (TOCTOU fix, parameterized queries), code tidied (parallel Vec helper, body parser comments), and test coverage gaps closed (fixture extraction, file-backed round-trip, transmute guard, suggestion proptests)
**Depends on**: Phase 41
**Requirements**: TIDY-01, TIDY-02, TIDY-03, TIDY-04, TIDY-05, TIDY-06, TIDY-07, TIDY-08
**Success Criteria** (what must be TRUE):
  1. `catalog_insert` and `catalog_delete` use single write lock for check-and-mutate (no TOCTOU race)
  2. All persistence SQL uses `duckdb_prepare`/`duckdb_bind_varchar` parameterized queries (no string interpolation)
  3. `SemanticViewDefinition` has an `inferred_types()` helper method expressing the parallel Vec invariant
  4. Body parser `unwrap()` calls have safety invariant comments
  5. `sql_gen.rs` test fixtures extracted to `expand/test_helpers.rs` with shared builders
  6. File-backed catalog round-trip verified via sqllogictest
  7. Transmute layout guard test catches `Value`/`duckdb_value` size mismatches
  8. `suggest_closest` property tested (suggestions are always valid names)
**Plans**: 3 plans
Plans:
- [ ] 42-01-PLAN.md -- Persistence correctness: TOCTOU fix + parameterized queries + body parser comments
- [ ] 42-02-PLAN.md -- Code tidying: parallel Vec helper + test fixture extraction
- [ ] 42-03-PLAN.md -- New test coverage: persistence round-trip + transmute guard + suggestion proptests

## Progress

**Execution Order:**
Phases execute in numeric order: 37 -> 38 -> 39 -> 40 -> 41 -> 42
(Phases 40 and 41 both depend on Phase 39; Phase 41 is independent of Phase 40)
Phase 42: Plans 01 and 02 run in parallel (Wave 1), Plan 03 depends on Plan 01 (Wave 2)

| Phase | Milestone | Plans Complete | Status | Completed |
|-------|-----------|----------------|--------|-----------|
| 37. Extract Shared Utilities | v0.5.5 | 0/1 | Complete    | 2026-04-01 |
| 38. Module Directory Splits | v0.5.5 | 0/2 | Complete    | 2026-04-01 |
| 39. Metadata Storage | v0.5.5 | 1/1 | Complete    | 2026-04-02 |
| 40. SHOW Command Alignment | v0.5.5 | 2/2 | Complete    | 2026-04-02 |
| 41. DESCRIBE Rewrite | v0.5.5 | 2/2 | Complete    | 2026-04-02 |
| 42. Refactor & Test Reorg | v0.5.5 | 0/3 | Complete    | 2026-04-05 |
