---
phase: 20-extended-ddl-statements
plan: 02
type: execute
wave: 2
depends_on: ["20-01"]
files_modified:
  - cpp/src/shim.cpp
  - src/parse.rs
  - test/sql/phase20_extended_ddl.test
autonomous: true
requirements: [DDL-07, DDL-08]

must_haves:
  truths:
    - "DESCRIBE SEMANTIC VIEW x shows dimensions, metrics, and types"
    - "SHOW SEMANTIC VIEWS lists all defined semantic views"
    - "Mutating DDL (CREATE, DROP, etc.) still works after the C++ refactor"
  artifacts:
    - path: "cpp/src/shim.cpp"
      provides: "Result-forwarding sv_ddl_bind and sv_ddl_execute that handle arbitrary column schemas"
      contains: "sv_rewrite_ddl_rust"
    - path: "src/parse.rs"
      provides: "New FFI function sv_rewrite_ddl_rust that returns rewritten SQL without executing"
      contains: "sv_rewrite_ddl_rust"
    - path: "test/sql/phase20_extended_ddl.test"
      provides: "Integration tests for DESCRIBE SEMANTIC VIEW and SHOW SEMANTIC VIEWS"
  key_links:
    - from: "cpp/src/shim.cpp"
      to: "src/parse.rs"
      via: "sv_rewrite_ddl_rust FFI -- C++ calls Rust to get rewritten SQL, then C++ executes it"
      pattern: "sv_rewrite_ddl_rust"
    - from: "cpp/src/shim.cpp"
      to: "duckdb_query on sv_ddl_conn"
      via: "sv_ddl_bind executes rewritten SQL and captures result schema + data"
      pattern: "duckdb_query.*sv_ddl_conn"
---

<objective>
Refactor the C++ sv_ddl_bind/sv_ddl_execute pipeline to forward arbitrary result sets from rewritten SQL, enabling DESCRIBE SEMANTIC VIEW and SHOW SEMANTIC VIEWS to return their full multi-column output.

Purpose: The current C++ pipeline returns a single VARCHAR "view_name" column, which works for mutating DDL but cannot convey the multi-column results from describe_semantic_view() (6 columns) or list_semantic_views() (2 columns). This plan changes the architecture so that C++ executes the rewritten SQL and dynamically forwards the result schema and data.

Output: Working native DDL for DESCRIBE SEMANTIC VIEW and SHOW SEMANTIC VIEWS. Full Phase 20 complete.
</objective>

<execution_context>
@/Users/paul/.claude/get-shit-done/workflows/execute-plan.md
@/Users/paul/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/20-extended-ddl-statements/20-RESEARCH.md
@.planning/phases/20-extended-ddl-statements/20-01-SUMMARY.md

<interfaces>
<!-- Key types and contracts from Plan 01 output (Rust side) -->

From src/parse.rs (after Plan 01):
```rust
pub const PARSE_NOT_OURS: u8 = 0;
pub const PARSE_DETECTED: u8 = 1;

#[derive(Debug, Clone, Copy, PartialEq)]
pub enum DdlKind {
    Create,
    CreateOrReplace,
    CreateIfNotExists,
    Drop,
    DropIfExists,
    Describe,
    Show,
}

pub fn detect_ddl_kind(query: &str) -> Option<DdlKind>;
pub fn detect_semantic_view_ddl(query: &str) -> u8;
pub fn rewrite_ddl(query: &str) -> Result<String, String>;
pub fn extract_ddl_name(query: &str) -> Result<Option<String>, String>;
```

From cpp/src/shim.cpp (current, to be refactored):
```cpp
// sv_ddl_bind currently:
//   1. Declares one VARCHAR "view_name" output column
//   2. Calls sv_execute_ddl_rust (Rust rewrites + executes + returns name)
//   3. Stores name in SvDdlBindData
//
// sv_ddl_execute currently:
//   1. Emits one row with bind_data.view_name
//
// This needs to change to:
//   sv_ddl_bind:
//     1. Call sv_rewrite_ddl_rust (Rust rewrites, does NOT execute)
//     2. Execute rewritten SQL on sv_ddl_conn via duckdb_query
//     3. Read result column count, names, types from duckdb_result
//     4. Declare matching output columns on the TableFunction
//     5. Store result data in bind data for emission
//   sv_ddl_execute:
//     1. Emit all result rows from stored data

// Existing FFI declarations:
extern "C" uint8_t sv_parse_rust(const char *query, size_t query_len);
extern "C" uint8_t sv_execute_ddl_rust(
    const char *query_ptr, size_t query_len,
    duckdb_connection exec_conn,
    char *name_out, size_t name_out_len,
    char *error_out, size_t error_out_len);
```

From src/ddl/describe.rs (already registered, output schema):
```
describe_semantic_view('name') returns 6 VARCHAR columns:
  name, base_table, tables_json, dimensions_json, metrics_json, relationships_json
```

From src/ddl/list.rs (already registered, output schema):
```
list_semantic_views() returns 2 VARCHAR columns:
  name, base_table
```
</interfaces>
</context>

<tasks>

<task type="auto">
  <name>Task 1: Add sv_rewrite_ddl_rust FFI and refactor C++ to forward result sets</name>
  <files>src/parse.rs, cpp/src/shim.cpp</files>
  <action>
    **Rust side (src/parse.rs):**

    Add a new FFI function `sv_rewrite_ddl_rust` that returns the rewritten SQL string WITHOUT executing it. This replaces `sv_execute_ddl_rust` as the primary FFI entry point for the DDL pipeline.

    ```rust
    #[cfg(feature = "extension")]
    #[no_mangle]
    pub extern "C" fn sv_rewrite_ddl_rust(
        query_ptr: *const u8,
        query_len: usize,
        sql_out: *mut u8,
        sql_out_len: usize,
        error_out: *mut u8,
        error_out_len: usize,
    ) -> u8
    ```

    - On success: writes the rewritten SQL to `sql_out` (null-terminated), returns 0
    - On failure: writes the error message to `error_out` (null-terminated), returns 1
    - Uses `rewrite_ddl(query)` internally
    - Keep `sv_execute_ddl_rust` for backward compatibility but it is no longer called from C++

    **C++ side (cpp/src/shim.cpp):**

    1. Add FFI declaration for `sv_rewrite_ddl_rust`.

    2. Replace `SvDdlBindData` with a new struct that stores:
       - The full result set as a vector of rows (each row = vector of string values)
       - Column names and types (vector of string names, vector of LogicalType)
       - This captures the entire result from `duckdb_query` so sv_ddl_execute can emit it

    3. Refactor `sv_ddl_bind`:
       a. Extract the query text from `input.inputs[0]` (unchanged)
       b. Call `sv_rewrite_ddl_rust` to get the rewritten SQL string (NOT sv_execute_ddl_rust)
       c. Execute the rewritten SQL on `sv_ddl_conn` via `duckdb_query`
       d. Read the result metadata: `duckdb_column_count`, `duckdb_column_name`, `duckdb_column_type`
       e. Map DuckDB column types to LogicalTypes and push into `return_types` and `names`
       f. Read all result rows using `duckdb_value_varchar` for each column (convert everything to VARCHAR for simplicity -- the underlying functions already return VARCHAR columns anyway)
       g. Store rows in the new bind data struct
       h. Call `duckdb_destroy_result` to clean up

    4. Refactor `sv_ddl_execute`:
       a. Track current row index in `SvDdlGlobalState`
       b. Emit rows from the stored data, setting cardinality per chunk
       c. Handle the case where there are 0 result rows (e.g., DROP IF EXISTS on nonexistent)

    5. Refactor `SvDdlGlobalState` to track `idx_t offset` instead of `bool done`.

    **Design note:** Converting all columns to VARCHAR simplifies the C++ code significantly. The underlying describe_semantic_view() and list_semantic_views() functions already return VARCHAR columns. The mutating DDL functions (create_semantic_view, drop_semantic_view) also return VARCHAR. So VARCHAR forwarding is lossless for all 7 DDL forms. If future DDL functions return non-VARCHAR types, this can be revisited.

    **Backward compatibility:** The mutating DDL forms (DROP, CREATE OR REPLACE, etc.) will now return their function's actual output columns instead of a synthetic "view_name" column. For `drop_semantic_view('x')`, this returns one row with `view_name` VARCHAR = "x". For `create_semantic_view(...)`, this returns one row with `view_name` VARCHAR = "name". The end-user output is functionally identical -- they see the same information. The column name may change from the hardcoded "view_name" to whatever the function returns, but this is a cosmetic improvement (accurate column names).

    **Anti-patterns to avoid:**
    - Do NOT re-execute the SQL twice (once in Rust, once in C++) -- execute ONLY in C++ via duckdb_query
    - Do NOT modify sv_parse_stub or sv_plan_function -- the parse/plan path is unchanged
    - Do NOT add new connections -- use existing sv_ddl_conn
    - Do NOT use DuckDB C++ internal APIs (ClientContext, etc.) -- use the C API (duckdb_query, duckdb_value_varchar, etc.) which is available through the amalgamation
  </action>
  <verify>
    <automated>cd /Users/paul/Documents/Dev/Personal/duckdb-semantic-views && cargo test && just build</automated>
  </verify>
  <done>
    - sv_rewrite_ddl_rust FFI function exists in src/parse.rs
    - sv_ddl_bind calls sv_rewrite_ddl_rust then duckdb_query, forwarding result schema dynamically
    - sv_ddl_execute emits all rows from stored result data
    - cargo test passes (Rust unit tests for rewrite)
    - just build succeeds (C++ compiles)
  </done>
</task>

<task type="auto">
  <name>Task 2: Add DESCRIBE and SHOW integration tests and verify full suite</name>
  <files>test/sql/phase20_extended_ddl.test</files>
  <action>
    Append to the existing test/sql/phase20_extended_ddl.test file (created in Plan 01) with tests for DDL-07 and DDL-08. Add these AFTER the existing Plan 01 tests but BEFORE the cleanup section.

    DDL-07 (DESCRIBE SEMANTIC VIEW):
    - Setup: Create a view `CREATE SEMANTIC VIEW desc_test (tables := [...], dimensions := [...], metrics := [...])`
    - `DESCRIBE SEMANTIC VIEW desc_test` should return a row with columns including `name` = 'desc_test' and `base_table` = source table name
    - Verify with: `query TT` selecting name and base_table from the DESCRIBE result
    - Also test case insensitivity: `describe semantic view desc_test`
    - Error case: `DESCRIBE SEMANTIC VIEW nonexistent` should error with `does not exist`
    - Verify normal DESCRIBE still works: `DESCRIBE sv_test` (the source table) should NOT be intercepted
    - Clean up: `DROP SEMANTIC VIEW desc_test`

    DDL-08 (SHOW SEMANTIC VIEWS):
    - Setup: Create two views for listing
    - `SHOW SEMANTIC VIEWS` should return rows with (name, base_table) columns
    - Verify with: `query TT rowsort` showing both views
    - Also test case insensitivity: `show semantic views`
    - Verify normal SHOW works: `SHOW TABLES` should NOT be intercepted
    - Clean up: DROP both views

    Full lifecycle test:
    - CREATE SEMANTIC VIEW lifecycle_test (...)
    - DESCRIBE SEMANTIC VIEW lifecycle_test -- verify it exists
    - SHOW SEMANTIC VIEWS -- verify it appears in the list
    - CREATE OR REPLACE SEMANTIC VIEW lifecycle_test (...new metrics...)
    - DESCRIBE SEMANTIC VIEW lifecycle_test -- verify updated definition
    - DROP SEMANTIC VIEW lifecycle_test
    - DESCRIBE SEMANTIC VIEW lifecycle_test -- should error
    - SHOW SEMANTIC VIEWS -- should not include lifecycle_test

    Run the full test suite: `just build && just test-all`
  </action>
  <verify>
    <automated>cd /Users/paul/Documents/Dev/Personal/duckdb-semantic-views && just build && just test-all</automated>
  </verify>
  <done>
    - DESCRIBE SEMANTIC VIEW returns correct columns (name, base_table, JSON fields) (DDL-07)
    - SHOW SEMANTIC VIEWS lists all defined views (DDL-08)
    - Case insensitive detection works for DESCRIBE and SHOW
    - Normal DESCRIBE and SHOW are not intercepted
    - Error on nonexistent view for DESCRIBE
    - Full lifecycle test passes (create -> describe -> show -> replace -> describe -> drop -> verify gone)
    - `just test-all` passes (all Rust + sqllogictest + DuckLake CI)
    - All existing tests (phase2_ddl.test, phase16_parser.test) still pass
  </done>
</task>

</tasks>

<verification>
- `cargo test` -- all Rust unit tests pass
- `just build && just test-sql` -- all sqllogictest files pass including updated phase20_extended_ddl.test
- `just test-all` -- full suite green
- `DESCRIBE SEMANTIC VIEW x` returns the describe_semantic_view output columns
- `SHOW SEMANTIC VIEWS` returns the list_semantic_views output columns
- `DROP SEMANTIC VIEW x`, `CREATE OR REPLACE`, `CREATE IF NOT EXISTS` still work after the C++ refactor
- Existing phase16_parser.test and phase2_ddl.test still pass unchanged
</verification>

<success_criteria>
- User can run `DESCRIBE SEMANTIC VIEW name` and see dimensions, metrics, and types (DDL-07)
- User can run `SHOW SEMANTIC VIEWS` and see all defined semantic views (DDL-08)
- All 6 DDL requirements (DDL-03 through DDL-08) work end-to-end via native SQL syntax
- `just test-all` passes with zero failures
</success_criteria>

<output>
After completion, create `.planning/phases/20-extended-ddl-statements/20-02-SUMMARY.md`
</output>
