---
phase: 62
plan: 03
type: execute
wave: 2
depends_on: ["62-02"]
files_modified:
  - src/parse.rs
  - cpp/src/shim.cpp
  - src/lib.rs
autonomous: true
requirements: []
notes: |
  Wave 2 — Caret restoration. Re-introduce `parse_function` (renamed from
  legacy) purely as the error-reporting layer. parser_override keeps the success
  path (transactional rewrite + re-parse). For every error case, parser_override
  defers (DISPLAY_ORIGINAL_ERROR); the default parser fails on the unrecognized
  prefix; DuckDB calls `parse_function`; `parse_function` re-runs validation and
  returns DISPLAY_EXTENSION_ERROR with `error_location` byte offset.
  ParserException::SyntaxError formats `LINE 1: ... ^` for free.

  Per RESEARCH §2 Q1: ParseError::position is ALREADY a byte offset into the
  user's input — pass straight through to ParserExtensionParseResult::error_location.

must_haves:
  truths:
    - "parser_override returns DISPLAY_ORIGINAL_ERROR for ALL error cases (no more synthesised SELECT error)"
    - "src/parse.rs no longer contains `sql_throwing` function"
    - "src/parse.rs exports `sv_parse_function_rust` returning rc=0/1/2/3"
    - "cpp/src/shim.cpp registers `parse_function = sv_parse_stub` AND `plan_function = sv_plan_unreachable`"
    - "cpp/src/shim.cpp `sv_parse_stub` writes `result.error_location = position` for rc=1 / rc=3 with valid position"
    - "Caret rendering works: malformed CREATE produces `Parser Error: ... LINE 1: ... ^` with caret at the offending token"
    - "rc=3 actionable error fires for `disable_peg_parser` + CREATE without re-arming FALLBACK"
    - "All v0.8.x transactional behaviour preserved — `just test-all` green"
  artifacts:
    - path: "src/parse.rs"
      provides: "sv_parse_function_rust (rc=0/1/2/3); sql_throwing deleted; parser_override error path returns rc=2"
      contains: "sv_parse_function_rust"
    - path: "cpp/src/shim.cpp"
      provides: "sv_parse_stub + sv_plan_unreachable + SemanticViewParseData; parse_function registered alongside parser_override"
      contains: "sv_parse_stub"
  key_links:
    - from: "DuckDB Parser::ParseQuery default-parse failure"
      to: "sv_parse_stub"
      via: "registered ext.parse_function"
      pattern: "ext\\.parse_function = sv_parse_stub"
    - from: "sv_parse_stub"
      to: "ParserException::SyntaxError caret rendering"
      via: "ParserExtensionParseResult{error, error_location}"
      pattern: "result\\.error_location = position"
    - from: "sv_parse_function_rust"
      to: "ParseError::position byte offset"
      via: "validate_and_rewrite (unchanged)"
      pattern: "ParseError.*position|position_out"
---

<objective>
Restore `LINE 1: … ^` caret rendering for all CREATE/DROP/ALTER validation
errors WITHOUT regressing the v0.8.x transactional DDL pipeline. The mechanism:

1. parser_override KEEPS the success path (rewrite to native SQL, re-parse on
   caller's connection — every transactional win preserved).
2. parser_override now returns DISPLAY_ORIGINAL_ERROR for ALL error cases
   (rc=2 from Rust). The synthesised `SELECT error('...')` workaround is gone.
3. DuckDB's default parser fails on the unrecognized DDL prefix.
4. DuckDB calls `parse_function` (newly registered); our `sv_parse_stub`
   re-runs validation, returns DISPLAY_EXTENSION_ERROR with `error_location =
   ParseError::position` (byte offset into user input).
5. `ParserException::SyntaxError(query, msg, error_location)` formats the
   caret automatically.

This resolves TECH-DEBT 22.

Q1 confirmed by RESEARCH: `validate_and_rewrite` already tracks positions in
the user's input string — no remapping needed. Pass `ParseError::position`
straight through.

rc encoding (sv_parse_function_rust):
- 0 — success/unreachable (parser_override should have produced rewritten SQL; defensive fallback returns rc=1 with internal-error)
- 1 — ours-but-invalid (malformed DDL, near-miss with suggestion); message + position written
- 2 — not-ours (defer to DuckDB's default error)
- 3 — valid-but-override-disabled (e.g. `disable_peg_parser` + CREATE w/o re-arming FALLBACK); actionable hint + position=0

Purpose: TECH-DEBT 22 resolved; user sees parser-quality errors with caret.

Output: parse_function registered + sv_parse_stub + new Rust FFI; sql_throwing
deleted; caret tests in test_caret_position.py B1/B2/B3 produce non-None
caret columns when run by Plan 04.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/STATE.md
@.planning/phases/62-caret-restoration-lru-removal/62-RESEARCH.md
@.planning/phases/62-caret-restoration-lru-removal/62-VALIDATION.md
@_notes/v0.8.0_phase_62_ultraplan.md
@CLAUDE.md
@TECH-DEBT.md

<interfaces>
<!-- ParserExtensionParseResult constructors (cpp/include/parser_extension_compat.hpp:67-87) -->
```cpp
struct ParserExtensionParseResult {
    ParserExtensionParseResult();                                                   // DISPLAY_ORIGINAL_ERROR
    explicit ParserExtensionParseResult(string error_p);                            // DISPLAY_EXTENSION_ERROR
    explicit ParserExtensionParseResult(unique_ptr<ParserExtensionParseData> data); // PARSE_SUCCESSFUL
    string error;
    unique_ptr<ParserExtensionParseData> parse_data;
    ParserExtensionResultType type;
    optional_idx error_location;  // public; assignable post-construction
};
typedef ParserExtensionParseResult (*parse_function_t)(ParserExtensionInfo *info, const string &query);
typedef ParserExtensionPlanResult  (*plan_function_t) (ParserExtensionInfo *info, ClientContext &context, ...);
```

<!-- New Rust FFI signature -->
```rust
#[no_mangle]
pub unsafe extern "C" fn sv_parse_function_rust(
    ctx_ptr: *const std::ffi::c_void,
    query_ptr: *const u8, query_len: usize,
    error_out: *mut u8, error_out_len: usize,
    position_out: *mut u32,
) -> u8;  // 0=success-unreachable, 1=ours-invalid, 2=not-ours, 3=valid-but-override-disabled
```

<!-- ParseError already has the right shape (src/errors.rs:11-16) -->
```rust
pub struct ParseError {
    pub message: String,
    /// 0-based byte offset into the original query string.
    pub position: Option<usize>,
}
```

<!-- detect_near_miss + detect_ddl_kind helpers (src/parse.rs:946, src/parse.rs around 936) -->
```rust
fn detect_near_miss(query: &str) -> Option<NearMissHit>;
fn detect_ddl_kind(query: &str) -> Option<DdlKind>;
```
</interfaces>
</context>

<tasks>

<task type="auto" tdd="true">
  <name>Task 1: Add `sv_parse_function_rust` (rc=0/1/2/3) + delete `sql_throwing`; parser_override error path returns rc=2</name>
  <files>src/parse.rs</files>
  <behavior>
    Three new unit tests pin the rc encoding contract (B9, B10, B11 from RESEARCH §6):

    - `parse::tests::sv_parse_function_rust_returns_2_for_select` — input `"SELECT 1;"` returns rc=2 (not ours; defer to DuckDB's default parser).
    - `parse::tests::sv_parse_function_rust_returns_2_for_invalid_utf8` — input is invalid UTF-8 bytes; returns rc=2 (defer rather than panic).
    - `parse::tests::sv_parse_function_rust_returns_1_with_position_for_malformed_create` — input `"CREATE SEMANTIC VIEW v AS TABLES "` (missing `(`) returns rc=1; `position_out` is set to a non-MAX value pointing at the byte offset of the validation failure; `error_out` contains the validator's message text.
    - `parse::tests::sv_parse_function_rust_returns_1_for_near_miss` — input `"CRETAE SEMANTIC VIEW v AS TABLES (t);"` returns rc=1 with suggestion text containing `Did you mean` and position=0 (start of CRETAE).
    - `parse::tests::sv_parser_override_rust_returns_2_for_validation_failure` — input is malformed CREATE; with the rewritten override path, the error branch now returns rc=2 (was rc=1 with synthesised SQL via `sql_throwing` in v0.8.x). This is the change that lets parse_function take over.

    Existing tests preserved: `validate_and_rewrite_*` proptests, the round-trip tests added in Plan 02, and any `sv_parser_override_rust_*` tests that exercised the success path (rc=0 with rewritten SQL).
  </behavior>
  <read_first>
    - src/parse.rs lines 1620–1700 (`write_error_to_buffer` at line 1631 — currently `#[allow(dead_code)]`; becomes the live error-emit path)
    - src/parse.rs lines ~2540–2620 (current `sv_parser_override_rust` + `sql_throwing` to delete)
    - src/parse.rs lines 940–1000 (`detect_near_miss`, `detect_ddl_kind` if present, `validate_and_rewrite`)
    - src/errors.rs lines 11–16 (`ParseError::position` byte-offset contract)
    - .planning/phases/62-caret-restoration-lru-removal/62-RESEARCH.md §2 Q1 (positions are user-input byte offsets — pass through)
    - .planning/phases/62-caret-restoration-lru-removal/62-RESEARCH.md §2 Q3 (rc=3 actionable error vector list — message text)
    - .planning/phases/62-caret-restoration-lru-removal/62-RESEARCH.md §3 Risk C (rc=3 should set position=0)
    - .planning/phases/62-caret-restoration-lru-removal/62-RESEARCH.md §4 Refinement 2 (rc=3 wording)
    - _notes/v0.8.0_phase_62_ultraplan.md "src/parse.rs" section steps 4–7
  </read_first>
  <action>
    Sequence:

    1. **Add `sv_parse_function_rust` FFI entry** (place near `sv_parser_override_rust`):
       ```rust
       /// Phase 62 — sole purpose is error reporting with caret rendering.
       /// Called by DuckDB's `parse_function` after the default parser fails on
       /// an unrecognized prefix. Re-runs validation and returns rc:
       ///   0 — success/unreachable (defensive — see RESEARCH §3)
       ///   1 — ours-but-invalid; error_out + position_out populated
       ///   2 — not-ours; defer (DISPLAY_ORIGINAL_ERROR on the C++ side)
       ///   3 — valid DDL but parser_override didn't fire (override setting disabled);
       ///       actionable hint written + position_out = 0
       #[no_mangle]
       pub unsafe extern "C" fn sv_parse_function_rust(
           ctx_ptr: *const std::ffi::c_void,
           query_ptr: *const u8, query_len: usize,
           error_out: *mut u8, error_out_len: usize,
           position_out: *mut u32,
       ) -> u8 {
           std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
               // ctx_ptr is currently unused on this path (validation does not
               // need the catalog) but we keep it in the signature for symmetry
               // with sv_parser_override_rust and future-proofing.
               let _ = ctx_ptr;

               // Initialize position_out to UINT32_MAX (no-position sentinel).
               if !position_out.is_null() {
                   *position_out = u32::MAX;
               }

               // UTF-8 check; on failure defer rather than synthesise an error.
               let bytes = std::slice::from_raw_parts(query_ptr, query_len);
               let query = match std::str::from_utf8(bytes) {
                   Ok(s) => s,
                   Err(_) => return 2_u8,
               };

               // Is this a DDL we recognize at the prefix level?
               if detect_ddl_kind(query).is_none() {
                   // Not a recognized DDL prefix — try near-miss detection.
                   if let Some(hit) = detect_near_miss(query) {
                       write_error_to_buffer(error_out, error_out_len, &hit.message);
                       if !position_out.is_null() {
                           *position_out = u32::try_from(hit.position).unwrap_or(u32::MAX);
                       }
                       return 1_u8;
                   }
                   return 2_u8;  // not ours
               }

               // Recognized DDL prefix — re-run validation.
               match validate_and_rewrite(query) {
                   Ok(Some(_rewritten)) => {
                       // Valid DDL but we got here — parser_override must not have fired.
                       // This is the rc=3 actionable-error path. Position=0 (caret on first char).
                       let msg = "semantic_views: parser_override is not active for \
                                  this connection (allow_parser_override_extension is \
                                  'DEFAULT' or 'STRICT'). Re-enable with: \
                                  SET allow_parser_override_extension='FALLBACK';";
                       write_error_to_buffer(error_out, error_out_len, msg);
                       if !position_out.is_null() {
                           *position_out = 0;
                       }
                       3_u8
                   }
                   Ok(None) => {
                       // detect_ddl_kind matched but validate_and_rewrite returned None —
                       // unreachable for a matched prefix. Defensive: emit internal error.
                       write_error_to_buffer(
                           error_out, error_out_len,
                           "semantic_views: internal error — recognized prefix produced no rewrite",
                       );
                       1_u8
                   }
                   Err(parse_err) => {
                       write_error_to_buffer(error_out, error_out_len, &parse_err.message);
                       if !position_out.is_null() {
                           *position_out = parse_err.position
                               .and_then(|p| u32::try_from(p).ok())
                               .unwrap_or(u32::MAX);
                       }
                       1_u8
                   }
               }
           })).unwrap_or(2)
       }
       ```

    2. **Rewrite `sv_parser_override_rust` error branch** — currently rc=1 with `sql_throwing`-synthesised SELECT. Change to rc=2:
       ```rust
       // Was (v0.8.x):
       //     Err(err) => {
       //         let sql = sql_throwing(&err.message);
       //         /* publish sql; */ return 0;
       //     }
       // Now (Phase 62):
       Err(_err) => {
           // Defer to default parser → parse_function produces caret-rendered error.
           // The error message will be re-derived by sv_parse_function_rust.
           return 2;
       }
       ```

       Same change for `Ok(None)` from validate (truly not ours): already rc=2.

       The `error_out` parameter on `sv_parser_override_rust` is now unused on every code path. KEEP the parameter in the signature for FFI compatibility with the unchanged C++ caller (Plan 02 left it). A future cleanup can drop it; not in scope here.

    3. **Delete `sql_throwing`** (lines ~2602–2620): the entire function body and its call sites in `sv_parser_override_rust` go away. Once removed, `cargo clippy --no-default-features` should not surface a `dead_code` warning for it.

    4. **`write_error_to_buffer`** (line 1631): remove the `#[allow(dead_code)]` attribute — it now has live callers in `sv_parse_function_rust` (and was already used by `sv_parser_override_rust` for the catalog-evicted error in v0.8.1, which Plan 02 deleted; reconfirm no other dead_code attributes remain).

    5. **Add the 5 unit tests** described in `<behavior>`. They should be `#[test]` functions in the existing `mod tests { ... }` block of `src/parse.rs`. Use raw `unsafe` calls to `sv_parse_function_rust` and `sv_parser_override_rust` with stack-allocated buffers, matching the style of any existing FFI round-trip tests added in Plan 02.

    6. **detect_near_miss return shape** — verify it carries position. If the existing struct doesn't have `position`, expose it (e.g. add `pub position: usize`) and populate at the existing call sites. Per RESEARCH §2 Q1: "`detect_near_miss` returns `position: Some(trim_offset)`" — confirm in code and adapt accordingly.

    7. **Public re-export** — confirm `validate_and_rewrite` is reachable from `tests/parse_proptest.rs` (Plan 01 added a proptest using it); if not already, add a `pub` or `#[cfg(test)] pub` re-export in `src/lib.rs`.
  </action>
  <verify>
    <automated>cd /Users/paul/Documents/Dev/Personal/duckdb-semantic-views && cargo test --no-default-features parse:: 2>&1 | tail -30</automated>
  </verify>
  <acceptance_criteria>
    - `rg "fn sql_throwing" src/parse.rs` returns ZERO hits
    - `src/parse.rs` contains `pub unsafe extern \"C\" fn sv_parse_function_rust`
    - `src/parse.rs` `sv_parse_function_rust` body contains `if detect_ddl_kind(query).is_none()` AND `validate_and_rewrite(query)` AND `*position_out =`
    - `src/parse.rs` `sv_parse_function_rust` rc=3 path contains the substring `SET allow_parser_override_extension='FALLBACK'`
    - `src/parse.rs` `sv_parser_override_rust` `Err(_)` arm returns `2` (NOT calls `sql_throwing`)
    - `cargo test sv_parse_function_rust_returns_2_for_select` exits 0
    - `cargo test sv_parse_function_rust_returns_2_for_invalid_utf8` exits 0
    - `cargo test sv_parse_function_rust_returns_1_with_position_for_malformed_create` exits 0
    - `cargo test sv_parse_function_rust_returns_1_for_near_miss` exits 0
    - `cargo test sv_parser_override_rust_returns_2_for_validation_failure` exits 0
    - `cargo test --no-default-features` exits 0 (full suite stays green)
  </acceptance_criteria>
  <done>sv_parse_function_rust shipped with full rc=0/1/2/3 contract; sql_throwing deleted; parser_override defers all errors to rc=2; 5 new unit tests green.</done>
</task>

<task type="auto">
  <name>Task 2: Add `sv_parse_stub` + `sv_plan_unreachable` + `SemanticViewParseData` to C++ shim; register parse_function alongside parser_override</name>
  <files>cpp/src/shim.cpp</files>
  <read_first>
    - cpp/src/shim.cpp full file as it stands after Plan 02 (lines 1–211, post-Wave-1 edits — `SemanticViewsParserInfo` holds `rust_state`, `sv_register_parser_hooks` takes `(db_handle, catalog_conn, is_file_backed)`)
    - cpp/include/parser_extension_compat.hpp lines 67–155 (full ParserExtension declaration — including parse_function, plan_function, parser_override fields)
    - cpp/include/duckdb.cpp lines 347279–347304 (`ParserExtension::Parse` invocation flow — confirms parse_function is called after default parser fails)
    - .planning/phases/62-caret-restoration-lru-removal/62-RESEARCH.md §3 Risk F (static_assert was added in Plan 01 Task 1)
    - _notes/v0.8.0_phase_62_ultraplan.md "cpp/src/shim.cpp" steps 1, 3, 4, 5, 6
  </read_first>
  <action>
    Sequence:

    1. **Restore `SemanticViewParseData : ParserExtensionParseData`** structurally (the type `parse_function` produces on PARSE_SUCCESSFUL — but we never return that variant, so it's structurally needed only for layout / type-system reasons):
       ```cpp
       struct SemanticViewParseData : public ParserExtensionParseData {
           string query;
           explicit SemanticViewParseData(string q) : query(std::move(q)) {}

           unique_ptr<ParserExtensionParseData> Copy() const override {
               return make_uniq<SemanticViewParseData>(query);
           }
           string ToString() const override {
               return query;
           }
       };
       ```

       Place near the top of the anonymous namespace, alongside `SvOwnedBuffer`.

    2. **Add forward declaration** for the new Rust FFI entry (alongside the existing `extern "C"` block):
       ```cpp
       extern "C" {
           // ... existing (sv_make_override_context, sv_drop_override_context, sv_parser_override_rust, sv_free_buffer) ...

           uint8_t sv_parse_function_rust(
               const void *ctx_ptr,
               const char *query_ptr, size_t query_len,
               char *error_buf, size_t error_buf_len,
               uint32_t *position_out);
       }
       ```

    3. **Add `sv_parse_stub`** (the parse_function callback):
       ```cpp
       static ParserExtensionParseResult sv_parse_stub(
           ParserExtensionInfo *info, const string &query) {
           auto *sv_info = dynamic_cast<SemanticViewsParserInfo *>(info);
           if (!sv_info || !sv_info->rust_state) {
               return ParserExtensionParseResult();  // DISPLAY_ORIGINAL_ERROR
           }

           char error_buf[1024];
           memset(error_buf, 0, sizeof(error_buf));
           uint32_t position = UINT32_MAX;

           uint8_t rc = sv_parse_function_rust(
               sv_info->rust_state,
               query.c_str(), query.size(),
               error_buf, sizeof(error_buf),
               &position);

           switch (rc) {
               case 2:  // not-ours
                   return ParserExtensionParseResult();  // DISPLAY_ORIGINAL_ERROR

               case 1:  // ours-but-invalid
               case 3: {  // valid-but-override-disabled (actionable hint)
                   ParserExtensionParseResult result(string(error_buf));
                   if (position != UINT32_MAX) {
                       result.error_location = optional_idx(position);
                   }
                   return result;
               }

               case 0: {
                   // Defensive — should be unreachable. parser_override would have
                   // produced rewritten SQL on the success path. If we land here,
                   // the override didn't fire AND validation succeeded — equivalent
                   // to rc=3 but with a different message.
                   ParserExtensionParseResult result(string(
                       "semantic_views: internal error — parse_function reached \
                        success branch (please report this bug)"));
                   return result;
               }

               default: {
                   // Defensive — unknown rc. Display as extension error so the
                   // user sees something rather than the default parser's
                   // syntax error.
                   ParserExtensionParseResult result(string(
                       "semantic_views: internal error — unknown rc from \
                        sv_parse_function_rust"));
                   return result;
               }
           }
       }
       ```

    4. **Add `sv_plan_unreachable`** (required because parse_function has a sibling plan_function pointer in `ParserExtension`, even though sv_parse_stub never returns PARSE_SUCCESSFUL so plan_function should not run):
       ```cpp
       static ParserExtensionPlanResult sv_plan_unreachable(
           ParserExtensionInfo * /*info*/,
           ClientContext & /*context*/,
           unique_ptr<ParserExtensionParseData> /*parse_data*/) {
           throw InternalException(
               "semantic_views: sv_plan_unreachable called — parse_function \
                should never return PARSE_SUCCESSFUL");
       }
       ```

       (NOTE: signature must match `plan_function_t` in parser_extension_compat.hpp. If it differs from the above guess, match it exactly. `ParserExtensionPlanResult` may be `void` or a struct — check the header.)

    5. **Update `sv_register_parser_hooks`** to register all three callbacks:
       ```cpp
       ParserExtension ext;
       ext.parser_override = sv_parser_override;
       ext.parse_function  = sv_parse_stub;          // NEW — Phase 62
       ext.plan_function   = sv_plan_unreachable;    // NEW — required sibling
       std::shared_ptr<ParserExtensionInfo> info_std(
           new SemanticViewsParserInfo(rust_state));
       ext.parser_info = duckdb::shared_ptr<ParserExtensionInfo>(info_std);
       ```

    6. **Update the `sv_parser_override` rc=1 branch** — Plan 02 left it intact for a working interim; now that parse_function takes over error reporting, parser_override should:
       - Return `ParserOverrideResult()` (DISPLAY_ORIGINAL_ERROR) for rc=2 (unchanged).
       - Return `ParserOverrideResult()` for rc=1 too (NEW — defer to default parser → parse_function will produce the caret-rendered error).
       - rc=0 success path unchanged.

       Specifically:
       ```cpp
       if (rc == 2) {
           return ParserOverrideResult();
       }
       if (rc == 1) {
           // Phase 62: defer to default parser; sv_parse_stub will re-derive
           // the error and return DISPLAY_EXTENSION_ERROR with caret position.
           return ParserOverrideResult();
       }
       // rc == 0: rewritten SQL, re-parse on default parser.
       // ... existing logic ...
       ```

       Per Plan 03 Task 1, the Rust side now ALWAYS returns rc=2 on the error path, so the rc=1 branch in C++ is defensive — but it must still match the contract.
  </action>
  <verify>
    <automated>cd /Users/paul/Documents/Dev/Personal/duckdb-semantic-views && just build 2>&1 | tail -20 && just test-sql 2>&1 | tail -15</automated>
  </verify>
  <acceptance_criteria>
    - `cpp/src/shim.cpp` contains `struct SemanticViewParseData`
    - `cpp/src/shim.cpp` contains `static ParserExtensionParseResult sv_parse_stub`
    - `cpp/src/shim.cpp` contains `static ParserExtensionPlanResult sv_plan_unreachable` (or equivalent matching the typedef)
    - `cpp/src/shim.cpp` contains `ext.parse_function = sv_parse_stub`
    - `cpp/src/shim.cpp` contains `ext.plan_function = sv_plan_unreachable`
    - `cpp/src/shim.cpp` `sv_parse_stub` body contains `result.error_location = optional_idx(position)` (or equivalent assignment with type-safe construction)
    - `just build` exits 0
    - `just test-sql` exits 0 (NOTE: error TEXT may now arrive as `Parser Error:` prefix instead of `Invalid Input Error:` — per spike report this is non-breaking for existing tests since matchers are substring; if any test breaks, halt and assess)
  </acceptance_criteria>
  <done>parse_function registered alongside parser_override; sv_parse_stub passes rc=1/3 errors with caret position; just build green; sqllogictest green.</done>
</task>

<task type="auto">
  <name>Task 3: Manual caret smoke + full quality gate run</name>
  <files>src/lib.rs</files>
  <read_first>
    - src/lib.rs (post-Wave-1 state — ensure no leftover LRU references; nothing should change here unless a regression surfaces)
    - test/integration/test_caret_position.py (the 3 existing tests — currently they assert message text; per Plan 01 Task 3 they ALSO call `extract_caret_position` but skip if it returns None. After Plan 03 lands, the skip should NO LONGER be hit — the caret IS present.)
    - .planning/phases/62-caret-restoration-lru-removal/62-VALIDATION.md "Manual-Only Verifications" section
  </read_first>
  <action>
    This task is a verification-and-glue task, not new code (assuming the above
    two tasks land cleanly). Steps:

    1. **Re-run quality gate end-to-end:**
       - `cargo test --no-default-features` — all unit tests + proptests green.
       - `just build` — extension binary builds.
       - `just test-sql` — sqllogictest green. Pay special attention to:
         - `test/sql/v080_transactional_ddl.test` — the BEGIN/CREATE/ROLLBACK + INSERT OR REPLACE row-count cases must remain byte-identical.
         - `test/sql/peg_compat.test` — the existing "re-arm FALLBACK after disable_peg_parser" workaround case still works (Plan 04 will add a complementary test for the rc=3 actionable error).
       - `just test-caret` — IMPORTANT: the 3 existing tests should now produce an error with caret rendering. Per the Wave 0 staging in Plan 01 Task 3, those tests skip if `extract_caret_position` returns None. With Phase 62 caret restored, the helper should now return a non-None column. Plan 04 Task 1 flips the skip to `assert ... is not None`; for Wave 2 we just confirm the TEST DOES NOT SKIP (i.e. caret IS there). If it still skips, the integration is broken — debug before continuing.
       - `just test-adbc` — transactional DDL via ADBC unchanged.
       - `just test-multi-db` — pre-existing 2-DB test passes.
       - `just test-concurrent` — race shape preserved.
       - `just test-large-view` — large rewritten DDL still works.
       - `just test-vtab-crash` — unchanged.

    2. **Manual caret smoke** (per VALIDATION.md "Manual-Only Verifications"):
       Run an interactive Python session (no CLI is shipped per project notes — use Python `duckdb` package):
       ```python
       import duckdb, os
       conn = duckdb.connect(config={"allow_unsigned_extensions": "true",
                                      "extension_directory": "build/debug"})
       conn.execute("FORCE INSTALL 'build/debug/semantic_views.duckdb_extension'")
       conn.execute("LOAD semantic_views")
       try:
           conn.execute("CREATE SEMANTIC VIEW bad AS TABLES (missing_table) DIMENSIONS (TBLES x);")
       except Exception as e:
           print(repr(e))  # Confirm "Parser Error: ... LINE 1: ... ^" with caret near TBLES
       ```

       Confirm visually that:
       - The error string starts with `Parser Error:` (was `Invalid Input Error:` in v0.8.1).
       - The string contains `LINE 1: ` followed by the user's input.
       - The next line contains a `^` aligned with the column where TBLES (or other offending token) starts.

       Capture the output and paste into `62-03-SUMMARY.md` for evidence.

    3. **If any test fails, debug:**
       - Most likely: `position` byte-offset off-by-one. Compare `ParseError::position` against the expected token offset.
       - Possible: `optional_idx` construction wrong in C++. Verify `optional_idx(uint32_t)` is the correct constructor — it may need `optional_idx(idx_t)` or similar; check `cpp/include/duckdb.cpp`.
       - Possible: parse_function not being called because parser_override returned `PARSE_SUCCESSFUL` somehow. Add `fprintf(stderr, ...)` in `sv_parse_stub` to confirm invocation, then remove.

    4. **Verify no production-code regressions** by running:
       ```bash
       just test-all 2>&1 | tee /tmp/wave2_test_all.log; echo "exit=$?"
       ```

       Then `tail -50 /tmp/wave2_test_all.log` to inspect the final summary.
  </action>
  <verify>
    <automated>cd /Users/paul/Documents/Dev/Personal/duckdb-semantic-views && just test-all 2>&1 | tee /tmp/wave2_test_all.log | tail -30; echo "EXIT=$?"</automated>
  </verify>
  <acceptance_criteria>
    - `just test-all` exits 0
    - `just test-caret` runs and the 3 existing tests do NOT take the `pytest.skip("...not yet restored")` branch (i.e. `extract_caret_position` returns non-None for malformed DDL)
    - Manual smoke shows `Parser Error: ... LINE 1: ... ^` rendering with caret aligned to offending token
    - `rg "Invalid Input Error.*semantic" /tmp/wave2_test_all.log` returns ZERO hits (the runtime-error wrapper class is gone)
    - `rg "sql_throwing" src/` returns ZERO hits
    - `rg "parser_override_catalog" src/ cpp/src/` returns ZERO hits
  </acceptance_criteria>
  <done>Caret rendering restored end-to-end; manual smoke verified; full quality gate green; TECH-DEBT 22 functionally resolved.</done>
</task>

</tasks>

<threat_model>
## Trust Boundaries

| Boundary | Description |
|----------|-------------|
| User input → `sv_parse_function_rust` | UTF-8 validation; defer (rc=2) on invalid bytes rather than crash. Existing FFI UTF-8 validation from Phase 60 covers the parser_override path; this same protection extends to parse_function path via the same `std::str::from_utf8` check. |
| `position` byte offset → C++ `error_location` field → DuckDB caret rendering | Position is a `u32` (was `usize` in `ParseError::position`); cast saturates to UINT32_MAX on overflow. DuckDB's `ParserException::SyntaxError` rendering is responsible for bounds-checking the position against the query length — outside our trust boundary. |

## STRIDE Threat Register

| Threat ID | Category | Component | Disposition | Mitigation Plan |
|-----------|----------|-----------|-------------|-----------------|
| T-62-W2-01 | Tampering | `error_location` byte offset → out-of-bounds caret render | mitigate | Position is derived from `ParseError::position` which is bounds-checked at the validator level (it's an offset into the input string the validator just read). UINT32_MAX sentinel signals "no position" — DuckDB skips caret rendering. |
| T-62-W2-02 | DoS | Crafted input causing repeated re-validation between parser_override → default parser → parse_function | accept | Worst case is 2x validation cost on error paths (parser_override defers, parse_function re-validates). The user error case is rare; success path runs validation exactly once. No amplification factor exceeds 2. |
| T-62-W2-03 | Information disclosure | `error_buf` content from `validate_and_rewrite` errors | accept | Validator messages contain user-supplied identifiers. This is identical to v0.8.x message exposure (synthesised SELECT error already echoed user input). No regression in disclosure surface. |
| T-62-W2-04 | Spoofing | Forged `ctx_ptr` causing arbitrary memory access | mitigate | `sv_parse_function_rust` does not currently dereference `ctx_ptr` (validation does not need the catalog). The `let _ = ctx_ptr;` suppression is intentional — defense in depth via non-use. |

No high-severity threats. Phase modifies parser hooks and error-reporting path; threat surface is small (interior architecture, no new untrusted input paths beyond what v0.8.x already handled).
</threat_model>

<verification>
- `cargo test --no-default-features` exits 0 (5 new sv_parse_function_rust tests + sv_parser_override_rust deferral test green)
- `just build` exits 0
- `just test-sql` exits 0 — including v080_transactional_ddl + peg_compat byte-identical
- `just test-caret` exits 0 — 3 existing tests do NOT skip; caret rendering active
- `just test-adbc` exits 0
- `just test-multi-db` exits 0
- `just test-concurrent` exits 0
- `just test-all` exits 0
- Manual smoke: `Parser Error: ... LINE 1: ... ^` rendering visible and aligned
- Grep guards: zero hits for `sql_throwing`, `parser_override_catalog`, `db_token`, `set_catalog_for_parser_override`
</verification>

<success_criteria>
- Caret rendering restored: `LINE 1: … ^` visible for CREATE/DROP/ALTER validation errors
- TECH-DEBT 22 functionally resolved (Plan 04 marks it ✅ in TECH-DEBT.md)
- TECH-DEBT 20 functionally resolved (LRU module is gone since Plan 02; no new code introduces it)
- All v0.8.x transactional behaviour preserved (sqllogictest + ADBC + concurrent green)
- rc=3 actionable error path implemented and ready for Plan 04 sqllogictest fixture
- `sql_throwing` deleted; `write_error_to_buffer` is now live (no `#[allow(dead_code)]`)
</success_criteria>

<output>
After completion, create `.planning/phases/62-caret-restoration-lru-removal/62-03-SUMMARY.md` documenting:
- The exact `sv_parse_function_rust` signature shipped
- The exact `sv_parse_stub` C++ implementation
- The manual caret-smoke output (paste of the rendered error)
- `just test-all` result summary
</output>
