# Review: DuckDB API usage in the glue code, C API equivalence, and a migration design

Date: 2026-07-22.
Baseline: duckdb-r 1.5.4.9012, vendored DuckDB v1.5.5 (`src/duckdb/`), glue code `src/*.cpp` (~6,800 lines) + `src/include/*.hpp`.

## 1. Executive summary

The glue code exposes **63 `.Call()` entry points** and registers six kinds of
objects *into* the engine (a table function, two replacement scans, a cast, a
custom progress display, and the rfuns extension). Its DuckDB API consumption
falls into two very different halves:

- **The DBI half** (database/connection/statement/results, data-frame scan,
  Arrow import/export, rfuns): 24 entry points, ~5,200 lines. **Nearly every
  API used here has a direct, non-deprecated C API equivalent in v1.5.5.**
  The exceptions are narrow and enumerated in §5: table-function *filter*
  pushdown (Arrow scans only), the `INITIALIZE_ON_SCHEDULE` thread-affinity
  flag, `DUCKDB_TYPE_POINTER` missing from the C type enum (`r_string`),
  VARIANT decoding helpers (C API section exists but is empty in 1.5.5),
  error *location* metadata, and non-deprecated creation of streaming
  results.

- **The relational half** (`rapi_rel_*`, `rapi_expr_*`, ALTREP relations —
  duckplyr's backend): 39 entry points, ~1,600 lines. **The C API has no
  relational or expression-building surface at all** — this half constructs
  `ParsedExpression` trees field-by-field, composes 11 `Relation` subclasses,
  subclasses `Relation` itself (`AltrepDataFrameRelation`), and deparses
  relations to SQL via `GetQueryNode()->ToString()`. None of this is
  expressible over `duckdb.h` today.

**Recommended target**: a **two-layer glue**. Port the DBI half to the plain C
API (`duckdb.h` only), quarantine the relational half as an explicitly-marked
C++ residue, and drive an upstream RFC for a C relational/expression API —
seeded with the precise 39-entry-point requirement list in §7.6 — to
eventually eliminate the residue. Estimated effort for the port itself:
**11–16 engineering weeks** (§8), yielding an ~80 % reduction of the
internal-header surface that vendor bumps can break, materially better
interrupt/progress architecture, and a path toward version-tolerant linking
against stock `libduckdb`.

## 2. Why this matters (current coupling)

- The glue compiles against **71 vendored C++ headers, ~37 of which are not
  even part of the released `duckdb.hpp` amalgamation** (documented in
  `AGENTS.md` and `configure`): `GenericExecutor`, Arrow integration
  internals, core-functions extension internals, planner filters, catalog
  entries.
- Consequently the opt-in system-`libduckdb` fast path must **hard-verify the
  same upstream commit** (`configure` greps `DUCKDB_SOURCE_ID` out of the
  `.so`); any C++ ABI drift is undefined behavior.
- `BRANCHES.md` codifies the maintenance reality: *“Glue code may change when
  DuckDB's C++ API shifts, so it is updated together with vendoring
  commits.”* The C API, by contrast, is autogenerated, versioned
  (`src/duckdb/src/include/duckdb.h`, 551 `DUCKDB_C_API` functions in
  v1.5.5), ABI-stable, and the boundary every other non-C++ binding
  (Node-neo, Go, Rust, Julia, ODBC) builds on.

## 3. Inventory: what the glue actually uses

### 3.1 Entry-point census (63)

| Family | Count | Entry points |
|---|---|---|
| DB lifecycle | 8 | `startup`, `shutdown`, `connect`, `disconnect`, `connection_valid`, `lock`, `unlock`, `is_locked` |
| Statements | 4 | `prepare`, `bind`, `execute`, `release` |
| Arrow out | 5 | `execute_arrow`, `fetch_arrow_array`, `fetch_arrow_stream_into`, `record_batch`, `list_arrow` |
| Registration | 4 | `register_df`, `unregister_df`, `register_arrow`, `unregister_arrow` |
| Utilities | 3 | `ptr_to_str`, `adbc_init_func`, `load_rfuns` |
| Expressions | 9 | `expr_reference`, `expr_constant`, `expr_function`, `expr_comparison`, `expr_operator`, `expr_window`, `expr_set_alias`, `expr_tostring`, `get_null_SEXP_ptr` |
| Relations | 30 | `rel_from_df`, `rel_from_altrep_df`, `rel_from_sql`, `rel_from_table`, `rel_from_table_function`, `rel_project`, `rel_filter`, `rel_aggregate`, `rel_order`, `rel_join`, `rel_limit`, `rel_distinct`, `rel_union_all`, `rel_set_intersect`, `rel_set_diff`, `rel_set_symdiff`, `rel_to_df`, `rel_to_altrep`, `rel_to_sql`, `rel_to_table`, `rel_to_view`, `rel_to_csv`, `rel_to_parquet`, `rel_insert`, `rel_sql`, `rel_names`, `rel_alias`, `rel_set_alias`, `rel_tostring`, `rel_explain` |

The relational family (39 = expressions + relations) is **62 % of the entry
points but only ~24 % of the code**; the DBI family is the inverse — narrow
surface, deep implementation. duckplyr's generated conformance corpus
(`tests/testthat/test-rel_api.R`) exercises 20 of the 39 heavily
(`expr_reference` 1716×, `expr_set_alias` 1300×, `rel_project` 339×,
`rel_to_altrep` 222×, …), which both freezes their semantics and provides an
excellent safety net.

### 3.2 Objects registered into the engine

| Object | Where | Mechanism (C++) |
|---|---|---|
| `r_dataframe_scan` table function | `scan.cpp:682`, registered `database.cpp:84–91` | `TableFunction` subclass; system catalog + manual `MetaTransaction` |
| Environment replacement scan | `register.cpp:67`, installed `database.cpp:63–67` | `DBConfig::replacement_scans` callback synthesizing `TableFunctionRef` AST |
| Arrow replacement scan | `register.cpp:280`, installed `database.cpp:59–61` | same |
| `r_string → VARCHAR` cast | `database.cpp:15–20,95–96` | `CastFunctionSet::RegisterCastFunction`, `GenericExecutor` |
| Progress display | `connection.cpp:17–51` | `ProgressBarDisplay` virtual subclass via `ClientConfig::display_create_func` |
| rfuns extension (21 function sets) | `rfuns.cpp`, loaded `utils.cpp:312–321` | `DuckDB::LoadStaticExtension<RfunsExtension>()`, `ExtensionLoader` |
| `arrow_scan` catalog patch | `database.cpp:71–78` | System-catalog surgery: force `INITIALIZE_ON_SCHEDULE` on every overload |
| `AltrepDataFrameRelation` | `altrepdataframe_relation.cpp` | `Relation` subclass (`EXTENSION_RELATION`) with `GetQueryNode()` override |

### 3.3 Depth histogram (which headers could break)

- **MAIN-PUBLIC** (`DuckDB`, `Connection`, `PreparedStatement`,
  `QueryResult`, `ClientContext` members): bulk of DBI half — all covered by
  the C API.
- **COMMON** (`Value`, `Vector` + Flat/Constant/Unified/List/Struct/Array/Map
  accessors, `LogicalType`, validity words, `string_t`, Arrow converters,
  `ErrorData`, enum dictionary): heavily used — covered by the C value /
  logical-type / data-chunk / vector interfaces.
- **PARSER-INTERNAL** (7 `ParsedExpression` subclasses, `TableRef`,
  `QueryNode::ToString`, `OrderModifier`/`OrderByNode`,
  `Create*FunctionInfo`): relational half + registration — **no C
  equivalent** except the replacement-scan shape.
- **PLANNER-INTERNAL** (`TableFilterSet`, `ConstantFilter`,
  `ConjunctionFilter`): Arrow filter pushdown only — **no C equivalent**.
- **CATALOG-INTERNAL** (system catalog, `CatalogTransaction`,
  `TableFunctionCatalogEntry`): startup only — replaceable by
  `duckdb_register_*` + one upstream ask (§5.3).
- **FUNCTION-INTERNAL** (table-function plumbing, `CastFunctionSet`,
  `ArrowTableFunction` factory protocol, aggregate bind machinery,
  executor templates): scan/rfuns/arrow — mostly covered; gaps in §5.
- **EXTENSION-INTERNAL** (`ExtensionLoader`, core_functions `sum_helpers`):
  rfuns — replaceable (`RegularAdd` is two lines to inline).

## 4. The C API available in vendored v1.5.5

551 functions. Relevant capabilities (all verified in
`src/duckdb/src/include/duckdb.h`):

- **Lifecycle**: `duckdb_create_instance_cache` / `duckdb_get_or_create_from_cache`
  (matches the R driver's by-path DB caching), `duckdb_open_ext` + config,
  `duckdb_connect`, `duckdb_interrupt`, `duckdb_query_progress`,
  `duckdb_connection_get_client_context`, `duckdb_connection_get_arrow_options`.
- **Statements**: `duckdb_extract_statements`, `duckdb_prepare_extracted_statement`,
  `duckdb_prepare_error`, `duckdb_nparams`, `duckdb_parameter_name`,
  `duckdb_param_logical_type`, `duckdb_prepared_statement_column_count/name/logical_type`
  (pre-execution result schema — exactly what `construct_retlist` needs).
- **Execution**: `duckdb_execute_prepared` (materialized);
  **pending interface** `duckdb_pending_prepared` +
  `duckdb_pending_execute_task` + `duckdb_execute_pending` — incremental,
  client-driven, interruptible execution; `duckdb_fetch_chunk` on any result.
  ⚠️ Creation of *streaming* results (`duckdb_pending_prepared_streaming`,
  `duckdb_execute_prepared_streaming`, `duckdb_result_is_streaming`,
  `duckdb_stream_fetch_chunk`) is **deprecated-guarded** — see §5.2.
- **Values/types**: complete `duckdb_create_*` coverage — every timestamp
  variant, `time_tz`, `time_ns`, hugeint/uhugeint, decimal, enum, bit,
  bignum, uuid, interval, blob, list/struct/map/array/union values;
  `duckdb_create_enum_type/list_type/array_type/struct_type/union_type/decimal_type`;
  `duckdb_logical_type_get_alias` / `set_alias`. `DUCKDB_TYPE_GEOMETRY` (40)
  and `DUCKDB_TYPE_VARIANT` (41) exist. **`POINTER` does not** (§5.3).
- **Chunks/vectors**: data + validity pointers, string assignment
  (`duckdb_vector_assign_string_element_len`, `duckdb_unsafe_…` skipping
  UTF-8 validation), list child + `reserve`/`set_size`, struct/array
  children, `duckdb_slice_vector`, selection vectors,
  `duckdb_vector_reference_value/vector`. `duckdb_string_t` layout is public
  in the header (16-byte inline/pointer union).
- **Scalar functions**: full — sets/overloads, varargs, volatility, bind
  callback with `duckdb_scalar_function_bind_get_argument` (+ the new
  read-only **expression interface**: `duckdb_expression_return_type`,
  `is_foldable`, `fold` — enough to resolve constant arguments like `na.rm`
  or `%in%` tables at bind time), bind data, per-call errors, client context.
- **Aggregate functions**: sets, state size/init/update/combine/finalize,
  destructor, special null handling, extra info. **No bind callback** (§5.4).
- **Table functions**: bind (result columns, named parameters, cardinality),
  global/local init, `duckdb_init_get_column_count/index` (projection
  pushdown), `duckdb_init_set_max_threads`, per-call errors, client context.
  **No filter pushdown, no `to_string`, no `INITIALIZE_ON_SCHEDULE`** (§5).
- **Replacement scans**: `duckdb_add_replacement_scan` with function name +
  scalar parameters — exactly the shape both R replacement scans synthesize.
  No `external_dependency` hook (only matters for `rel_from_sql`, which stays
  in the residue anyway).
- **Casts**: `duckdb_create_cast_function` + source/target types + implicit
  cost + `duckdb_register_cast_function`. (Two internal reviews previously
  claimed this was missing — it is present in 1.5.5.)
- **Arrow**: modern, non-deprecated conversion —
  `duckdb_to_arrow_schema`, `duckdb_data_chunk_to_arrow`,
  `duckdb_schema_from_arrow`, `duckdb_data_chunk_from_arrow`, with
  `duckdb_arrow_options` from connection or result. (The old
  `duckdb_query_arrow` / `duckdb_arrow_scan` family is deprecated.)
- **Errors**: `duckdb_error_data` with `duckdb_error_data_error_type` /
  `message`; `duckdb_result_error_type`; the `duckdb_error_type` enum maps
  1:1 to C++ `ExceptionType` (incl. `INTERRUPT`) — the R condition-class
  mapping survives. Error *location* (`AddErrorLocation` → `ExtraInfo`) has
  no C surface (§5.5).
- Also available, currently unused but useful: appender (chunk-wise),
  table description, profiling info, threading/task API, custom config
  options, VFS, copy functions, catalog read API, logging;
  `duckdb_geometry_type_get_crs` (covers the `crs` attribute for wk).
- **Absent entirely**: any relational/expression *construction* API.
  `registerFunction`-style registration is per-database (system catalog,
  `ALTER_ON_CONFLICT`) — verified in `src/duckdb/src/main/capi/scalar_function-c.cpp:534–540`.

## 5. Equivalence map and gaps

### 5.1 Subsystem verdicts

| Subsystem (files) | C API verdict | Notes |
|---|---|---|
| DB lifecycle, config (`database.cpp`, `connection.cpp`) | ✅ Full | `duckdb_open_ext`+config / instance cache; Dual-wrapper lock semantics are glue-side; replacement-scan registration moves from `DBConfig` to post-open `duckdb_add_replacement_scan` |
| Progress display (`connection.cpp`, `r_progress_bar_display.hpp`) | 🔁 Replace, don't port | Engine-driven virtual display → poll `duckdb_query_progress` from the pending-task loop (§7.3). Removes R-evaluation-on-engine-thread hazard |
| Interrupts (`signal.cpp`) | ✅ Full (improved) | `duckdb_interrupt` == `ClientContext::Interrupt` (same atomic). Pending-task loop makes SIGINT swapping unnecessary on the C path (§7.3) |
| Prepare/bind/execute (`statement.cpp`) | ✅ Full | extract/prepare/introspect/bind all covered; typed NULLs via `duckdb_bind_null`; multi-row bind loop unchanged |
| Result → R (`transform.cpp`, allocate/decorate) | ✅ Full except VARIANT | Mechanical rewrite against `duckdb_data_chunk`/`duckdb_vector`; `Vector::Slice` per-row child views become offset arithmetic; enum dictionary, decimal widths, geometry CRS all exposed. VARIANT: §5.6 |
| Value layer (`utils.cpp` `SexpToValue`/`ValueToSexp`) | ✅ Full | `duckdb_create_*` complete incl. LIST/STRUCT/UNION creation; UNION *extraction* has no direct value accessor — read tag+member through a 1-row chunk via `duckdb_struct_vector_get_child` (mirrors today's bounce-through-a-Vector design, `utils.cpp:291–292`) |
| Data-frame scan (`scan.cpp`) | ✅ ~Full | bind/init/local-init/exec/projection/named-params/cardinality/max_threads all covered. Regressions: zero-copy strings (§5.7), `to_string` label (cosmetic), rowid sentinel (behavioral no-op: fill nothing + set cardinality) |
| DF registration + env replacement scan (`register.cpp` part) | ✅ Full | View DDL via SQL with pointer smuggled as UBIGINT parameter; env scan = C replacement scan calling R on the client thread (binds are synchronous) |
| Arrow import (`register.cpp` arrow part) | ⚠️ Portable with regressions | Rewrite as own C table function + `duckdb_data_chunk_from_arrow`. Loses: filter pushdown (planner `TableFilter` → R Arrow expressions today), and needs bind-time stream production for R thread affinity (§5.8) |
| Arrow export (`statement.cpp` arrow part) | ✅ Full | `duckdb_to_arrow_schema` + `duckdb_data_chunk_to_arrow` + hand-rolled `ArrowArrayStream` shim (the current `ResultArrowArrayStreamWrapper` is exactly that); arrow options from connection/result replace `ClientProperties` |
| rfuns (`rfuns.cpp`) | ✅ Portable with rework | Scalar/aggregate sets exist; executor templates → hand-rolled loops (3 already exist in-file); `na.rm` bind-dispatch needs redesign (§5.4); registration becomes per-database `duckdb_register_*` right after open (drop `ExtensionLoader`) |
| `r_string` experimental mode | ⚠️ Blocked on 1 enum | Cast registration ✅; `DUCKDB_TYPE_POINTER` missing → use UBIGINT+alias `r_string`, or 1-line upstream addition (§5.3) |
| Relational + expressions (`relational.cpp`) | ❌ No C API | 7 expression classes built field-by-field (incl. full window frames), 11 relation classes, `setop_all` mutation, `GetQueryNode()->ToString()` (`rel_to_sql`), `Relation::Query`, bind-only `Columns()` (`rel_names`) |
| ALTREP relations (`reltoaltrep.cpp`, `altrepdataframe_relation.cpp`) | ❌ No C API | `Relation` subclass with post-materialization source swap; `LimitRelation` push for row/cell budgets; column-selective `ColumnDataCollection::Chunks({i})` re-scan; `max_expression_depth` toggle (that one is portable via `SET`) |
| ADBC (`utils.cpp:14`) | ✅ Already C | `duckdb_adbc_init` is a C symbol today |

### 5.2 Streaming results (small upstream wart)

`ConvertOpts::streaming` (used by the `dbSendQueryArrow` plan) maps to
`duckdb_pending_prepared_streaming`, which still exists but sits behind the
`DUCKDB_API_NO_DEPRECATED` guard, with no non-deprecated replacement for
*creating* a streaming result (`duckdb_fetch_chunk` itself is fine and
unified). Choices: use the deprecated symbol knowingly; or materialize
(memory regression only); or upstream ask (§6 #4).

### 5.3 Catalog/type asks

- `DUCKDB_TYPE_POINTER` is absent from `duckdb_type` — `r_string` is
  `LogicalType(POINTER)` + alias `"r_string"` (`utils.cpp:97–101`). Workaround:
  UBIGINT+alias (same 8-byte physical layout; the cast and the transform key
  on the alias, `transform.cpp:411`, `types.cpp:306`). Clean fix: trivial
  upstream enum addition.
- The `arrow_scan` catalog patch (`database.cpp:71–78`) disappears in the
  target design (own table function), but the *reason* for it —
  “global init must run on the scheduling thread because it calls R” — has no
  C-API expression (§5.8).
- System-catalog registration of `r_dataframe_scan` becomes
  `duckdb_register_table_function(conn, …)` — per-database, alter-on-conflict;
  no manual `MetaTransaction` needed.

### 5.4 Aggregate `na.rm` bind-dispatch

Today `r_base::sum/min/max` read the `na.rm` constant from the bound
`Expression` at bind time and *swap the implementation function*
(`rfuns.cpp:483–500,1055–1088`). The C aggregate interface has no bind
callback. Options, in order of preference:

1. Read `na.rm` per-chunk from the second argument vector in `update` (it is
   constant-foldable, so the branch costs one load per chunk).
2. Register distinct function sets and rewrite the call in R/duckplyr.
3. Upstream ask: aggregate bind callback (same shape as the scalar one).

Two latent `Combine` bugs (source state dropped when both states are set:
`rfuns.cpp:444–453`, `1038–1043`) and a broken `na.rm` detection (the
`ToString()` comparison never matches SQL literals, which bind as
`CAST('t' AS BOOLEAN)`) were found here; they are fixed independently of
the port in #2404, #2405, and #2407. The `na.rm` regression is itself a
data point for this review: the string comparison presumably matched under
an earlier vendored DuckDB and broke silently when the binder's rendering
changed.

### 5.5 Structured errors

`rapi_error_with_context(ErrorData)` currently ships
`Type()`/`Message()`/`RawMessage()`/`ExtraInfo()` (incl. query location from
`AddErrorLocation`, `statement.cpp:81–83`) into classed R conditions. The C
API preserves the error *type* everywhere (`duckdb_error_data_error_type`,
`duckdb_result_error_type`) but not raw-vs-annotated message or the extra-info
map. Minor regression; upstream ask (§6 #6) or accept.

### 5.6 VARIANT and GEOMETRY

- GEOMETRY: fine — payload arrives as WKB blob (`DUCKDB_TYPE_GEOMETRY`), CRS
  via `duckdb_geometry_type_get_crs` → the `wk_wkb` + `crs` attribute path
  survives.
- VARIANT: `transform.cpp:705–719` uses `Vector::RecursiveToUnifiedFormat` +
  `VariantUtils::ConvertVariantToValue` — deep internals. The C header
  already contains an (empty) “Variant Helpers” section; until it fills in,
  either keep a ~50-line C++ residue for VARIANT columns or convert via SQL
  (`CAST(v AS JSON)`) at a semantic cost. Upstream ask (§6 #5).

### 5.7 Zero-copy strings in the scan

`RStringSexpType::Convert` writes `string_t(CHAR(x))` pointing directly into
R's CHARSXP pool (`types.cpp:433`, written at `scan.cpp:218–222`). The
official C write path copies (`duckdb_vector_assign_string_element_len`).
`duckdb_string_t`'s layout *is* public in `duckdb.h`, so a hand-written
non-owning assignment is possible but outside the documented ownership
contract. Plan: port with the copying API, benchmark string-heavy scans, and
only then decide between (a) accept the copy, (b) manual `duckdb_string_t`
writes, (c) upstream ask for an external-string assign. (The current
zero-copy design also has a latent lifetime hazard if a vector outlives the
pinned data frame — copying is *safer*, not just slower.)

### 5.8 Arrow-scan thread affinity

R Arrow streams must be produced on the R thread. Today that is guaranteed by
patching `arrow_scan` to `INITIALIZE_ON_SCHEDULE`. A C table function cannot
request that; but its **bind** phase always runs on the client (R) thread.
Design: produce the `ArrowArrayStream` at bind time into the bind data;
global init consumes it; a re-scan of the same node (rare plan shapes)
re-binds or errors with a clear message. Alternative: tiny upstream ask
(`duckdb_table_function_set_init_on_schedule`).

### 5.9 The relational half — why there is genuinely no C path today

Concrete blockers, with the duckplyr call-corpus weight behind them:

1. **Expression builders**: `ColumnRefExpression` (multi-part), typed-NULL
   `ConstantExpression`, `FunctionExpression` (+ `order_bys`, `filter`),
   `ComparisonExpression`, `ConjunctionExpression`, `OperatorExpression`
   (IN/NOT IN), and fully-specified `WindowExpression` (partitions, orders,
   8 boundary kinds, `start_expr`/`end_expr`/`offset_expr`/`default_expr`,
   `IGNORE NULLS`-style fields) — `relational.cpp:133–549`.
2. **Relation algebra**: Project/Filter/Aggregate/Order/Join(+ref types:
   regular/cross/positional/asof)/CrossProduct/SetOp(+`setop_all`)/Limit/
   Distinct/TableFunctionRelation/Alias — plus `Relation::Query(name, sql)`.
3. **Deparse**: `rel_to_sql` = `GetQueryNode()->ToString()`; `rel_tostring`;
   `rel_explain` enums.
4. **Bind-only schema**: `Relation::Columns()` without execution
   (`rel_names`, ALTREP shell construction).
5. **ALTREP machinery**: budgeted materialization by pushing
   `LimitRelation(max_rows+1)`, sticky `mat_error`, column-selective
   re-scan `Collection().Chunks({col})`, and `AltrepDataFrameRelation` —
   a user-defined `Relation` whose source is swapped to a
   `TableFunctionRelation` over the materialized data frame after first
   materialization.
6. **`rel_from_sql` CTE pinning** via `ExternalDependency` on replacement-scanned
   `TableFunctionRef`s.

Items 1–4 are exactly the shape of a C relational API proposal; items 5–6 are
R-specific but implementable *on top of* 1–4 (the source swap can become
“rebuild the R-level relation handle over `r_dataframe_scan`”, which the R
side can do without engine subclassing — `reltoaltrep.cpp` already has all
the state to do so).

## 6. Upstream asks (ranked by size)

1. `DUCKDB_TYPE_POINTER` in the C type enum (+ `duckdb_create_pointer`) — trivial; unblocks `r_string` cleanly.
2. `duckdb_table_function_set_init_on_schedule(bool)` (or documented bind-thread guarantee) — small; removes the Arrow bind-time-production compromise.
3. Error-data parity: `duckdb_prepared_statement_error_data`, raw message + extra-info (error location) accessors — small.
4. Non-deprecated creation of streaming results (or un-deprecating `duckdb_pending_prepared_streaming`) — small, policy question.
5. Variant Helpers — the section already exists in `duckdb.h`; filling it unblocks VARIANT→R without residue — medium.
6. Table-function **filter pushdown** for the C API (expose filters as the existing C `duckdb_expression` read-only handles) — medium; restores Arrow predicate pushdown; benefits every C-API binding.
7. Aggregate bind callback in the C API — small/medium; cleanest `na.rm` fix.
8. **C relational + expression-building API** — large; the strategic unblock for the residue. duckdb-r can seed the RFC with §5.9 as the concrete requirements list (few other bindings have a battle-tested, generated 4,000-case conformance corpus to contribute).

## 7. Target design

### 7.1 Architecture

```
src/
  core/          # C-API-only glue — includes duckdb.h, R, cpp11; NEVER duckdb C++ headers
    db.cpp       # startup/shutdown/lock/unlock, config, instance cache, replacement-scan reg.
    conn.cpp     # connect/disconnect/valid; per-DB registration of scan/casts/rfuns
    stmt.cpp     # prepare/bind/execute/release; pending-task execution loop
    fetch.cpp    # chunk → R (replaces transform.cpp; same allocate/decorate/transform split)
    values.cpp   # SexpToValue/ValueToSexp over duckdb_value
    scan_df.cpp  # r_dataframe_scan as C table function
    arrow_in.cpp # r_arrow_scan C table function + registry (replaces register.cpp arrow part)
    arrow_out.cpp# ArrowArrayStream shim over to_arrow_schema/data_chunk_to_arrow
    rfuns.cpp    # C scalar/aggregate function sets
    errors.cpp   # duckdb_error_data → classed R conditions
  residue/       # C++ glue — the ONLY place allowed to include duckdb C++ headers
    relational.cpp reltoaltrep.cpp altrepdataframe_relation.cpp (+ minimal shims)
```

Enforced boundary: a CI lint that fails if `core/` includes anything from
`src/duckdb/src/include/duckdb/` other than `duckdb.h`. The residue keeps the
same-commit constraint; `core/` compiles against any ≥ pinned C-API version.

R-visible behavior, classes, external-pointer semantics (`DualWrapper`
lock/unlock, GC-warning deleters, `registered_dfs` pinning, `ConvertOpts`)
stay identical — they are glue-side constructs.

### 7.2 Handle model

- `DBWrapper` holds `duckdb_database` (plus the existing SEXP slots and the
  arrow registry); `DualWrapper` semantics unchanged (the strong/weak
  shared-ptr dance is entirely ours).
- `ConnWrapper` holds `duckdb_connection` + a `shared_ptr<DBWrapper>`;
  connection outliving guarantees replace the current
  `shared_ptr<ClientContext>` sharing for interrupts.
- `RStatement` holds `duckdb_prepared_statement` + bound `duckdb_value`s.
- `RQueryResult` holds `duckdb_result` (+ the Arrow stream shim state
  machine, as today).

### 7.3 Execution loop: interrupts + progress unified (design improvement)

Replace both the SIGINT-handler swap *and* the engine-driven progress display
with one client-driven loop:

```
duckdb_pending_prepared(stmt, &pending)
while (!duckdb_pending_execution_is_finished(state)) {
  state = duckdb_pending_execute_task(pending)
  every ~100ms:
    - R_CheckUserInterrupt() under R_ToplevelExec  → on interrupt: duckdb_interrupt(conn), drain, raise R interrupt condition
    - poll duckdb_query_progress(conn)             → call the R progress callback (on the R thread!)
}
duckdb_execute_pending(pending, &result)
```

This removes the two most delicate aspects of the current design: R
evaluation on engine execution threads (progress display,
`connection.cpp:36–47`) and process-global SIGINT handler swapping
(`signal.cpp`), while preserving the observable contracts (interrupt raises
the same condition; the reentrancy error for nested queries moves into the
wrapper). The `cpp11::message("")` RStudio/Windows workaround is kept at the
raise site. The relational residue keeps `ScopedInterruptHandler` until it,
too, is ported.

### 7.4 Scan and registration specifics

- Bind: harvest column pointers exactly as today (bind runs on the R
  thread); ENUM levels via `duckdb_create_enum_type`; matrices via
  `duckdb_create_array_type`; `map_list_of` cell-walk unchanged.
- Exec: raw writes through `duckdb_vector_get_data` + validity; strings via
  `assign_string_element_len` first (benchmark, then §5.7); lists via
  `duckdb_list_vector_reserve/set_size` + child writes (replacing the slow
  `Value`-based `ListVector::PushBack` — a *speedup* opportunity).
- Parallelism: `duckdb_init_set_max_threads` + the same mutex-guarded range
  claiming in global init data.
- Registration: `duckdb_register_table_function` at open;
  `CREATE OR REPLACE TEMPORARY VIEW "<name>" AS FROM r_dataframe_scan(<ptr>, …)`
  via a bound statement or literal; env/arrow replacement scans via
  `duckdb_add_replacement_scan` (pointer parameters as UBIGINT values).

### 7.5 rfuns port

Register per-database right after open (system catalog,
alter-on-conflict — verified). Scalar kernels: hand-rolled loops over
`duckdb_vector` (the three existing word-level validity loops in
`rfuns.cpp:238–268,307–339,814–843` translate 1:1). `%in%`: resolve the RHS
list at bind via the C expression interface (`is_foldable` + `fold`) — an
improvement over today's `CONSTANT_VECTOR` check. `na.rm`: per-chunk read
(§5.4). Drop the `dispatch()` debug helpers or reimplement via
`duckdb_functions()` SQL. (The `Combine` and `na.rm`-binding bugs found
during this review are fixed ahead of the port in #2404, #2405, #2407; the
consumed-result crash in the Arrow entry points in #2406; the Arrow OR/IN
pushdown failure — the review initially classified the `CONJUNCTION_OR`
cast as benign, but it turned out to hide hard query failures for
`OPTIONAL_FILTER`/`IN_FILTER` — in #2410; and the `BinaryTypeAssert`
copy-paste in #2411.)

### 7.6 The residue and its retirement path

The residue is ~1,600 lines serving 39 entry points, all relational. Its
retirement is gated on upstream ask #8. Until then it still forces vendored
builds and the same-commit rule — but the exposed churn surface shrinks from
71 internal headers to the ~15 the relational half uses (main/relation,
parser expressions, `ColumnDataCollection`), and every other subsystem
becomes vendor-bump-proof. When the C relational API lands, `reltoaltrep`'s
source-swap trick is rebuilt R-side (rebuild the relation over
`r_dataframe_scan(materialized_df)` — no engine subclassing needed), and the
package can finally offer **version-tolerant linking against stock
libduckdb** (today's dev-only fast path, minus the same-commit lock, and
without shipping 1,700 vendored sources in every build).

### 7.7 Explicitly rejected alternative: SQL-string backend for duckplyr

Reimplementing `rel_*`/`expr_*` by generating SQL text would eliminate the
residue today, and `rel_to_sql` proves relations *can* deparse. Rejected as
the primary plan because: literal-rendering fidelity (typed NULLs, NaN vs NA,
blob/timestamp literals, quoting), alias semantics derived from
`GetName()`, window-frame corner cases, and the ALTREP budget/limit push are
all places where text round-trips can silently diverge from the bound
relation tree that duckplyr's 4,000-case corpus locks in. Worth revisiting
only if upstream declines a C relational API.

## 8. Migration plan, effort, and risk

Phases (each lands green on the full suite — DBItest, ~40 test files, the
rel_api corpus stays on the untouched residue until the end):

| Phase | Scope | Est. |
|---|---|---|
| 0 | Module split, include-lint CI, micro-benchmarks (scan/fetch, string-heavy), pin C-API version | 1 wk |
| 1 | Result path: `fetch.cpp` (transform port) + result handling in `stmt.cpp` | 2–3 wk |
| 2 | Prepare/bind/execute, pending loop, interrupts+progress redesign, `errors.cpp` | 2–3 wk |
| 3 | `scan_df.cpp` + registration + env replacement scan | 2–3 wk |
| 4 | Arrow in/out (`arrow_in.cpp`, `arrow_out.cpp`, nanoarrow paths) | 1.5–2 wk |
| 5 | `rfuns.cpp` port (incl. `na.rm` redesign) | 2–3 wk |
| 6 | DB lifecycle swap, delete dead C++ glue, docs | 1 wk |
| Σ | **Phases 0–6** | **11–16 wk** |
| U | Upstream track (asks §6 #1–7 early; #8 RFC) — parallel | ongoing |
| 7 | Relational port once #8 exists; drop residue; version-tolerant linking | +3–4 wk |

Key risks and mitigations:

- **Conversion regressions** (type corner cases): the existing suite is
  strong; add golden tests for VARIANT/GEOMETRY/TIME_TZ/DECIMAL widths before
  phase 1.
- **Performance**: string-copy in scan and fetch-loop overhead — measured in
  phase 0 benchmarks, decided with data (§5.7). List-cell writes get
  *faster*.
- **Behavioral deltas**: progress/interrupt timing (poll-based), Arrow
  rescan semantics (§5.8), error location loss (§5.5) — each is small,
  user-visible, and should be listed in NEWS.
- **Deprecated-API dependence**: only streaming creation; tracked as
  upstream ask.
- **Two glue dialects during migration**: bounded by the phase structure
  (each phase fully replaces a subsystem; no long-lived dual paths except
  the phase-1 opt-in switch).

## 9. Bottom line

- **Do plain C equivalents exist for all entry points?** For 24 of 63 entry
  points (the entire DBI/scan/Arrow/rfuns surface): essentially yes — with
  six narrow, well-understood gaps, none of which blocks the port (worst
  case: keep ~50 lines of residue for VARIANT and accept Arrow filter
  pushdown loss until upstreamed). For the 39 relational/expression entry
  points that duckplyr depends on: no, and nothing close exists in v1.5.5.
- **Refactoring effort**: 11–16 weeks to a two-layer package whose
  vendor-bump-fragile surface shrinks by ~80 %, with a cleaner
  interrupt/progress architecture as a side effect; plus an upstream
  engagement whose largest item (C relational API) is the gate to full
  C-API purity and version-tolerant linking against stock libduckdb.
