Adaptive row-group estimation duration is very high

This is a follow-up to #59. Table writes spend substantial time on adaptive row-group estimation when `max_row_group_size` is not configured ## Context While investigating the performance regression, I compared two `dbt run` executions of a materialized table: - **Fixed row-group size:** `max_row_group_size: 8000000` - **Adaptive row-group selection:** configuration removed, allowing duckrun to estimate the row-group size The fixed setting is now configured centrally in the dbt project: ```yaml models: dbt_project: +max_row_group_size: 8000000 ``` ## Environment - Windows / Microsoft Fabric OneLake - duckrun `0.4.48` - dbt `1.12.2` - `threads: 1` ## Observed timings ### Fixed `max_row_group_size: 8000000` First run: | Phase | Time | |---|---:| | `write_deltalake` | 49.93 s | | `delta plugin store` | 80.56 s | | Total wall-clock time | 154.53 s | | `estimated_rows (planner)` | not present | ### Adaptive row-group selection The following run used the same model with the explicit row-group setting removed: | Phase | Time | |---|---:| | `estimated_rows (planner)` | 106.83 s | | `prior_row_count (log replay)` | 14.64 s | | `write_deltalake` | 48.19 s | | `delta plugin store` | 220.01 s | | Total wall-clock time | 328.50 s | The actual Parquet write duration was almost identical in these two runs. The large difference was before or around the write, especially in adaptive row-count estimation. As an additional check, a second fixed-size run after the adaptive run still did not report an `estimated_rows` phase. It took 236.20 s wall-clock, with 116.51 s in `delta plugin store`. This run had only 2.90 GiB available RAM and a different Delta-log state, so it should not be treated as a controlled repetition. ## Why this looks suspicious The adaptive path appears to spend more than 100 seconds estimating the number of rows before choosing a row-group size, even though the model is a regular dbt table write and the actual Parquet write takes about 50–63 seconds. The fixed configuration skips this planner phase entirely. This suggests that adaptive row-group selection can dominate the write path for some tables and storage states. The current measurements are not yet a fully controlled benchmark. The runs were sequential, the Delta log grew between runs, and available memory varied. Therefore the numbers demonstrate a strong signal, but do not yet isolate row-group estimation from all other metadata and cache effects. ## Possible root causes Potential contributors include: 1. Adaptive row-group selection performs an expensive source scan or row-count estimation. 2. The estimation path replays Delta metadata or opens the target relation repeatedly. 3. The estimated row count is not reused across the subsequent write phases. 4. Fixed `max_row_group_size` bypasses the expensive estimation path, but the adaptive path has no comparable fast path for ordinary dbt writes. ## Suggested investigation Please investigate whether the adaptive row-group planner can be made cheaper or avoided when sufficient information is already available. Suggested checks: - Add phase-level timings around adaptive row-count estimation and row-group selection. - Verify whether the planner scans the complete source relation or replays Delta metadata unnecessarily. - Check whether row counts from DuckDB statistics, source materialization, or Delta metadata can be reused. - Avoid repeated target/log discovery during one write operation. - Confirm that an explicitly configured `max_row_group_size` bypasses all adaptive estimation, as observed above. - Compare the resulting physical Parquet row groups and write quality for fixed and adaptive modes. ## Reproduction proposal Run a controlled benchmark with at least three repetitions per variant: 1. Use two equivalent target tables or restore the same target snapshot before each run. 2. Alternate the execution order to avoid systematically favoring one variant. 3. Keep duckrun, dbt, Python, available-memory conditions, and source data constant. 4. Capture: - total wall-clock time, - adaptive row-count estimation time, - Delta-log replay/discovery time, - Parquet write time, - number and size of generated row groups, - Delta-log version before and after the write. 5. Report median and range rather than a single run. ## Expected outcome For a table write where the row-group size is not explicitly configured, adaptive selection should not add a multi-minute planning phase or otherwise make the write substantially slower than the fixed configuration. If adaptive estimation is intentionally retained, it should be bounded, cached, and demonstrably beneficial compared with the fixed `max_row_group_size` path.