ITADN

json2 flush fails with "Failed to align JSON array, reason: expect struct array" when a JSON column has inconsistent native types across bulk parts

#8320Openkillme2008 创建于 2026-06-17
C-bug
K
killme2008commented
## What type of bug is this? Crash / panic during flush. ## What subsystems are affected? Storage Engine (mito2), JSON v2 data type. ## What happened Flushing a table with a JSON (json2) column can fail with: ``` Failed to align JSON array, reason: expect struct array ``` The error originates in `JsonArray::try_align` at `src/datatypes/src/vectors/json/array.rs:117-119`, where `self.inner.as_struct_opt()` returns `None`. Because the JSON vector builder unwraps the build result (`to_vector` -> `try_build().unwrap_or_else(|e| panic!(...))`), this surfaces as a hard failure during flush. ## Root cause A json2 column's **physical** Arrow type depends on the inferred native type (`src/datatypes/src/types/json_type.rs:130` `JsonNativeType::as_arrow_type`): | JSON native type | Arrow physical type | |---|---| | `Object{...}` | `Struct` | | `Null` | `Null` (NullArray) | | `Bool` / `Number` | `Boolean` / `Int64` / `UInt64` / `Float64` | | `String` | `Utf8View` | | `Array` | `List` | | `Variant` | `Binary` | So a JSON column is only a `StructArray` when its inferred native type is `Object`. `JsonArray::try_align` (`array.rs:106`) only knows how to align **struct -> struct**: it calls `as_struct_opt()` and bails with `"expect struct array"` if the source array is not a struct. It cannot promote a non-struct array (NullArray / scalar / `Variant` Binary) into a merged object schema. During flush, `align_parts` (`src/mito2/src/memtable/bulk/part.rs:483`) merges each JSON column's `JsonType` across all bulk parts into one unified type (line 500-507), then calls `JsonArray::from(column).try_align(merged.as_arrow_type())` on **every** part's column (line 530-535). When parts infer different native types for the same column, the merged type is typically promoted to `Object` (struct), but at least one part's physical column is still non-struct -> `try_align` fails. ## Reproduce conditions Same JSON column, across bulk parts (or rows within a flush), inferred as inconsistent native types. Most likely triggers: 1. One part has the column **entirely NULL** -> inferred `JsonNativeType::Null` -> `NullArray`; another part has objects -> merged type `Object` -> aligning the NullArray to the struct target fails. 2. Mixed **scalar and object** JSON values across parts (e.g. one part is a string/number/`true`/`Variant`, another is an object). To confirm a specific occurrence, enable the `common_telemetry::trace!` at `array.rs:111` ("Try aligning JSON array {from} to {to}") to see the non-struct source physical type. ## Affected versions - **v1.1.0+** (and nightlies after 2026-04-23, commit `209880b991` "feat: json2 flush (#8011)"). - **Not** affected: v1.0.0 / v1.0.1 / v1.0.2 (released before #8011; do not contain this alignment path). ## Suggested fix directions 1. Make `try_align` handle non-struct sources: at minimum, when the source is a `NullArray`, build an all-null `StructArray` matching the target fields and row count. For scalar/`Variant` sources where the target is a struct, this indicates the merge step should have converged to `Variant` instead. 2. Fix `JsonType::merge` so that irreconcilable native types (`Object` vs non-`Object`) converge to `Variant` (Binary), so the alignment target is not a struct and `try_align` takes the `try_cast` branch. Needs a closer look at `JsonType::merge`'s rules for `Object` vs `Null` / scalar to decide whether the fix belongs in merge or in `try_align`.
0 条评论