json2 flush fails with "Failed to align JSON array, reason: expect struct array" when a JSON column has inconsistent native types across bulk parts
C-bug
## What type of bug is this?
Crash / panic during flush.
## What subsystems are affected?
Storage Engine (mito2), JSON v2 data type.
## What happened
Flushing a table with a JSON (json2) column can fail with:
```
Failed to align JSON array, reason: expect struct array
```
The error originates in `JsonArray::try_align` at `src/datatypes/src/vectors/json/array.rs:117-119`, where `self.inner.as_struct_opt()` returns `None`. Because the JSON vector builder unwraps the build result (`to_vector` -> `try_build().unwrap_or_else(|e| panic!(...))`), this surfaces as a hard failure during flush.
## Root cause
A json2 column's **physical** Arrow type depends on the inferred native type (`src/datatypes/src/types/json_type.rs:130` `JsonNativeType::as_arrow_type`):
| JSON native type | Arrow physical type |
|---|---|
| `Object{...}` | `Struct` |
| `Null` | `Null` (NullArray) |
| `Bool` / `Number` | `Boolean` / `Int64` / `UInt64` / `Float64` |
| `String` | `Utf8View` |
| `Array` | `List` |
| `Variant` | `Binary` |
So a JSON column is only a `StructArray` when its inferred native type is `Object`.
`JsonArray::try_align` (`array.rs:106`) only knows how to align **struct -> struct**: it calls `as_struct_opt()` and bails with `"expect struct array"` if the source array is not a struct. It cannot promote a non-struct array (NullArray / scalar / `Variant` Binary) into a merged object schema.
During flush, `align_parts` (`src/mito2/src/memtable/bulk/part.rs:483`) merges each JSON column's `JsonType` across all bulk parts into one unified type (line 500-507), then calls `JsonArray::from(column).try_align(merged.as_arrow_type())` on **every** part's column (line 530-535). When parts infer different native types for the same column, the merged type is typically promoted to `Object` (struct), but at least one part's physical column is still non-struct -> `try_align` fails.
## Reproduce conditions
Same JSON column, across bulk parts (or rows within a flush), inferred as inconsistent native types. Most likely triggers:
1. One part has the column **entirely NULL** -> inferred `JsonNativeType::Null` -> `NullArray`; another part has objects -> merged type `Object` -> aligning the NullArray to the struct target fails.
2. Mixed **scalar and object** JSON values across parts (e.g. one part is a string/number/`true`/`Variant`, another is an object).
To confirm a specific occurrence, enable the `common_telemetry::trace!` at `array.rs:111` ("Try aligning JSON array {from} to {to}") to see the non-struct source physical type.
## Affected versions
- **v1.1.0+** (and nightlies after 2026-04-23, commit `209880b991` "feat: json2 flush (#8011)").
- **Not** affected: v1.0.0 / v1.0.1 / v1.0.2 (released before #8011; do not contain this alignment path).
## Suggested fix directions
1. Make `try_align` handle non-struct sources: at minimum, when the source is a `NullArray`, build an all-null `StructArray` matching the target fields and row count. For scalar/`Variant` sources where the target is a struct, this indicates the merge step should have converged to `Variant` instead.
2. Fix `JsonType::merge` so that irreconcilable native types (`Object` vs non-`Object`) converge to `Variant` (Binary), so the alignment target is not a struct and `try_align` takes the `try_cast` branch.
Needs a closer look at `JsonType::merge`'s rules for `Object` vs `Null` / scalar to decide whether the fix belongs in merge or in `try_align`.
0 条评论