ITADN

Generic Tables cannot store or expose structured schema metadata

#5128Openhelmanofer 创建于 2026-07-23
stale
H
helmanofercommented
### Describe the bug Polaris Generic Tables can register non-Iceberg table formats, but their API model has no first-class schema field. The create/load payload only exposes `name`, `format`, `base-location`, `doc`, and string `properties`. As a result, a catalog integration cannot populate ordered columns, types, or required/nullability metadata for a Generic Table. Clients and catalog UIs can render a structured schema for Iceberg tables from `metadata.schemas`, but equivalent metadata cannot be represented for Generic Tables. ### To Reproduce 1. Create a Generic Table for any non-Iceberg format. 2. Include its schema as JSON in a string property because no schema field exists. 3. Load the Generic Table through the Polaris API. 4. Observe that schema information is available only as an opaque property; there is no structured schema in the response and no schema update operation. Example workaround property: ```json { "external.schema-json": "{\"columns\":[{\"name\":\"customer_id\",\"type\":\"long\",\"nullable\":true}]}" } ``` ### Actual Behavior Column metadata must be serialized into arbitrary string properties. It is not validated, typed, queryable, interoperable, or exposed through the same schema model used for Iceberg tables. A schema change also requires replacing the Generic Table metadata rather than updating a structured schema. ### Expected Behavior Generic Tables should support an optional structured schema in their create/load/update API, including: - ordered field names; - field types, including nested types; - required/nullability information; and - schema evolution or replacement. This would let non-Iceberg formats participate as proper catalog tables rather than name/location registrations with opaque properties. ### Additional context This affects any integration cataloging external formats whose physical system remains authoritative for schema. Encoding schema JSON in properties preserves information but does not provide catalog-level schema semantics. ### System information Polaris Catalog Version: 1.5.0 Setup: Generic Table backed by an external non-Iceberg system
3 条评论