Generic Tables cannot store or expose structured schema metadata
stale
### Describe the bug
Polaris Generic Tables can register non-Iceberg table formats, but their API model has no first-class schema field. The create/load payload only exposes `name`, `format`, `base-location`, `doc`, and string `properties`.
As a result, a catalog integration cannot populate ordered columns, types, or required/nullability metadata for a Generic Table. Clients and catalog UIs can render a structured schema for Iceberg tables from `metadata.schemas`, but equivalent metadata cannot be represented for Generic Tables.
### To Reproduce
1. Create a Generic Table for any non-Iceberg format.
2. Include its schema as JSON in a string property because no schema field exists.
3. Load the Generic Table through the Polaris API.
4. Observe that schema information is available only as an opaque property; there is no structured schema in the response and no schema update operation.
Example workaround property:
```json
{
"external.schema-json": "{\"columns\":[{\"name\":\"customer_id\",\"type\":\"long\",\"nullable\":true}]}"
}
```
### Actual Behavior
Column metadata must be serialized into arbitrary string properties. It is not validated, typed, queryable, interoperable, or exposed through the same schema model used for Iceberg tables. A schema change also requires replacing the Generic Table metadata rather than updating a structured schema.
### Expected Behavior
Generic Tables should support an optional structured schema in their create/load/update API, including:
- ordered field names;
- field types, including nested types;
- required/nullability information; and
- schema evolution or replacement.
This would let non-Iceberg formats participate as proper catalog tables rather than name/location registrations with opaque properties.
### Additional context
This affects any integration cataloging external formats whose physical system remains authoritative for schema. Encoding schema JSON in properties preserves information but does not provide catalog-level schema semantics.
### System information
Polaris Catalog Version: 1.5.0
Setup: Generic Table backed by an external non-Iceberg system
3 条评论