ITADN

feat: `OpenAPISchemaNormalizer` should strip `pattern` keyword, like it does `enum`

#3554Openedgarrmondragon 创建于 2026-03-10
kind/Feature
E
edgarrmondragoncommented
## Summary `OpenAPISchemaNormalizer` (in `singer_sdk/schema/source.py`) already removes the `enum` keyword from OpenAPI schemas because it is a validation-only constraint that has no meaning in Singer's data-exchange model. The `pattern` keyword is in the same category — it is a string-validation constraint that Singer taps and targets do not use — but it is currently left in the schema. This causes a concrete failure when an OpenAPI spec contains a `pattern` value that is valid in the spec's intended regex dialect (e.g. Java/PCRE with named groups `(?<name>...)`) but is not valid ECMA 262 / Python regex. The jsonschema meta-schema validator then rejects the Singer schema at test time with: ``` jsonschema.exceptions.SchemaError: 'data:(?<mime>image\/(.+?));base64,(?<data>.*)' is not a 'regex' ``` Even when the pattern is syntactically valid Python regex, keeping it in Singer schemas produces false-negative validation failures for records whose values are perfectly acceptable for ELT purposes but happen not to match the constraint. ## Concrete example `WorkspaceResource` in a Spring Boot / Springdoc OpenAPI spec: ```json "imageUrl": { "type": "string", "maxLength": 699052, "minLength": 0, "pattern": "data:(?<mime>image\\/(.+?));base64,(?<data>.*)" } ``` `(?<name>...)` is a Java named-capture-group — valid PCRE, invalid ECMA 262 and invalid Python `re`. When `OpenAPISchema.fetch_schema` returns this schema and the SDK's built-in test suite validates it, `test_tap_stream_schema_is_valid` fails because `jsonschema` checks the `pattern` value against the `regex` format. The workaround currently required in tap code: ```python class MyOpenAPISchema(OpenAPISchema): def fetch_schema(self, key): schema = super().fetch_schema(key) for prop in schema.get("properties", {}).values(): if isinstance(prop, dict) and prop.get("pattern"): try: re.compile(prop["pattern"]) except re.error: prop.pop("pattern") return schema ``` ## Proposed fix Add `handle_pattern` to `OpenAPISchemaNormalizer`, mirroring the existing `handle_enum`: ```python def handle_pattern(self, schema: Schema) -> Schema: """Handle pattern values in a JSON schema. Args: schema: A JSON schema. Returns: The schema with ``pattern`` removed. """ schema.pop("pattern", None) return schema ``` And call it from `normalize_schema`, alongside `handle_enum`: ```python # Remove 'pattern' keyword (validation-only, not used in Singer data exchange) if "pattern" in result: result = self.handle_pattern(result) ``` ## Rationale Singer schemas describe the **shape** of data for transport between taps and targets. Validation-constraint keywords like `enum`, `pattern`, `minimum`, `maximum`, `minLength`, `maxLength`, `multipleOf`, etc. are meaningful in an OpenAPI context (to validate API request/response payloads) but have no role in Singer's ELT pipeline. Stripping `enum` is already established precedent; `pattern` should follow. The overridable `handle_pattern` hook keeps the same design as `handle_enum`, so tap authors who want to preserve patterns can override it. ## Environment - Singer SDK version: `0.54.0a5` - Reproduced while building a tap against a Spring Boot API whose OpenAPI spec uses Java-style named-capture-group patterns. - Related: #3552 (`additionalProperties` $ref not resolved — filed separately, now fixed in 0.54.0a5)
0 条评论