feat: `OpenAPISchemaNormalizer` should strip `pattern` keyword, like it does `enum`
kind/Feature
## Summary
`OpenAPISchemaNormalizer` (in `singer_sdk/schema/source.py`) already removes the `enum` keyword from OpenAPI schemas because it is a validation-only constraint that has no meaning in Singer's data-exchange model. The `pattern` keyword is in the same category — it is a string-validation constraint that Singer taps and targets do not use — but it is currently left in the schema.
This causes a concrete failure when an OpenAPI spec contains a `pattern` value that is valid in the spec's intended regex dialect (e.g. Java/PCRE with named groups `(?<name>...)`) but is not valid ECMA 262 / Python regex. The jsonschema meta-schema validator then rejects the Singer schema at test time with:
```
jsonschema.exceptions.SchemaError: 'data:(?<mime>image\/(.+?));base64,(?<data>.*)' is not a 'regex'
```
Even when the pattern is syntactically valid Python regex, keeping it in Singer schemas produces false-negative validation failures for records whose values are perfectly acceptable for ELT purposes but happen not to match the constraint.
## Concrete example
`WorkspaceResource` in a Spring Boot / Springdoc OpenAPI spec:
```json
"imageUrl": {
"type": "string",
"maxLength": 699052,
"minLength": 0,
"pattern": "data:(?<mime>image\\/(.+?));base64,(?<data>.*)"
}
```
`(?<name>...)` is a Java named-capture-group — valid PCRE, invalid ECMA 262 and invalid Python `re`. When `OpenAPISchema.fetch_schema` returns this schema and the SDK's built-in test suite validates it, `test_tap_stream_schema_is_valid` fails because `jsonschema` checks the `pattern` value against the `regex` format.
The workaround currently required in tap code:
```python
class MyOpenAPISchema(OpenAPISchema):
def fetch_schema(self, key):
schema = super().fetch_schema(key)
for prop in schema.get("properties", {}).values():
if isinstance(prop, dict) and prop.get("pattern"):
try:
re.compile(prop["pattern"])
except re.error:
prop.pop("pattern")
return schema
```
## Proposed fix
Add `handle_pattern` to `OpenAPISchemaNormalizer`, mirroring the existing `handle_enum`:
```python
def handle_pattern(self, schema: Schema) -> Schema:
"""Handle pattern values in a JSON schema.
Args:
schema: A JSON schema.
Returns:
The schema with ``pattern`` removed.
"""
schema.pop("pattern", None)
return schema
```
And call it from `normalize_schema`, alongside `handle_enum`:
```python
# Remove 'pattern' keyword (validation-only, not used in Singer data exchange)
if "pattern" in result:
result = self.handle_pattern(result)
```
## Rationale
Singer schemas describe the **shape** of data for transport between taps and targets. Validation-constraint keywords like `enum`, `pattern`, `minimum`, `maximum`, `minLength`, `maxLength`, `multipleOf`, etc. are meaningful in an OpenAPI context (to validate API request/response payloads) but have no role in Singer's ELT pipeline. Stripping `enum` is already established precedent; `pattern` should follow.
The overridable `handle_pattern` hook keeps the same design as `handle_enum`, so tap authors who want to preserve patterns can override it.
## Environment
- Singer SDK version: `0.54.0a5`
- Reproduced while building a tap against a Spring Boot API whose OpenAPI spec uses Java-style named-capture-group patterns.
- Related: #3552 (`additionalProperties` $ref not resolved — filed separately, now fixed in 0.54.0a5)
0 条评论