feat: Exclude context keys following informal private naming convention from state partitions automatically
kind/Featurebreaking changevaluestream/SDKStream DAG
### Feature scope
Taps (catalog, state, tests, etc.)
### Description
Currently, in a parent stream class that implements
```py
def get_child_context(self, record, context):
return {
"order_id": record["id"],
"order": record,
}
```
child stream classes have to set
```py
state_partitioning_keys = ["order_id"]
```
to avoid other items from being set in state (this whole record in this case).
It would be convenient if something like
```py
def get_child_context(self, record, context):
return {
"order_id": record["id"],
"_order": record,
}
```
was supported, so that `state_partitioning_keys` would not need to be defined for child stream classes.
Additionally, it would also be good (especially in this case) to avoid logging out private context items in standard and metric logs. It is possible to obfuscate the contents by defining a custom dict-like class
```py
# important that this doesn't subclass dict directly to ensure json.dumps calls do not expose the contents
class hiddendict(collections.UserDict):
def __repr__(self):
return "***"
```
```py
def get_child_context(self, record, context):
return {
"order_id": record["id"],
"order": hiddendict(record),
}
```
but may be simpler to exclude private context items outright.
1 条评论