Feature request: aws_ses source, add the remaining GET-reachable tables
enhancementteam/data-warehousefeature/pipeline-sources
### Feature request
Expand the `aws_ses` source from 4 tables to 11. Every new table uses a GET operation, so the existing SigV4 client already reaches all of them.
**Current state:** #79766 shipped the source on 2026-08-10 with `account`, `configuration_sets`, `email_identities` and `suppressed_destinations`. That was the starting scope named in #79107. The build environment held no AWS credentials, so nobody ran a live sync before the merge, and the source shipped as `ALPHA`.
**Why now:** a customer runs the source against a live AWS account today, and the four tables sync. They ask for the widest table coverage we can offer. SESv2 exposes 44 read operations. We use 5 of them.
Every shape below comes from botocore's bundled service model, `sesv2/2019-09-27`.
## Tables to add
Six tables are one new entry each in `AWS_SES_ENDPOINTS` (`settings.py`). The existing pagination and fan-out code handles them with no change.
| Table | List path | `result_key` | Detail path | `item_name_key` | `name_column` | `primary_key` |
|---|---|---|---|---|---|---|
| `email_templates` | `/v2/email/templates` | `TemplatesMetadata` | `/v2/email/templates/{name}` | `TemplateName` | `template_name` | `["template_name"]` |
| `contact_lists` | `/v2/email/contact-lists` | `ContactLists` | `/v2/email/contact-lists/{name}` | `ContactListName` | `contact_list_name` | `["contact_list_name"]` |
| `dedicated_ip_pools` | `/v2/email/dedicated-ip-pools` | `DedicatedIpPools` | `/v2/email/dedicated-ip-pools/{name}` | `None` | `pool_name` | `["pool_name"]` |
| `dedicated_ips` | `/v2/email/dedicated-ips` | `DedicatedIps` | none | n/a | n/a | `["ip"]` |
| `custom_verification_email_templates` | `/v2/email/custom-verification-email-templates` | `CustomVerificationEmailTemplates` | `/v2/email/custom-verification-email-templates/{name}` | `TemplateName` | `template_name` | `["template_name"]` |
| `multi_region_endpoints` | `/v2/email/multi-region-endpoints` | `MultiRegionEndpoints` | none | n/a | n/a | `["endpoint_id"]` |
Per-table notes:
1. `email_templates`: the list response carries only `TemplateName` and `CreatedTimestamp`. `GetEmailTemplate` adds `TemplateContent`, which holds `Subject`, `Text` and `Html`. The fan-out is what makes this table useful. Set `timestamp_columns=("created_timestamp",)`.
2. `contact_lists`: `GetContactList` adds `Topics`, `Description`, `CreatedTimestamp` and `Tags`. `Topics` and `Tags` are lists, so `_flatten` keeps them whole. Set `timestamp_columns=("created_timestamp", "last_updated_timestamp")`.
3. `dedicated_ip_pools`: the list returns plain strings, exactly like `ListConfigurationSets`, so `item_name_key` stays `None`. `GetDedicatedIpPool` returns a nested `DedicatedIpPool` structure, which flattens to `dedicated_ip_pool_*` columns.
4. `dedicated_ips`: one row per IP, with `Ip`, `WarmupStatus`, `WarmupPercentage` and `PoolName`. An account with no dedicated IPs returns an empty list. That is not an error, and the table must stay reachable in the schema picker.
5. `custom_verification_email_templates`: the list already carries every field except `TemplateContent`. Keep the fan-out anyway, because the template body is the point of the table.
6. `multi_region_endpoints`: `PageSize` here is `PageSizeV2`, which AWS bounds at 1000. Every other endpoint uses an unbounded `MaxItems`. Set `page_size=1000` for this table and 100 for the rest. `Regions` is a list and stays whole. Set `timestamp_columns=("created_timestamp", "last_updated_timestamp")`.
## The seventh table needs a small code change
`configuration_set_event_destinations` does not fit the current fan-out shape.
- List: `/v2/email/configuration-sets`, `result_key=ConfigurationSets`. This reuses the walk that `configuration_sets` already performs.
- Detail: `/v2/email/configuration-sets/{name}/event-destinations`.
- The detail response is `{"EventDestinations": [...]}`, which is a list. Today `_fanout_page_rows` runs `row.update(normalize_row(endpoint_config, detail))`, so the whole list would land in one `event_destinations` column, with one row per configuration set.
- Wanted: one row per configuration set and event destination pair. Set `primary_key=["configuration_set_name", "name"]`.
- Change: add a `detail_result_key` field to `AwsSesEndpointConfig`. When a config sets it, `_fanout_page_rows` iterates `detail[detail_result_key]` and emits one row per element. When a config leaves it `None`, the current single-row merge behavior stays.
- Each `EventDestination` carries `Name`, `Enabled`, `MatchingEventTypes`, and one of `KinesisFirehoseDestination`, `CloudWatchDestination`, `SnsDestination`, `EventBridgeDestination` or `PinpointDestination`. `_flatten` turns those into columns such as `sns_destination_topic_arn`, which is the wanted shape.
This table earns its place. It reports where SES publishes per-message events today, per configuration set. SESv2 serves no per-message data itself, so this table is how a customer finds the pipeline that does.
## Sync behavior
None of the seven endpoints accepts a server-side time filter. So all seven stay full refresh, like the three existing non-suppression tables. Keep `sort_mode="desc"`, and add no `INCREMENTAL_FIELDS` entries.
SESv2 read APIs throttle at roughly one request per second, and `TRANSPORT_RETRY` already gives 429 extra headroom. Four of the new tables fan out at one detail request per item. These lists are small in practice, so the request cost is acceptable.
## Deliberately excluded
- `ListTagsForResource` requires a `ResourceArn`, so it fans out over every identity, configuration set, template and pool. The request cost is high and the value is low.
- `GetEmailIdentityPolicies` adds nothing. `GetEmailIdentity` already returns `Policies`, and `email_identities` stores it whole through `raw_keys`.
- `GetDedicatedIp` and `GetSuppressedDestination` are single-item reads of rows the list tables already return.
- The POST-shaped list operations need request-body signing first. #86804 tracks them.
- The VDM sending metrics need both body signing and a synthesized table design. #86806 tracks them.
- The Deliverability Dashboard family needs the paid AWS subscription, at $1,250 per month. Those tables return nothing on a normal account, so they read as broken.
- Per-message send, bounce, open and click stay impossible. SESv2 has no list operation for messages. SES publishes those events to an event destination instead.
## Definition of done
1. The seven tables appear in the schema picker and sync rows against a live AWS account.
2. `canonical_descriptions.py` and `ENDPOINT_DESCRIPTIONS` carry a description for every new table and column.
3. `SOURCES.md` reports the new table count.
4. `probe_endpoint_permissions` reports a clear reason for each new table when the IAM policy denies it.
5. The source doc at `contents/docs/cdp/sources/aws-ses.md` on posthog.com lists the new tables.
Implementation guide: `.agents/skills/implementing-warehouse-sources/SKILL.md`.
0 条评论