Feature request: Klaviyo suppression & consent status coverage
enhancementteam/data-warehousefeature/pipeline-sources
<!-- req-ref: 1786125059.910269 -->
## What & why
Teams migrating email/SMS sending off Klaviyo onto a new platform need a complete suppression and consent export before cutting over. Fivetran's Klaviyo connector ships dedicated `GLOBAL_EXCLUSION` and `LIST_EXCLUSION` tables for exactly this reason. PostHog's Klaviyo data warehouse source has no equivalent, and today the only place suppression/consent data can show up is nested inside the `profiles` table's `subscriptions` object.
Without a complete unsubscribe/suppression/SMS-consent export, a migration to a new sending platform risks messaging users who already opted out. That's a CAN-SPAM/TCPA compliance risk, which is why closing this gap is the top-priority ask from the reporting user, not a nice-to-have.
## Implementation plan
**Current state (already in the codebase — verify before doing anything else):**
`products/warehouse_sources/backend/temporal/data_imports/sources/klaviyo/settings.py` already sets, on the `profiles` endpoint config:
```python
extra_params={"additional-fields[profile]": "subscriptions"},
```
Per Klaviyo's [Get Profiles](https://developers.klaviyo.com/en/reference/get_profiles) docs, this `additional-fields[profile]=subscriptions` param adds a `subscriptions` object to every profile in the API response, keyed by channel (`email`, `sms`, `mobile_push`). For `email`, this includes `subscriptions.email.marketing.suppression` (array of suppression reasons, e.g. `USER_SUPPRESSED`, `INVALID_EMAIL`, `UNSUBSCRIBE`) and `subscriptions.email.marketing.list_suppressions` (per-list suppression entries) nested under the same object. The comment in `settings.py` (lines 224-228) confirms this was deliberately added and notes it carries no rate-limit penalty (unlike `predictive_analytics`, which drops the rate limit from 75/s to 10/s and stays excluded).
So the core compliance-relevant data — email marketing suppression status and list-level suppression — **should already be landing in the `profiles` table** for any team syncing that table today. This needs to be confirmed against a real Klaviyo account with suppressed profiles (or Klaviyo's sandbox/test data) before concluding anything is actually missing:
1. Run a sync against an account with at least one globally suppressed profile and one profile with a list-specific suppression, and inspect the raw synced row. Confirm `subscriptions.email.marketing.suppression` and `subscriptions.email.marketing.list_suppressions` (and the SMS equivalents under `subscriptions.sms.marketing`, if present) actually appear in the row's properties/payload column, not just in the raw API response.
2. If the fields are present and complete, the remaining work is likely: (a) making this data easy to find/query (e.g. documentation, or promoting key sub-fields like `is_suppressed`/`suppression_reasons` to top-level columns instead of leaving them buried in a nested JSON blob), and (b) confirming SMS consent (`subscriptions.sms.marketing.consent`) is covered the same way — the "subscriptions" additional-fields value is a single opt-in per profile that isn't channel-scoped, so it should already include SMS, but this hasn't been explicitly verified against a live account with SMS consent activity.
**Known gap that is *not* covered by the existing `additional-fields[profile]=subscriptions` param:**
The `list_profiles` and `segment_profiles` fan-out tables (which walk `/lists/{list_id}/profiles` and `/segments/{segment_id}/profiles`) explicitly restrict their profile payload via `extra_params={"fields[profile]": "joined_group_at"}` — a sparse fieldset that returns *only* `joined_group_at` and excludes `subscriptions` entirely (see the comment on the `profiles` endpoint: "The list_profiles/segment_profiles fan-outs are unaffected"). This means per-list suppression status is not visible from those tables today. That's the closest analogue to Fivetran's `LIST_EXCLUSION` table, and it's currently a real gap:
- Decide whether closing it requires changing `list_profiles`/`segment_profiles` to also request `additional-fields[profile]=subscriptions` (increases per-row payload size for a fan-out that already costs one paginated request per list/segment — check whether Klaviyo's per-page size cap or rate limit changes when both `fields[profile]` and `additional-fields[profile]` are combined), or whether it's simpler to rely on `profiles.subscriptions.email.marketing.list_suppressions` (which already carries the list ID per suppression entry) and treat that as the join-table equivalent, avoiding a second per-list API cost.
- **Recommendation:** prefer relying on `profiles.subscriptions.*.marketing.list_suppressions` over adding a dedicated per-list/per-segment suppression table. It's the lower-risk change (no new fan-out, no new rate-limit exposure) and, if the field actually contains list-scoped suppression entries as documented, it fully covers the compliance need without a schema change. Only add a dedicated table if step 1 above shows `list_suppressions` is missing list-level granularity that a migrating team actually needs.
**Concrete steps once the audit above confirms what's missing:**
- If nothing is missing: no `settings.py` change needed for `profiles` — add/update the table description to call out that `subscriptions` (including suppression and list suppression) is included by default, so support and users don't have to read the source code to discover it.
- If SMS consent or list-level suppression turns out to be incomplete: extend the `extra_params` on the affected endpoint(s) accordingly, following the existing pattern in `KLAVIYO_ENDPOINTS["profiles"]`.
- No changes should be needed to the existing incremental/pagination logic for `profiles` — `additional-fields[profile]` only changes the shape of each row's payload, not the `updated`/`created` incremental cursor or `page[size]` pagination.
- Tests to add: a fixture-based test asserting the `profiles` request is built with `additional-fields[profile]=subscriptions` in its query params (guard against a future edit accidentally dropping it), and a test asserting a fixture profile response containing `subscriptions.email.marketing.suppression` and `list_suppressions` round-trips into the synced row's columns/properties as expected.
## Open questions/risks
- Unconfirmed: does combining `additional-fields[profile]=subscriptions` with `fields[profile]=joined_group_at` on the `list_profiles`/`segment_profiles` fan-outs change Klaviyo's rate-limit cost or the effective page size cap, if that combination is attempted? Verify against Klaviyo's rate-limit docs or a live test call before shipping any change there.
- Unconfirmed: whether `subscriptions.sms.marketing` and `subscriptions.mobile_push.marketing` are populated with the same completeness as `subscriptions.email.marketing` for accounts that use SMS/push — needs verification against a live account with SMS consent/suppression activity, not just docs.
0 条评论