Feature request: Google Analytics configurable custom reports
enhancementteam/data-warehousefeature/pipeline-sources
<!-- req-ref: 1786124993.424429 -->
## What & why
The Google Analytics (GA4) data warehouse source currently only exposes 10 fixed report tables (`website_overview`, `daily_active_users`, `weekly_active_users`, `four_weekly_active_users`, `devices`, `locations`, `pages`, `traffic_sources`, `user_acquisition`, `events`). Each is a hardcoded `dimensions`/`metrics` combination. Users who need a different combination — for example campaign-grain acquisition, a landing-page report, key events, custom dimensions, or demographics — have no way to get it without a code change to add another fixed table.
GA4's own Data API (`runReport`) already supports arbitrary dimension × metric combinations per request — this isn't an API limitation. [Fivetran's GA4 connector](https://fivetran.com/docs/connectors/applications/google-analytics-4) solves this by letting users define their own custom report tables (up to 9 dimensions × 10 metrics each, date-keyed and synced incrementally), instead of shipping only a fixed table set.
Letting users define their own dimension/metric combinations, the way GA4's API and Fivetran's connector both support, removes the need to add and maintain a new fixed table for every new report shape someone asks for. This is the highest-leverage single change for this source: it would subsume most other report-shape requests (campaign-grain acquisition, landing-page report, key events, custom dimensions, demographics) rather than requiring a separate fixed table per request.
## Implementation plan
**Research findings**
- GA4 Data API `runReport`: [API schema reference](https://developers.google.com/analytics/devguides/reporting/data/v1/api-schema). Per Google's documented limits, a single `runReport` request supports up to 9 dimensions and 10 metrics. The current source already calls `runReport` with a fixed `dimensions`/`metrics` list per table — see `_run_report()` in `products/warehouse_sources/backend/temporal/data_imports/sources/google_analytics/google_analytics.py`. Extending it to accept a caller-supplied dimension/metric list needs no API-side change.
- Fivetran's GA4 connector: [custom reports docs](https://fivetran.com/docs/connectors/applications/google-analytics-4). This is the target UX pattern — the user names a report, picks up to 9 dimensions and up to 10 metrics from GA4's schema, and the connector syncs it as its own table, date-keyed and incremental, alongside (not instead of) the standard tables.
**Current structure (for context)**
- `products/warehouse_sources/backend/temporal/data_imports/sources/google_analytics/settings.py` — `GOOGLE_ANALYTICS_REPORT_SCHEMAS`, a `dict[str, GoogleAnalyticsReportSchema]` mapping a fixed table name to a fixed `dimensions`/`metrics`/`primary_key` combination. This is the dict a "custom report" would need to extend or supplement at runtime instead of at code-authoring time.
- `products/warehouse_sources/backend/temporal/data_imports/sources/google_analytics/google_analytics.py` — `google_analytics_source()` looks up a schema by `resource_name` in `GOOGLE_ANALYTICS_REPORT_SCHEMAS` and calls `_run_report()` with its `dimensions`/`metrics`. `get_schemas()` in `source.py` builds the list of syncable tables directly from that same dict.
- `products/warehouse_sources/backend/temporal/data_imports/sources/google_analytics/source.py` — `GoogleAnalyticsSource.get_source_config` (the `SourceConfig` fields definition, i.e. this source's config serializer/schema) currently only declares the OAuth integration field and `property_id`. Any user-defined report config would be declared here, and `GoogleAnalyticsSourceConfig` (generated from `products/warehouse_sources/backend/temporal/data_imports/sources/generated_configs/googleanalytics.py`) would need a corresponding field to carry it — regenerate via `hogli build:openapi` after the serializer change.
- No existing warehouse source in this repo has a "user-defined table schema" mechanism to copy from — a repo-wide check (Google Ads' GAQL usage, and a broad search across `products/warehouse_sources/backend/temporal/data_imports/sources` for `custom_report`/`user_defined`/dynamic-schema patterns) found none. This is a genuinely new mechanism for the source framework, not a variation on an existing one.
**Concrete steps (proposed minimal viable slice)**
Rather than building full multi-table configurable reporting in one PR, scope the first slice to **one configurable custom-report table per connection**:
1. Add a new config field (e.g. `custom_report_dimensions` / `custom_report_metrics`, or a single structured field) to the source's `SourceConfig` in `source.py`, and to the generated config dataclass in `generated_configs/googleanalytics.py` (via `hogli build:openapi` after the serializer/schema change).
2. In `settings.py`, either extend `GOOGLE_ANALYTICS_REPORT_SCHEMAS` to be built dynamically per-config (fixed tables + one optional custom one) or add a separate path that constructs a `GoogleAnalyticsReportSchema` from the user's config at sync time. `date` must remain a mandatory dimension (as all fixed tables enforce today) so day-grained incremental sync keeps working.
3. In `google_analytics.py`, thread the user-supplied dimensions/metrics through to `_run_report()` for the custom table's `resource_name`, respecting GA4's 9-dimension / 10-metric request limit (validate and surface a clear error if exceeded, matching the style of existing validation in `validate_credentials()`).
4. In `source.py`, include the custom table (when configured) in `get_schemas()` output so it shows up in the schema picker like any fixed table.
5. If a frontend config form exists for this source's setup flow (check for a Google Analytics-specific component under `products/warehouse_sources/frontend/` or the generic source-config form that renders `SourceConfig.fields`), confirm the new field(s) render sensibly — a multi-select or dimension/metric picker is likely needed rather than a plain text field, since users should be picking from GA4's actual dimension/metric names (available via the existing `get_property_metadata()` call in `google_analytics.py`, already used for credential validation).
6. Tests: extend `products/warehouse_sources/backend/temporal/data_imports/sources/google_analytics/tests/test_google_analytics.py` and `test_google_analytics_source.py` to cover a configured custom report syncing correctly, the dimension/metric limit validation, and `get_schemas()` including/excluding the custom table based on config presence.
**Open questions/risks**
- This is the largest and most product-judgment-heavy item in the GA4 source wishlist — a maintainer should confirm scope and UX (single custom table vs. multiple, structured picker vs. free text, whether dimension/metric names are validated against `get_property_metadata()` at config time) before an agent attempts full implementation.
- A first PR might reasonably deliver a narrower slice still — e.g. just adding the fixed campaign-grain acquisition table and/or landing-page report as additional entries in `GOOGLE_ANALYTICS_REPORT_SCHEMAS` (no new config surface) — rather than the full configurable-report engine described above. That narrower campaign-grain/landing-page work is noted here as a design option and has **not** been filed as its own separate issue this round.
- Decide whether the custom report reuses the same incremental/resumable machinery (`ResumableSourceManager`, `_iter_chunks`, lookback window) as the fixed tables, or needs its own — the plan above assumes reuse, since the underlying `_run_report()` call shape is identical.
0 条评论