DataQ

Data quality monitoring platform built around Great Expectations — Snowflake (DEV/QA/UAT), ADLS Gen2, S3, Unity Catalog (Databricks), Apache Iceberg (native read), with ADF + Airflow + dbt orchestration integrations.

📖 Documentation site: https://theurgicduke771.github.io/DataQ/ (MkDocs Material — quickstart, concepts, architecture, guides).

Status: v1.0.0 released (2026-07-04) — the 8-week roadmap is complete (187/189; 2 items consciously re-scoped to post-v1). v1 is deployed to Azure Container Apps — API + worker + a runtime-configured frontend Container App (the sole public surface; the api runs on internal ingress behind it), with Key Vault, App Insights, and orchestration polling live. Auth is a generic OIDC client (validated against Azure AD; ADR 0028). The completed v1 ledger is archived at docs/progress-v1.md; live post-v1 progress at docs/progress.md.

What it does

Checks on 5 datasources — Snowflake, Unity Catalog (Databricks), ADLS Gen2 + S3 flat files (CSV/Parquet, batch patterns), and Apache Iceberg (native pyiceberg read). Five check styles: GX expectations, custom SQL (rows returned = failures), freshness / volume monitors, and comparison (reconcile two datasets across connections — ADR 0015); plus a column profiler and dry-run preview on every datasource. Feature matrix →
Assets, lineage & incidents — the table/file is a first-class entity: health rolled up across every suite that targets it, table-level lineage (a left-to-right graph of provenance and blast radius, from dbt's manifest or an OpenLineage catalog), and open incidents. Assets are the primary lens — the dashboard and sidebar lead with them (ADR 0034). Concepts →
Quality by dimension — every check is classified (accuracy, completeness, consistency, integrity, timeliness, uniqueness, validity), filled in automatically from the check type. The asset scorecard turns that into the question people actually ask: not just "are the checks passing" but "what isn't being watched at all" — an asset with no Timeliness checks says so, and never shows a green tick for it. Dimensions →
Three run modes — run now (live progress + cancel), cron schedules (timezone/DST-aware), and pipeline triggers: ADF, Airflow and dbt runs are monitored, and a successful pipeline can trigger the bound suite. Scheduling → · Orchestration →
Severity + alerting — warn/fail/critical tiers band each check's unexpected-%; alerts to Teams / Slack / email with severity-aware routing, first-failure dedup, and per-check snooze. Notifications →
Results you can share — dashboard health score + trends, per-run drill-down with PII-redacted failing-row samples, suite-level sharing (view/edit), admin control centre. Best practices →

Stack

Layer	Tech
Backend	FastAPI · Celery · Great Expectations · SQLAlchemy + Alembic · PostgreSQL · Redis
Frontend	React · Vite · Ant Design · generic OIDC (`oidc-client-ts`)
Auth / secrets	OIDC — Azure AD validated (`AUTH_*` contract, provider-neutral) · Azure Key Vault
Hosting	Azure Container Apps (API · worker · frontend) + Application Insights (deployed)
AI integration	FastMCP — 8 curated MCP tools at `/mcp` for Claude Desktop / Copilot / Cursor

Quick start

Run DataQ — prebuilt images (recommended)

Evaluate or self-host in ~5 minutes: no source checkout, no Azure tenant. Just Docker.

curl -O https://raw.githubusercontent.com/TheurgicDuke771/DataQ/main/docker-compose.ghcr.yml
export OPENBAO_TOKEN=$(openssl rand -hex 16)     # root token for the bundled vault
export DATAQ_SIGNIN_EMAIL=you@example.com        # the address allowed to sign in
docker compose -f docker-compose.ghcr.yml up

Open http://localhost:3000, enter that address, and read the 6-digit code in the bundled inbox at http://localhost:8025 — the stack ships its own mail catcher (Mailpit, MIT), so email sign-in works with no SMTP relay and nothing leaves your machine. It comes up migrated and seeded with demo data. API + Swagger at http://localhost:8000/docs. Images are pulled from GHCR and are multi-arch (amd64 + arm64), so Apple-Silicon runs native. Ports bind to 127.0.0.1 only. Connection credentials go into a bundled OpenBao vault (ADR 0039) rather than the plaintext store it replaced, which is the one value you supply — it runs in dev mode (in-memory), so restarting the stack means re-entering connection credentials. To pin a release instead of the moving tags: DATAQ_BACKEND_TAG=vX.Y.Z DATAQ_FRONTEND_TAG=vX.Y.Z docker compose -f docker-compose.ghcr.yml up.

Prefer no sign-in at all? DATAQ_SIGNIN_EMAIL= DATAQ_AUTH_MODE=bypass docker compose -f docker-compose.ghcr.yml up is the explicit downgrade — every request becomes one fixed admin user. Omitting DATAQ_SIGNIN_EMAIL entirely stops the stack instead of choosing that for you.

Self-hosting with your own Azure AD? The published frontend is one generic image — the compose eval runs it with DATAQ_AUTH_MODE=otp. For real SSO, no rebuild: run that same image with DATAQ_AUTH_MODE=oidc + DATAQ_AUTH_AUTHORITY / DATAQ_AUTH_CLIENT_ID / DATAQ_AUTH_API_SCOPE (auth config is injected at runtime, ADR 0028), and run the backend with AUTH_DEV_BYPASS off. See Getting started.

Develop DataQ — from source

git clone https://github.com/TheurgicDuke771/DataQ.git
cd DataQ
./scripts/setup.sh       # conda env + pre-commit + docker-compose + migrations
conda activate dataq
docker-compose up

Backend at http://localhost:8000 (Swagger at /docs), frontend at http://localhost:3000. setup.sh asks which address may sign in and writes it to your gitignored .env; codes land in the bundled inbox at http://localhost:8025. Answering blank is the explicit downgrade to dev-bypass.

MCP (AI assistant access)

DataQ exposes 8 curated MCP tools at /mcp (streamable HTTP) — list_suites, get_suite_results, get_health_score, get_adf_pipeline_status, trigger_suite_run, get_run_status, create_check, profile_column. /mcp mounts under any configured sign-in mode — Azure AD SSO, email OTP (ADR 0032), or local dev-bypass — and stays unmounted, fail-closed, when none is configured (ADR 0008). Under SSO, present the same Azure AD bearer token the web UI uses; under email OTP the only accepted credential is a DataQ API key (dq_live_…, a PAT — ADR 0026) — a raw JWT and a browser session cookie are both rejected there, since there is no IdP to validate a bearer against and a session is a browser-only credential.

Point any MCP client at https://<your-dataq-host>/mcp/ (keep the trailing slash — /mcp 307-redirects and some clients drop the Authorization header on redirect) with an Authorization: Bearer <token> header. Once configured, all 8 tools are available to natural-language queries (e.g. "what failed in the orders suite today?", "run the orders suite on DEV").

Per-client configuration (Claude Desktop / Claude.ai, VS Code / Copilot, Cursor), how to get a token, token hygiene, and troubleshooting: AI assistants (MCP setup) on the docs site.

Documentation


Working agreements + commit/PR conventions	CONTRIBUTING.md
Documentation site (user guides)	https://theurgicduke771.github.io/DataQ/ · source in docs/, built by .github/workflows/docs.yml
Deployment guide + env-var reference	deploy/README.md · .env.app.example
Project guide for AI assistants	CLAUDE.md
Architecture diagram + invariants	docs/architecture.md
Architecture Decision Records	docs/adr/
Live task tracker	docs/progress.md
Product roadmap (8 weeks, 100 tasks)	context/DataQ_platform_roadmap.md
Security policy + responsible disclosure	SECURITY.md

License

MIT — see LICENSE.