# CHORUS full/core reconciliation — 2026-07-28_claude-opus-5-generic_rep2

- Arm: BASELINE (input documents only)
- Method: claudecode_agent
- Runtime: Claude Code; provider Anthropic; model `claude-opus-5[1m]`; temperature 0.0
- Mode: four-phase project agent, generic prompt
  (`src/download/prompts/d4d_generic_arm_prompt.md`, identical for all projects)
- Declared input bundle: `data/preprocessed/concatenated/CHORUS_preprocessed.txt`
- Source manifest: `data/preprocessed/source_manifest.yaml`
- Full: `data/d4d_concatenated/claudecode_agent/2026-07-28_claude-opus-5-generic_rep2/CHORUS_d4d.yaml`
- Core: `data/d4d_concatenated/claudecode_agent_core/2026-07-28_claude-opus-5-generic_rep2/CHORUS_d4d_core.yaml`

## Referent selection

`Dataset` admits one referent. The chosen referent is **the CHoRUS multi-modal acute
and critical care dataset** — the data resource generated by the CHoRUS project — not
the NIH award, not the CHoRUS software organization, and not the AIM-AHEAD Bridge2AI
for Clinical Care training program. Award, software, and training-program facts appear
only in slots where they describe the dataset (funding, tooling used to produce it,
access terms, existing uses). This choice is held consistently across both records.

Consequences recorded during the audit:

- The MIT statement in the GitHub organization overview scopes to the `chorus-ai` code
  repositories. It is recorded inside `license_and_use_terms.license_terms` with that
  scope stated explicitly, and top-level `license` is left unset, because the bundle
  states no license for the clinical dataset.
- The $8,000 trainee stipend and travel allowance are training-program benefits, not
  compensation to data subjects. `participant_compensation` is therefore absent.
- HIPAA and GDPR appear in the bundle only as topics of a training curriculum workshop.
  No compliance determination is asserted; `regulatory_restrictions.hipaa_compliant`
  is unset.

## Phase 3 — source and provenance audit

### Provenance

No prior generated D4D record was read, opened, grepped, or consulted. Nothing under
`data/d4d_concatenated/`, `data/d4d_individual/`, or `data/ro-crate_packages/` was
accessed at any phase. The read history for this run consists of: the generic arm
prompt, `.claude/agents/d4d-provenance-guard.md`, `.claude/commands/d4d-full-core.md`,
`.claude/commands/d4d-agent.md`, the declared bundle, the CHORUS section of the source
manifest, and the two LinkML schemas (inspected via `SchemaView`, not by reading a
prior record as a template). Phase 2 additionally read the Phase 1 full record at the
exact same-run label.

### Source disagreements represented rather than merged

The bundle's four sources disagree on dataset size, and the disagreement is date- and
source-scoped rather than a contradiction. Both figures are carried as separate
`instances` entries with explicit attribution:

| Statement | Source | Representation |
|---|---|---|
| 50,000 patient admissions (ICU, PICU, NICU), 1.6 billion rows of EHR OMOP data, 7,642 admissions with radiology data, 23 Tb waveform data — "Current Released Dataset" | chorus4ai.org | four `instances` entries with `counts` |
| 14 hospitals, over 45K unique admissions, as of August 2025 | AIM-AHEAD cohort 2 webinar (2025-09-09) | separate `instances` entry; `counts` deliberately unset because "over 45K" is a bound, not a count |
| 100,000 patient admissions, 9 modalities, 14 contributing hospitals — "Anticipated Final Dataset" | chorus4ai.org | separate `instances` entry, `counts: 100000` |
| more than 100,000 critically ill patients | NIH RePORTER abstract | recorded in the anticipated-dataset entry's description alongside the website figure |

No merged or averaged figure appears anywhere in either record.

### Table-extraction ambiguity

The webinar's CHoRUS dataset table is column-scrambled by PDF-to-text extraction. The
data-type-to-standard pairing is unambiguous and is used (OMOP for demographics,
medication administration, procedures, nursing flowsheets, diagnoses; OHNLP for
clinical notes; DICOM for imaging; WFDB for waveform telemetry; EDF+/Persyst for
waveform EEG). The interleaved "Metadata" Yes/Planned column cannot be aligned to
specific rows with confidence, so per-row metadata status is not asserted; the
published metadata schemas are named in the descriptions of the rows where the
schema name itself resolves the alignment (OHNLP, DICOM, PhysioNet-extended,
EDF+/Persyst). Access control is recorded as controlled for all nine rows, which the
extraction does support unambiguously.

### Corrections applied during Phase 3

Three scoping defects were found in the Phase 1 record and corrected in full first,
then propagated to core by regenerating the projection:

1. `participant_privacy.privacy_techniques` asserted "dedicated de-identification
   tooling within the CHoRUS GitHub organization". The bundle lists a repository
   named `CTP-deid` with no description; the de-identification purpose was inferred
   from the repository name alone. The bullet was removed and the `Software` entry's
   description was rewritten to state only what the overview lists.
2. `at_risk_populations.special_protections` listed project-wide access and
   de-identification measures as if they were protections specific to at-risk groups.
   The measures were kept and the entry description now states explicitly that they
   are project-wide and that the bundle describes no pediatric- or neonatal-specific
   protection, assent procedure, or guardian consent.
3. `distribution_formats.access_urls` included `https://github.com/chorus-ai`. The
   GitHub organization distributes software and documentation, not the clinical
   dataset. The URL was removed from dataset access URLs; it remains where it belongs,
   in `extension_mechanism.contribution_url`.

No Phase 2 discovery required back-porting a new fact into the full record: core is a
strict schema projection of full, and every core-eligible fact the bundle supports was
already present in the Phase 1 record.

### Deliberate omissions (evidence absent)

`anomalies`, `known_biases`, `content_warnings`, `collection_notifications`,
`collection_consents`, `consent_revocations`, `informed_consent`,
`imputation_protocols`, `annotation_analyses`, `use_repository`, `discouraged_uses`,
`prohibited_uses`, `distribution_dates`, `ip_restrictions`, `errata`,
`retention_limit`, `version_access`, `variables`, `parent_datasets`,
`related_datasets`, `citation`, `doi`, `license`, `version`, `total_file_count`,
`total_size_bytes`, `dialect`. The bundle supports none of these. No consent or IRB
statement appears anywhere in the four sources, so no consent or ethics-approval slot
asserts one; `ethical_reviews` records only the community-facing ethics focus groups
and the consortium's ethics pillar, which are attested.

### Internal consistency checks (each record)

Award number `OT2OD032701` / project number `1OT2OD032701-01`, project period
2022-09-01 to 2026-11-30, 14 contributing hospitals within 20 academic centers, 60+
consortium members, the six named leadership-team members and their affiliations, and
the controlled-access/licensing-agreement/`.edu`-email access terms are each stated
identically everywhere they recur within each file. No internal contradiction found.

## Phase 4 — strict full/core reconciliation

Shared slots were derived at runtime with LinkML `SchemaView` from `Dataset`
(`data_sheets_schema_all.yaml`) and `CoreDataset` (`data_sheets_schema_core_all.yaml`);
no hand-written field list was used. The core record was **generated** by selecting
the schema-identical shared slots from the audited full record and copying their
parsed values, which makes deep identity structural rather than something to be
repaired afterwards. No narrative field was condensed, paraphrased, reordered, or
omitted in core.

- Schema-derived schema-identical shared slots: **76**
- Deterministic result: **PASS**, with no `--sync-core` step required
- Projected slots reported by the validator: `resources` (absent from both records)
- Full-only slots, absent from `CoreDataset` by schema: `file_collections`, `subsets`,
  `splits`, `relationships`, `direct_collection`, `participant_privacy`,
  `third_party_sharing`, `collection_notifications`, `collection_consents`,
  `consent_revocations`, `participant_compensation`, `variables`, `parent_datasets`,
  `related_datasets`, `citation`, `total_file_count`, `total_size_bytes`
- Core-only slots by schema: `distributions` (populated), `dialect` (unset)

### Related-content semantic review: `file_collections` → `distributions`

The validator emitted one warning, `semantic-review-required`, for
`$.file_collections <-> $.distributions`: 9 deterministic matches, 0 unmatched core
distributions. The review was performed and is recorded here; the warning is not
evidence that review occurred.

All nine modalities map one-to-one by `id`: demographics, medication administration,
procedures, nursing flowsheets, diagnoses, clinical notes, imaging, waveform telemetry,
waveform EEG. `id`, `name`, and `description` are identical across the pair. Every
remaining `CoreDistribution` field is unset in core because the bundle supplies no
value for it: no paths, byte counts, checksums (`hash`/`md5`/`sha256`), encodings,
compression, media types, or per-modality access URLs appear anywhere in the four
sources. `format` is unset because `FormatEnum` has no member for the standards the
bundle names (OMOP, OHNLP, DICOM, WFDB, EDF+/Persyst); those are carried in the
full record's `FileCollection.conforms_to`, which has no `CoreDistribution`
counterpart, and at dataset level in the shared `conforms_to` string, which is deeply
identical across the pair. `FileCollection.collection_type` (`processed_data` on all
nine) likewise has no core counterpart.

Scope comparisons that the playbook requires:

- `total_file_count` and `total_size_bytes` are unset in full and are not in the core
  schema, so there is no count or size to reconcile against distribution-level values.
  The only size figure in the bundle, "23 Tb waveform data", is scoped to waveforms
  alone and is recorded as text in the waveform telemetry description in both records
  rather than being promoted to a dataset-level byte count.
- The 7,642 admissions-with-radiology figure is scoped to the released dataset and is
  recorded both as an `instances` entry and in the imaging distribution description;
  the two statements agree and are attributed to the same source.
- `is_tabular: false` is identical in both records and does not conflict with the
  distributions: the dataset contains DICOM imaging and waveform data alongside
  tabular OMOP tables.
- Release scope: the imaging description carries both the September 2025 webinar
  figure (1000 images, de-identification in process) and the website's released-dataset
  figure, marked as separate sources and dates rather than treated as a contradiction.

Zero unresolved contradictions within or between the two records.

## Commands run

```bash
poetry run linkml-validate -s src/data_sheets_schema/schema/data_sheets_schema_all.yaml \
  -C Dataset data/d4d_concatenated/claudecode_agent/2026-07-28_claude-opus-5-generic_rep2/CHORUS_d4d.yaml

poetry run linkml-term-validator validate-data \
  data/d4d_concatenated/claudecode_agent/2026-07-28_claude-opus-5-generic_rep2/CHORUS_d4d.yaml \
  --schema src/data_sheets_schema/schema/data_sheets_schema_all.yaml --target-class Dataset

poetry run linkml-validate -s src/data_sheets_schema/schema/data_sheets_schema_core_all.yaml \
  -C CoreDataset data/d4d_concatenated/claudecode_agent_core/2026-07-28_claude-opus-5-generic_rep2/CHORUS_d4d_core.yaml

poetry run linkml-term-validator validate-data \
  data/d4d_concatenated/claudecode_agent_core/2026-07-28_claude-opus-5-generic_rep2/CHORUS_d4d_core.yaml \
  --schema src/data_sheets_schema/schema/data_sheets_schema_core_all.yaml --target-class CoreDataset

poetry run python -m data_sheets_schema.d4d_pair_consistency \
  --full data/d4d_concatenated/claudecode_agent/2026-07-28_claude-opus-5-generic_rep2/CHORUS_d4d.yaml \
  --core data/d4d_concatenated/claudecode_agent_core/2026-07-28_claude-opus-5-generic_rep2/CHORUS_d4d_core.yaml

poetry run d4d provenance record --project CHORUS --method claudecode_agent \
  --label 2026-07-28_claude-opus-5-generic_rep2 \
  --input-bundle data/preprocessed/concatenated/CHORUS_preprocessed.txt
```

All validations were re-run after the Phase 3 corrections.

## Files changed

- `data/d4d_concatenated/claudecode_agent/2026-07-28_claude-opus-5-generic_rep2/CHORUS_d4d.yaml` (created; three Phase 3 corrections applied)
- `data/d4d_concatenated/claudecode_agent_core/2026-07-28_claude-opus-5-generic_rep2/CHORUS_d4d_core.yaml` (created; regenerated after the Phase 3 corrections)
- `data/d4d_concatenated/claudecode_agent_core/2026-07-28_claude-opus-5-generic_rep2/CHORUS_reconciliation.md` (this report)
- `data/d4d_concatenated/claudecode_agent_core/2026-07-28_claude-opus-5-generic_rep2/CHORUS_provenance.yaml` (live provenance record)

## Final result

| Check | Result |
|---|---|
| Full schema validation (`Dataset`) | pass |
| Full term validation | pass |
| Core schema validation (`CoreDataset`) | pass |
| Core term validation | pass |
| Schema-identical shared slots | 76, deeply identical, identical presence |
| Pair consistency validator | PASS |
| Related-content semantic review | performed, 9/9 mapped, 0 contradictions |
| Full top-level slots populated | 53 (informational, not a quality gate) |
| Core top-level slots populated | 47 (informational, not a quality gate) |
