# CHORUS full/core reconciliation

- **Run label:** `2026-08-07_claude-opus-5-claudecode-generic-v3_rep3`
- **Arm:** BASELINE (input documents only)
- **Mode:** four-phase project agent, generic prompt
- **Runtime / provider / model:** Claude Code / Anthropic / `claude-opus-5`
- **Reasoning effort:** `high` (observed from `$CLAUDE_EFFORT`)
- **Temperature:** 0.0
- **Declared input bundle:** `data/preprocessed/concatenated/CHORUS_preprocessed.txt`
- **Source manifest:** `data/preprocessed/source_manifest.yaml`
- **Full record:** `data/d4d_concatenated/claudecode_agent/2026-08-07_claude-opus-5-claudecode-generic-v3_rep3/CHORUS_d4d.yaml`
- **Core record:** `data/d4d_concatenated/claudecode_agent_core/2026-08-07_claude-opus-5-claudecode-generic-v3_rep3/CHORUS_d4d_core.yaml`

## Referent choice

`Dataset` admits one referent. The declared bundle describes two distinct things: the
CHoRUS critical care dataset, and the AIM-AHEAD Bridge2AI for Clinical Care Training
Program (Cohort 2) that grants access to it. **The referent chosen is the CHoRUS
dataset** — the multi-centre, multimodal, controlled-access critical care resource
generated by the CHoRUS data generation project. It is the only entity the bundle
describes with composition, standards, access controls and provenance.

Training-programme material (trainee eligibility, the $8,000 stipend, travel
allowances, application deadlines, mentorship matching, curriculum) is **not** recorded
as a dataset property. The only webinar content admitted is what the source states about
the dataset itself: its modality inventory and standards, its snapshot statistics, and
its access procedure (registration, provisioned compute, signed licensing agreement,
`.edu` email requirement). This scoping decision is stated in the full record's
`source_caveats` and held identically in both records.

`id` is set to the project's canonical URL `https://chorus4ai.org/` because the bundle
publishes no dataset identifier, accession or DOI.

## Phase 3 — source and provenance audit

### Provenance boundary

- Factual inputs were the declared bundle and `data/preprocessed/source_manifest.yaml`
  only. Structure was derived at runtime from the LinkML schemas using `SchemaView`
  (`Dataset` in `data_sheets_schema_all.yaml`; `CoreDataset` in
  `data_sheets_schema_core_all.yaml`), not from any example record.
- No prior D4D record — from any arm, label or date — was read, opened, grepped or
  consulted. Nothing under `data/d4d_concatenated/` was read other than this run's own
  two outputs, and no `*_crate_d4d.yaml` or `*_crate_mapped_d4d.yaml` was touched. This
  run is a replicate; the run it replicates was not inspected.
- No `d4d:docExample` value was copied. Every populated value traces to the bundle.
- No live web content was fetched.

### Source disagreements — represented, not resolved

| Fact | Webinar (Sept 2025 slides, "as of August 2025") | chorus4ai.org | Treatment |
|---|---|---|---|
| Admission count | 14 hospitals, "over 45K unique admissions" | "Current Released Dataset": 50,000 admissions from ICU, PICU, NICU | Two separate `instances` entries, each attributed to its source; the divergence is stated in the dataset-level and per-instance `source_caveats`. Neither figure was chosen over the other and they were not averaged or merged. |
| Total admissions target | — | "Anticipated Final Dataset": 100,000 | Recorded as a third, explicitly anticipated `instances` entry, corroborated by NIH RePORTER ("more than 100,000 critically ill patients"). |

The "over 45K" instance carries **no** `counts` value: the source gives a lower bound,
and an integer slot cannot express one. The bound is stated in that instance's
`description`.

### Shape and slot-filling findings, and the corrections made

1. **`notes` used for evidence commentary.** The `CHoRUS Consortium` creator carried an
   explanation of why `credit_roles` is unpopulated in `notes`. Under the slot-filling
   contract that is evidence commentary and belongs in `source_caveats`. **Corrected**
   in the full record and propagated to core. The full record now uses `notes` nowhere.
2. **Overstated repository count.** Three descriptions said "28 public repositories".
   The GitHub overview states `Repositories 28` and separately marks individual repos
   `Public`; it does not state that all 28 are public. **Corrected** to "28
   repositories" in all three places.
3. **Title scope.** `title` holds the NIH award / GitHub README project title. The
   sources give the resource itself only the informal names "CHoRUS Dataset" and
   "Bridge2AI for Clinical Care Dataset" and publish no formal dataset title. A caveat
   naming this, and the basis for the `id` value, was **added** to `source_caveats`.

### Mis-scoping actively avoided

- **Licences.** MIT (GitHub organization, `chorus_waveform`, `chorus-extract-upload`,
  `UF-Geocoding`) and Apache-2.0 (`Chorus_SOP`) attach to CHoRUS *code*. Top-level
  `license` is therefore **left empty**, and the software licences are recorded under
  `ip_restrictions` with their scope stated explicitly, and on the individual `Software`
  objects. The dataset's own terms — controlled access, signed licensing agreement,
  `.edu` email — sit in `license_and_use_terms`.
- **HIPAA / GDPR / IRB.** These appear in the bundle only as *topics of the AIM-AHEAD
  training curriculum*, never as determinations about CHoRUS. `hipaa_compliant`,
  `irb_approval`, `ethics_review_board` and `regulatory_compliance` are all left
  unpopulated, with the reason recorded in `source_caveats`.
- **Award period vs. clinical timeframe.** `collection_timeframes` carries
  `2022-09-01` / `2026-11-30`, and its name and `timeframe_details` state plainly that
  these are the NIH award dates during which acquisition occurs, not the clinical
  timeframe covered by the retrospective records, which no source states.
- **`data_use_permission` enum.** Not populated. The access procedure (`.edu` email,
  signed agreement) does not map cleanly onto any `DataUsePermissionEnum` value, and
  guessing an enum is worse than omitting it. The terms are stated in full in
  `license_terms`.
- **`identifiable_elements_present`.** Not populated. De-identification is described as
  in process, not complete; neither `true` nor `false` is supported. The state of the
  work is described in `is_deidentified.deidentification_details`.
- **`principal_investigator`.** `Creator.principal_investigator` has range `Person` and
  is *not* inlined — it takes an identifier reference. The bundle supplies no Person
  registry and the record has nowhere to define one, so writing a reference would create
  a dangling identifier. The PI role is stated in the Rosenthal creator's `description`
  instead. The same reasoning applies to `contact_person` on `LicenseAndUseTerms` and
  `EthicalReview`, and to `governance_committee_contact`.

### Verbatim transcriptions retained (with caveats)

- `cmccrary@mgh.havard.edu` — the domain appears misspelled in chorus4ai.org; reproduced
  as published rather than silently corrected, and flagged in `source_caveats` and on the
  maintainer entry.
- `"This repoitory is under review for potential modification in compliance with
  Administration directives."` — typographical error present in the source; quoted
  verbatim under `regulatory_restrictions` and flagged.
- `www.bridge2ai.org/chorus` — reproduced without a scheme, as published.
- Keywords are the NIH RePORTER "Preferred terms" list for award 1OT2OD032701-01,
  reproduced in full and in source order rather than curated, so the list is an
  observation rather than a selection.

### Extraction table caveat

The webinar's data-type table is partially garbled by PDF-to-text extraction. The
data-type → data-standard pairing and the `Controlled` access-control value for every
row are unambiguous and are recorded. The per-row alignment of the "Metadata" and
"Published metadata schema" columns is only partly recoverable; those values are
therefore recorded at the level the text supports and the limit is stated in
`source_caveats`.

### Slots deliberately left empty

`file_collections`, `total_file_count`, `total_size_bytes`, `subsets`, `variables`,
`anomalies`, `content_warnings`, `informed_consent`, `collection_consents`,
`collection_notifications`, `consent_revocations`, `participant_compensation`,
`imputation_protocols`, `annotation_analyses`, `discouraged_uses`, `prohibited_uses`,
`errata`, `retention_limit`, `distribution_dates`, `parent_datasets`, `doi`,
`download_url`, `version`, `license`, `citation`, `is_tabular`, `language`,
`publisher`, `created_on`, `issued`, `last_updated_on`, `resources`.

The bundle supports none of them. Two are worth naming: there is no file-level
inventory anywhere in the bundle (no paths, sizes, counts or checksums), so
`file_collections` is empty in full and `distributions` is empty in core; and no dated
release exists, so `distribution_dates` is empty while the two named dataset states are
recorded under `version_access` where they belong.

### Post-correction validation

Both files were re-validated after every correction; results in the final section.

## Phase 4 — strict full/core reconciliation

### Shared-slot derivation

Shared slots were derived at runtime with LinkML `SchemaView` over `Dataset` and
`CoreDataset`. No hand-written field list was used. The core record was constructed by
projecting the Phase-3-audited full record through the runtime-derived `CoreDataset`
slot inventory, so schema-identical slots are byte-for-byte the same content rather
than a re-authored paraphrase.

**Validator result: `PASS: 78 schema-identical slots; projected slots=['resources']`.**

### Full-only slots (no `CoreDataset` counterpart)

Six populated top-level slots exist in `Dataset` and not in `CoreDataset`, so they are
correctly absent from core rather than divergent:

| Full-only slot | Where the content survives in core |
|---|---|
| `relationships` | Modality-to-admission organisation is described in the relevant `instances` entries. |
| `splits` | The sequestered holdout test set appears in `purposes`, `tasks` and `intended_uses`. |
| `direct_collection` | Retrospective, non-direct collection is described in `acquisition_methods` (`was_reported_by_subjects: false`) and `collection_mechanisms`. |
| `third_party_sharing` | Controlled release to approved external users is described in `license_and_use_terms` and `regulatory_restrictions`. |
| `participant_privacy` | Privacy techniques, tokenization, local note retention and the controlled enclave appear in `is_deidentified`, `data_protection_impacts` and `at_risk_populations.special_protections`. |
| `related_datasets` | The Bridge2AI part-of relationship is stated in the dataset `description` and in `purposes`. |

No factual content is lost by the projection; only its structural home changes.

### Core-only slots

`distributions` (`CoreDistribution`) and `dialect` (`FormatDialect`) exist only in
`CoreDataset`. Both are left empty: the bundle contains no file-level inventory (no
paths, byte counts, checksums, encodings or media types) and no tabular dialect
information. This is consistent with `file_collections` being empty in full.

### Projected slot

`resources` is `Dataset` in full and `CoreDataset` in core. It is absent from **both**
records — the bundle describes no component sub-datasets distinct from the modality
inventory already carried in `instances` — so coverage is trivially equal and the
projection holds.

### Related, non-identical representation review

- **`file_collections` → `distributions`:** both empty; nothing to map, no conflict.
- **`total_file_count` / `total_size_bytes` vs. distribution-level values:** all unset in
  both records; no scope mismatch is possible.
- **`dialect`, formats, `is_tabular`:** `dialect` unset; `is_tabular` unset in both,
  which is the correct answer for a resource mixing tabular OMOP data with DICOM
  imaging and WFDB/EDF+ waveforms. `distribution_formats` is deeply identical across the
  pair (five entries: OMOP, OHNLP, DICOM, WFDB, EDF+/Persyst) and agrees with
  `conforms_to` and with the per-modality `instances`.
- **Top-level identity / version / access facts:** `id`, `name`, `title`, `page`,
  `status`, `conforms_to`, `was_derived_from` are identical across the pair and agree
  with `version_access` (two named dataset states, no formal version or DOI),
  `license_and_use_terms` and `regulatory_restrictions` (controlled access on every
  modality) and `updates` (ongoing accrual toward the anticipated final dataset).
- **Historical vs. current release:** the August 2025 webinar snapshot, the current
  released dataset and the anticipated final dataset are three distinct scopes, each
  labelled with its scope in both records. They are not treated as a contradiction, and
  the same three-way distinction appears identically in `instances`, `version_access`,
  `updates` and `known_limitations` in both files.

**Unresolved contradictions within or between the two records: none.**

### Slot counts (informational metadata, not a quality gate)

| Record | Populated top-level slots |
|---|---|
| Full (`Dataset`) | 58 |
| Core (`CoreDataset`) | 52 |

The difference is exactly the six full-only slots listed above.

## Commands run

```bash
poetry run linkml-validate -s src/data_sheets_schema/schema/data_sheets_schema_all.yaml \
  -C Dataset data/d4d_concatenated/claudecode_agent/2026-08-07_claude-opus-5-claudecode-generic-v3_rep3/CHORUS_d4d.yaml

poetry run linkml-term-validator validate-data \
  data/d4d_concatenated/claudecode_agent/2026-08-07_claude-opus-5-claudecode-generic-v3_rep3/CHORUS_d4d.yaml \
  --schema src/data_sheets_schema/schema/data_sheets_schema_all.yaml --target-class Dataset

poetry run linkml-validate -s src/data_sheets_schema/schema/data_sheets_schema_core_all.yaml \
  -C CoreDataset data/d4d_concatenated/claudecode_agent_core/2026-08-07_claude-opus-5-claudecode-generic-v3_rep3/CHORUS_d4d_core.yaml

poetry run linkml-term-validator validate-data \
  data/d4d_concatenated/claudecode_agent_core/2026-08-07_claude-opus-5-claudecode-generic-v3_rep3/CHORUS_d4d_core.yaml \
  --schema src/data_sheets_schema/schema/data_sheets_schema_core_all.yaml --target-class CoreDataset

poetry run python -m data_sheets_schema.d4d_pair_consistency \
  --full data/d4d_concatenated/claudecode_agent/2026-08-07_claude-opus-5-claudecode-generic-v3_rep3/CHORUS_d4d.yaml \
  --core data/d4d_concatenated/claudecode_agent_core/2026-08-07_claude-opus-5-claudecode-generic-v3_rep3/CHORUS_d4d_core.yaml

poetry run d4d provenance record --project CHORUS --method claudecode_agent \
  --label 2026-08-07_claude-opus-5-claudecode-generic-v3_rep3 \
  --input-bundle data/preprocessed/concatenated/CHORUS_preprocessed.txt \
  --prompt src/download/prompts/d4d_generic_arm_prompt.md

poetry run d4d runs validate --label 2026-08-07_claude-opus-5-claudecode-generic-v3_rep3 --project CHORUS

poetry run d4d runs check --strict --label 2026-08-07_claude-opus-5-claudecode-generic-v3_rep3
```

`--sync-core` was not needed: the core record was projected from the audited full
record, so the pair was already deeply identical on every schema-identical slot when the
independent check was first run.

## Files changed by this run

- `data/d4d_concatenated/claudecode_agent/2026-08-07_claude-opus-5-claudecode-generic-v3_rep3/CHORUS_d4d.yaml` (created)
- `data/d4d_concatenated/claudecode_agent_core/2026-08-07_claude-opus-5-claudecode-generic-v3_rep3/CHORUS_d4d_core.yaml` (created)
- `data/d4d_concatenated/claudecode_agent_core/2026-08-07_claude-opus-5-claudecode-generic-v3_rep3/CHORUS_reconciliation.md` (created)
- `data/d4d_concatenated/claudecode_agent_core/2026-08-07_claude-opus-5-claudecode-generic-v3_rep3/CHORUS_provenance.yaml` (created by `d4d provenance record` / `d4d runs validate`)

No file outside this run's CHORUS-prefixed outputs was modified.

## Final results

| Check | Result |
|---|---|
| Full — LinkML schema validation (`Dataset`) | PASS ("No issues found") |
| Full — ontology term validation | PASS |
| Core — LinkML schema validation (`CoreDataset`) | PASS ("No issues found") |
| Core — ontology term validation | PASS |
| Full/core pair consistency | PASS — 78 schema-identical slots, projected slots `['resources']` |
| Phase 3 corrections applied | 3 (see above), all re-validated |
| Unresolved full/core divergence | none |
