# CHORUS full/core reconciliation — 2026-07-28_claude-opus-5-generic_rep3

- **Project:** CHORUS
- **Arm:** BASELINE (input documents only)
- **Prompt:** `src/download/prompts/d4d_generic_arm_prompt.md` (identical text for all projects)
- **Runtime / provider / model:** Claude Code / Anthropic / claude-opus-5[1m]
- **Mode:** four-phase project agent, generic prompt
- **Declared input bundle:** `data/preprocessed/concatenated/CHORUS_preprocessed.txt`
- **Manifest:** `data/preprocessed/source_manifest.yaml`
- **Full:** `data/d4d_concatenated/claudecode_agent/2026-07-28_claude-opus-5-generic_rep3/CHORUS_d4d.yaml`
- **Core:** `data/d4d_concatenated/claudecode_agent_core/2026-07-28_claude-opus-5-generic_rep3/CHORUS_d4d_core.yaml`

## Pinned referent

`Dataset` admits one referent. The pinned referent is **the CHoRUS dataset** — the
multi-modal critical care dataset generated by the CHoRUS for Equitable AI data
generation project — not the NIH award, not the CHoRUS GitHub software
organization, and not the AIM-AHEAD Bridge2AI for Clinical Care training program.

This is the referent the bundle best supports: the project website publishes an
explicit dataset snapshot ("Current Released Dataset" / "Anticipated Final
Dataset") and the cohort 2 webinar publishes a per-modality dataset table with
standards and access control. The award, the software organization, and the
training program appear in the record only in slots that describe the dataset's
funding, tooling, access terms, and existing uses, and each such statement is
scoped in place to say which entity it is about. The referent is held identically
in both records.

## Phase 3 — source and provenance audit

### Provenance boundary

- Factual inputs read: the declared bundle and `data/preprocessed/source_manifest.yaml`.
- Structural inputs read: `data_sheets_schema_all.yaml` (class `Dataset`) and
  `data_sheets_schema_core_all.yaml` (class `CoreDataset`), resolved with LinkML
  `SchemaView` rather than by reading YAML by eye. Nested shapes, ranges,
  cardinality, inlining and enum ranges were taken from the induced classes.
- Instruction inputs read: the generic arm prompt, `.claude/agents/d4d-provenance-guard.md`,
  `.claude/commands/d4d-full-core.md`, `.claude/commands/d4d-agent.md`.
- **No previously generated D4D record was read, from any arm, label, or date.**
  Nothing under `data/d4d_concatenated/` was opened except this run's own outputs;
  the only directory listing taken there was of directory *names*, to confirm the
  run label was unused. No `*_crate_d4d.yaml` or `*_crate_mapped_d4d.yaml` was read.
  No web content was fetched.
- The core record's declared full input carries this run's exact label.

### Source disagreements found and how they are represented

1. **Admission count of the released data.** The cohort 2 webinar states that as of
   August 2025 the dataset "covers 14 different hospitals with over 45K unique
   admissions"; chorus4ai.org's "Current Released Dataset" panel states "50,000
   patient admissions from ICU, PICU, and NICU". Neither source dates its figure in
   a way that makes one strictly supersede the other, and the units differ ("unique
   admissions" vs "patient admissions"). Both are recorded as separate `instances`
   entries, each attributed to its source and its as-of scope, and the divergence is
   stated in the August 2025 entry's description. Neither was silently selected.
2. **Patients vs admissions in the target figure.** The NIH RePORTER abstract
   describes acquiring data "from more than 100,000 critically ill patients"; the
   website's "Anticipated Final Dataset" reports "100,000 patient admissions". These
   are different units for a similar magnitude. Both are recorded as separate
   `instances` entries with their own `instance_type` and source attribution; they
   are not merged.
3. **Imaging volume.** The webinar reports "currently 1000 images available" as of
   August 2025; the website reports "7,642 admissions with Radiology Data". These
   count different things (images vs admissions with radiology data) and are kept as
   distinct instances; both are also named in the imaging resource description.
4. **MIT license scope.** The GitHub organization README states "This project is
   licensed under the MIT License", and individual repositories show MIT and
   Apache-2.0 badges. Every dataset modality is simultaneously marked "Controlled"
   access, and dataset access requires a signed licensing agreement. These are two
   distinct entities. The top-level `license` slot is left unpopulated, and
   `license_and_use_terms.license_terms` states explicitly that MIT applies to the
   organization's software repositories rather than to the clinical dataset.
5. **Training-program eligibility vs dataset restrictions.** U.S. citizenship /
   permanent residency, W-9, degree and prior-programming requirements belong to
   AIM-AHEAD training-program admission, not to dataset access. They are recorded
   only in `regulatory_restrictions.other_compliance` with that scoping stated. The
   requirements that the sources do attach to dataset access — signed licensing
   agreement, `.edu` email address, registration and compute provisioning — are
   recorded in `license_and_use_terms` and `intended_uses.usage_notes`.
6. **IRB / HIPAA.** "Navigating IRB, Data Compliance, and Quality Assurance" and
   "HIPAA/GDPR compliance for OMOP/FHIR data" appear in the bundle only as titles of
   training curriculum sessions. They are **not** evidence of a dataset IRB approval
   or HIPAA determination. `human_subject_research.irb_approval`,
   `ethics_review_board` and `regulatory_restrictions.hipaa_compliant` are therefore
   left unpopulated, and `regulatory_compliance` says the sources name no IRB
   determination for the dataset.

### Extraction defect in the source bundle

The webinar's data-type table is column-scrambled by PDF text extraction. Data type
→ data standard adjacency survives intact and is used (Demographics / Medication
administration / Procedures / Nursing flowsheets / Diagnoses → OMOP; Clinical notes
→ OHNLP; Imaging → DICOM; Waveform telemetry → WFDB; Waveform EEG → EDF+ and
Persyst), as does the uniform "Controlled" access column and the set of published
metadata schemas. The **Metadata** column's Yes/Planned values cannot be assigned to
specific rows reliably, so no per-modality metadata-availability claim is made; the
ambiguity itself is recorded in `known_limitations`.

### Corrections applied during Phase 3

Corrections were made to the full record first, then the core record was
regenerated from the corrected full record.

1. `machine_annotation_tools` originally listed `privacy_scan_tool` and `CTP-deid`
   alongside the OHNLP toolkit. Those two are de-identification/privacy tools, not
   annotation tools, and are already recorded under `participant_privacy`. Removed;
   the slot now names only the OHNLP toolkit.
2. `subpopulations` originally carried a second entry ("Cohort balance and
   diversity") whose content was sampling *intent*, already present verbatim in
   `sampling_strategies`, and which asserted `subpopulation_elements_present: true`
   for something that is not an identified subpopulation. Removed. The remaining
   entry (ICU / PICU / NICU) is directly evidenced.
3. `external_resources[0].archival: false` removed. No source states an archival
   guarantee either way; the availability caveat that does exist (the site's
   administrative-review notice) is recorded in `future_guarantees` instead.

Nothing was back-ported from Phase 2 to Phase 1: the core field inventory is a
subset of the full inventory except for `distributions` and `dialect`, neither of
which the bundle supports, so Phase 2 surfaced no fact absent from the full record.

### Interpretive judgments recorded here for transparency

Three boolean slots encode a reading of the evidence rather than a quoted
statement, and are flagged as such:

- `acquisition_methods[0].was_directly_observed: true` — provider documentation,
  nursing flowsheets, bedside monitors and hospital EEG systems are instruments and
  direct clinical observation under the slot's definition.
- `acquisition_methods[0].was_inferred_derived: true` — note tokenization, OMOP
  standardization, and DeGauss geocoding produce derived content.
- `direct_collection[0].is_direct: false` — "Retrospective data collection" from
  hospital systems is not collection from the individuals described.

`known_biases[0].bias_type` was deliberately left unpopulated: the sources speak of
"bias" generally and of cohort balance, but do not name a bias category, and
selecting an enum value would be inference.

### Identifier minting

The bundle supplies no dataset DOI, accession, or persistent identifier. Required
`id` values are minted as URIs anchored on the evidenced project site
(`https://chorus4ai.org/`, with `#…` fragments for the nine modality resources,
organizations and persons). Software `id`s use `https://github.com/chorus-ai/<repo>`,
composed mechanically from the repository names the GitHub organization overview
lists. These are structural identifiers, not factual claims; no identifier asserts
anything the bundle does not state.

## Phase 4 — strict full/core reconciliation

Shared slots were derived at runtime from `Dataset` and `CoreDataset` via LinkML
`SchemaView`; no hand-written field list was used. The core record was produced by
filtering the validated full record's parsed content to the induced `CoreDataset`
slot inventory, so schema-identical slots are byte-for-byte the same parsed values —
core condenses, paraphrases, reorders and omits nothing.

- Full top-level slots populated: **54** (of 94 induced `Dataset` slots)
- Core top-level slots populated: **48** (of 79 induced `CoreDataset` slots)
- Schema-identical shared slots checked by the validator: **76** — all present-in-both
  or absent-in-both, all deeply identical
- Projected slots: **`resources`** (`Dataset` in full, `CoreDataset` in core)

### Full-only slots (correctly absent from core)

`relationships`, `splits`, `direct_collection`, `participant_privacy`,
`third_party_sharing`, `related_datasets` — none of these exist on `CoreDataset`.
Each is populated in the full record and is dropped from the core projection by the
schema, not by editorial choice.

### Core-only slots (correctly unpopulated)

`distributions` (`CoreDistribution`) and `dialect` (`FormatDialect`) exist only on
`CoreDataset`. The bundle contains no file inventory, path, checksum, byte count,
download URL, or delimiter/dialect information, so both are left empty. This is
consistent with `file_collections`, `total_file_count` and `total_size_bytes` also
being empty in the full record — the full→core `file_collections`→`distributions`
mapping has an empty domain and an empty range, with no conflict.

### `resources` projection

Nine modality resources are matched by `id`, with equal coverage in both records:

| id fragment | name | conforms_to |
|---|---|---|
| `#modality-demographics` | Demographics | OMOP |
| `#modality-medication-administration` | Medication administration | OMOP |
| `#modality-procedures` | Procedures | OMOP |
| `#modality-nursing-flowsheets` | Nursing flowsheets | OMOP |
| `#modality-diagnoses` | Diagnoses | OMOP |
| `#modality-clinical-notes` | Clinical notes | OHNLP |
| `#modality-imaging` | Imaging | DICOM |
| `#modality-waveform-telemetry` | Waveform telemetry | WFDB |
| `#modality-waveform-eeg` | Waveform EEG | EDF+ and Persyst |

Every nested slot used (`id`, `name`, `conforms_to`, `description`) is
schema-identical between `Dataset` and `CoreDataset`, and each is deeply identical
across the pair. No full-only nested slot was used on any resource, so nothing was
dropped in the projection.

### Semantic review of related, non-identical content

- **`file_collections` → `distributions`:** empty on both sides; no conflict.
- **`total_file_count` / `total_size_bytes` vs distribution-level values:** all
  empty; the one published volume figure ("23 Tb waveform data") is recorded as an
  `instances` entry and in the waveform telemetry resource description, in both
  records identically, and is not converted to a byte count because the source's
  unit ("Tb") is ambiguous between bits and bytes.
- **`dialect`, formats, `is_tabular`:** `dialect` and `is_tabular` are unpopulated
  in both. The dataset spans tabular OMOP content, DICOM imaging and binary
  waveforms, and no source asserts tabularity, so no value is inferred. Format
  statements live in `distribution_formats` and in each resource's `conforms_to`,
  and the two agree: OMOP / OHNLP / DICOM / WFDB / EDF+ and Persyst, matching the
  top-level `conforms_to` summary.
- **Top-level identity, version and access facts vs resources and repeated
  statements:** "controlled access" is asserted at top level, in
  `confidential_elements`, in `license_and_use_terms`, in `regulatory_restrictions`
  and on every one of the nine resources — consistent throughout. "14
  data-contributing hospitals" appears in `description`, `known_limitations`,
  `sampling_strategies`, `data_collectors`, `updates` and `version_access` with the
  same value. The 20-institution / 60+ member figures appear only where the sources
  state them (`creators`, `data_collectors`).
- **Historical vs current release:** the record distinguishes the August 2025 status
  report, the current released dataset, and the anticipated final dataset by
  explicit scope in every slot that mentions a count. These differing values are
  represented as different scopes, not as a contradiction.
- **`version_access`:** the bundle publishes no version identifier or DOI for the
  dataset, so `latest_version_doi` is empty and `versions_available` records the two
  published snapshots verbatim.

### Commands run

```bash
poetry run linkml-validate -s src/data_sheets_schema/schema/data_sheets_schema_all.yaml \
  -C Dataset data/d4d_concatenated/claudecode_agent/2026-07-28_claude-opus-5-generic_rep3/CHORUS_d4d.yaml

poetry run linkml-term-validator validate-data \
  data/d4d_concatenated/claudecode_agent/2026-07-28_claude-opus-5-generic_rep3/CHORUS_d4d.yaml \
  --schema src/data_sheets_schema/schema/data_sheets_schema_all.yaml --target-class Dataset

poetry run linkml-validate -s src/data_sheets_schema/schema/data_sheets_schema_core_all.yaml \
  -C CoreDataset data/d4d_concatenated/claudecode_agent_core/2026-07-28_claude-opus-5-generic_rep3/CHORUS_d4d_core.yaml

poetry run linkml-term-validator validate-data \
  data/d4d_concatenated/claudecode_agent_core/2026-07-28_claude-opus-5-generic_rep3/CHORUS_d4d_core.yaml \
  --schema src/data_sheets_schema/schema/data_sheets_schema_core_all.yaml --target-class CoreDataset

poetry run python -m data_sheets_schema.d4d_pair_consistency \
  --full data/d4d_concatenated/claudecode_agent/2026-07-28_claude-opus-5-generic_rep3/CHORUS_d4d.yaml \
  --core data/d4d_concatenated/claudecode_agent_core/2026-07-28_claude-opus-5-generic_rep3/CHORUS_d4d_core.yaml

poetry run d4d provenance record --project CHORUS --method claudecode_agent \
  --label 2026-07-28_claude-opus-5-generic_rep3 \
  --input-bundle data/preprocessed/concatenated/CHORUS_preprocessed.txt
```

`--sync-core` was **not** needed: the pair passed the independent consistency check
on the first run, because core was derived from the audited full record by
schema-driven projection rather than by re-authoring.

### Files changed

- `data/d4d_concatenated/claudecode_agent/2026-07-28_claude-opus-5-generic_rep3/CHORUS_d4d.yaml` (created; 3 Phase 3 corrections applied)
- `data/d4d_concatenated/claudecode_agent_core/2026-07-28_claude-opus-5-generic_rep3/CHORUS_d4d_core.yaml` (created; regenerated after Phase 3 corrections)
- `data/d4d_concatenated/claudecode_agent_core/2026-07-28_claude-opus-5-generic_rep3/CHORUS_provenance.yaml` (created by `d4d provenance record`, `record_mode: live`)
- `data/d4d_concatenated/claudecode_agent_core/2026-07-28_claude-opus-5-generic_rep3/CHORUS_reconciliation.md` (this report)

## Final results

| Check | Result |
|---|---|
| Full schema validation (`Dataset`) | PASS — no issues found |
| Full ontology term validation | PASS |
| Core schema validation (`CoreDataset`) | PASS — no issues found |
| Core ontology term validation | PASS |
| Schema-derived pair consistency | PASS — 76 schema-identical slots; projected slots `['resources']` |
| Provenance record | present, `record_mode: live` |
| Prior-D4D factual reuse | none — no prior generated record was read |
| Unresolved contradictions | none |
