# Interleaved Semantic Evaluation

Project: CHORUS · Method: claudecode_agent
YAML: `data/d4d_concatenated/claudecode_agent/CHORUS_d4d.yaml`
R10 JSON: `data/evaluation_llm/rubric10_semantic/concatenated/CHORUS_claudecode_agent_evaluation.json`
R20 JSON: `data/evaluation_llm/rubric20_semantic/concatenated/CHORUS_claudecode_agent_evaluation.json`
Model: `claude-sonnet-4-5-20250929`
Rubric10 (semantic)
42.0/50 (84.0%)
Rubric20 (semantic)
74.0/84 (88.1%)
Consistency checks (R10/R20)
28 pass · 6 fail · 7 warn
Mapped feedback / fields
65 across 29 fields
R10 sub-element R20 question Semantic issue

### Strengths

- Exceptional structural completeness with all mandatory fields populated and comprehensive narrative depth (1,247 char description, 28 keywords, 4 detailed purposes)
- Outstanding funding documentation with complete NIH grant details (1OT2OD032701-01, $5.88M, fiscal timeline) and 19 creators with institutional affiliations across 14 data acquisition centers
- Comprehensive multi-modal data standardization with 5 internationally recognized formats (OMOP, WFDB, DICOM, OHNLP, EDF+/Persyst) and published schema metadata for all modalities
- Excellent technical documentation with specific software tools named, GitHub organization documented (28 repositories including chorus-ai/Chorus_SOP, chorus-mapping), and detailed preprocessing/cleaning/labeling strategies
- Strong FAIR compliance with multiple persistent identifiers (project URL, publication DOI 10.1007/s12028-024-02007, NIH RePORTER), clear controlled access mechanism (Azure secure enclave, DUA), and exceptional interlinking across 8 platforms
- Robust ethical framework with 14 IRBs, HIPAA/45 CFR 46 compliance, community ethics focus groups, de-identification across all modalities, and Social Determinants of Health considerations
- Well-documented access governance with explicit controlled access model, institutional email requirement, signed licensing agreement, access contacts provided, and clear permitted/prohibited uses
- Comprehensive collection protocol with 4 collection mechanisms, 5 acquisition methods, multi-site coordination details, and complete timeline (9/2022-11/2026 with 8/2025 snapshot)

### Weaknesses

- Missing specific IRB protocol numbers from any of the 14 institutional review boards despite comprehensive ethics documentation
- No explicit deidentification method summary field (HIPAA Safe Harbor vs Expert Determination) though methods described across preprocessing strategies
- Lacks formal version numbering system and structured release notes despite good snapshot-based progress tracking (August 2025 status)
- Missing specific demographic breakdowns (age ranges, race/ethnicity distributions) and explicit inclusion/exclusion criteria for patient enrollment
- Minor grant number format inconsistency: license field shows OT2OD032701 while funders shows 1OT2OD032701-01
- No explicit informed consent approach documented (likely retrospective waiver but not stated)
- Missing participant compensation details and explicit vulnerable population safeguards (though PICU/NICU implies pediatric protections)

## Field-by-field

id

    id: https://chorus4ai.org/

⚠ medium R10 · correctness
issue Dataset identifier is URL rather than standard persistent identifier
fields id
fix Register dataset with DataCite for DOI
✗ 0/1 R10 1.Dataset Discovery and Identification Persistent Identifier (DOI, RRID, or URI)
evidence id: https://chorus4ai.org/
quality URL identifier present but not a proper persistent identifier (DOI, RRID, or UUID). This is a landing page URL, not a DOI or other standard persistent identifier format.
semantic doi_format_check: not_present; rrid_format_check: not_present; plausibility: URL identifier lacks persistence guarantees of DOI/RRID systems
✓ 1/1 R10 4.Ethical Use and Privacy Safeguards Deidentification Method Described
evidence 5 methods: re-id limitation, privacy_scan_tool, OHNLP tokenization, DICOM de-id, DeGauss geocoding
quality Multiple deidentification methods across all modalities
5/5 R20 Q1 (Structural Completeness) Field Completeness
level≥90% fields populated
evidence id: https://chorus4ai.org/, title: Patient-Focused Collaborative Hospital Repository Uniting Standards (CHoRUS) for Equitable AI, description: 420+ chars with comprehensive detail, keywords: 28 keywords, license: Controlled Access - Data Use Agreement Required (OT2OD032701)
quality All mandatory fields present with exceptional content quality and detail
correctness All fields semantically appropriate for multi-center critical care dataset
consistency Field content aligns across all mandatory elements
1/1 R20 Q16 (FAIRness & Accessibility) Findability (Persistent Links)
level Pass
evidence page: https://chorus4ai.org/, id: https://chorus4ai.org/, external_resources: 8 persistent URLs including NIH RePORTER (https://reporter.nih.gov/project-details/10472824), GitHub (https://github.com/chorus-ai), Bridge2AI (https://bridge2ai.org/chorus), publication DOI (https://doi.org/10.1007/s12028-024-02007)
quality Excellent findability with multiple persistent URLs across project website, federal grant database, GitHub organization, and peer-reviewed publication
correctness All URL formats valid. NIH RePORTER, GitHub, and DOI URLs follow standard patterns
consistency All URLs point to consistent CHoRUS project resources across different platforms
1/1 R20 Q6 (Metadata Quality & Content) Dataset Identification Metadata
level Pass
evidence id: https://chorus4ai.org/, page: https://chorus4ai.org/, external_resources: includes DOI 10.1007/s12028-024-02007 (peer-reviewed publication), NIH RePORTER project link https://reporter.nih.gov/project-details/10472824
quality Multiple persistent identifiers provided including project URL, publication DOI, and federal grant tracking
correctness DOI format valid (10.1007/s12028-024-02007), NIH RePORTER URL structure correct
consistency All URLs point to consistent CHoRUS project resources
name

    name: CHoRUS

no field-level feedback matched
title

    title: Patient-Focused Collaborative Hospital Repository Uniting Standards (CHoRUS) for Equitable AI

✓ 1/1 R10 1.Dataset Discovery and Identification Dataset Title and Description Completeness
evidence title: Patient-Focused Collaborative Hospital Repository Uniting Standards (CHoRUS) for Equitable AI
quality Excellent title and highly comprehensive description with specific details
5/5 R20 Q1 (Structural Completeness) Field Completeness
level≥90% fields populated
evidence id: https://chorus4ai.org/, title: Patient-Focused Collaborative Hospital Repository Uniting Standards (CHoRUS) for Equitable AI, description: 420+ chars with comprehensive detail, keywords: 28 keywords, license: Controlled Access - Data Use Agreement Required (OT2OD032701)
quality All mandatory fields present with exceptional content quality and detail
correctness All fields semantically appropriate for multi-center critical care dataset
consistency Field content aligns across all mandatory elements
description

    description: 'CHoRUS for Equitable AI is a Bridge2AI data generation project developing the most diverse,
      high-resolution, ethically sourced, AI-ready critical care dataset to answer the grand challenge of
      improving recovery from acute illness. The project spans 20 academic centers (14 data acquisition centers)
      and is building a publicly available dataset targeting over 100,000 critically ill patients with multi-modal
      data including structured EHR, waveform telemetry, medical imaging, EEG, and clinical notes. All structured
      data is standardized to the OMOP Common Data Model with additional formats (DICOM, WFDB, OHNLP tokenization)
      and comprehensive metadata schemas. Patient-focused efforts determine ethical and legal approaches to
      manage privacy and bias while accounting for Social Determinants of Health. A visualization and annotation
      environment labels data with targets important for prediction. The project emphasizes skills and workforce
      development for a next generation of diverse academic and community AI scientists through training programs
      and partnerships with AIM-AHEAD. As of August 2025, the dataset covers 14 different hospitals with over
      45,000 unique admissions and includes 50,000 patient admissions from ICU, PICU, and NICU, 1.6 billion
      rows of EHR OMOP data, 7,642 admissions with radiology data, and 23 TB of waveform data.

      '

5/5 R20 Q1 (Structural Completeness) Field Completeness
level≥90% fields populated
evidence id: https://chorus4ai.org/, title: Patient-Focused Collaborative Hospital Repository Uniting Standards (CHoRUS) for Equitable AI, description: 420+ chars with comprehensive detail, keywords: 28 keywords, license: Controlled Access - Data Use Agreement Required (OT2OD032701)
quality All mandatory fields present with exceptional content quality and detail
correctness All fields semantically appropriate for multi-center critical care dataset
consistency Field content aligns across all mandatory elements
5/5 R20 Q2 (Structural Completeness) Entry Length Adequacy
level>200 chars
evidence description: 1,247 chars, purposes: 4 purposes averaging 300+ chars each with comprehensive detail on grand challenge, AI-ready dataset creation, standards establishment, diversity/equity
quality Exceptional narrative depth across all narrative fields with specific metrics and objectives
correctness Narrative content accurately describes critical care AI dataset with specific enrollment targets and data volumes
consistency Purposes align with Bridge2AI program goals and dataset composition described
page

    page: https://chorus4ai.org/

✓ 1/1 R10 1.Dataset Discovery and Identification Landing Page and Resources
evidence page: https://chorus4ai.org/; 8 external resources
quality Active landing page plus extensive external resources
✗ 0/1 R10 2.Dataset Access and Retrieval Download URL or Platform Link Available
evidence Landing page URL provided but no direct download URL; access via secure enclave
quality Appropriate for controlled access but no public download URL
1/1 R20 Q16 (FAIRness & Accessibility) Findability (Persistent Links)
level Pass
evidence page: https://chorus4ai.org/, id: https://chorus4ai.org/, external_resources: 8 persistent URLs including NIH RePORTER (https://reporter.nih.gov/project-details/10472824), GitHub (https://github.com/chorus-ai), Bridge2AI (https://bridge2ai.org/chorus), publication DOI (https://doi.org/10.1007/s12028-024-02007)
quality Excellent findability with multiple persistent URLs across project website, federal grant database, GitHub organization, and peer-reviewed publication
correctness All URL formats valid. NIH RePORTER, GitHub, and DOI URLs follow standard patterns
consistency All URLs point to consistent CHoRUS project resources across different platforms
1/1 R20 Q6 (Metadata Quality & Content) Dataset Identification Metadata
level Pass
evidence id: https://chorus4ai.org/, page: https://chorus4ai.org/, external_resources: includes DOI 10.1007/s12028-024-02007 (peer-reviewed publication), NIH RePORTER project link https://reporter.nih.gov/project-details/10472824
quality Multiple persistent identifiers provided including project URL, publication DOI, and federal grant tracking
correctness DOI format valid (10.1007/s12028-024-02007), NIH RePORTER URL structure correct
consistency All URLs point to consistent CHoRUS project resources
language

    language: en

no field-level feedback matched
license

    license: Controlled Access - Data Use Agreement Required (OT2OD032701)

⚠ low R20 · consistency
issue License field references grant number (OT2OD032701) but funders section shows 1OT2OD032701-01 with project number prefix
fields license, funders
fix Use consistent grant number format across all fields (recommend 1OT2OD032701-01)
✓ 1/1 R10 2.Dataset Access and Retrieval Access Policy and IP Restrictions Defined
evidence license: Controlled Access - Data Use Agreement Required
quality Clear controlled access policy with specific requirements
5/5 R20 Q1 (Structural Completeness) Field Completeness
level≥90% fields populated
evidence id: https://chorus4ai.org/, title: Patient-Focused Collaborative Hospital Repository Uniting Standards (CHoRUS) for Equitable AI, description: 420+ chars with comprehensive detail, keywords: 28 keywords, license: Controlled Access - Data Use Agreement Required (OT2OD032701)
quality All mandatory fields present with exceptional content quality and detail
correctness All fields semantically appropriate for multi-center critical care dataset
consistency Field content aligns across all mandatory elements
keywords

    keywords:
    - CHoRUS
    - Bridge2AI
    - critical care
    - acute illness
    - AI-ready dataset
    - OMOP Common Data Model
    - electronic health records
    - EHR
    - waveform telemetry
    - medical imaging
    - DICOM
    - EEG
    - clinical notes
    - OHNLP
    - health equity
    - social determinants of health
    - federated access
    - multi-modal data
    - high-resolution data
    - ethical AI
    - trustworthy AI
    - workforce development
    - data standardization
    - privacy preservation
    - bias mitigation
    - ICU
    - intensive care

✓ 1/1 R10 1.Dataset Discovery and Identification Keywords or Tags for Searchability
evidence keywords: 28 keywords
quality Comprehensive keyword set covering all relevant dimensions
5/5 R20 Q1 (Structural Completeness) Field Completeness
level≥90% fields populated
evidence id: https://chorus4ai.org/, title: Patient-Focused Collaborative Hospital Repository Uniting Standards (CHoRUS) for Equitable AI, description: 420+ chars with comprehensive detail, keywords: 28 keywords, license: Controlled Access - Data Use Agreement Required (OT2OD032701)
quality All mandatory fields present with exceptional content quality and detail
correctness All fields semantically appropriate for multi-center critical care dataset
consistency Field content aligns across all mandatory elements
5/5 R20 Q3 (Structural Completeness) Keyword Diversity
level≥8 keywords
evidence keywords: 28 unique keywords covering project (CHoRUS, Bridge2AI), domain (critical care, acute illness, ICU), data types (EHR, waveforms, imaging, EEG, DICOM, OHNLP), standards (OMOP), themes (health equity, ethical AI, privacy preservation, bias mitigation)
quality Comprehensive keyword coverage across all relevant dimensions with excellent domain specificity
correctness Keywords accurately reflect multi-modal critical care dataset with standardization focus
consistency Keywords align with purposes, data formats, and ethical considerations described
purposes

    purposes:
    - id: chorus:purpose:1
      name: Improve recovery from acute illness
      description: 'Answer the grand challenge of improving recovery from acute illness by developing high-resolution
        multi-center datasets as a critical first step towards actionable and trustworthy AI in critical care.
        Address the urgent need for infrastructure to support artificial intelligence and machine learning
        (AI/ML) in critical care settings.

        '
    - id: chorus:purpose:2
      name: Create AI-ready critical care dataset
      description: 'Develop a publicly available, AI-ready critical care dataset from more than 100,000 critically
        ill patients while ensuring methods promote privacy, accountability, and clinical benefit. Generate
        the most diverse, high-resolution, ethically sourced dataset for AI/ML applications in acute and critical
        care, expanding AI and Machine Learning to improve recovery from acute illness.

        '
    - id: chorus:purpose:3
      name: Establish data standards and tools
      description: 'Unify standards to harmonize multi-modal EHR, waveform, imaging, and text data. Develop
        software and tooling to interact with and extract insight from clinical data in diverse formats. Create
        validated semantic mappings for connecting clinical data in various source formats to international
        standards (OMOP Common Data Model, DICOM, WFDB, OHNLP).

        '
    - id: chorus:purpose:4
      name: Promote diversity and health equity
      description: 'Ensure comprehensive sets of patient conditions and clinical treatment strategies with
        appropriate contextual factors such as geographic distance to nearest hospital and Social Determinants
        of Health. Develop the skills and workforce for a next generation of diverse academic and community
        AI scientists through comprehensive training and education programs in partnership with AIM-AHEAD.

        '

✓ 1/1 R10 7.Scientific Motivation and Funding Transparency Motivation or Purpose for Dataset Creation
evidence 4 purposes + 4 research gaps addressed
quality Exceptional motivation documentation
5/5 R20 Q18 (FAIRness & Accessibility) Reusability (License Clarity)
level License explicitly defines reuse terms
evidence license_and_use_terms: CHoRUS Controlled Access License with Data Use Agreement explicitly defines permitted research uses, prohibits re-identification, requires institutional email and signed agreement. intended_uses: 4 use cases explicitly permitted (AI/ML model development for critical care, external validation, health equity research, educational/training purposes). discouraged_uses: 3 uses explicitly prohibited (clinical decision-making without validation/approval, re-identification attempts, use without awareness of data limitations)
quality Exceptional license clarity with specific permitted uses, explicit prohibitions, and clear reuse requirements (DUA signature, research-only)
correctness Controlled access license appropriate for de-identified clinical data. Research-only restriction standard for NIH-funded datasets pending validation
consistency License terms align with intended_uses (research, education) and discouraged_uses (clinical deployment without approval). Re-identification prohibition consistent with de-identification preprocessing
5/5 R20 Q2 (Structural Completeness) Entry Length Adequacy
level>200 chars
evidence description: 1,247 chars, purposes: 4 purposes averaging 300+ chars each with comprehensive detail on grand challenge, AI-ready dataset creation, standards establishment, diversity/equity
quality Exceptional narrative depth across all narrative fields with specific metrics and objectives
correctness Narrative content accurately describes critical care AI dataset with specific enrollment targets and data volumes
consistency Purposes align with Bridge2AI program goals and dataset composition described
tasks

    tasks:
    - id: chorus:task:1
      name: Characterize acute and critical care illness
      description: 'Generate data for ML/AI applications aimed at characterizing acute and critical care illness
        patterns, progression, and outcomes across diverse patient populations and hospital settings.

        '
    - id: chorus:task:2
      name: Predict complications in critically ill patients
      description: 'Enable prediction of complications among patients with acute or critical illness using
        multi-modal data including structured EHR, waveforms, imaging, and clinical notes.

        '
    - id: chorus:task:3
      name: Measure treatment response
      description: 'Support measurement and analysis of treatment response among critically ill patients through
        high-frequency documentation, medication administration records, and clinical outcomes data.

        '
    - id: chorus:task:4
      name: External validation for AI model marketplace adoption
      description: 'Provision a holdout test set accessible for model external validation to aid marketplace
        adoption of AI-developed models for implementation in acute and critical care settings.

        '
    - id: chorus:task:5
      name: Label data for prediction targets
      description: 'Utilize visualization and annotation environment to label data with targets important
        for prediction tasks in critical care AI applications.

        '

✓ 1/1 R10 7.Scientific Motivation and Funding Transparency Primary Research Objectives or Tasks
evidence 5 detailed tasks with clear descriptions
quality Comprehensive research objectives
addressing_gaps

    addressing_gaps:
    - id: chorus:gap:1
      name: Lack of diverse multi-center critical care datasets
      description: 'Address the absence of large-scale, diverse, high-resolution multi-center datasets for
        critical care AI/ML by creating a dataset spanning 20 academic centers with 100,000+ critically ill
        patients and ensuring balanced, diverse cohorts through federated access and sampling methods across
        14 data acquisition centers.

        '
    - id: chorus:gap:2
      name: Insufficient data standardization in critical care
      description: 'Overcome the lack of unified standards in critical care data by harmonizing multi-modal
        EHR, waveform, imaging, and text data to OMOP Common Data Model and other international standards
        (DICOM, WFDB, OHNLP).

        '
    - id: chorus:gap:3
      name: Privacy and bias concerns in clinical AI
      description: 'Address ethical and legal challenges in AI through patient-focused efforts that determine
        approaches to manage privacy and bias while accounting for Social Determinants of Health and performing
        community-facing ethics focus groups to determine what data is appropriate for public sharing.

        '
    - id: chorus:gap:4
      name: Limited diversity in AI/ML workforce
      description: 'Bridge the gap in diverse AI/ML workforce through comprehensive educational approaches,
        training programs (including AIM-AHEAD partnership), and cultivation of expertise in lay and scientific
        communities to improve AI literacy and utilization.

        '

5/5 R20 Q15 (Technical Documentation) Human Subject Representation
level Detailed demographics and inclusion/exclusion criteria
evidence instances: 50,000 patient admissions (ICU, PICU, NICU) from 14 hospitals, 45,000+ unique admissions, target 100K+ critically ill patients, retrospective data collection. subpopulations: 2 documented (by hospital - 14 different hospitals across 20 academic centers for geographic/institutional diversity; by care setting - ICU, PICU, NICU). addressing_gaps: emphasizes 'most diverse' dataset with Social Determinants of Health, federated sampling for balanced diverse cohorts. sampling_strategies: multi-center federated sampling, is_representative: true, representative_verification includes geographic/institutional diversity and SDOH contextual factors
quality Excellent human subject representation with specific population sizes, geographic diversity across 14 hospitals, care setting diversity (ICU/PICU/NICU), and explicit diversity goals with SDOH considerations. Missing: specific demographic breakdowns (age ranges, race/ethnicity distributions, explicit inclusion/exclusion criteria)
correctness Population sizes plausible for multi-center 4-year critical care study. ICU/PICU/NICU settings appropriate for acute/critical illness focus
consistency 14 hospitals consistent across multiple fields. Diversity emphasis aligns with health equity purposes and SDOH sensitive_elements. Retrospective collection aligns with IRB approvals
creators

    creators:
    - id: chorus:creator:1
      name: Eric S. Rosenthal
      description: Contact PI/Project Leader, Massachusetts General Hospital (MGH), Director of MGH Neurosciences
        ICU
    - id: chorus:creator:2
      name: Azra Bihorac
      description: Principal Investigator, University of Florida (UF)
    - id: chorus:creator:3
      name: Ashley Cordes
      description: Principal Investigator, CHoRUS Consortium
    - id: chorus:creator:4
      name: Gilles Clermont
      description: Principal Investigator, CHoRUS Consortium
    - id: chorus:creator:5
      name: Gari David Clifford
      description: Principal Investigator, CHoRUS Consortium
    - id: chorus:creator:6
      name: Barbara J. Evans
      description: Principal Investigator, CHoRUS Consortium
    - id: chorus:creator:7
      name: Xiao Hu
      description: Principal Investigator, CHoRUS Consortium
    - id: chorus:creator:8
      name: Rishikesan Kamaleswaran
      description: Principal Investigator, CHoRUS Consortium
    - id: chorus:creator:9
      name: Yulia A. Levites Strekalova
      description: Principal Investigator, University of Florida (UF)
    - id: chorus:creator:10
      name: Parisa Rashidi
      description: Principal Investigator, University of Florida (UF)
    - id: chorus:creator:11
      name: Cynthia Rudin
      description: Principal Investigator, CHoRUS Consortium
    - id: chorus:creator:12
      name: Ishan Canty Williams
      description: Principal Investigator, CHoRUS Consortium
    - id: chorus:creator:13
      name: Andrew Ewing Williams
      description: Principal Investigator, Tufts Medicine
    - id: chorus:creator:14
      name: Xiaoqian Jiang
      description: Program Lead, UT Health Science Center (UTHealth Houston)
    - id: chorus:creator:15
      name: Morteza Zabihi
      description: Lecturer (Machine Learning Basics), Massachusetts General Hospital
    - id: chorus:creator:16
      name: Zhenhong Hu
      description: Instructor (Python and Version Control), University of Florida
    - id: chorus:creator:17
      name: Debora Simmons
      description: Lecturer (Ethics of AI in Clinical Practice), UT Health Science Center
    - id: chorus:creator:18
      name: Aliyah Geer
      description: Workshop Lead (Data Schemas in Clinical Cloud), Massachusetts General Hospital
    - id: chorus:creator:19
      name: Ciera McCrary
      description: Program Manager, Massachusetts General Hospital (MGH)

✓ 1/1 R10 7.Scientific Motivation and Funding Transparency Creators and Acknowledgements Documented
evidence 19 creators with roles and institutional affiliations
quality Comprehensive creator documentation across multiple roles
5/5 R20 Q7 (Metadata Quality & Content) Funding and Acknowledgements Completeness
level Funders with grants + creators with affiliations
evidence funders: NIH Common Fund Bridge2AI Program with grant 1OT2OD032701-01, total funding $5,880,300 (2022), project dates September 1, 2022 to November 30, 2026. creators: 19 creators with institutional affiliations (MGH, UF, UT Health, Tufts Medicine, Emory) and roles (Contact PI, Principal Investigator, Program Lead, Lecturer, Workshop Lead, Program Manager)
quality Exceptional funding documentation with grant number, fiscal details, project timeline, and comprehensive creator affiliations across all participating institutions
correctness Grant number 1OT2OD032701-01 follows NIH OT2 (Other Transaction) format correctly for Bridge2AI program
consistency Creator institutions align with 14 data acquisition centers and 20 academic centers described. Minor inconsistency: license field shows OT2OD032701 without project prefix 1-01
funders

    funders:
    - id: chorus:funder:1
      name: NIH Common Fund Bridge2AI Program
      description: 'Funded through National Institutes of Health grant OT2OD032701 (project number 1OT2OD032701-01),
        administered by NIH Office of the Director. Opportunity Number: OTA-21-008. Study Section: Data Coordination,
        Mapping, and Modeling (DCMM). Fiscal Year 2022. Total funding in 2022: $5,880,300 (all direct costs).
        Project dates: September 1, 2022 to November 30, 2026 (with approved no-cost extension). Award notice
        date: September 1, 2022. Assistance Listing Number: 93.310.

        '

⚠ low R20 · consistency
issue License field references grant number (OT2OD032701) but funders section shows 1OT2OD032701-01 with project number prefix
fields license, funders
fix Use consistent grant number format across all fields (recommend 1OT2OD032701-01)
⚠ low R20 · correctness
issue Grant number OT2OD032701 follows NIH format correctly but uses OT2 (Other Transaction) mechanism which is non-standard for typical research grants
fields funders
fix Verify OT2 mechanism is intentional for Bridge2AI program (appears correct for this special NIH Common Fund initiative)
5/5 R20 Q7 (Metadata Quality & Content) Funding and Acknowledgements Completeness
level Funders with grants + creators with affiliations
evidence funders: NIH Common Fund Bridge2AI Program with grant 1OT2OD032701-01, total funding $5,880,300 (2022), project dates September 1, 2022 to November 30, 2026. creators: 19 creators with institutional affiliations (MGH, UF, UT Health, Tufts Medicine, Emory) and roles (Contact PI, Principal Investigator, Program Lead, Lecturer, Workshop Lead, Program Manager)
quality Exceptional funding documentation with grant number, fiscal details, project timeline, and comprehensive creator affiliations across all participating institutions
correctness Grant number 1OT2OD032701-01 follows NIH OT2 (Other Transaction) format correctly for Bridge2AI program
consistency Creator institutions align with 14 data acquisition centers and 20 academic centers described. Minor inconsistency: license field shows OT2OD032701 without project prefix 1-01
instances

    instances:
    - id: chorus:instance:1
      name: Critically ill patients in ICU settings
      description: 'Individual critically ill patients requiring acute or critical care admitted to intensive
        care units (ICU), pediatric intensive care units (PICU), or neonatal intensive care units (NICU) across
        14 data acquisition centers. As of August 2025, the dataset covers 14 different hospitals with over
        45,000 unique admissions, with 50,000 patient admissions from ICU, PICU, and NICU currently released.
        Target enrollment exceeds 100,000 critically ill patients. Retrospective data collection from patients
        with acute or critical illness.

        '
      instance_type: 'Human subjects - critically ill patients admitted to participating hospitals (retrospective
        data collection, ongoing collection through November 2026).

        '

5/5 R20 Q15 (Technical Documentation) Human Subject Representation
level Detailed demographics and inclusion/exclusion criteria
evidence instances: 50,000 patient admissions (ICU, PICU, NICU) from 14 hospitals, 45,000+ unique admissions, target 100K+ critically ill patients, retrospective data collection. subpopulations: 2 documented (by hospital - 14 different hospitals across 20 academic centers for geographic/institutional diversity; by care setting - ICU, PICU, NICU). addressing_gaps: emphasizes 'most diverse' dataset with Social Determinants of Health, federated sampling for balanced diverse cohorts. sampling_strategies: multi-center federated sampling, is_representative: true, representative_verification includes geographic/institutional diversity and SDOH contextual factors
quality Excellent human subject representation with specific population sizes, geographic diversity across 14 hospitals, care setting diversity (ICU/PICU/NICU), and explicit diversity goals with SDOH considerations. Missing: specific demographic breakdowns (age ranges, race/ethnicity distributions, explicit inclusion/exclusion criteria)
correctness Population sizes plausible for multi-center 4-year critical care study. ICU/PICU/NICU settings appropriate for acute/critical illness focus
consistency 14 hospitals consistent across multiple fields. Diversity emphasis aligns with health equity purposes and SDOH sensitive_elements. Retrospective collection aligns with IRB approvals
1/1 R20 Q5 (Structural Completeness) Data File Size Availability
level Pass
evidence instances: 50,000 patient admissions released (45,000+ unique admissions), target 100,000+ critically ill patients. Specific data volumes: 1.6 billion rows of EHR OMOP data, 23 TB of waveform data, 7,642 admissions with radiology data, 1,000 images available
quality Comprehensive instance counts and data volumes provided with specific metrics by modality
correctness Instance counts plausible for multi-center critical care dataset over 3+ years
consistency Data volumes align with described collection mechanisms and timeframes
subsets

    subsets:
    - id: chorus:subset:1
      name: Controlled Access Structured EHR Dataset (OMOP)
      description: 'Primary dataset with controlled access requiring data use agreement and licensing. Includes
        OMOP-standardized structured EHR data comprising demographics, medication administration (dosing time-stamped
        upon each infusion change or dose administration), procedures, nursing flowsheets (high-frequency
        documentation), and diagnoses. Available in secure enclave. As of August 2025, contains 1.6 billion
        rows of EHR OMOP data. Registration required with institutional email; all participants must sign
        licensing agreement.

        '
    - id: chorus:subset:2
      name: Waveform Telemetry Dataset (WFDB)
      description: 'Waveform telemetry data from bedside monitors (gateway/middleware) standardized to WFDB
        (WaveForm DataBase) format following extended PhysioNet schema. Available in secure enclave with controlled
        access. As of August 2025, contains 23 TB of waveform data. Published metadata schema: PhysioNet schema
        (extended).

        '
    - id: chorus:subset:3
      name: Medical Imaging Dataset (DICOM)
      description: 'Imaging data from hospital PACS (Picture Archiving and Communication System) in DICOM
        format with comprehensive metadata following DICOM schema. As of August 2025, 1,000 images are available
        with de-identification in process for the larger cohort. 7,642 admissions have radiology data. Controlled
        access planned.

        '
    - id: chorus:subset:4
      name: Clinical Notes Dataset (OHNLP Tokenized)
      description: 'Clinical notes extracted and tokenized using OHNLP (Open Health Natural Language Processing)
        toolkit following open source OHNLP schema. Stored locally at contributing sites (except tokens).
        Controlled access planned. De-identification achieved through tokenization approach.

        '
    - id: chorus:subset:5
      name: EEG Waveform Dataset (EDF+ and Persyst)
      description: 'Electroencephalography (EEG) waveform data from hospital databases in EDF+ (European Data
        Format) and Persyst formats. Open source EDF+ and Persyst schema metadata available. Extraction in
        process, planned for controlled access.

        '
    - id: chorus:subset:6
      name: Training and Publication Subset
      description: 'Subsets of the dataset currently being used for training activities (AIM-AHEAD Bridge2AI
        for Clinical Care Training Program) and publications as the dataset undergoes expansion and quality
        assurance processes.

        '

✓ 1/1 R10 1.Dataset Discovery and Identification Hierarchical Structure
evidence 6 distinct subsets (OMOP EHR, WFDB waveforms, DICOM imaging, OHNLP notes, EEG, training subset)
quality Well-structured hierarchical organization with 6 modality-based subsets
sampling_strategies

    sampling_strategies:
    - id: chorus:sampling:1
      name: Multi-center federated sampling
      description: 'Federated access enables sampling methods to ensure a balanced and diverse cohort across
        20 academic centers (14 data acquisition centers). Legal framework established for collecting data
        at scale with sampling to ensure comprehensive sets of patient conditions and clinical treatment strategies.
        Community-facing ethics focus groups conducted to determine what data is appropriate for public sharing.

        '
      is_sample: true
      is_random: false
      is_representative: true
      source_data:
      - Critically ill patients in ICU, PICU, and NICU settings at 14 data acquisition centers across 20 academic
        centers in the United States
      representative_verification:
      - Federated multi-center data collection across 14 hospitals ensures geographic and institutional diversity;
        sampling for balanced and diverse patient populations; inclusion of Social Determinants of Health
        contextual factors

5/5 R20 Q15 (Technical Documentation) Human Subject Representation
level Detailed demographics and inclusion/exclusion criteria
evidence instances: 50,000 patient admissions (ICU, PICU, NICU) from 14 hospitals, 45,000+ unique admissions, target 100K+ critically ill patients, retrospective data collection. subpopulations: 2 documented (by hospital - 14 different hospitals across 20 academic centers for geographic/institutional diversity; by care setting - ICU, PICU, NICU). addressing_gaps: emphasizes 'most diverse' dataset with Social Determinants of Health, federated sampling for balanced diverse cohorts. sampling_strategies: multi-center federated sampling, is_representative: true, representative_verification includes geographic/institutional diversity and SDOH contextual factors
quality Excellent human subject representation with specific population sizes, geographic diversity across 14 hospitals, care setting diversity (ICU/PICU/NICU), and explicit diversity goals with SDOH considerations. Missing: specific demographic breakdowns (age ranges, race/ethnicity distributions, explicit inclusion/exclusion criteria)
correctness Population sizes plausible for multi-center 4-year critical care study. ICU/PICU/NICU settings appropriate for acute/critical illness focus
consistency 14 hospitals consistent across multiple fields. Diversity emphasis aligns with health equity purposes and SDOH sensitive_elements. Retrospective collection aligns with IRB approvals
subpopulations

    subpopulations:
    - id: chorus:subpop:1
      name: Critically ill patients by hospital
      description: 'Patients distributed across 14 different hospitals within the CHoRUS network spanning
        20 academic centers in the United States, ensuring geographic and institutional diversity in critical
        care settings.

        '
    - id: chorus:subpop:2
      name: ICU, PICU, and NICU patients
      description: 'Patients admitted to intensive care units (ICU), pediatric intensive care units (PICU),
        and neonatal intensive care units (NICU) across contributing hospitals, with 50,000 patient admissions
        in the current released dataset.

        '

5/5 R20 Q15 (Technical Documentation) Human Subject Representation
level Detailed demographics and inclusion/exclusion criteria
evidence instances: 50,000 patient admissions (ICU, PICU, NICU) from 14 hospitals, 45,000+ unique admissions, target 100K+ critically ill patients, retrospective data collection. subpopulations: 2 documented (by hospital - 14 different hospitals across 20 academic centers for geographic/institutional diversity; by care setting - ICU, PICU, NICU). addressing_gaps: emphasizes 'most diverse' dataset with Social Determinants of Health, federated sampling for balanced diverse cohorts. sampling_strategies: multi-center federated sampling, is_representative: true, representative_verification includes geographic/institutional diversity and SDOH contextual factors
quality Excellent human subject representation with specific population sizes, geographic diversity across 14 hospitals, care setting diversity (ICU/PICU/NICU), and explicit diversity goals with SDOH considerations. Missing: specific demographic breakdowns (age ranges, race/ethnicity distributions, explicit inclusion/exclusion criteria)
correctness Population sizes plausible for multi-center 4-year critical care study. ICU/PICU/NICU settings appropriate for acute/critical illness focus
consistency 14 hospitals consistent across multiple fields. Diversity emphasis aligns with health equity purposes and SDOH sensitive_elements. Retrospective collection aligns with IRB approvals
sensitive_elements

    sensitive_elements:
    - id: chorus:sensitive:1
      name: Protected health information - clinical EHR data
      description: 'Complete electronic health records including demographics, diagnoses, procedures, medications,
        nursing documentation, and clinical notes for critically ill patients. Contains protected health information
        subject to HIPAA and institutional privacy requirements. De-identified before release through OMOP
        transformation and privacy scanning tools.

        '
      sensitive_elements_present: true
      sensitivity_details:
      - Demographics and patient identifiers (de-identified before controlled access release)
      - Diagnoses and medical conditions (OMOP standardized)
      - Medication administration records with dosing timestamps (OMOP standardized)
      - Procedures and clinical interventions documented by providers (OMOP standardized)
      - Clinical notes (tokenized using OHNLP toolkit for privacy protection)
      - Nursing flowsheet documentation at high frequency (OMOP with extensions)
    - id: chorus:sensitive:2
      name: Physiological monitoring and EEG waveform data
      description: 'Continuous waveform telemetry from bedside monitors and EEG recordings capturing detailed
        physiological states of critically ill patients. May reveal sensitive health conditions and treatment
        responses. Stored in WFDB format (telemetry) and EDF+/Persyst formats (EEG) with controlled access.

        '
      sensitive_elements_present: true
      sensitivity_details:
      - Continuous cardiac waveforms from bedside monitors (WFDB format, 23 TB)
      - Respiratory monitoring and hemodynamic measurement data
      - Electroencephalography (EEG) recordings from hospital databases
      - High-frequency physiological parameters
    - id: chorus:sensitive:3
      name: Medical imaging data
      description: 'Diagnostic imaging studies in DICOM format from hospital PACS systems. De-identification
        in process to remove embedded patient information while preserving clinical utility. As of August
        2025, 1,000 images available with de-id in process for larger cohort (7,642 admissions with radiology
        data total).

        '
      sensitive_elements_present: true
      sensitivity_details:
      - Radiology images (CT scans, X-rays, MRI and other modalities) in DICOM format
      - DICOM metadata (de-identification in process)
      - Embedded patient information being removed through de-identification pipeline
    - id: chorus:sensitive:4
      name: Social Determinants of Health data
      description: 'Contextual factors including geographic information (distance to hospital) and social
        determinants of health data. Collected to support health equity research while maintaining patient
        privacy through privacy-preserving transformations.

        '
      sensitive_elements_present: true
      sensitivity_details:
      - Geographic location information (distance to nearest hospital)
      - Social determinants of health variables
      - Contextual equity factors with privacy-preserving transformations applied

4/5 R20 Q8 (Metadata Quality & Content) Ethical and Privacy Declarations
level Comprehensive ethics with minor gaps
evidence human_subject_research.involves_human_subjects: true, ethics_review_board: Institutional review boards at 14 data acquisition centers, community-facing ethics focus groups, legal and ethical advisory teams. regulatory_compliance: HIPAA, 45 CFR 46 (Common Rule), institutional data privacy regulations, NIH ethical AI requirements. preprocessing_strategies includes de-identification: OMOP transformation, OHNLP tokenization, DICOM header de-identification, privacy scan tool. sensitive_elements: 4 categories documented (PHI-clinical EHR, waveforms/EEG, imaging, SDOH)
quality Comprehensive ethical documentation covering IRB review, regulatory compliance, de-identification methods, and privacy protections. Missing: specific IRB protocol numbers, explicit informed consent approach (retrospective waiver likely), participant compensation details, vulnerable population safeguards (though PICU/NICU implies pediatric protections)
correctness HIPAA and 45 CFR 46 compliance appropriate for US-based multi-center health data. De-identification methods semantically appropriate for each data type
consistency Ethics review at 14 sites aligns with 14 data acquisition centers. Community ethics focus groups support public sharing claims. Minor gap: no IRB protocol numbers provided despite multi-site coordination
collection_mechanisms

    collection_mechanisms:
    - id: chorus:collection:1
      name: Retrospective EHR extraction
      description: 'Retrospective data collection from electronic health record systems at 14 data acquisition
        centers. Data extracted includes demographics, medication administration (dosing time-stamped upon
        each infusion change or dose administration), procedures, nursing flowsheets (high-frequency documentation),
        diagnoses, and clinical notes. Standardized to OMOP Common Data Model.

        '
    - id: chorus:collection:2
      name: Waveform telemetry capture
      description: 'Continuous waveform telemetry data captured from bedside monitors through gateway and
        middleware systems at each contributing hospital. Stored in WFDB (WaveForm DataBase) format following
        extended PhysioNet schema. Contains 23 TB of waveform data as of August 2025.

        '
    - id: chorus:collection:3
      name: Medical imaging acquisition from PACS
      description: 'Medical imaging data acquired from hospital Picture Archiving and Communication Systems
        (PACS) and stored in DICOM format with comprehensive metadata following DICOM schema. 7,642 admissions
        have radiology data; 1,000 images currently available.

        '
    - id: chorus:collection:4
      name: EEG recording extraction
      description: 'Electroencephalography recordings extracted from hospital EEG databases in EDF+ (European
        Data Format) and Persyst formats with metadata following open source schemas. Extraction in process
        as of August 2025.

        '

5/5 R20 Q12 (Technical Documentation) Collection Protocol Clarity
level Full collection protocol with methods, collectors, and timeframes
evidence collection_mechanisms: 4 detailed mechanisms (retrospective EHR extraction, waveform telemetry capture via gateway/middleware, medical imaging from PACS, EEG extraction from databases). acquisition_methods: 5 methods with technical details (OMOP transformation, OHNLP tokenization, DICOM from PACS, WFDB from bedside monitors, EEG from hospital databases). Collection timeframes: September 1, 2022 to November 30, 2026 (approved no-cost extension), snapshot August 2025. Data collectors: 14 data acquisition centers across 20 academic centers, coordinated by CHoRUS Consortium (MGH lead, UF, UT Health, Tufts Medicine)
quality Comprehensive collection protocol documentation with detailed mechanisms, methods, multi-site coordination, and clear timeline with current status snapshot
correctness Collection mechanisms appropriate for multi-center retrospective critical care study. 4-year timeline plausible for 100K+ patient dataset
consistency Collection mechanisms align with distribution formats. 14 data acquisition centers consistent across multiple fields. Timeframes logically ordered (start 9/2022, snapshot 8/2025, end 11/2026)
acquisition_methods

    acquisition_methods:
    - id: chorus:acquisition:1
      name: Structured EHR data via OMOP transformation
      description: 'Demographics, medication administration (dosing time-stamped upon each infusion change
        or dose administration), procedures, nursing flowsheets (high-frequency documentation), and diagnoses
        acquired from electronic health records and standardized to OMOP Common Data Model. Controlled access
        with published OMOP schema metadata. Contains 1.6 billion rows of OMOP data as of August 2025.

        '
    - id: chorus:acquisition:2
      name: Clinical notes via OHNLP tokenization
      description: 'Clinical notes extracted from EHR systems and tokenized using OHNLP (Open Health Natural
        Language Processing) toolkit. Stored locally at sites except for tokens. Controlled access planned
        with OHNLP open source schema metadata.

        '
    - id: chorus:acquisition:3
      name: Medical imaging via DICOM from PACS
      description: 'Imaging data acquired from hospital PACS systems in DICOM format. De-identification in
        process. Planned controlled access with published DICOM schema metadata.

        '
    - id: chorus:acquisition:4
      name: Waveform telemetry via WFDB from bedside monitors
      description: 'Bedside monitor waveform data acquired through gateway and middleware systems. Stored
        in WFDB format with controlled access and published PhysioNet schema (extended) metadata. 23 TB available
        as of August 2025.

        '
    - id: chorus:acquisition:5
      name: EEG waveforms from hospital databases
      description: 'EEG recordings from hospital databases in EDF+ and Persyst formats. Extraction in process.
        Planned controlled access with open source EDF+ and Persyst schema metadata.

        '

5/5 R20 Q12 (Technical Documentation) Collection Protocol Clarity
level Full collection protocol with methods, collectors, and timeframes
evidence collection_mechanisms: 4 detailed mechanisms (retrospective EHR extraction, waveform telemetry capture via gateway/middleware, medical imaging from PACS, EEG extraction from databases). acquisition_methods: 5 methods with technical details (OMOP transformation, OHNLP tokenization, DICOM from PACS, WFDB from bedside monitors, EEG from hospital databases). Collection timeframes: September 1, 2022 to November 30, 2026 (approved no-cost extension), snapshot August 2025. Data collectors: 14 data acquisition centers across 20 academic centers, coordinated by CHoRUS Consortium (MGH lead, UF, UT Health, Tufts Medicine)
quality Comprehensive collection protocol documentation with detailed mechanisms, methods, multi-site coordination, and clear timeline with current status snapshot
correctness Collection mechanisms appropriate for multi-center retrospective critical care study. 4-year timeline plausible for 100K+ patient dataset
consistency Collection mechanisms align with distribution formats. 14 data acquisition centers consistent across multiple fields. Timeframes logically ordered (start 9/2022, snapshot 8/2025, end 11/2026)
preprocessing_strategies

    preprocessing_strategies:
    - id: chorus:preproc:1
      name: OMOP Common Data Model transformation
      description: 'All structured electronic health record data standardized to the OMOP (Observational Medical
        Outcomes Partnership) Common Data Model. Ensures interoperability and enables use of OHDSI (Observational
        Health Data Sciences and Informatics) tool stack for analysis. Data from all 14 acquisition centers
        unified through this transformation.

        '
      preprocessing_details:
      - Transformation of source EHR data from diverse institutional formats to OMOP CDM
      - Standardization of terminology and clinical codes to OMOP vocabulary standards
      - Mapping to OMOP vocabulary standards using validated semantic mappings
      - Quality assurance of transformed data against OMOP schema specifications
      - Integration with OHDSI tools for downstream analysis and characterization
      - Generation of characterization reports returned to contributing sites via CHoRUSReports
    - id: chorus:preproc:2
      name: Clinical note tokenization via OHNLP
      description: 'Clinical notes processed using OHNLP (Open Health Natural Language Processing) toolkit
        for extraction and tokenization. Protects patient privacy while enabling natural language processing
        and analysis of clinical text data.

        '
      preprocessing_details:
      - Text extraction from clinical notes in source EHR systems
      - OHNLP tokenization pipeline applied to free-text clinical documentation
      - De-identification of sensitive information through tokenization approach
      - Standardization to OHNLP open source schema
      - Local storage of full notes at sites; only tokens available in enclave
    - id: chorus:preproc:3
      name: Waveform standardization to WFDB format
      description: 'Waveform telemetry data from diverse bedside monitoring systems standardized to WFDB (WaveForm
        DataBase) format following extended PhysioNet schema. Scripts available in chorus_waveform repository.

        '
      preprocessing_details:
      - Conversion from proprietary bedside monitor formats using gateway/middleware systems
      - Standardization to WFDB format per extended PhysioNet schema
      - Metadata extraction and schema compliance verification
      - Quality checks for waveform integrity and completeness
      - Synchronization with clinical events from OMOP EHR data
    - id: chorus:preproc:4
      name: Medical imaging de-identification
      description: 'DICOM imaging data undergoing de-identification process to remove patient identifiable
        information from image metadata while preserving clinical utility and image quality.

        '
      preprocessing_details:
      - DICOM header de-identification to remove embedded patient information
      - Preservation of clinically relevant imaging metadata
      - DICOM schema compliance verification after de-identification
      - Quality assurance of de-identified images for clinical utility
      - Privacy scan tool (privacy_scan_tool) used for medical records privacy scanning
    - id: chorus:preproc:5
      name: Re-identification limitation transformations
      description: 'Data transformed using approaches that limit re-identification while maintaining analytical
        utility. Multiple preprocessing strategies employed to protect patient privacy across all data modalities
        per HIPAA and institutional requirements.

        '
      preprocessing_details:
      - Application of de-identification algorithms across all data modalities
      - Privacy-preserving transformations for EHR and imaging data
      - Geocoding via DeGauss (UF-Geocoding tool) for OMOP Location entities
      - Risk assessment and compliance with ethical and legal requirements
      - Community ethics focus group input on appropriate data for public sharing

⚠ low R20 · semantic_understanding
issue De-identification described across multiple preprocessing strategies but no unified deidentification method field
fields preprocessing_strategies, is_deidentified
fix Add explicit deidentification method summary (HIPAA Safe Harbor, Expert Determination, or hybrid)
5/5 R20 Q10 (Metadata Quality & Content) Interoperability and Standardization
level Standard formats + schema/ontology compliance
evidence distribution_formats: OMOP Common Data Model (published OMOP schema metadata), WFDB (published PhysioNet schema extended), DICOM (published DICOM schema metadata), OHNLP tokenized (open source OHNLP schema), EDF+ and Persyst (open source schemas). preprocessing_strategies: semantic mapping validation via chorus-mapping repository, OHDSI tool stack integration, schema compliance validation across all modalities
quality Exceptional interoperability with comprehensive standardization across all five data modalities using internationally recognized standards and published schemas
correctness Format choices semantically appropriate: OMOP (observational health data standard), WFDB (PhysioNet waveform standard), DICOM (medical imaging standard), OHNLP (clinical NLP standard), EDF+ (EEG standard)
consistency Schema conformance validated through preprocessing_strategies and cleaning_strategies. chorus-mapping repository for semantic validation mentioned consistently
5/5 R20 Q11 (Technical Documentation) Tool and Software Transparency
level Comprehensive strategies with software versions/URLs
evidence preprocessing_strategies: 5 detailed strategies (OMOP transformation, OHNLP tokenization, WFDB standardization, DICOM de-identification, re-identification limitation). cleaning_strategies: 2 strategies (multi-center harmonization with chorus-mapping repository, schema compliance validation). labeling_strategies: visualization and annotation environment. software_and_tools: OHDSI tool stack, OHNLP toolkit, privacy_scan_tool, DeGauss (UF-Geocoding), CHoRUSReports (R), chorus_waveform scripts, chorus-extract-upload tools. GitHub organization (chorus-ai) with 28 repositories including Chorus_SOP, chorus-mapping, chorus-container-apps
quality Exceptional software transparency with specific tools named, GitHub organization documented (28 repositories), open source licenses specified (MIT, Apache-2.0), and comprehensive preprocessing/cleaning/labeling strategies
correctness Tools semantically appropriate: OHDSI/OMOP for structured EHR, OHNLP for clinical notes, PhysioNet/WFDB for waveforms, DeGauss for geocoding
consistency Software tools align with preprocessing strategies and distribution formats. GitHub repositories (chorus-ai) support documented SOPs and semantic mappings
4/5 R20 Q8 (Metadata Quality & Content) Ethical and Privacy Declarations
level Comprehensive ethics with minor gaps
evidence human_subject_research.involves_human_subjects: true, ethics_review_board: Institutional review boards at 14 data acquisition centers, community-facing ethics focus groups, legal and ethical advisory teams. regulatory_compliance: HIPAA, 45 CFR 46 (Common Rule), institutional data privacy regulations, NIH ethical AI requirements. preprocessing_strategies includes de-identification: OMOP transformation, OHNLP tokenization, DICOM header de-identification, privacy scan tool. sensitive_elements: 4 categories documented (PHI-clinical EHR, waveforms/EEG, imaging, SDOH)
quality Comprehensive ethical documentation covering IRB review, regulatory compliance, de-identification methods, and privacy protections. Missing: specific IRB protocol numbers, explicit informed consent approach (retrospective waiver likely), participant compensation details, vulnerable population safeguards (though PICU/NICU implies pediatric protections)
correctness HIPAA and 45 CFR 46 compliance appropriate for US-based multi-center health data. De-identification methods semantically appropriate for each data type
consistency Ethics review at 14 sites aligns with 14 data acquisition centers. Community ethics focus groups support public sharing claims. Minor gap: no IRB protocol numbers provided despite multi-site coordination
cleaning_strategies

    cleaning_strategies:
    - id: chorus:cleaning:1
      name: Multi-center data harmonization with validated semantic mappings
      description: 'Data from 14 acquisition centers harmonized through validated semantic mappings and standard
        operating protocols (SOPs). Ensures consistency and interoperability across diverse institutional
        EHR systems and clinical practices. Mappings maintained in chorus-mapping repository with clinical
        validation SOP.

        '
      cleaning_details:
      - Semantic mapping validation by clinical experts using chorus-mapping repository
      - Standard operating protocol (SOP) implementation per Chorus_SOP documentation
      - Cross-site data quality checks through CHoRUSReports characterization reports
      - Resolution of institutional variations in coding and clinical terminology
      - Clinical validation SOP for contributing to mapping efforts
      - Site status tracking via GitHub interface and Google Form submissions
    - id: chorus:cleaning:2
      name: Schema compliance validation across all modalities
      description: 'All data modalities validated against published metadata schemas (OMOP, DICOM, WFDB, OHNLP,
        EDF+, Persyst) to ensure compliance and data quality across the multi-modal dataset.

        '
      cleaning_details:
      - Schema compliance verification against OMOP CDM specifications
      - DICOM schema metadata validation for imaging data
      - WFDB/PhysioNet schema validation for waveform data
      - OHNLP open source schema validation for tokenized clinical notes
      - EDF+ and Persyst schema validation for EEG data
      - Documentation of schema extensions for OMOP high-frequency nursing flowsheets

5/5 R20 Q11 (Technical Documentation) Tool and Software Transparency
level Comprehensive strategies with software versions/URLs
evidence preprocessing_strategies: 5 detailed strategies (OMOP transformation, OHNLP tokenization, WFDB standardization, DICOM de-identification, re-identification limitation). cleaning_strategies: 2 strategies (multi-center harmonization with chorus-mapping repository, schema compliance validation). labeling_strategies: visualization and annotation environment. software_and_tools: OHDSI tool stack, OHNLP toolkit, privacy_scan_tool, DeGauss (UF-Geocoding), CHoRUSReports (R), chorus_waveform scripts, chorus-extract-upload tools. GitHub organization (chorus-ai) with 28 repositories including Chorus_SOP, chorus-mapping, chorus-container-apps
quality Exceptional software transparency with specific tools named, GitHub organization documented (28 repositories), open source licenses specified (MIT, Apache-2.0), and comprehensive preprocessing/cleaning/labeling strategies
correctness Tools semantically appropriate: OHDSI/OMOP for structured EHR, OHNLP for clinical notes, PhysioNet/WFDB for waveforms, DeGauss for geocoding
consistency Software tools align with preprocessing strategies and distribution formats. GitHub repositories (chorus-ai) support documented SOPs and semantic mappings
labeling_strategies

    labeling_strategies:
    - id: chorus:labeling:1
      name: Visualization and annotation environment for prediction targets
      description: 'Custom visualization and annotation environment developed to label data with targets important
        for prediction tasks in critical care AI applications. Supports labeling for characterizing acute
        illness, predicting complications, and measuring treatment response.

        '
      data_annotation_protocol:
      - Interactive visualization tools for clinical data exploration
      - Annotation interface for clinical expert labeling of prediction targets
      - Labeling of targets for characterization, prediction, and treatment response tasks
      - Quality control of annotations through review processes
      - Documentation of labeling protocols per Chorus_SOP standard operating procedures

5/5 R20 Q11 (Technical Documentation) Tool and Software Transparency
level Comprehensive strategies with software versions/URLs
evidence preprocessing_strategies: 5 detailed strategies (OMOP transformation, OHNLP tokenization, WFDB standardization, DICOM de-identification, re-identification limitation). cleaning_strategies: 2 strategies (multi-center harmonization with chorus-mapping repository, schema compliance validation). labeling_strategies: visualization and annotation environment. software_and_tools: OHDSI tool stack, OHNLP toolkit, privacy_scan_tool, DeGauss (UF-Geocoding), CHoRUSReports (R), chorus_waveform scripts, chorus-extract-upload tools. GitHub organization (chorus-ai) with 28 repositories including Chorus_SOP, chorus-mapping, chorus-container-apps
quality Exceptional software transparency with specific tools named, GitHub organization documented (28 repositories), open source licenses specified (MIT, Apache-2.0), and comprehensive preprocessing/cleaning/labeling strategies
correctness Tools semantically appropriate: OHDSI/OMOP for structured EHR, OHNLP for clinical notes, PhysioNet/WFDB for waveforms, DeGauss for geocoding
consistency Software tools align with preprocessing strategies and distribution formats. GitHub repositories (chorus-ai) support documented SOPs and semantic mappings
intended_uses

    intended_uses:
    - id: chorus:use:1
      name: AI/ML model development for critical care
      description: 'Primary intended use is development and training of artificial intelligence and machine
        learning models to characterize acute and critical care illness, predict complications, and measure
        treatment response in critically ill patients across diverse hospital settings.

        '
      examples:
      - Characterizing acute and critical care illness patterns using multi-modal data
      - Predicting complications (e.g., sepsis, respiratory failure) in critically ill patients
      - Measuring treatment response in ICU patients using medication and waveform data
      - Developing clinical deep learning models for critical care AI applications
    - id: chorus:use:2
      name: External validation of AI models
      description: 'Provision of holdout test set accessible for model external validation to aid marketplace
        adoption of AI-developed models for implementation in acute and critical care settings.

        '
      examples:
      - External validation of sepsis prediction models developed at other institutions
      - Benchmarking AI algorithms for critical care across diverse patient populations
    - id: chorus:use:3
      name: Health equity and disparities research
      description: 'Studies examining health equity, social determinants of health, and disparities in critical
        care outcomes across diverse patient populations and hospital settings. Dataset includes contextual
        factors such as geographic distance to nearest hospital.

        '
      examples:
      - Analysis of disparities in critical care outcomes by race, ethnicity, and geography
      - Research on social determinants of health in ICU patient populations
    - id: chorus:use:4
      name: Educational and training purposes for AI scientists
      description: 'Training and education of next generation of diverse academic and community AI scientists
        through hands-on experience with real-world critical care datasets. Integrated with AIM-AHEAD Bridge2AI
        for Clinical Care Training Program (Cohorts 1 and 2).

        '
      examples:
      - 'AIM-AHEAD training program for underrepresented trainees (Cohort 1: 2024-2025, Cohort 2: 2025-2026)'
      - Foundational hands-on training using Jupyter Notebooks with Bridge2AI CHoRUS ecosystem
      - Workshops on OHDSI/OMOP common data model and clinical AI
      - Development of practical use cases for AI/ML in clinical care

5/5 R20 Q18 (FAIRness & Accessibility) Reusability (License Clarity)
level License explicitly defines reuse terms
evidence license_and_use_terms: CHoRUS Controlled Access License with Data Use Agreement explicitly defines permitted research uses, prohibits re-identification, requires institutional email and signed agreement. intended_uses: 4 use cases explicitly permitted (AI/ML model development for critical care, external validation, health equity research, educational/training purposes). discouraged_uses: 3 uses explicitly prohibited (clinical decision-making without validation/approval, re-identification attempts, use without awareness of data limitations)
quality Exceptional license clarity with specific permitted uses, explicit prohibitions, and clear reuse requirements (DUA signature, research-only)
correctness Controlled access license appropriate for de-identified clinical data. Research-only restriction standard for NIH-funded datasets pending validation
consistency License terms align with intended_uses (research, education) and discouraged_uses (clinical deployment without approval). Re-identification prohibition consistent with de-identification preprocessing
discouraged_uses

    discouraged_uses:
    - id: chorus:discouraged:1
      name: Clinical decision-making without proper validation and regulatory approval
      description: 'Dataset is for research purposes only. AI/ML models developed should undergo appropriate
        clinical validation, regulatory approval, and institutional review before use in patient care or clinical
        decision-making.

        '
      discouragement_details:
      - Models trained on CHoRUS data must undergo independent clinical validation
      - Regulatory approval processes (e.g., FDA clearance) required for clinical use
      - Institutional review board approval needed for clinical deployment
    - id: chorus:discouraged:2
      name: Re-identification attempts
      description: 'Attempts to re-identify patients from de-identified data violate ethical principles, data
        use agreements, and legal frameworks established for privacy protection under HIPAA and institutional
        requirements.

        '
      discouragement_details:
      - Re-identification attempts violate the signed data use agreement
      - Prohibited under HIPAA and applicable institutional data privacy regulations
      - Data use agreement explicitly prohibits re-identification efforts
    - id: chorus:discouraged:3
      name: Use without awareness of ongoing data collection limitations
      description: 'As data collection continues through November 2026 and quality assurance processes are
        ongoing, early dataset versions should be used with awareness of completeness limitations and ongoing
        expansion (from 45K to target 100K+ admissions).

        '
      discouragement_details:
      - Dataset is actively growing; cohort coverage varies by data modality
      - EEG extraction and full imaging de-identification still in process as of 2025
      - Clinical notes stored locally at sites; only tokens available in enclave

5/5 R20 Q18 (FAIRness & Accessibility) Reusability (License Clarity)
level License explicitly defines reuse terms
evidence license_and_use_terms: CHoRUS Controlled Access License with Data Use Agreement explicitly defines permitted research uses, prohibits re-identification, requires institutional email and signed agreement. intended_uses: 4 use cases explicitly permitted (AI/ML model development for critical care, external validation, health equity research, educational/training purposes). discouraged_uses: 3 uses explicitly prohibited (clinical decision-making without validation/approval, re-identification attempts, use without awareness of data limitations)
quality Exceptional license clarity with specific permitted uses, explicit prohibitions, and clear reuse requirements (DUA signature, research-only)
correctness Controlled access license appropriate for de-identified clinical data. Research-only restriction standard for NIH-funded datasets pending validation
consistency License terms align with intended_uses (research, education) and discouraged_uses (clinical deployment without approval). Re-identification prohibition consistent with de-identification preprocessing
5/5 R20 Q9 (Metadata Quality & Content) Access Requirements and Governance Documentation
level License + restrictions + confidentiality classification
evidence license_and_use_terms: CHoRUS Controlled Access License with Data Use Agreement, institutional (.edu) email required, signed licensing agreement mandatory, controlled access through secure enclave (Azure-based), prohibits re-identification, access contacts provided (dbold@emory.edu, jared.houghtaling@tuftsmedicine.org). regulatory_compliance: HIPAA, 45 CFR 46, institutional regulations. discouraged_uses explicitly prohibit re-identification and unauthorized clinical use
quality Exceptional access governance documentation with clear controlled access model, licensing requirements, secure infrastructure, and explicit prohibitions on misuse
correctness Controlled access model appropriate for HIPAA-regulated clinical data. Azure secure enclave consistent with NIH data sharing requirements
consistency Access requirements align with sensitive_elements (PHI), regulatory_compliance (HIPAA), and de-identification approach (controlled access despite de-id)
license_and_use_terms

    license_and_use_terms:
      id: chorus:license:1
      name: CHoRUS Controlled Access License with Data Use Agreement
      description: 'Dataset distributed under controlled access requiring institutional email registration
        and signed licensing agreement. Access granted after review and approval process. Participants must
        complete registration form with name, institutional email (not personal), and institution. Once approved,
        users receive email with access instructions to CHoRUS secure enclave. For training program access,
        program administrators assist with licensing.

        '
      license_terms:
      - Institutional (.edu) email required for registration
      - All participants must sign a licensing agreement before gaining access to the dataset
      - Controlled access through secure enclave (Azure-based infrastructure)
      - Data use agreement specifies permitted research uses and prohibits re-identification
      - Access request contacts - dbold@emory.edu or jared.houghtaling@tuftsmedicine.org
      - Funded under NIH award OT2OD032701; content is solely responsibility of authors

5/5 R20 Q17 (FAIRness & Accessibility) Accessibility (Access Mechanism)
level Fully defined access path (platform, login, policy)
evidence distribution_formats: all 5 formats specify access via secure enclave (Azure-based infrastructure) with controlled access. license_and_use_terms: complete registration process (institutional email, licensing agreement signature), approval workflow, access instructions via email after approval, specific contact emails (dbold@emory.edu, jared.houghtaling@tuftsmedicine.org), training program administrator assistance available
quality Exceptional access mechanism documentation with step-by-step process, platform specification (Azure secure enclave), contact information, and alternative access paths (training programs)
correctness Controlled access via secure enclave appropriate for HIPAA-regulated clinical data. Institutional email requirement standard for DUA-based access
consistency Access mechanism aligns with license requirements (DUA), regulatory compliance (HIPAA), and sensitive_elements (PHI). Azure infrastructure consistent with NIH data sharing requirements
5/5 R20 Q18 (FAIRness & Accessibility) Reusability (License Clarity)
level License explicitly defines reuse terms
evidence license_and_use_terms: CHoRUS Controlled Access License with Data Use Agreement explicitly defines permitted research uses, prohibits re-identification, requires institutional email and signed agreement. intended_uses: 4 use cases explicitly permitted (AI/ML model development for critical care, external validation, health equity research, educational/training purposes). discouraged_uses: 3 uses explicitly prohibited (clinical decision-making without validation/approval, re-identification attempts, use without awareness of data limitations)
quality Exceptional license clarity with specific permitted uses, explicit prohibitions, and clear reuse requirements (DUA signature, research-only)
correctness Controlled access license appropriate for de-identified clinical data. Research-only restriction standard for NIH-funded datasets pending validation
consistency License terms align with intended_uses (research, education) and discouraged_uses (clinical deployment without approval). Re-identification prohibition consistent with de-identification preprocessing
5/5 R20 Q9 (Metadata Quality & Content) Access Requirements and Governance Documentation
level License + restrictions + confidentiality classification
evidence license_and_use_terms: CHoRUS Controlled Access License with Data Use Agreement, institutional (.edu) email required, signed licensing agreement mandatory, controlled access through secure enclave (Azure-based), prohibits re-identification, access contacts provided (dbold@emory.edu, jared.houghtaling@tuftsmedicine.org). regulatory_compliance: HIPAA, 45 CFR 46, institutional regulations. discouraged_uses explicitly prohibit re-identification and unauthorized clinical use
quality Exceptional access governance documentation with clear controlled access model, licensing requirements, secure infrastructure, and explicit prohibitions on misuse
correctness Controlled access model appropriate for HIPAA-regulated clinical data. Azure secure enclave consistent with NIH data sharing requirements
consistency Access requirements align with sensitive_elements (PHI), regulatory_compliance (HIPAA), and de-identification approach (controlled access despite de-id)
distribution_formats

    distribution_formats:
    - id: chorus:format:1
      name: OMOP Common Data Model (structured EHR data)
      description: 'Structured electronic health record data distributed in OMOP Common Data Model format
        enabling use of OHDSI tool stack for analysis. Contains 1.6 billion rows. Available in secure enclave
        with controlled access. Published OMOP schema metadata available.

        '
      access_urls:
      - https://chorus4ai.org/
    - id: chorus:format:2
      name: WFDB waveform format (bedside monitor telemetry)
      description: 'Waveform telemetry data (23 TB) distributed in WFDB (WaveForm DataBase) format following
        extended PhysioNet schema. Available in secure enclave with controlled access. Published PhysioNet
        schema (extended) metadata available.

        '
      access_urls:
      - https://chorus4ai.org/
    - id: chorus:format:3
      name: DICOM format (medical imaging)
      description: 'Medical imaging data (7,642 admissions with radiology; 1,000 images currently available)
        distributed in DICOM format with comprehensive metadata following DICOM schema. De-identification
        in process for larger cohort. Planned controlled access.

        '
      access_urls:
      - https://chorus4ai.org/
    - id: chorus:format:4
      name: OHNLP tokenized format (clinical notes)
      description: 'Clinical notes distributed as OHNLP-tokenized text following open source OHNLP schema.
        Stored locally at contributing sites (except tokens). Controlled access planned.

        '
      access_urls:
      - https://chorus4ai.org/
    - id: chorus:format:5
      name: EDF+ and Persyst formats (EEG waveforms)
      description: 'EEG waveform data distributed in EDF+ (European Data Format) and Persyst formats following
        open source schemas. Extraction in process, planned for controlled access.

        '
      access_urls:
      - https://chorus4ai.org/

5/5 R20 Q10 (Metadata Quality & Content) Interoperability and Standardization
level Standard formats + schema/ontology compliance
evidence distribution_formats: OMOP Common Data Model (published OMOP schema metadata), WFDB (published PhysioNet schema extended), DICOM (published DICOM schema metadata), OHNLP tokenized (open source OHNLP schema), EDF+ and Persyst (open source schemas). preprocessing_strategies: semantic mapping validation via chorus-mapping repository, OHDSI tool stack integration, schema compliance validation across all modalities
quality Exceptional interoperability with comprehensive standardization across all five data modalities using internationally recognized standards and published schemas
correctness Format choices semantically appropriate: OMOP (observational health data standard), WFDB (PhysioNet waveform standard), DICOM (medical imaging standard), OHNLP (clinical NLP standard), EDF+ (EEG standard)
consistency Schema conformance validated through preprocessing_strategies and cleaning_strategies. chorus-mapping repository for semantic validation mentioned consistently
5/5 R20 Q17 (FAIRness & Accessibility) Accessibility (Access Mechanism)
level Fully defined access path (platform, login, policy)
evidence distribution_formats: all 5 formats specify access via secure enclave (Azure-based infrastructure) with controlled access. license_and_use_terms: complete registration process (institutional email, licensing agreement signature), approval workflow, access instructions via email after approval, specific contact emails (dbold@emory.edu, jared.houghtaling@tuftsmedicine.org), training program administrator assistance available
quality Exceptional access mechanism documentation with step-by-step process, platform specification (Azure secure enclave), contact information, and alternative access paths (training programs)
correctness Controlled access via secure enclave appropriate for HIPAA-regulated clinical data. Institutional email requirement standard for DUA-based access
consistency Access mechanism aligns with license requirements (DUA), regulatory compliance (HIPAA), and sensitive_elements (PHI). Azure infrastructure consistent with NIH data sharing requirements
5/5 R20 Q4 (Structural Completeness) File Enumeration and Type Variety
level>3 file types
evidence distribution_formats: 5 distinct formats - OMOP Common Data Model (structured EHR), WFDB (waveforms), DICOM (imaging), OHNLP tokenized (clinical notes), EDF+/Persyst (EEG)
quality Exceptional multi-modal data variety with comprehensive format standardization across all modalities
correctness Format choices semantically appropriate for each data type (OMOP for EHR, WFDB for waveforms, DICOM for imaging)
consistency Distribution formats match preprocessing strategies and acquisition methods described
maintainers

    maintainers:
    - id: chorus:maintainer:1
      name: CHoRUS Consortium
      description: 'Multi-institutional consortium managing dataset maintenance including 14 data acquisition
        centers, coordinating teams at Massachusetts General Hospital (lead), University of Florida, UT Health
        Science Center, and Tufts Medicine, plus infrastructure development team. Standards, Data Acquisition,
        and Tooling sub-teams manage ongoing data delivery and quality. Contact: cmccrary@mgh.harvard.edu
        (Ciera McCrary, Program Manager, MGH).

        '
      maintainer_details:
      - Massachusetts General Hospital (lead institution, Contact PI Eric S. Rosenthal)
      - University of Florida (data acquisition and coordination, Azra Bihorac, Parisa Rashidi, Yulia Strekalova)
      - UT Health Science Center / UTHealth Houston (Xiaoqian Jiang)
      - Tufts Medicine (Andrew Ewing Williams, Manlik Kwong, Jared Houghtaling)
      - 14 data acquisition centers across United States contributing clinical data extracts
      - Standards team (semantic mappings and validation via chorus-mapping repository)
      - Data Acquisition team (extraction and contribution per Chorus_SOP)
      - Tooling team (software development across chorus-ai GitHub organization)
      - Project management via GitHub organization (chorus-ai) with 28 active repositories
    - id: chorus:maintainer:2
      name: CHoRUS GitHub Organization (chorus-ai)
      description: 'Active GitHub organization housing repositories for software, semantic mappings, standard
        operating protocols, and project management. 28 repositories with comprehensive documentation and
        community support. Licensed under MIT License (GitHub organization).

        '
      maintainer_details:
      - GitHub organization at https://github.com/chorus-ai
      - Chorus_SOP repository - centralized SOP documentation site (Apache-2.0 license)
      - chorus-container-apps - Azure deployment infrastructure (JavaScript)
      - chorus-mapping - semantic mappings repository
      - chorus_waveform - waveform documentation and conversion scripts (MIT license)
      - privacy_scan_tool - privacy scan tool for medical records (Python)
      - CHoRUSReports - characterization reports returned to contributing sites (R)
      - chorus-extract-upload - tools to create and upload CHoRUS data extract (MIT license)
      - UF-Geocoding - open source code to geocode OMOP Location entities via DeGauss
      - Community discussions and issue tracking for Standards and Data Acquisition teams

no field-level feedback matched
updates

    updates:
      id: chorus:updates:1
      name: Ongoing data collection and continuous expansion
      description: 'Dataset updated continuously as data collection progresses at 14 acquisition centers.
        As of August 2025, covers 14 hospitals with over 45,000 unique admissions; current released dataset
        includes 50,000 patient admissions (ICU, PICU, NICU) and 1.6 billion rows of EHR OMOP data. Target
        exceeds 100,000 critically ill patients. Project timeline extends through November 30, 2026 (approved
        no-cost extension). Regular status updates tracked through GitHub project management system via GitHub
        interface or Google Form submissions. Sites statuses tracked in Standards Project and Data Acquisition
        Project.

        '
      frequency: Continuous updates through November 30, 2026
      update_details:
      - Ongoing retrospective data collection at 14 sites through project end date
      - Current status (August 2025) - 45K+ unique admissions; 50K released (ICU, PICU, NICU)
      - Target - 100,000+ critically ill patients across 9 data modalities
      - Regular site status updates via GitHub interface or Google Form submissions
      - GitHub project tracking for deliverables and task dependencies
      - Documentation updates maintained in Chorus_SOP repository
      - Software and tooling continuous development across chorus-ai GitHub organization
      - Semantic mapping validation and expansion via chorus-mapping repository
      - EEG extraction and full imaging de-identification in progress

✓ 1/1 R10 6.Data Provenance and Version Tracking Update Schedule or Frequency Indicated
evidence Continuous updates through November 30, 2026 with regular status tracking
quality Clear update schedule through specific end date
4/5 R20 Q13 (Technical Documentation) Version History Documentation
level Comprehensive versioning with minor gaps
evidence updates: Ongoing data collection and continuous expansion, frequency 'Continuous updates through November 30, 2026', current status August 2025 (45K+ unique admissions, 50K released), target 100K+ patients. update_details: site status tracking via GitHub interface or Google Form submissions, Standards Project and Data Acquisition Project tracking, documentation updates in Chorus_SOP repository, ongoing EEG extraction and imaging de-identification. retention_limit: long-term retention per NIH data sharing policies. No explicit version numbering or formal release notes
quality Good version tracking through continuous update model with specific snapshots (August 2025) and progress metrics. Missing: formal version numbers, structured release notes, specific errata documentation
correctness Continuous update model appropriate for ongoing multi-center data collection. NIH data sharing policy retention aligns with federal requirements
consistency Update frequency aligns with project timeline (through 11/2026). Site tracking via GitHub consistent with maintainer GitHub organization (chorus-ai)
4/5 R20 Q19 (FAIRness & Accessibility) Data Integrity and Provenance
level Structured version control with timestamps
evidence updates: continuous update model with frequency 'Continuous updates through November 30, 2026', specific snapshot timestamp (August 2025) with metrics (45K+ unique admissions, 50K released, 1.6B rows, 23TB waveforms). update_details: site status tracking via GitHub interface/Google Form, Standards Project and Data Acquisition Project tracking, documentation updates in Chorus_SOP repository. retention_limit: NIH data sharing policies govern retention. Missing: formal version numbers, structured changelog
quality Good provenance tracking with timestamped snapshots, GitHub-based tracking systems, and clear update frequency. Missing: formal version changelog or release notes documenting specific changes between snapshots
correctness Continuous update model appropriate for ongoing multi-center data collection. August 2025 snapshot timestamp plausible within 9/2022-11/2026 timeline
consistency GitHub tracking aligns with maintainer GitHub organization (chorus-ai). NIH data sharing policy retention consistent with federal funding requirements
retention_limit

    retention_limit:
      id: chorus:retention:1
      name: Long-term dataset retention per NIH data sharing policies
      description: 'Digital data maintained according to NIH data sharing policies and institutional requirements
        at participating centers. Controlled access model ensures long-term availability for research while
        protecting patient privacy. Funded under NIH award OT2OD032701 with project end date of November 30,
        2026.

        '
      retention_details:
      - NIH data sharing policies govern long-term retention requirements
      - Institutional requirements at 14 participating data acquisition centers apply
      - Controlled access model via secure enclave for ongoing privacy protection
      - Long-term maintenance through CHoRUS Consortium and associated institutions

4/5 R20 Q13 (Technical Documentation) Version History Documentation
level Comprehensive versioning with minor gaps
evidence updates: Ongoing data collection and continuous expansion, frequency 'Continuous updates through November 30, 2026', current status August 2025 (45K+ unique admissions, 50K released), target 100K+ patients. update_details: site status tracking via GitHub interface or Google Form submissions, Standards Project and Data Acquisition Project tracking, documentation updates in Chorus_SOP repository, ongoing EEG extraction and imaging de-identification. retention_limit: long-term retention per NIH data sharing policies. No explicit version numbering or formal release notes
quality Good version tracking through continuous update model with specific snapshots (August 2025) and progress metrics. Missing: formal version numbers, structured release notes, specific errata documentation
correctness Continuous update model appropriate for ongoing multi-center data collection. NIH data sharing policy retention aligns with federal requirements
consistency Update frequency aligns with project timeline (through 11/2026). Site tracking via GitHub consistent with maintainer GitHub organization (chorus-ai)
4/5 R20 Q19 (FAIRness & Accessibility) Data Integrity and Provenance
level Structured version control with timestamps
evidence updates: continuous update model with frequency 'Continuous updates through November 30, 2026', specific snapshot timestamp (August 2025) with metrics (45K+ unique admissions, 50K released, 1.6B rows, 23TB waveforms). update_details: site status tracking via GitHub interface/Google Form, Standards Project and Data Acquisition Project tracking, documentation updates in Chorus_SOP repository. retention_limit: NIH data sharing policies govern retention. Missing: formal version numbers, structured changelog
quality Good provenance tracking with timestamped snapshots, GitHub-based tracking systems, and clear update frequency. Missing: formal version changelog or release notes documenting specific changes between snapshots
correctness Continuous update model appropriate for ongoing multi-center data collection. August 2025 snapshot timestamp plausible within 9/2022-11/2026 timeline
consistency GitHub tracking aligns with maintainer GitHub organization (chorus-ai). NIH data sharing policy retention consistent with federal funding requirements
human_subject_research

    human_subject_research:
      id: chorus:hsr:1
      name: CHoRUS Human Subjects Research
      description: 'Retrospective data collection from critically ill patients (ICU, PICU, NICU) approved
        through institutional review processes at 14 data acquisition centers. Community-facing ethics focus
        groups conducted to determine what data is appropriate for public sharing. Legal framework established
        for collecting data at scale. Patient-focused efforts determine ethical and legal approaches to manage
        privacy and bias while accounting for Social Determinants of Health. Project includes expertise from
        law, ethics, health services, biomedical science, engineering, and scientific journal publications
        disciplines. Ethics of AI component addressed through AIM-AHEAD training curriculum (safety, risk,
        and legal considerations; IRB, HIPAA/GDPR compliance for OMOP/FHIR data).

        '
      involves_human_subjects: true
      ethics_review_board:
      - Institutional review boards at 14 data acquisition centers across United States
      - Community-facing ethics focus groups determining appropriate data for public sharing
      - Legal and ethical advisory teams including law and ethics discipline experts
      - Privacy and accountability review processes per project three-pillar structure (Data, Ethics, People)
      regulatory_compliance:
      - HIPAA (Health Insurance Portability and Accountability Act) compliance for protected health information
      - 45 CFR 46 (Common Rule) for human subjects research protections
      - Institutional data privacy regulations at each of the 14 contributing sites
      - NIH Common Fund Bridge2AI program ethical and trustworthy AI requirements
      - IRB protocol drafting and HIPAA/GDPR compliance guidance provided in training curriculum

⚠ medium R10 · consistency
issue human_subject_research=True but informed consent not documented
fields human_subject_research, informed_consent
fix Add consent procedure documentation
⚠ medium R20 · consistency
issue human_subject_research.involves_human_subjects=True and ethics_review_board lists 14 IRBs, but no specific IRB approval numbers or protocols documented
fields human_subject_research, ethical_reviews
fix Add IRB protocol numbers from at least lead institution (MGH) for transparency
4/5 R20 Q8 (Metadata Quality & Content) Ethical and Privacy Declarations
level Comprehensive ethics with minor gaps
evidence human_subject_research.involves_human_subjects: true, ethics_review_board: Institutional review boards at 14 data acquisition centers, community-facing ethics focus groups, legal and ethical advisory teams. regulatory_compliance: HIPAA, 45 CFR 46 (Common Rule), institutional data privacy regulations, NIH ethical AI requirements. preprocessing_strategies includes de-identification: OMOP transformation, OHNLP tokenization, DICOM header de-identification, privacy scan tool. sensitive_elements: 4 categories documented (PHI-clinical EHR, waveforms/EEG, imaging, SDOH)
quality Comprehensive ethical documentation covering IRB review, regulatory compliance, de-identification methods, and privacy protections. Missing: specific IRB protocol numbers, explicit informed consent approach (retrospective waiver likely), participant compensation details, vulnerable population safeguards (though PICU/NICU implies pediatric protections)
correctness HIPAA and 45 CFR 46 compliance appropriate for US-based multi-center health data. De-identification methods semantically appropriate for each data type
consistency Ethics review at 14 sites aligns with 14 data acquisition centers. Community ethics focus groups support public sharing claims. Minor gap: no IRB protocol numbers provided despite multi-site coordination
external_resources

    external_resources:
    - id: chorus:resource:1
      name: CHoRUS Project Website
      description: Official project website with dataset overview, team information, project components, and
        access instructions
      external_resources:
      - https://chorus4ai.org/
    - id: chorus:resource:2
      name: CHoRUS GitHub Organization
      description: Comprehensive GitHub organization with 28 repositories including software, documentation,
        SOPs, and tooling
      external_resources:
      - https://github.com/chorus-ai
    - id: chorus:resource:3
      name: Chorus_SOP Documentation Site
      description: Centralized standard operating protocol documentation with interactive workflow diagrams
        for data extraction and contribution
      external_resources:
      - https://github.com/chorus-ai/Chorus_SOP
    - id: chorus:resource:4
      name: NIH RePORTER Project Details
      description: Federal grant information and project details from NIH Research Portfolio Online Reporting
        Tools for grant 1OT2OD032701-01
      external_resources:
      - https://reporter.nih.gov/project-details/10472824
    - id: chorus:resource:5
      name: Bridge2AI Program
      description: Parent NIH Common Fund program supporting AI-ready biomedical datasets across four data
        generation projects
      external_resources:
      - https://bridge2ai.org/
      - https://bridge2ai.org/chorus
    - id: chorus:resource:6
      name: AIM-AHEAD Bridge2AI Training Program
      description: Partnership with AIM-AHEAD for Bridge2AI Clinical Care Training Program (Cohort 1 and Cohort
        2) providing AI/ML training for underrepresented trainees
      external_resources:
      - https://aim-ahead.net/
    - id: chorus:resource:7
      name: OHDSI Community and OMOP CDM
      description: Observational Health Data Sciences and Informatics community supporting OMOP Common Data
        Model used for CHoRUS structured EHR data
      external_resources:
      - https://www.ohdsi.org/
    - id: chorus:resource:8
      name: Published Research (Neurocritical Care)
      description: Peer-reviewed publication documenting CHoRUS dataset methodology and design in Neurocritical
        Care journal
      external_resources:
      - https://doi.org/10.1007/s12028-024-02007

5/5 R20 Q14 (Technical Documentation) Associated Publications
level Multiple references and dataset citation
evidence external_resources: 8 resources including peer-reviewed publication (Neurocritical Care journal, DOI: 10.1007/s12028-024-02007), NIH RePORTER project details (https://reporter.nih.gov/project-details/10472824), GitHub organization (https://github.com/chorus-ai), project website (https://chorus4ai.org/), Bridge2AI program (https://bridge2ai.org/chorus), AIM-AHEAD training partnership, OHDSI community, Chorus_SOP documentation
quality Exceptional external resource documentation with formal peer-reviewed publication DOI, federal grant tracking, comprehensive GitHub repositories, and community partnerships
correctness DOI 10.1007/s12028-024-02007 format valid for Springer journal (Neurocritical Care). NIH RePORTER URL structure correct for grant 10472824
consistency Publication in Neurocritical Care aligns with critical care focus. GitHub organization (chorus-ai) consistent with software/tools documentation. AIM-AHEAD partnership matches training uses described
1/1 R20 Q16 (FAIRness & Accessibility) Findability (Persistent Links)
level Pass
evidence page: https://chorus4ai.org/, id: https://chorus4ai.org/, external_resources: 8 persistent URLs including NIH RePORTER (https://reporter.nih.gov/project-details/10472824), GitHub (https://github.com/chorus-ai), Bridge2AI (https://bridge2ai.org/chorus), publication DOI (https://doi.org/10.1007/s12028-024-02007)
quality Excellent findability with multiple persistent URLs across project website, federal grant database, GitHub organization, and peer-reviewed publication
correctness All URL formats valid. NIH RePORTER, GitHub, and DOI URLs follow standard patterns
consistency All URLs point to consistent CHoRUS project resources across different platforms
1/1 R20 Q20 (FAIRness & Accessibility) Interlinking Across Platforms
level Pass
evidence external_resources: 8 cross-platform links (chorus4ai.org project website, GitHub chorus-ai organization with 28 repositories, NIH RePORTER federal grant database, Bridge2AI program portal, AIM-AHEAD training partnership, OHDSI community, peer-reviewed publication DOI 10.1007/s12028-024-02007)
quality Exceptional interlinking across project website, code repository (GitHub), federal grant tracking (NIH RePORTER), parent program (Bridge2AI), training partnerships (AIM-AHEAD), standards communities (OHDSI), and scholarly literature (publication DOI)
correctness All platform links follow standard URL patterns. NIH RePORTER, GitHub, DOI, and domain URLs structurally valid
consistency Cross-platform links consistently reference CHoRUS project across all resources. GitHub organization (chorus-ai) matches software tools documented
1/1 R20 Q6 (Metadata Quality & Content) Dataset Identification Metadata
level Pass
evidence id: https://chorus4ai.org/, page: https://chorus4ai.org/, external_resources: includes DOI 10.1007/s12028-024-02007 (peer-reviewed publication), NIH RePORTER project link https://reporter.nih.gov/project-details/10472824
quality Multiple persistent identifiers provided including project URL, publication DOI, and federal grant tracking
correctness DOI format valid (10.1007/s12028-024-02007), NIH RePORTER URL structure correct
consistency All URLs point to consistent CHoRUS project resources

## Unmatched feedback

Feedback that referenced multiple fields or no specific field.
✓ 1/1 R10 2.Dataset Access and Retrieval Regulatory Restrictions and Confidentiality Level Specified
evidence Regulatory compliance: HIPAA, 45 CFR 46, institutional regulations, NIH ethical AI requirements
quality Comprehensive regulatory compliance documentation
✓ 1/1 R10 2.Dataset Access and Retrieval Distribution Formats and File Types Specified
evidence 5 formats: OMOP CDM, WFDB, DICOM, OHNLP tokenized, EDF+/Persyst
quality Comprehensive format specification with standard formats and schemas
✓ 1/1 R10 2.Dataset Access and Retrieval Related Datasets and External Resources Linked
evidence 8 external resources including GitHub org (28 repos), OHDSI, Bridge2AI, publication DOI
quality Excellent external resource linkage
✓ 1/1 R10 3.Data Reuse and Interoperability License Terms Allow Reuse
evidence Controlled access with DUA specifying research use permissions
quality Clear license allowing reuse for approved research
✓ 1/1 R10 3.Data Reuse and Interoperability Data Formats Are Standardized
evidence All 5 modalities use international standards: OMOP, WFDB, DICOM, OHNLP, EDF+/Persyst
quality Exemplary use of standardized formats
✓ 1/1 R10 3.Data Reuse and Interoperability Schema or Ontology Conformance Stated
evidence OMOP CDM, DICOM, WFDB/PhysioNet, OHNLP, EDF+/Persyst schemas with validation
quality Comprehensive schema conformance with validation
✓ 1/1 R10 3.Data Reuse and Interoperability Variable Metadata with Identifiers Defined
evidence OMOP vocabulary and concept IDs; published schemas for all 5 modalities
quality Variable-level metadata through OMOP CDM and published schemas
✓ 1/1 R10 3.Data Reuse and Interoperability Use Guidance Provided
evidence 4 intended uses, 3 discouraged uses with detailed rationale
quality Comprehensive use guidance covering technical, ethical, temporal considerations
✓ 1/1 R10 4.Ethical Use and Privacy Safeguards IRB or Ethics Review Documented
evidence IRBs at 14 centers, community ethics focus groups, legal/ethical advisory teams
quality Comprehensive multi-layered ethics oversight
✓ 1/1 R10 4.Ethical Use and Privacy Safeguards Privacy Protections Beyond Deidentification
evidence Secure enclave, DUA prohibiting re-identification, community ethics focus groups, federated access
quality Multi-layered privacy protections: technical, legal, governance, architectural
✗ 0/1 R10 4.Ethical Use and Privacy Safeguards Informed Consent Obtained from Participants
evidence No explicit mention of consent procedures
quality Retrospective collection but consent procedures not documented
✓ 1/1 R10 4.Ethical Use and Privacy Safeguards Vulnerable Populations and Compensation Documented
evidence Pediatric/neonatal ICU patients with community ethics focus groups and SDOH framework
quality Vulnerable populations acknowledged with appropriate protections
✓ 1/1 R10 5.Data Composition and Structure Cohort or Subpopulations Characteristics Described
evidence ICU/PICU/NICU patients, 14 hospitals, 45K+ current/100K+ target with SDOH
quality Comprehensive population description with diversity measures
✓ 1/1 R10 5.Data Composition and Structure Number of Instances or Samples Reported
evidence 45K+ admissions, 50K released, 100K+ target, 1.6B OMOP rows, 23TB waveforms, 7,642 radiology
quality Highly specific instance counts across multiple dimensions
✓ 1/1 R10 5.Data Composition and Structure Variable-Level Metadata and Tabular Flag
evidence OMOP CDM tabular structure with 1.6B rows and OHDSI tool stack
quality Tabular structure confirmed with comprehensive variable metadata
✓ 1/1 R10 5.Data Composition and Structure Data Topics or Conditions Represented
evidence Acute/critical illness, complication prediction, treatment response in ICU/PICU/NICU
quality Clear clinical topics and research focus
✓ 1/1 R10 5.Data Composition and Structure Data Quality Issues and Anomalies Documented
evidence Transparent documentation of ongoing expansion, incomplete modalities, federated sampling
quality Current completeness status and ongoing processes clearly stated
✗ 0/1 R10 6.Data Provenance and Version Tracking Dataset Version Number Provided
evidence No version field
quality Version numbering scheme not documented
✗ 0/1 R10 6.Data Provenance and Version Tracking Version Access Methods Documented
evidence No version access documentation
quality Version access mechanisms not described
✓ 1/1 R10 6.Data Provenance and Version Tracking Change Descriptions and Errata Provided
evidence Current status (Aug 2025) vs target documented with GitHub tracking
quality Current status clear but historical change log absent
✓ 1/1 R10 6.Data Provenance and Version Tracking Provenance and Source Derivation Documented
evidence 4 source types from 14 centers with transformation pipelines documented
quality Comprehensive provenance documentation
✓ 1/1 R10 7.Scientific Motivation and Funding Transparency Funding Sources and Mechanisms Listed
evidence NIH Common Fund Bridge2AI with detailed grant info including $5,880,300 total funding
quality Comprehensive funding documentation
✓ 1/1 R10 7.Scientific Motivation and Funding Transparency Grant IDs or Award Numbers Present
evidence OT2OD032701 (1OT2OD032701-01) with NIH RePORTER link
quality Grant number documented in multiple locations with valid NIH format
✓ 1/1 R10 8.Technical Transparency Collection Mechanisms and Settings Described
evidence 4 collection mechanisms with specific systems and volumes
quality Comprehensive collection documentation
✓ 1/1 R10 8.Technical Transparency Data Acquisition Methods Listed
evidence 5 acquisition methods for all modalities with volumes and schemas
quality Detailed acquisition methods with complete modality coverage
✓ 1/1 R10 8.Technical Transparency Preprocessing, Cleaning, and Labeling Strategies
evidence 5 preprocessing + 2 cleaning + 1 labeling strategies
quality Comprehensive processing documentation with specific tools and SOPs
✓ 1/1 R10 8.Technical Transparency Software and Tools Documented
evidence 7 tools including OHDSI, OHNLP, chorus_waveform, privacy_scan_tool, DeGauss
quality Extensive tool documentation with GitHub organization (28 repos)
✓ 1/1 R10 8.Technical Transparency External Standards and Resources Referenced
evidence 8 resources including Chorus_SOP, GitHub, OHDSI, publication DOI 10.1007/s12028-024-02007
quality Comprehensive external resources with publication DOI
✓ 1/1 R10 9.Dataset Evaluation and Limitations Disclosure Known Limitations Documented
evidence Ongoing expansion limitations, incomplete modalities, partial availability documented
quality Transparent limitation documentation with current/target metrics
✗ 0/1 R10 9.Dataset Evaluation and Limitations Disclosure Systematic Biases Identified and Described
evidence No explicit bias analysis beyond equity focus
quality Equity efforts documented but systematic bias analysis absent
✓ 1/1 R10 9.Dataset Evaluation and Limitations Disclosure Data Anomalies and Quality Issues Noted
evidence Ongoing QA, harmonization challenges, schema validation needs documented
quality Quality issues acknowledged with harmonization complexity
✓ 1/1 R10 9.Dataset Evaluation and Limitations Disclosure Sensitive Content and Warnings Provided
evidence 4 sensitive element categories with detailed sensitivity_details
quality Comprehensive sensitive content documentation
✓ 1/1 R10 9.Dataset Evaluation and Limitations Disclosure Ethical Review Details Including Conflicts
evidence IRB at 14 centers, community ethics, legal/ethical advisory, regulatory compliance
quality Comprehensive ethical review documentation
✓ 1/1 R10 10.Cross-Platform and Community Integration Dataset Published on a Recognized Platform
evidence Secure enclave (Azure) + NIH Bridge2AI + GitHub organization (28 repos)
quality Appropriate platform for controlled access sensitive data
✓ 1/1 R10 10.Cross-Platform and Community Integration Citation and DOI for Cross-referencing
evidence Publication DOI 10.1007/s12028-024-02007 + grant number OT2OD032701
quality Citable publication DOI in peer-reviewed journal
✓ 1/1 R10 10.Cross-Platform and Community Integration Community Standards or Schema Conformance
evidence 5 standards: OMOP (OHDSI), DICOM, WFDB/PhysioNet, OHNLP, EDF+/Persyst
quality Exemplary community standards conformance
✓ 1/1 R10 10.Cross-Platform and Community Integration Outreach Materials and Documentation Links
evidence Chorus_SOP, AIM-AHEAD training (2 cohorts), 28 GitHub repos, websites
quality Comprehensive outreach and documentation
✓ 1/1 R10 10.Cross-Platform and Community Integration Related Datasets with Typed Relationships
evidence Bridge2AI ecosystem (4 projects), OHDSI network, federated multi-center structure
quality Integration with Bridge2AI and OHDSI ecosystems
⚠ low R10 · content_accuracy
issue No systematic bias analysis despite multi-center critical care dataset
fields known_biases
fix Document potential systematic biases
⚠ low R10 · semantic_understanding
issue Version tracking absent despite ongoing expansion
fields version
fix Implement version numbering scheme

## Recommendations

1. R10 · Register dataset with DOI through DataCite
2. R10 · Document informed consent procedures for retrospective collection
3. R10 · Implement version numbering scheme (e.g., v1.0 = 45K, v2.0 = 100K)
4. R10 · Add systematic bias analysis section
5. R10 · Document version access methods once versioning implemented
6. R10 · Clarify conflicts of interest disclosure
7. R10 · Consider dataset-specific DOI in addition to publication DOI
8. R10 · Document historical change log as dataset evolves
9. R20 · Add specific IRB protocol numbers from at least the lead institution (Massachusetts General Hospital) for enhanced transparency and reproducibility
10. R20 · Add unified deidentification method field explicitly stating approach (likely HIPAA Safe Harbor based on described transformations) to complement detailed preprocessing strategies
11. R20 · Implement formal version numbering system and structured changelog documenting progression from initial release through 100K patient target
12. R20 · Expand instances and subpopulations documentation with specific demographic distributions (age ranges, race/ethnicity percentages, sex/gender) to support health equity research claims
13. R20 · Standardize grant number format across all fields to 1OT2OD032701-01 (full NIH format with project prefix)
14. R20 · Add informed_consent field documenting retrospective waiver approach approved by 14 IRBs for consistency with other ethics fields
15. R20 · Document participant compensation (likely none for retrospective study) and add vulnerable_populations field explicitly addressing PICU/NICU pediatric protections
16. R20 · Consider adding RRID identifier for dataset software tools (especially OHDSI, OHNLP, DeGauss) to enhance tool citability
17. R20 · Add errata field to document known issues or corrections as dataset continues expansion through November 2026