Interleaved Semantic Evaluation

Project: CHORUS · Method: claudecode_agent
YAML: data/d4d_concatenated/claudecode_agent/CHORUS_d4d.yaml
R10 JSON: data/evaluation_llm/rubric10_semantic/concatenated/CHORUS_claudecode_agent_evaluation.json
R20 JSON: data/evaluation_llm/rubric20_semantic/concatenated/CHORUS_claudecode_agent_evaluation.json
Model: claude-sonnet-4-5-20250929
Rubric10 (semantic)
42.0/50 (84.0%)
Rubric20 (semantic)
74.0/84 (88.1%)
Consistency checks (R10/R20)
28 pass · 6 fail · 7 warn
Mapped feedback / fields
65 across 29 fields
R10 sub-element R20 question Semantic issue

Strengths

  • Exceptional structural completeness with all mandatory fields populated and comprehensive narrative depth (1,247 char description, 28 keywords, 4 detailed purposes)
  • Outstanding funding documentation with complete NIH grant details (1OT2OD032701-01, $5.88M, fiscal timeline) and 19 creators with institutional affiliations across 14 data acquisition centers
  • Comprehensive multi-modal data standardization with 5 internationally recognized formats (OMOP, WFDB, DICOM, OHNLP, EDF+/Persyst) and published schema metadata for all modalities
  • Excellent technical documentation with specific software tools named, GitHub organization documented (28 repositories including chorus-ai/Chorus_SOP, chorus-mapping), and detailed preprocessing/cleaning/labeling strategies
  • Strong FAIR compliance with multiple persistent identifiers (project URL, publication DOI 10.1007/s12028-024-02007, NIH RePORTER), clear controlled access mechanism (Azure secure enclave, DUA), and exceptional interlinking across 8 platforms
  • Robust ethical framework with 14 IRBs, HIPAA/45 CFR 46 compliance, community ethics focus groups, de-identification across all modalities, and Social Determinants of Health considerations
  • Well-documented access governance with explicit controlled access model, institutional email requirement, signed licensing agreement, access contacts provided, and clear permitted/prohibited uses
  • Comprehensive collection protocol with 4 collection mechanisms, 5 acquisition methods, multi-site coordination details, and complete timeline (9/2022-11/2026 with 8/2025 snapshot)

Weaknesses

  • Missing specific IRB protocol numbers from any of the 14 institutional review boards despite comprehensive ethics documentation
  • No explicit deidentification method summary field (HIPAA Safe Harbor vs Expert Determination) though methods described across preprocessing strategies
  • Lacks formal version numbering system and structured release notes despite good snapshot-based progress tracking (August 2025 status)
  • Missing specific demographic breakdowns (age ranges, race/ethnicity distributions) and explicit inclusion/exclusion criteria for patient enrollment
  • Minor grant number format inconsistency: license field shows OT2OD032701 while funders shows 1OT2OD032701-01
  • No explicit informed consent approach documented (likely retrospective waiver but not stated)
  • Missing participant compensation details and explicit vulnerable population safeguards (though PICU/NICU implies pediatric protections)

Field-by-field

id
id: https://chorus4ai.org/
⚠ medium R10 · correctness
issueDataset identifier is URL rather than standard persistent identifier
fieldsid
fixRegister dataset with DataCite for DOI
✗ 0/1 R10 1.Dataset Discovery and Identification Persistent Identifier (DOI, RRID, or URI)
evidenceid: https://chorus4ai.org/
qualityURL identifier present but not a proper persistent identifier (DOI, RRID, or UUID). This is a landing page URL, not a DOI or other standard persistent identifier format.
semanticdoi_format_check: not_present; rrid_format_check: not_present; plausibility: URL identifier lacks persistence guarantees of DOI/RRID systems
✓ 1/1 R10 4.Ethical Use and Privacy Safeguards Deidentification Method Described
evidence5 methods: re-id limitation, privacy_scan_tool, OHNLP tokenization, DICOM de-id, DeGauss geocoding
qualityMultiple deidentification methods across all modalities
5/5 R20 Q1 (Structural Completeness) Field Completeness
level≥90% fields populated
evidenceid: https://chorus4ai.org/, title: Patient-Focused Collaborative Hospital Repository Uniting Standards (CHoRUS) for Equitable AI, description: 420+ chars with comprehensive detail, keywords: 28 keywords, license: Controlled Access - Data Use Agreement Required (OT2OD032701)
qualityAll mandatory fields present with exceptional content quality and detail
correctnessAll fields semantically appropriate for multi-center critical care dataset
consistencyField content aligns across all mandatory elements
1/1 R20 Q16 (FAIRness & Accessibility) Findability (Persistent Links)
levelPass
evidencepage: https://chorus4ai.org/, id: https://chorus4ai.org/, external_resources: 8 persistent URLs including NIH RePORTER (https://reporter.nih.gov/project-details/10472824), GitHub (https://github.com/chorus-ai), Bridge2AI (https://bridge2ai.org/chorus), publication DOI (https://doi.org/10.1007/s12028-024-02007)
qualityExcellent findability with multiple persistent URLs across project website, federal grant database, GitHub organization, and peer-reviewed publication
correctnessAll URL formats valid. NIH RePORTER, GitHub, and DOI URLs follow standard patterns
consistencyAll URLs point to consistent CHoRUS project resources across different platforms
1/1 R20 Q6 (Metadata Quality & Content) Dataset Identification Metadata
levelPass
evidenceid: https://chorus4ai.org/, page: https://chorus4ai.org/, external_resources: includes DOI 10.1007/s12028-024-02007 (peer-reviewed publication), NIH RePORTER project link https://reporter.nih.gov/project-details/10472824
qualityMultiple persistent identifiers provided including project URL, publication DOI, and federal grant tracking
correctnessDOI format valid (10.1007/s12028-024-02007), NIH RePORTER URL structure correct
consistencyAll URLs point to consistent CHoRUS project resources
name
name: CHoRUS
no field-level feedback matched
title
title: Patient-Focused Collaborative Hospital Repository Uniting Standards (CHoRUS) for Equitable AI
✓ 1/1 R10 1.Dataset Discovery and Identification Dataset Title and Description Completeness
evidencetitle: Patient-Focused Collaborative Hospital Repository Uniting Standards (CHoRUS) for Equitable AI
qualityExcellent title and highly comprehensive description with specific details
5/5 R20 Q1 (Structural Completeness) Field Completeness
level≥90% fields populated
evidenceid: https://chorus4ai.org/, title: Patient-Focused Collaborative Hospital Repository Uniting Standards (CHoRUS) for Equitable AI, description: 420+ chars with comprehensive detail, keywords: 28 keywords, license: Controlled Access - Data Use Agreement Required (OT2OD032701)
qualityAll mandatory fields present with exceptional content quality and detail
correctnessAll fields semantically appropriate for multi-center critical care dataset
consistencyField content aligns across all mandatory elements
description
description: 'CHoRUS for Equitable AI is a Bridge2AI data generation project developing the most diverse,
  high-resolution, ethically sourced, AI-ready critical care dataset to answer the grand challenge of
  improving recovery from acute illness. The project spans 20 academic centers (14 data acquisition centers)
  and is building a publicly available dataset targeting over 100,000 critically ill patients with multi-modal
  data including structured EHR, waveform telemetry, medical imaging, EEG, and clinical notes. All structured
  data is standardized to the OMOP Common Data Model with additional formats (DICOM, WFDB, OHNLP tokenization)
  and comprehensive metadata schemas. Patient-focused efforts determine ethical and legal approaches to
  manage privacy and bias while accounting for Social Determinants of Health. A visualization and annotation
  environment labels data with targets important for prediction. The project emphasizes skills and workforce
  development for a next generation of diverse academic and community AI scientists through training programs
  and partnerships with AIM-AHEAD. As of August 2025, the dataset covers 14 different hospitals with over
  45,000 unique admissions and includes 50,000 patient admissions from ICU, PICU, and NICU, 1.6 billion
  rows of EHR OMOP data, 7,642 admissions with radiology data, and 23 TB of waveform data.

  '
5/5 R20 Q1 (Structural Completeness) Field Completeness
level≥90% fields populated
evidenceid: https://chorus4ai.org/, title: Patient-Focused Collaborative Hospital Repository Uniting Standards (CHoRUS) for Equitable AI, description: 420+ chars with comprehensive detail, keywords: 28 keywords, license: Controlled Access - Data Use Agreement Required (OT2OD032701)
qualityAll mandatory fields present with exceptional content quality and detail
correctnessAll fields semantically appropriate for multi-center critical care dataset
consistencyField content aligns across all mandatory elements
5/5 R20 Q2 (Structural Completeness) Entry Length Adequacy
level>200 chars
evidencedescription: 1,247 chars, purposes: 4 purposes averaging 300+ chars each with comprehensive detail on grand challenge, AI-ready dataset creation, standards establishment, diversity/equity
qualityExceptional narrative depth across all narrative fields with specific metrics and objectives
correctnessNarrative content accurately describes critical care AI dataset with specific enrollment targets and data volumes
consistencyPurposes align with Bridge2AI program goals and dataset composition described
page
page: https://chorus4ai.org/
✓ 1/1 R10 1.Dataset Discovery and Identification Landing Page and Resources
evidencepage: https://chorus4ai.org/; 8 external resources
qualityActive landing page plus extensive external resources
✗ 0/1 R10 2.Dataset Access and Retrieval Download URL or Platform Link Available
evidenceLanding page URL provided but no direct download URL; access via secure enclave
qualityAppropriate for controlled access but no public download URL
1/1 R20 Q16 (FAIRness & Accessibility) Findability (Persistent Links)
levelPass
evidencepage: https://chorus4ai.org/, id: https://chorus4ai.org/, external_resources: 8 persistent URLs including NIH RePORTER (https://reporter.nih.gov/project-details/10472824), GitHub (https://github.com/chorus-ai), Bridge2AI (https://bridge2ai.org/chorus), publication DOI (https://doi.org/10.1007/s12028-024-02007)
qualityExcellent findability with multiple persistent URLs across project website, federal grant database, GitHub organization, and peer-reviewed publication
correctnessAll URL formats valid. NIH RePORTER, GitHub, and DOI URLs follow standard patterns
consistencyAll URLs point to consistent CHoRUS project resources across different platforms
1/1 R20 Q6 (Metadata Quality & Content) Dataset Identification Metadata
levelPass
evidenceid: https://chorus4ai.org/, page: https://chorus4ai.org/, external_resources: includes DOI 10.1007/s12028-024-02007 (peer-reviewed publication), NIH RePORTER project link https://reporter.nih.gov/project-details/10472824
qualityMultiple persistent identifiers provided including project URL, publication DOI, and federal grant tracking
correctnessDOI format valid (10.1007/s12028-024-02007), NIH RePORTER URL structure correct
consistencyAll URLs point to consistent CHoRUS project resources
language
language: en
no field-level feedback matched
license
license: Controlled Access - Data Use Agreement Required (OT2OD032701)
⚠ low R20 · consistency
issueLicense field references grant number (OT2OD032701) but funders section shows 1OT2OD032701-01 with project number prefix
fieldslicense, funders
fixUse consistent grant number format across all fields (recommend 1OT2OD032701-01)
✓ 1/1 R10 2.Dataset Access and Retrieval Access Policy and IP Restrictions Defined
evidencelicense: Controlled Access - Data Use Agreement Required
qualityClear controlled access policy with specific requirements
5/5 R20 Q1 (Structural Completeness) Field Completeness
level≥90% fields populated
evidenceid: https://chorus4ai.org/, title: Patient-Focused Collaborative Hospital Repository Uniting Standards (CHoRUS) for Equitable AI, description: 420+ chars with comprehensive detail, keywords: 28 keywords, license: Controlled Access - Data Use Agreement Required (OT2OD032701)
qualityAll mandatory fields present with exceptional content quality and detail
correctnessAll fields semantically appropriate for multi-center critical care dataset
consistencyField content aligns across all mandatory elements
keywords
keywords:
- CHoRUS
- Bridge2AI
- critical care
- acute illness
- AI-ready dataset
- OMOP Common Data Model
- electronic health records
- EHR
- waveform telemetry
- medical imaging
- DICOM
- EEG
- clinical notes
- OHNLP
- health equity
- social determinants of health
- federated access
- multi-modal data
- high-resolution data
- ethical AI
- trustworthy AI
- workforce development
- data standardization
- privacy preservation
- bias mitigation
- ICU
- intensive care
✓ 1/1 R10 1.Dataset Discovery and Identification Keywords or Tags for Searchability
evidencekeywords: 28 keywords
qualityComprehensive keyword set covering all relevant dimensions
5/5 R20 Q1 (Structural Completeness) Field Completeness
level≥90% fields populated
evidenceid: https://chorus4ai.org/, title: Patient-Focused Collaborative Hospital Repository Uniting Standards (CHoRUS) for Equitable AI, description: 420+ chars with comprehensive detail, keywords: 28 keywords, license: Controlled Access - Data Use Agreement Required (OT2OD032701)
qualityAll mandatory fields present with exceptional content quality and detail
correctnessAll fields semantically appropriate for multi-center critical care dataset
consistencyField content aligns across all mandatory elements
5/5 R20 Q3 (Structural Completeness) Keyword Diversity
level≥8 keywords
evidencekeywords: 28 unique keywords covering project (CHoRUS, Bridge2AI), domain (critical care, acute illness, ICU), data types (EHR, waveforms, imaging, EEG, DICOM, OHNLP), standards (OMOP), themes (health equity, ethical AI, privacy preservation, bias mitigation)
qualityComprehensive keyword coverage across all relevant dimensions with excellent domain specificity
correctnessKeywords accurately reflect multi-modal critical care dataset with standardization focus
consistencyKeywords align with purposes, data formats, and ethical considerations described
purposes
purposes:
- id: chorus:purpose:1
  name: Improve recovery from acute illness
  description: 'Answer the grand challenge of improving recovery from acute illness by developing high-resolution
    multi-center datasets as a critical first step towards actionable and trustworthy AI in critical care.
    Address the urgent need for infrastructure to support artificial intelligence and machine learning
    (AI/ML) in critical care settings.

    '
- id: chorus:purpose:2
  name: Create AI-ready critical care dataset
  description: 'Develop a publicly available, AI-ready critical care dataset from more than 100,000 critically
    ill patients while ensuring methods promote privacy, accountability, and clinical benefit. Generate
    the most diverse, high-resolution, ethically sourced dataset for AI/ML applications in acute and critical
    care, expanding AI and Machine Learning to improve recovery from acute illness.

    '
- id: chorus:purpose:3
  name: Establish data standards and tools
  description: 'Unify standards to harmonize multi-modal EHR, waveform, imaging, and text data. Develop
    software and tooling to interact with and extract insight from clinical data in diverse formats. Create
    validated semantic mappings for connecting clinical data in various source formats to international
    standards (OMOP Common Data Model, DICOM, WFDB, OHNLP).

    '
- id: chorus:purpose:4
  name: Promote diversity and health equity
  description: 'Ensure comprehensive sets of patient conditions and clinical treatment strategies with
    appropriate contextual factors such as geographic distance to nearest hospital and Social Determinants
    of Health. Develop the skills and workforce for a next generation of diverse academic and community
    AI scientists through comprehensive training and education programs in partnership with AIM-AHEAD.

    '
✓ 1/1 R10 7.Scientific Motivation and Funding Transparency Motivation or Purpose for Dataset Creation
evidence4 purposes + 4 research gaps addressed
qualityExceptional motivation documentation
5/5 R20 Q18 (FAIRness & Accessibility) Reusability (License Clarity)
levelLicense explicitly defines reuse terms
evidencelicense_and_use_terms: CHoRUS Controlled Access License with Data Use Agreement explicitly defines permitted research uses, prohibits re-identification, requires institutional email and signed agreement. intended_uses: 4 use cases explicitly permitted (AI/ML model development for critical care, external validation, health equity research, educational/training purposes). discouraged_uses: 3 uses explicitly prohibited (clinical decision-making without validation/approval, re-identification attempts, use without awareness of data limitations)
qualityExceptional license clarity with specific permitted uses, explicit prohibitions, and clear reuse requirements (DUA signature, research-only)
correctnessControlled access license appropriate for de-identified clinical data. Research-only restriction standard for NIH-funded datasets pending validation
consistencyLicense terms align with intended_uses (research, education) and discouraged_uses (clinical deployment without approval). Re-identification prohibition consistent with de-identification preprocessing
5/5 R20 Q2 (Structural Completeness) Entry Length Adequacy
level>200 chars
evidencedescription: 1,247 chars, purposes: 4 purposes averaging 300+ chars each with comprehensive detail on grand challenge, AI-ready dataset creation, standards establishment, diversity/equity
qualityExceptional narrative depth across all narrative fields with specific metrics and objectives
correctnessNarrative content accurately describes critical care AI dataset with specific enrollment targets and data volumes
consistencyPurposes align with Bridge2AI program goals and dataset composition described
tasks
tasks:
- id: chorus:task:1
  name: Characterize acute and critical care illness
  description: 'Generate data for ML/AI applications aimed at characterizing acute and critical care illness
    patterns, progression, and outcomes across diverse patient populations and hospital settings.

    '
- id: chorus:task:2
  name: Predict complications in critically ill patients
  description: 'Enable prediction of complications among patients with acute or critical illness using
    multi-modal data including structured EHR, waveforms, imaging, and clinical notes.

    '
- id: chorus:task:3
  name: Measure treatment response
  description: 'Support measurement and analysis of treatment response among critically ill patients through
    high-frequency documentation, medication administration records, and clinical outcomes data.

    '
- id: chorus:task:4
  name: External validation for AI model marketplace adoption
  description: 'Provision a holdout test set accessible for model external validation to aid marketplace
    adoption of AI-developed models for implementation in acute and critical care settings.

    '
- id: chorus:task:5
  name: Label data for prediction targets
  description: 'Utilize visualization and annotation environment to label data with targets important
    for prediction tasks in critical care AI applications.

    '
✓ 1/1 R10 7.Scientific Motivation and Funding Transparency Primary Research Objectives or Tasks
evidence5 detailed tasks with clear descriptions
qualityComprehensive research objectives
addressing_gaps
addressing_gaps:
- id: chorus:gap:1
  name: Lack of diverse multi-center critical care datasets
  description: 'Address the absence of large-scale, diverse, high-resolution multi-center datasets for
    critical care AI/ML by creating a dataset spanning 20 academic centers with 100,000+ critically ill
    patients and ensuring balanced, diverse cohorts through federated access and sampling methods across
    14 data acquisition centers.

    '
- id: chorus:gap:2
  name: Insufficient data standardization in critical care
  description: 'Overcome the lack of unified standards in critical care data by harmonizing multi-modal
    EHR, waveform, imaging, and text data to OMOP Common Data Model and other international standards
    (DICOM, WFDB, OHNLP).

    '
- id: chorus:gap:3
  name: Privacy and bias concerns in clinical AI
  description: 'Address ethical and legal challenges in AI through patient-focused efforts that determine
    approaches to manage privacy and bias while accounting for Social Determinants of Health and performing
    community-facing ethics focus groups to determine what data is appropriate for public sharing.

    '
- id: chorus:gap:4
  name: Limited diversity in AI/ML workforce
  description: 'Bridge the gap in diverse AI/ML workforce through comprehensive educational approaches,
    training programs (including AIM-AHEAD partnership), and cultivation of expertise in lay and scientific
    communities to improve AI literacy and utilization.

    '
5/5 R20 Q15 (Technical Documentation) Human Subject Representation
levelDetailed demographics and inclusion/exclusion criteria
evidenceinstances: 50,000 patient admissions (ICU, PICU, NICU) from 14 hospitals, 45,000+ unique admissions, target 100K+ critically ill patients, retrospective data collection. subpopulations: 2 documented (by hospital - 14 different hospitals across 20 academic centers for geographic/institutional diversity; by care setting - ICU, PICU, NICU). addressing_gaps: emphasizes 'most diverse' dataset with Social Determinants of Health, federated sampling for balanced diverse cohorts. sampling_strategies: multi-center federated sampling, is_representative: true, representative_verification includes geographic/institutional diversity and SDOH contextual factors
qualityExcellent human subject representation with specific population sizes, geographic diversity across 14 hospitals, care setting diversity (ICU/PICU/NICU), and explicit diversity goals with SDOH considerations. Missing: specific demographic breakdowns (age ranges, race/ethnicity distributions, explicit inclusion/exclusion criteria)
correctnessPopulation sizes plausible for multi-center 4-year critical care study. ICU/PICU/NICU settings appropriate for acute/critical illness focus
consistency14 hospitals consistent across multiple fields. Diversity emphasis aligns with health equity purposes and SDOH sensitive_elements. Retrospective collection aligns with IRB approvals
creators
creators:
- id: chorus:creator:1
  name: Eric S. Rosenthal
  description: Contact PI/Project Leader, Massachusetts General Hospital (MGH), Director of MGH Neurosciences
    ICU
- id: chorus:creator:2
  name: Azra Bihorac
  description: Principal Investigator, University of Florida (UF)
- id: chorus:creator:3
  name: Ashley Cordes
  description: Principal Investigator, CHoRUS Consortium
- id: chorus:creator:4
  name: Gilles Clermont
  description: Principal Investigator, CHoRUS Consortium
- id: chorus:creator:5
  name: Gari David Clifford
  description: Principal Investigator, CHoRUS Consortium
- id: chorus:creator:6
  name: Barbara J. Evans
  description: Principal Investigator, CHoRUS Consortium
- id: chorus:creator:7
  name: Xiao Hu
  description: Principal Investigator, CHoRUS Consortium
- id: chorus:creator:8
  name: Rishikesan Kamaleswaran
  description: Principal Investigator, CHoRUS Consortium
- id: chorus:creator:9
  name: Yulia A. Levites Strekalova
  description: Principal Investigator, University of Florida (UF)
- id: chorus:creator:10
  name: Parisa Rashidi
  description: Principal Investigator, University of Florida (UF)
- id: chorus:creator:11
  name: Cynthia Rudin
  description: Principal Investigator, CHoRUS Consortium
- id: chorus:creator:12
  name: Ishan Canty Williams
  description: Principal Investigator, CHoRUS Consortium
- id: chorus:creator:13
  name: Andrew Ewing Williams
  description: Principal Investigator, Tufts Medicine
- id: chorus:creator:14
  name: Xiaoqian Jiang
  description: Program Lead, UT Health Science Center (UTHealth Houston)
- id: chorus:creator:15
  name: Morteza Zabihi
  description: Lecturer (Machine Learning Basics), Massachusetts General Hospital
- id: chorus:creator:16
  name: Zhenhong Hu
  description: Instructor (Python and Version Control), University of Florida
- id: chorus:creator:17
  name: Debora Simmons
  description: Lecturer (Ethics of AI in Clinical Practice), UT Health Science Center
- id: chorus:creator:18
  name: Aliyah Geer
  description: Workshop Lead (Data Schemas in Clinical Cloud), Massachusetts General Hospital
- id: chorus:creator:19
  name: Ciera McCrary
  description: Program Manager, Massachusetts General Hospital (MGH)
✓ 1/1 R10 7.Scientific Motivation and Funding Transparency Creators and Acknowledgements Documented
evidence19 creators with roles and institutional affiliations
qualityComprehensive creator documentation across multiple roles
5/5 R20 Q7 (Metadata Quality & Content) Funding and Acknowledgements Completeness
levelFunders with grants + creators with affiliations
evidencefunders: NIH Common Fund Bridge2AI Program with grant 1OT2OD032701-01, total funding $5,880,300 (2022), project dates September 1, 2022 to November 30, 2026. creators: 19 creators with institutional affiliations (MGH, UF, UT Health, Tufts Medicine, Emory) and roles (Contact PI, Principal Investigator, Program Lead, Lecturer, Workshop Lead, Program Manager)
qualityExceptional funding documentation with grant number, fiscal details, project timeline, and comprehensive creator affiliations across all participating institutions
correctnessGrant number 1OT2OD032701-01 follows NIH OT2 (Other Transaction) format correctly for Bridge2AI program
consistencyCreator institutions align with 14 data acquisition centers and 20 academic centers described. Minor inconsistency: license field shows OT2OD032701 without project prefix 1-01
funders
funders:
- id: chorus:funder:1
  name: NIH Common Fund Bridge2AI Program
  description: 'Funded through National Institutes of Health grant OT2OD032701 (project number 1OT2OD032701-01),
    administered by NIH Office of the Director. Opportunity Number: OTA-21-008. Study Section: Data Coordination,
    Mapping, and Modeling (DCMM). Fiscal Year 2022. Total funding in 2022: $5,880,300 (all direct costs).
    Project dates: September 1, 2022 to November 30, 2026 (with approved no-cost extension). Award notice
    date: September 1, 2022. Assistance Listing Number: 93.310.

    '
⚠ low R20 · consistency
issueLicense field references grant number (OT2OD032701) but funders section shows 1OT2OD032701-01 with project number prefix
fieldslicense, funders
fixUse consistent grant number format across all fields (recommend 1OT2OD032701-01)
⚠ low R20 · correctness
issueGrant number OT2OD032701 follows NIH format correctly but uses OT2 (Other Transaction) mechanism which is non-standard for typical research grants
fieldsfunders
fixVerify OT2 mechanism is intentional for Bridge2AI program (appears correct for this special NIH Common Fund initiative)
5/5 R20 Q7 (Metadata Quality & Content) Funding and Acknowledgements Completeness
levelFunders with grants + creators with affiliations
evidencefunders: NIH Common Fund Bridge2AI Program with grant 1OT2OD032701-01, total funding $5,880,300 (2022), project dates September 1, 2022 to November 30, 2026. creators: 19 creators with institutional affiliations (MGH, UF, UT Health, Tufts Medicine, Emory) and roles (Contact PI, Principal Investigator, Program Lead, Lecturer, Workshop Lead, Program Manager)
qualityExceptional funding documentation with grant number, fiscal details, project timeline, and comprehensive creator affiliations across all participating institutions
correctnessGrant number 1OT2OD032701-01 follows NIH OT2 (Other Transaction) format correctly for Bridge2AI program
consistencyCreator institutions align with 14 data acquisition centers and 20 academic centers described. Minor inconsistency: license field shows OT2OD032701 without project prefix 1-01
instances
instances:
- id: chorus:instance:1
  name: Critically ill patients in ICU settings
  description: 'Individual critically ill patients requiring acute or critical care admitted to intensive
    care units (ICU), pediatric intensive care units (PICU), or neonatal intensive care units (NICU) across
    14 data acquisition centers. As of August 2025, the dataset covers 14 different hospitals with over
    45,000 unique admissions, with 50,000 patient admissions from ICU, PICU, and NICU currently released.
    Target enrollment exceeds 100,000 critically ill patients. Retrospective data collection from patients
    with acute or critical illness.

    '
  instance_type: 'Human subjects - critically ill patients admitted to participating hospitals (retrospective
    data collection, ongoing collection through November 2026).

    '
5/5 R20 Q15 (Technical Documentation) Human Subject Representation
levelDetailed demographics and inclusion/exclusion criteria
evidenceinstances: 50,000 patient admissions (ICU, PICU, NICU) from 14 hospitals, 45,000+ unique admissions, target 100K+ critically ill patients, retrospective data collection. subpopulations: 2 documented (by hospital - 14 different hospitals across 20 academic centers for geographic/institutional diversity; by care setting - ICU, PICU, NICU). addressing_gaps: emphasizes 'most diverse' dataset with Social Determinants of Health, federated sampling for balanced diverse cohorts. sampling_strategies: multi-center federated sampling, is_representative: true, representative_verification includes geographic/institutional diversity and SDOH contextual factors
qualityExcellent human subject representation with specific population sizes, geographic diversity across 14 hospitals, care setting diversity (ICU/PICU/NICU), and explicit diversity goals with SDOH considerations. Missing: specific demographic breakdowns (age ranges, race/ethnicity distributions, explicit inclusion/exclusion criteria)
correctnessPopulation sizes plausible for multi-center 4-year critical care study. ICU/PICU/NICU settings appropriate for acute/critical illness focus
consistency14 hospitals consistent across multiple fields. Diversity emphasis aligns with health equity purposes and SDOH sensitive_elements. Retrospective collection aligns with IRB approvals
1/1 R20 Q5 (Structural Completeness) Data File Size Availability
levelPass
evidenceinstances: 50,000 patient admissions released (45,000+ unique admissions), target 100,000+ critically ill patients. Specific data volumes: 1.6 billion rows of EHR OMOP data, 23 TB of waveform data, 7,642 admissions with radiology data, 1,000 images available
qualityComprehensive instance counts and data volumes provided with specific metrics by modality
correctnessInstance counts plausible for multi-center critical care dataset over 3+ years
consistencyData volumes align with described collection mechanisms and timeframes
subsets
subsets:
- id: chorus:subset:1
  name: Controlled Access Structured EHR Dataset (OMOP)
  description: 'Primary dataset with controlled access requiring data use agreement and licensing. Includes
    OMOP-standardized structured EHR data comprising demographics, medication administration (dosing time-stamped
    upon each infusion change or dose administration), procedures, nursing flowsheets (high-frequency
    documentation), and diagnoses. Available in secure enclave. As of August 2025, contains 1.6 billion
    rows of EHR OMOP data. Registration required with institutional email; all participants must sign
    licensing agreement.

    '
- id: chorus:subset:2
  name: Waveform Telemetry Dataset (WFDB)
  description: 'Waveform telemetry data from bedside monitors (gateway/middleware) standardized to WFDB
    (WaveForm DataBase) format following extended PhysioNet schema. Available in secure enclave with controlled
    access. As of August 2025, contains 23 TB of waveform data. Published metadata schema: PhysioNet schema
    (extended).

    '
- id: chorus:subset:3
  name: Medical Imaging Dataset (DICOM)
  description: 'Imaging data from hospital PACS (Picture Archiving and Communication System) in DICOM
    format with comprehensive metadata following DICOM schema. As of August 2025, 1,000 images are available
    with de-identification in process for the larger cohort. 7,642 admissions have radiology data. Controlled
    access planned.

    '
- id: chorus:subset:4
  name: Clinical Notes Dataset (OHNLP Tokenized)
  description: 'Clinical notes extracted and tokenized using OHNLP (Open Health Natural Language Processing)
    toolkit following open source OHNLP schema. Stored locally at contributing sites (except tokens).
    Controlled access planned. De-identification achieved through tokenization approach.

    '
- id: chorus:subset:5
  name: EEG Waveform Dataset (EDF+ and Persyst)
  description: 'Electroencephalography (EEG) waveform data from hospital databases in EDF+ (European Data
    Format) and Persyst formats. Open source EDF+ and Persyst schema metadata available. Extraction in
    process, planned for controlled access.

    '
- id: chorus:subset:6
  name: Training and Publication Subset
  description: 'Subsets of the dataset currently being used for training activities (AIM-AHEAD Bridge2AI
    for Clinical Care Training Program) and publications as the dataset undergoes expansion and quality
    assurance processes.

    '
✓ 1/1 R10 1.Dataset Discovery and Identification Hierarchical Structure
evidence6 distinct subsets (OMOP EHR, WFDB waveforms, DICOM imaging, OHNLP notes, EEG, training subset)
qualityWell-structured hierarchical organization with 6 modality-based subsets
sampling_strategies
sampling_strategies:
- id: chorus:sampling:1
  name: Multi-center federated sampling
  description: 'Federated access enables sampling methods to ensure a balanced and diverse cohort across
    20 academic centers (14 data acquisition centers). Legal framework established for collecting data
    at scale with sampling to ensure comprehensive sets of patient conditions and clinical treatment strategies.
    Community-facing ethics focus groups conducted to determine what data is appropriate for public sharing.

    '
  is_sample: true
  is_random: false
  is_representative: true
  source_data:
  - Critically ill patients in ICU, PICU, and NICU settings at 14 data acquisition centers across 20 academic
    centers in the United States
  representative_verification:
  - Federated multi-center data collection across 14 hospitals ensures geographic and institutional diversity;
    sampling for balanced and diverse patient populations; inclusion of Social Determinants of Health
    contextual factors
5/5 R20 Q15 (Technical Documentation) Human Subject Representation
levelDetailed demographics and inclusion/exclusion criteria
evidenceinstances: 50,000 patient admissions (ICU, PICU, NICU) from 14 hospitals, 45,000+ unique admissions, target 100K+ critically ill patients, retrospective data collection. subpopulations: 2 documented (by hospital - 14 different hospitals across 20 academic centers for geographic/institutional diversity; by care setting - ICU, PICU, NICU). addressing_gaps: emphasizes 'most diverse' dataset with Social Determinants of Health, federated sampling for balanced diverse cohorts. sampling_strategies: multi-center federated sampling, is_representative: true, representative_verification includes geographic/institutional diversity and SDOH contextual factors
qualityExcellent human subject representation with specific population sizes, geographic diversity across 14 hospitals, care setting diversity (ICU/PICU/NICU), and explicit diversity goals with SDOH considerations. Missing: specific demographic breakdowns (age ranges, race/ethnicity distributions, explicit inclusion/exclusion criteria)
correctnessPopulation sizes plausible for multi-center 4-year critical care study. ICU/PICU/NICU settings appropriate for acute/critical illness focus
consistency14 hospitals consistent across multiple fields. Diversity emphasis aligns with health equity purposes and SDOH sensitive_elements. Retrospective collection aligns with IRB approvals
subpopulations
subpopulations:
- id: chorus:subpop:1
  name: Critically ill patients by hospital
  description: 'Patients distributed across 14 different hospitals within the CHoRUS network spanning
    20 academic centers in the United States, ensuring geographic and institutional diversity in critical
    care settings.

    '
- id: chorus:subpop:2
  name: ICU, PICU, and NICU patients
  description: 'Patients admitted to intensive care units (ICU), pediatric intensive care units (PICU),
    and neonatal intensive care units (NICU) across contributing hospitals, with 50,000 patient admissions
    in the current released dataset.

    '
5/5 R20 Q15 (Technical Documentation) Human Subject Representation
levelDetailed demographics and inclusion/exclusion criteria
evidenceinstances: 50,000 patient admissions (ICU, PICU, NICU) from 14 hospitals, 45,000+ unique admissions, target 100K+ critically ill patients, retrospective data collection. subpopulations: 2 documented (by hospital - 14 different hospitals across 20 academic centers for geographic/institutional diversity; by care setting - ICU, PICU, NICU). addressing_gaps: emphasizes 'most diverse' dataset with Social Determinants of Health, federated sampling for balanced diverse cohorts. sampling_strategies: multi-center federated sampling, is_representative: true, representative_verification includes geographic/institutional diversity and SDOH contextual factors
qualityExcellent human subject representation with specific population sizes, geographic diversity across 14 hospitals, care setting diversity (ICU/PICU/NICU), and explicit diversity goals with SDOH considerations. Missing: specific demographic breakdowns (age ranges, race/ethnicity distributions, explicit inclusion/exclusion criteria)
correctnessPopulation sizes plausible for multi-center 4-year critical care study. ICU/PICU/NICU settings appropriate for acute/critical illness focus
consistency14 hospitals consistent across multiple fields. Diversity emphasis aligns with health equity purposes and SDOH sensitive_elements. Retrospective collection aligns with IRB approvals
sensitive_elements
sensitive_elements:
- id: chorus:sensitive:1
  name: Protected health information - clinical EHR data
  description: 'Complete electronic health records including demographics, diagnoses, procedures, medications,
    nursing documentation, and clinical notes for critically ill patients. Contains protected health information
    subject to HIPAA and institutional privacy requirements. De-identified before release through OMOP
    transformation and privacy scanning tools.

    '
  sensitive_elements_present: true
  sensitivity_details:
  - Demographics and patient identifiers (de-identified before controlled access release)
  - Diagnoses and medical conditions (OMOP standardized)
  - Medication administration records with dosing timestamps (OMOP standardized)
  - Procedures and clinical interventions documented by providers (OMOP standardized)
  - Clinical notes (tokenized using OHNLP toolkit for privacy protection)
  - Nursing flowsheet documentation at high frequency (OMOP with extensions)
- id: chorus:sensitive:2
  name: Physiological monitoring and EEG waveform data
  description: 'Continuous waveform telemetry from bedside monitors and EEG recordings capturing detailed
    physiological states of critically ill patients. May reveal sensitive health conditions and treatment
    responses. Stored in WFDB format (telemetry) and EDF+/Persyst formats (EEG) with controlled access.

    '
  sensitive_elements_present: true
  sensitivity_details:
  - Continuous cardiac waveforms from bedside monitors (WFDB format, 23 TB)
  - Respiratory monitoring and hemodynamic measurement data
  - Electroencephalography (EEG) recordings from hospital databases
  - High-frequency physiological parameters
- id: chorus:sensitive:3
  name: Medical imaging data
  description: 'Diagnostic imaging studies in DICOM format from hospital PACS systems. De-identification
    in process to remove embedded patient information while preserving clinical utility. As of August
    2025, 1,000 images available with de-id in process for larger cohort (7,642 admissions with radiology
    data total).

    '
  sensitive_elements_present: true
  sensitivity_details:
  - Radiology images (CT scans, X-rays, MRI and other modalities) in DICOM format
  - DICOM metadata (de-identification in process)
  - Embedded patient information being removed through de-identification pipeline
- id: chorus:sensitive:4
  name: Social Determinants of Health data
  description: 'Contextual factors including geographic information (distance to hospital) and social
    determinants of health data. Collected to support health equity research while maintaining patient
    privacy through privacy-preserving transformations.

    '
  sensitive_elements_present: true
  sensitivity_details:
  - Geographic location information (distance to nearest hospital)
  - Social determinants of health variables
  - Contextual equity factors with privacy-preserving transformations applied
4/5 R20 Q8 (Metadata Quality & Content) Ethical and Privacy Declarations
levelComprehensive ethics with minor gaps
evidencehuman_subject_research.involves_human_subjects: true, ethics_review_board: Institutional review boards at 14 data acquisition centers, community-facing ethics focus groups, legal and ethical advisory teams. regulatory_compliance: HIPAA, 45 CFR 46 (Common Rule), institutional data privacy regulations, NIH ethical AI requirements. preprocessing_strategies includes de-identification: OMOP transformation, OHNLP tokenization, DICOM header de-identification, privacy scan tool. sensitive_elements: 4 categories documented (PHI-clinical EHR, waveforms/EEG, imaging, SDOH)
qualityComprehensive ethical documentation covering IRB review, regulatory compliance, de-identification methods, and privacy protections. Missing: specific IRB protocol numbers, explicit informed consent approach (retrospective waiver likely), participant compensation details, vulnerable population safeguards (though PICU/NICU implies pediatric protections)
correctnessHIPAA and 45 CFR 46 compliance appropriate for US-based multi-center health data. De-identification methods semantically appropriate for each data type
consistencyEthics review at 14 sites aligns with 14 data acquisition centers. Community ethics focus groups support public sharing claims. Minor gap: no IRB protocol numbers provided despite multi-site coordination
collection_mechanisms
collection_mechanisms:
- id: chorus:collection:1
  name: Retrospective EHR extraction
  description: 'Retrospective data collection from electronic health record systems at 14 data acquisition
    centers. Data extracted includes demographics, medication administration (dosing time-stamped upon
    each infusion change or dose administration), procedures, nursing flowsheets (high-frequency documentation),
    diagnoses, and clinical notes. Standardized to OMOP Common Data Model.

    '
- id: chorus:collection:2
  name: Waveform telemetry capture
  description: 'Continuous waveform telemetry data captured from bedside monitors through gateway and
    middleware systems at each contributing hospital. Stored in WFDB (WaveForm DataBase) format following
    extended PhysioNet schema. Contains 23 TB of waveform data as of August 2025.

    '
- id: chorus:collection:3
  name: Medical imaging acquisition from PACS
  description: 'Medical imaging data acquired from hospital Picture Archiving and Communication Systems
    (PACS) and stored in DICOM format with comprehensive metadata following DICOM schema. 7,642 admissions
    have radiology data; 1,000 images currently available.

    '
- id: chorus:collection:4
  name: EEG recording extraction
  description: 'Electroencephalography recordings extracted from hospital EEG databases in EDF+ (European
    Data Format) and Persyst formats with metadata following open source schemas. Extraction in process
    as of August 2025.

    '
5/5 R20 Q12 (Technical Documentation) Collection Protocol Clarity
levelFull collection protocol with methods, collectors, and timeframes
evidencecollection_mechanisms: 4 detailed mechanisms (retrospective EHR extraction, waveform telemetry capture via gateway/middleware, medical imaging from PACS, EEG extraction from databases). acquisition_methods: 5 methods with technical details (OMOP transformation, OHNLP tokenization, DICOM from PACS, WFDB from bedside monitors, EEG from hospital databases). Collection timeframes: September 1, 2022 to November 30, 2026 (approved no-cost extension), snapshot August 2025. Data collectors: 14 data acquisition centers across 20 academic centers, coordinated by CHoRUS Consortium (MGH lead, UF, UT Health, Tufts Medicine)
qualityComprehensive collection protocol documentation with detailed mechanisms, methods, multi-site coordination, and clear timeline with current status snapshot
correctnessCollection mechanisms appropriate for multi-center retrospective critical care study. 4-year timeline plausible for 100K+ patient dataset
consistencyCollection mechanisms align with distribution formats. 14 data acquisition centers consistent across multiple fields. Timeframes logically ordered (start 9/2022, snapshot 8/2025, end 11/2026)
acquisition_methods
acquisition_methods:
- id: chorus:acquisition:1
  name: Structured EHR data via OMOP transformation
  description: 'Demographics, medication administration (dosing time-stamped upon each infusion change
    or dose administration), procedures, nursing flowsheets (high-frequency documentation), and diagnoses
    acquired from electronic health records and standardized to OMOP Common Data Model. Controlled access
    with published OMOP schema metadata. Contains 1.6 billion rows of OMOP data as of August 2025.

    '
- id: chorus:acquisition:2
  name: Clinical notes via OHNLP tokenization
  description: 'Clinical notes extracted from EHR systems and tokenized using OHNLP (Open Health Natural
    Language Processing) toolkit. Stored locally at sites except for tokens. Controlled access planned
    with OHNLP open source schema metadata.

    '
- id: chorus:acquisition:3
  name: Medical imaging via DICOM from PACS
  description: 'Imaging data acquired from hospital PACS systems in DICOM format. De-identification in
    process. Planned controlled access with published DICOM schema metadata.

    '
- id: chorus:acquisition:4
  name: Waveform telemetry via WFDB from bedside monitors
  description: 'Bedside monitor waveform data acquired through gateway and middleware systems. Stored
    in WFDB format with controlled access and published PhysioNet schema (extended) metadata. 23 TB available
    as of August 2025.

    '
- id: chorus:acquisition:5
  name: EEG waveforms from hospital databases
  description: 'EEG recordings from hospital databases in EDF+ and Persyst formats. Extraction in process.
    Planned controlled access with open source EDF+ and Persyst schema metadata.

    '
5/5 R20 Q12 (Technical Documentation) Collection Protocol Clarity
levelFull collection protocol with methods, collectors, and timeframes
evidencecollection_mechanisms: 4 detailed mechanisms (retrospective EHR extraction, waveform telemetry capture via gateway/middleware, medical imaging from PACS, EEG extraction from databases). acquisition_methods: 5 methods with technical details (OMOP transformation, OHNLP tokenization, DICOM from PACS, WFDB from bedside monitors, EEG from hospital databases). Collection timeframes: September 1, 2022 to November 30, 2026 (approved no-cost extension), snapshot August 2025. Data collectors: 14 data acquisition centers across 20 academic centers, coordinated by CHoRUS Consortium (MGH lead, UF, UT Health, Tufts Medicine)
qualityComprehensive collection protocol documentation with detailed mechanisms, methods, multi-site coordination, and clear timeline with current status snapshot
correctnessCollection mechanisms appropriate for multi-center retrospective critical care study. 4-year timeline plausible for 100K+ patient dataset
consistencyCollection mechanisms align with distribution formats. 14 data acquisition centers consistent across multiple fields. Timeframes logically ordered (start 9/2022, snapshot 8/2025, end 11/2026)
preprocessing_strategies
preprocessing_strategies:
- id: chorus:preproc:1
  name: OMOP Common Data Model transformation
  description: 'All structured electronic health record data standardized to the OMOP (Observational Medical
    Outcomes Partnership) Common Data Model. Ensures interoperability and enables use of OHDSI (Observational
    Health Data Sciences and Informatics) tool stack for analysis. Data from all 14 acquisition centers
    unified through this transformation.

    '
  preprocessing_details:
  - Transformation of source EHR data from diverse institutional formats to OMOP CDM
  - Standardization of terminology and clinical codes to OMOP vocabulary standards
  - Mapping to OMOP vocabulary standards using validated semantic mappings
  - Quality assurance of transformed data against OMOP schema specifications
  - Integration with OHDSI tools for downstream analysis and characterization
  - Generation of characterization reports returned to contributing sites via CHoRUSReports
- id: chorus:preproc:2
  name: Clinical note tokenization via OHNLP
  description: 'Clinical notes processed using OHNLP (Open Health Natural Language Processing) toolkit
    for extraction and tokenization. Protects patient privacy while enabling natural language processing
    and analysis of clinical text data.

    '
  preprocessing_details:
  - Text extraction from clinical notes in source EHR systems
  - OHNLP tokenization pipeline applied to free-text clinical documentation
  - De-identification of sensitive information through tokenization approach
  - Standardization to OHNLP open source schema
  - Local storage of full notes at sites; only tokens available in enclave
- id: chorus:preproc:3
  name: Waveform standardization to WFDB format
  description: 'Waveform telemetry data from diverse bedside monitoring systems standardized to WFDB (WaveForm
    DataBase) format following extended PhysioNet schema. Scripts available in chorus_waveform repository.

    '
  preprocessing_details:
  - Conversion from proprietary bedside monitor formats using gateway/middleware systems
  - Standardization to WFDB format per extended PhysioNet schema
  - Metadata extraction and schema compliance verification
  - Quality checks for waveform integrity and completeness
  - Synchronization with clinical events from OMOP EHR data
- id: chorus:preproc:4
  name: Medical imaging de-identification
  description: 'DICOM imaging data undergoing de-identification process to remove patient identifiable
    information from image metadata while preserving clinical utility and image quality.

    '
  preprocessing_details:
  - DICOM header de-identification to remove embedded patient information
  - Preservation of clinically relevant imaging metadata
  - DICOM schema compliance verification after de-identification
  - Quality assurance of de-identified images for clinical utility
  - Privacy scan tool (privacy_scan_tool) used for medical records privacy scanning
- id: chorus:preproc:5
  name: Re-identification limitation transformations
  description: 'Data transformed using approaches that limit re-identification while maintaining analytical
    utility. Multiple preprocessing strategies employed to protect patient privacy across all data modalities
    per HIPAA and institutional requirements.

    '
  preprocessing_details:
  - Application of de-identification algorithms across all data modalities
  - Privacy-preserving transformations for EHR and imaging data
  - Geocoding via DeGauss (UF-Geocoding tool) for OMOP Location entities
  - Risk assessment and compliance with ethical and legal requirements
  - Community ethics focus group input on appropriate data for public sharing
⚠ low R20 · semantic_understanding
issueDe-identification described across multiple preprocessing strategies but no unified deidentification method field
fieldspreprocessing_strategies, is_deidentified
fixAdd explicit deidentification method summary (HIPAA Safe Harbor, Expert Determination, or hybrid)
5/5 R20 Q10 (Metadata Quality & Content) Interoperability and Standardization
levelStandard formats + schema/ontology compliance
evidencedistribution_formats: OMOP Common Data Model (published OMOP schema metadata), WFDB (published PhysioNet schema extended), DICOM (published DICOM schema metadata), OHNLP tokenized (open source OHNLP schema), EDF+ and Persyst (open source schemas). preprocessing_strategies: semantic mapping validation via chorus-mapping repository, OHDSI tool stack integration, schema compliance validation across all modalities
qualityExceptional interoperability with comprehensive standardization across all five data modalities using internationally recognized standards and published schemas
correctnessFormat choices semantically appropriate: OMOP (observational health data standard), WFDB (PhysioNet waveform standard), DICOM (medical imaging standard), OHNLP (clinical NLP standard), EDF+ (EEG standard)
consistencySchema conformance validated through preprocessing_strategies and cleaning_strategies. chorus-mapping repository for semantic validation mentioned consistently
5/5 R20 Q11 (Technical Documentation) Tool and Software Transparency
levelComprehensive strategies with software versions/URLs
evidencepreprocessing_strategies: 5 detailed strategies (OMOP transformation, OHNLP tokenization, WFDB standardization, DICOM de-identification, re-identification limitation). cleaning_strategies: 2 strategies (multi-center harmonization with chorus-mapping repository, schema compliance validation). labeling_strategies: visualization and annotation environment. software_and_tools: OHDSI tool stack, OHNLP toolkit, privacy_scan_tool, DeGauss (UF-Geocoding), CHoRUSReports (R), chorus_waveform scripts, chorus-extract-upload tools. GitHub organization (chorus-ai) with 28 repositories including Chorus_SOP, chorus-mapping, chorus-container-apps
qualityExceptional software transparency with specific tools named, GitHub organization documented (28 repositories), open source licenses specified (MIT, Apache-2.0), and comprehensive preprocessing/cleaning/labeling strategies
correctnessTools semantically appropriate: OHDSI/OMOP for structured EHR, OHNLP for clinical notes, PhysioNet/WFDB for waveforms, DeGauss for geocoding
consistencySoftware tools align with preprocessing strategies and distribution formats. GitHub repositories (chorus-ai) support documented SOPs and semantic mappings
4/5 R20 Q8 (Metadata Quality & Content) Ethical and Privacy Declarations
levelComprehensive ethics with minor gaps
evidencehuman_subject_research.involves_human_subjects: true, ethics_review_board: Institutional review boards at 14 data acquisition centers, community-facing ethics focus groups, legal and ethical advisory teams. regulatory_compliance: HIPAA, 45 CFR 46 (Common Rule), institutional data privacy regulations, NIH ethical AI requirements. preprocessing_strategies includes de-identification: OMOP transformation, OHNLP tokenization, DICOM header de-identification, privacy scan tool. sensitive_elements: 4 categories documented (PHI-clinical EHR, waveforms/EEG, imaging, SDOH)
qualityComprehensive ethical documentation covering IRB review, regulatory compliance, de-identification methods, and privacy protections. Missing: specific IRB protocol numbers, explicit informed consent approach (retrospective waiver likely), participant compensation details, vulnerable population safeguards (though PICU/NICU implies pediatric protections)
correctnessHIPAA and 45 CFR 46 compliance appropriate for US-based multi-center health data. De-identification methods semantically appropriate for each data type
consistencyEthics review at 14 sites aligns with 14 data acquisition centers. Community ethics focus groups support public sharing claims. Minor gap: no IRB protocol numbers provided despite multi-site coordination
cleaning_strategies
cleaning_strategies:
- id: chorus:cleaning:1
  name: Multi-center data harmonization with validated semantic mappings
  description: 'Data from 14 acquisition centers harmonized through validated semantic mappings and standard
    operating protocols (SOPs). Ensures consistency and interoperability across diverse institutional
    EHR systems and clinical practices. Mappings maintained in chorus-mapping repository with clinical
    validation SOP.

    '
  cleaning_details:
  - Semantic mapping validation by clinical experts using chorus-mapping repository
  - Standard operating protocol (SOP) implementation per Chorus_SOP documentation
  - Cross-site data quality checks through CHoRUSReports characterization reports
  - Resolution of institutional variations in coding and clinical terminology
  - Clinical validation SOP for contributing to mapping efforts
  - Site status tracking via GitHub interface and Google Form submissions
- id: chorus:cleaning:2
  name: Schema compliance validation across all modalities
  description: 'All data modalities validated against published metadata schemas (OMOP, DICOM, WFDB, OHNLP,
    EDF+, Persyst) to ensure compliance and data quality across the multi-modal dataset.

    '
  cleaning_details:
  - Schema compliance verification against OMOP CDM specifications
  - DICOM schema metadata validation for imaging data
  - WFDB/PhysioNet schema validation for waveform data
  - OHNLP open source schema validation for tokenized clinical notes
  - EDF+ and Persyst schema validation for EEG data
  - Documentation of schema extensions for OMOP high-frequency nursing flowsheets
5/5 R20 Q11 (Technical Documentation) Tool and Software Transparency
levelComprehensive strategies with software versions/URLs
evidencepreprocessing_strategies: 5 detailed strategies (OMOP transformation, OHNLP tokenization, WFDB standardization, DICOM de-identification, re-identification limitation). cleaning_strategies: 2 strategies (multi-center harmonization with chorus-mapping repository, schema compliance validation). labeling_strategies: visualization and annotation environment. software_and_tools: OHDSI tool stack, OHNLP toolkit, privacy_scan_tool, DeGauss (UF-Geocoding), CHoRUSReports (R), chorus_waveform scripts, chorus-extract-upload tools. GitHub organization (chorus-ai) with 28 repositories including Chorus_SOP, chorus-mapping, chorus-container-apps
qualityExceptional software transparency with specific tools named, GitHub organization documented (28 repositories), open source licenses specified (MIT, Apache-2.0), and comprehensive preprocessing/cleaning/labeling strategies
correctnessTools semantically appropriate: OHDSI/OMOP for structured EHR, OHNLP for clinical notes, PhysioNet/WFDB for waveforms, DeGauss for geocoding
consistencySoftware tools align with preprocessing strategies and distribution formats. GitHub repositories (chorus-ai) support documented SOPs and semantic mappings
labeling_strategies
labeling_strategies:
- id: chorus:labeling:1
  name: Visualization and annotation environment for prediction targets
  description: 'Custom visualization and annotation environment developed to label data with targets important
    for prediction tasks in critical care AI applications. Supports labeling for characterizing acute
    illness, predicting complications, and measuring treatment response.

    '
  data_annotation_protocol:
  - Interactive visualization tools for clinical data exploration
  - Annotation interface for clinical expert labeling of prediction targets
  - Labeling of targets for characterization, prediction, and treatment response tasks
  - Quality control of annotations through review processes
  - Documentation of labeling protocols per Chorus_SOP standard operating procedures
5/5 R20 Q11 (Technical Documentation) Tool and Software Transparency
levelComprehensive strategies with software versions/URLs
evidencepreprocessing_strategies: 5 detailed strategies (OMOP transformation, OHNLP tokenization, WFDB standardization, DICOM de-identification, re-identification limitation). cleaning_strategies: 2 strategies (multi-center harmonization with chorus-mapping repository, schema compliance validation). labeling_strategies: visualization and annotation environment. software_and_tools: OHDSI tool stack, OHNLP toolkit, privacy_scan_tool, DeGauss (UF-Geocoding), CHoRUSReports (R), chorus_waveform scripts, chorus-extract-upload tools. GitHub organization (chorus-ai) with 28 repositories including Chorus_SOP, chorus-mapping, chorus-container-apps
qualityExceptional software transparency with specific tools named, GitHub organization documented (28 repositories), open source licenses specified (MIT, Apache-2.0), and comprehensive preprocessing/cleaning/labeling strategies
correctnessTools semantically appropriate: OHDSI/OMOP for structured EHR, OHNLP for clinical notes, PhysioNet/WFDB for waveforms, DeGauss for geocoding
consistencySoftware tools align with preprocessing strategies and distribution formats. GitHub repositories (chorus-ai) support documented SOPs and semantic mappings
intended_uses
intended_uses:
- id: chorus:use:1
  name: AI/ML model development for critical care
  description: 'Primary intended use is development and training of artificial intelligence and machine
    learning models to characterize acute and critical care illness, predict complications, and measure
    treatment response in critically ill patients across diverse hospital settings.

    '
  examples:
  - Characterizing acute and critical care illness patterns using multi-modal data
  - Predicting complications (e.g., sepsis, respiratory failure) in critically ill patients
  - Measuring treatment response in ICU patients using medication and waveform data
  - Developing clinical deep learning models for critical care AI applications
- id: chorus:use:2
  name: External validation of AI models
  description: 'Provision of holdout test set accessible for model external validation to aid marketplace
    adoption of AI-developed models for implementation in acute and critical care settings.

    '
  examples:
  - External validation of sepsis prediction models developed at other institutions
  - Benchmarking AI algorithms for critical care across diverse patient populations
- id: chorus:use:3
  name: Health equity and disparities research
  description: 'Studies examining health equity, social determinants of health, and disparities in critical
    care outcomes across diverse patient populations and hospital settings. Dataset includes contextual
    factors such as geographic distance to nearest hospital.

    '
  examples:
  - Analysis of disparities in critical care outcomes by race, ethnicity, and geography
  - Research on social determinants of health in ICU patient populations
- id: chorus:use:4
  name: Educational and training purposes for AI scientists
  description: 'Training and education of next generation of diverse academic and community AI scientists
    through hands-on experience with real-world critical care datasets. Integrated with AIM-AHEAD Bridge2AI
    for Clinical Care Training Program (Cohorts 1 and 2).

    '
  examples:
  - 'AIM-AHEAD training program for underrepresented trainees (Cohort 1: 2024-2025, Cohort 2: 2025-2026)'
  - Foundational hands-on training using Jupyter Notebooks with Bridge2AI CHoRUS ecosystem
  - Workshops on OHDSI/OMOP common data model and clinical AI
  - Development of practical use cases for AI/ML in clinical care
5/5 R20 Q18 (FAIRness & Accessibility) Reusability (License Clarity)
levelLicense explicitly defines reuse terms
evidencelicense_and_use_terms: CHoRUS Controlled Access License with Data Use Agreement explicitly defines permitted research uses, prohibits re-identification, requires institutional email and signed agreement. intended_uses: 4 use cases explicitly permitted (AI/ML model development for critical care, external validation, health equity research, educational/training purposes). discouraged_uses: 3 uses explicitly prohibited (clinical decision-making without validation/approval, re-identification attempts, use without awareness of data limitations)
qualityExceptional license clarity with specific permitted uses, explicit prohibitions, and clear reuse requirements (DUA signature, research-only)
correctnessControlled access license appropriate for de-identified clinical data. Research-only restriction standard for NIH-funded datasets pending validation
consistencyLicense terms align with intended_uses (research, education) and discouraged_uses (clinical deployment without approval). Re-identification prohibition consistent with de-identification preprocessing
discouraged_uses
discouraged_uses:
- id: chorus:discouraged:1
  name: Clinical decision-making without proper validation and regulatory approval
  description: 'Dataset is for research purposes only. AI/ML models developed should undergo appropriate
    clinical validation, regulatory approval, and institutional review before use in patient care or clinical
    decision-making.

    '
  discouragement_details:
  - Models trained on CHoRUS data must undergo independent clinical validation
  - Regulatory approval processes (e.g., FDA clearance) required for clinical use
  - Institutional review board approval needed for clinical deployment
- id: chorus:discouraged:2
  name: Re-identification attempts
  description: 'Attempts to re-identify patients from de-identified data violate ethical principles, data
    use agreements, and legal frameworks established for privacy protection under HIPAA and institutional
    requirements.

    '
  discouragement_details:
  - Re-identification attempts violate the signed data use agreement
  - Prohibited under HIPAA and applicable institutional data privacy regulations
  - Data use agreement explicitly prohibits re-identification efforts
- id: chorus:discouraged:3
  name: Use without awareness of ongoing data collection limitations
  description: 'As data collection continues through November 2026 and quality assurance processes are
    ongoing, early dataset versions should be used with awareness of completeness limitations and ongoing
    expansion (from 45K to target 100K+ admissions).

    '
  discouragement_details:
  - Dataset is actively growing; cohort coverage varies by data modality
  - EEG extraction and full imaging de-identification still in process as of 2025
  - Clinical notes stored locally at sites; only tokens available in enclave
5/5 R20 Q18 (FAIRness & Accessibility) Reusability (License Clarity)
levelLicense explicitly defines reuse terms
evidencelicense_and_use_terms: CHoRUS Controlled Access License with Data Use Agreement explicitly defines permitted research uses, prohibits re-identification, requires institutional email and signed agreement. intended_uses: 4 use cases explicitly permitted (AI/ML model development for critical care, external validation, health equity research, educational/training purposes). discouraged_uses: 3 uses explicitly prohibited (clinical decision-making without validation/approval, re-identification attempts, use without awareness of data limitations)
qualityExceptional license clarity with specific permitted uses, explicit prohibitions, and clear reuse requirements (DUA signature, research-only)
correctnessControlled access license appropriate for de-identified clinical data. Research-only restriction standard for NIH-funded datasets pending validation
consistencyLicense terms align with intended_uses (research, education) and discouraged_uses (clinical deployment without approval). Re-identification prohibition consistent with de-identification preprocessing
5/5 R20 Q9 (Metadata Quality & Content) Access Requirements and Governance Documentation
levelLicense + restrictions + confidentiality classification
evidencelicense_and_use_terms: CHoRUS Controlled Access License with Data Use Agreement, institutional (.edu) email required, signed licensing agreement mandatory, controlled access through secure enclave (Azure-based), prohibits re-identification, access contacts provided (dbold@emory.edu, jared.houghtaling@tuftsmedicine.org). regulatory_compliance: HIPAA, 45 CFR 46, institutional regulations. discouraged_uses explicitly prohibit re-identification and unauthorized clinical use
qualityExceptional access governance documentation with clear controlled access model, licensing requirements, secure infrastructure, and explicit prohibitions on misuse
correctnessControlled access model appropriate for HIPAA-regulated clinical data. Azure secure enclave consistent with NIH data sharing requirements
consistencyAccess requirements align with sensitive_elements (PHI), regulatory_compliance (HIPAA), and de-identification approach (controlled access despite de-id)
license_and_use_terms
license_and_use_terms:
  id: chorus:license:1
  name: CHoRUS Controlled Access License with Data Use Agreement
  description: 'Dataset distributed under controlled access requiring institutional email registration
    and signed licensing agreement. Access granted after review and approval process. Participants must
    complete registration form with name, institutional email (not personal), and institution. Once approved,
    users receive email with access instructions to CHoRUS secure enclave. For training program access,
    program administrators assist with licensing.

    '
  license_terms:
  - Institutional (.edu) email required for registration
  - All participants must sign a licensing agreement before gaining access to the dataset
  - Controlled access through secure enclave (Azure-based infrastructure)
  - Data use agreement specifies permitted research uses and prohibits re-identification
  - Access request contacts - dbold@emory.edu or jared.houghtaling@tuftsmedicine.org
  - Funded under NIH award OT2OD032701; content is solely responsibility of authors
5/5 R20 Q17 (FAIRness & Accessibility) Accessibility (Access Mechanism)
levelFully defined access path (platform, login, policy)
evidencedistribution_formats: all 5 formats specify access via secure enclave (Azure-based infrastructure) with controlled access. license_and_use_terms: complete registration process (institutional email, licensing agreement signature), approval workflow, access instructions via email after approval, specific contact emails (dbold@emory.edu, jared.houghtaling@tuftsmedicine.org), training program administrator assistance available
qualityExceptional access mechanism documentation with step-by-step process, platform specification (Azure secure enclave), contact information, and alternative access paths (training programs)
correctnessControlled access via secure enclave appropriate for HIPAA-regulated clinical data. Institutional email requirement standard for DUA-based access
consistencyAccess mechanism aligns with license requirements (DUA), regulatory compliance (HIPAA), and sensitive_elements (PHI). Azure infrastructure consistent with NIH data sharing requirements
5/5 R20 Q18 (FAIRness & Accessibility) Reusability (License Clarity)
levelLicense explicitly defines reuse terms
evidencelicense_and_use_terms: CHoRUS Controlled Access License with Data Use Agreement explicitly defines permitted research uses, prohibits re-identification, requires institutional email and signed agreement. intended_uses: 4 use cases explicitly permitted (AI/ML model development for critical care, external validation, health equity research, educational/training purposes). discouraged_uses: 3 uses explicitly prohibited (clinical decision-making without validation/approval, re-identification attempts, use without awareness of data limitations)
qualityExceptional license clarity with specific permitted uses, explicit prohibitions, and clear reuse requirements (DUA signature, research-only)
correctnessControlled access license appropriate for de-identified clinical data. Research-only restriction standard for NIH-funded datasets pending validation
consistencyLicense terms align with intended_uses (research, education) and discouraged_uses (clinical deployment without approval). Re-identification prohibition consistent with de-identification preprocessing
5/5 R20 Q9 (Metadata Quality & Content) Access Requirements and Governance Documentation
levelLicense + restrictions + confidentiality classification
evidencelicense_and_use_terms: CHoRUS Controlled Access License with Data Use Agreement, institutional (.edu) email required, signed licensing agreement mandatory, controlled access through secure enclave (Azure-based), prohibits re-identification, access contacts provided (dbold@emory.edu, jared.houghtaling@tuftsmedicine.org). regulatory_compliance: HIPAA, 45 CFR 46, institutional regulations. discouraged_uses explicitly prohibit re-identification and unauthorized clinical use
qualityExceptional access governance documentation with clear controlled access model, licensing requirements, secure infrastructure, and explicit prohibitions on misuse
correctnessControlled access model appropriate for HIPAA-regulated clinical data. Azure secure enclave consistent with NIH data sharing requirements
consistencyAccess requirements align with sensitive_elements (PHI), regulatory_compliance (HIPAA), and de-identification approach (controlled access despite de-id)
distribution_formats
distribution_formats:
- id: chorus:format:1
  name: OMOP Common Data Model (structured EHR data)
  description: 'Structured electronic health record data distributed in OMOP Common Data Model format
    enabling use of OHDSI tool stack for analysis. Contains 1.6 billion rows. Available in secure enclave
    with controlled access. Published OMOP schema metadata available.

    '
  access_urls:
  - https://chorus4ai.org/
- id: chorus:format:2
  name: WFDB waveform format (bedside monitor telemetry)
  description: 'Waveform telemetry data (23 TB) distributed in WFDB (WaveForm DataBase) format following
    extended PhysioNet schema. Available in secure enclave with controlled access. Published PhysioNet
    schema (extended) metadata available.

    '
  access_urls:
  - https://chorus4ai.org/
- id: chorus:format:3
  name: DICOM format (medical imaging)
  description: 'Medical imaging data (7,642 admissions with radiology; 1,000 images currently available)
    distributed in DICOM format with comprehensive metadata following DICOM schema. De-identification
    in process for larger cohort. Planned controlled access.

    '
  access_urls:
  - https://chorus4ai.org/
- id: chorus:format:4
  name: OHNLP tokenized format (clinical notes)
  description: 'Clinical notes distributed as OHNLP-tokenized text following open source OHNLP schema.
    Stored locally at contributing sites (except tokens). Controlled access planned.

    '
  access_urls:
  - https://chorus4ai.org/
- id: chorus:format:5
  name: EDF+ and Persyst formats (EEG waveforms)
  description: 'EEG waveform data distributed in EDF+ (European Data Format) and Persyst formats following
    open source schemas. Extraction in process, planned for controlled access.

    '
  access_urls:
  - https://chorus4ai.org/
5/5 R20 Q10 (Metadata Quality & Content) Interoperability and Standardization
levelStandard formats + schema/ontology compliance
evidencedistribution_formats: OMOP Common Data Model (published OMOP schema metadata), WFDB (published PhysioNet schema extended), DICOM (published DICOM schema metadata), OHNLP tokenized (open source OHNLP schema), EDF+ and Persyst (open source schemas). preprocessing_strategies: semantic mapping validation via chorus-mapping repository, OHDSI tool stack integration, schema compliance validation across all modalities
qualityExceptional interoperability with comprehensive standardization across all five data modalities using internationally recognized standards and published schemas
correctnessFormat choices semantically appropriate: OMOP (observational health data standard), WFDB (PhysioNet waveform standard), DICOM (medical imaging standard), OHNLP (clinical NLP standard), EDF+ (EEG standard)
consistencySchema conformance validated through preprocessing_strategies and cleaning_strategies. chorus-mapping repository for semantic validation mentioned consistently
5/5 R20 Q17 (FAIRness & Accessibility) Accessibility (Access Mechanism)
levelFully defined access path (platform, login, policy)
evidencedistribution_formats: all 5 formats specify access via secure enclave (Azure-based infrastructure) with controlled access. license_and_use_terms: complete registration process (institutional email, licensing agreement signature), approval workflow, access instructions via email after approval, specific contact emails (dbold@emory.edu, jared.houghtaling@tuftsmedicine.org), training program administrator assistance available
qualityExceptional access mechanism documentation with step-by-step process, platform specification (Azure secure enclave), contact information, and alternative access paths (training programs)
correctnessControlled access via secure enclave appropriate for HIPAA-regulated clinical data. Institutional email requirement standard for DUA-based access
consistencyAccess mechanism aligns with license requirements (DUA), regulatory compliance (HIPAA), and sensitive_elements (PHI). Azure infrastructure consistent with NIH data sharing requirements
5/5 R20 Q4 (Structural Completeness) File Enumeration and Type Variety
level>3 file types
evidencedistribution_formats: 5 distinct formats - OMOP Common Data Model (structured EHR), WFDB (waveforms), DICOM (imaging), OHNLP tokenized (clinical notes), EDF+/Persyst (EEG)
qualityExceptional multi-modal data variety with comprehensive format standardization across all modalities
correctnessFormat choices semantically appropriate for each data type (OMOP for EHR, WFDB for waveforms, DICOM for imaging)
consistencyDistribution formats match preprocessing strategies and acquisition methods described
maintainers
maintainers:
- id: chorus:maintainer:1
  name: CHoRUS Consortium
  description: 'Multi-institutional consortium managing dataset maintenance including 14 data acquisition
    centers, coordinating teams at Massachusetts General Hospital (lead), University of Florida, UT Health
    Science Center, and Tufts Medicine, plus infrastructure development team. Standards, Data Acquisition,
    and Tooling sub-teams manage ongoing data delivery and quality. Contact: cmccrary@mgh.harvard.edu
    (Ciera McCrary, Program Manager, MGH).

    '
  maintainer_details:
  - Massachusetts General Hospital (lead institution, Contact PI Eric S. Rosenthal)
  - University of Florida (data acquisition and coordination, Azra Bihorac, Parisa Rashidi, Yulia Strekalova)
  - UT Health Science Center / UTHealth Houston (Xiaoqian Jiang)
  - Tufts Medicine (Andrew Ewing Williams, Manlik Kwong, Jared Houghtaling)
  - 14 data acquisition centers across United States contributing clinical data extracts
  - Standards team (semantic mappings and validation via chorus-mapping repository)
  - Data Acquisition team (extraction and contribution per Chorus_SOP)
  - Tooling team (software development across chorus-ai GitHub organization)
  - Project management via GitHub organization (chorus-ai) with 28 active repositories
- id: chorus:maintainer:2
  name: CHoRUS GitHub Organization (chorus-ai)
  description: 'Active GitHub organization housing repositories for software, semantic mappings, standard
    operating protocols, and project management. 28 repositories with comprehensive documentation and
    community support. Licensed under MIT License (GitHub organization).

    '
  maintainer_details:
  - GitHub organization at https://github.com/chorus-ai
  - Chorus_SOP repository - centralized SOP documentation site (Apache-2.0 license)
  - chorus-container-apps - Azure deployment infrastructure (JavaScript)
  - chorus-mapping - semantic mappings repository
  - chorus_waveform - waveform documentation and conversion scripts (MIT license)
  - privacy_scan_tool - privacy scan tool for medical records (Python)
  - CHoRUSReports - characterization reports returned to contributing sites (R)
  - chorus-extract-upload - tools to create and upload CHoRUS data extract (MIT license)
  - UF-Geocoding - open source code to geocode OMOP Location entities via DeGauss
  - Community discussions and issue tracking for Standards and Data Acquisition teams
no field-level feedback matched
updates
updates:
  id: chorus:updates:1
  name: Ongoing data collection and continuous expansion
  description: 'Dataset updated continuously as data collection progresses at 14 acquisition centers.
    As of August 2025, covers 14 hospitals with over 45,000 unique admissions; current released dataset
    includes 50,000 patient admissions (ICU, PICU, NICU) and 1.6 billion rows of EHR OMOP data. Target
    exceeds 100,000 critically ill patients. Project timeline extends through November 30, 2026 (approved
    no-cost extension). Regular status updates tracked through GitHub project management system via GitHub
    interface or Google Form submissions. Sites statuses tracked in Standards Project and Data Acquisition
    Project.

    '
  frequency: Continuous updates through November 30, 2026
  update_details:
  - Ongoing retrospective data collection at 14 sites through project end date
  - Current status (August 2025) - 45K+ unique admissions; 50K released (ICU, PICU, NICU)
  - Target - 100,000+ critically ill patients across 9 data modalities
  - Regular site status updates via GitHub interface or Google Form submissions
  - GitHub project tracking for deliverables and task dependencies
  - Documentation updates maintained in Chorus_SOP repository
  - Software and tooling continuous development across chorus-ai GitHub organization
  - Semantic mapping validation and expansion via chorus-mapping repository
  - EEG extraction and full imaging de-identification in progress
✓ 1/1 R10 6.Data Provenance and Version Tracking Update Schedule or Frequency Indicated
evidenceContinuous updates through November 30, 2026 with regular status tracking
qualityClear update schedule through specific end date
4/5 R20 Q13 (Technical Documentation) Version History Documentation
levelComprehensive versioning with minor gaps
evidenceupdates: Ongoing data collection and continuous expansion, frequency 'Continuous updates through November 30, 2026', current status August 2025 (45K+ unique admissions, 50K released), target 100K+ patients. update_details: site status tracking via GitHub interface or Google Form submissions, Standards Project and Data Acquisition Project tracking, documentation updates in Chorus_SOP repository, ongoing EEG extraction and imaging de-identification. retention_limit: long-term retention per NIH data sharing policies. No explicit version numbering or formal release notes
qualityGood version tracking through continuous update model with specific snapshots (August 2025) and progress metrics. Missing: formal version numbers, structured release notes, specific errata documentation
correctnessContinuous update model appropriate for ongoing multi-center data collection. NIH data sharing policy retention aligns with federal requirements
consistencyUpdate frequency aligns with project timeline (through 11/2026). Site tracking via GitHub consistent with maintainer GitHub organization (chorus-ai)
4/5 R20 Q19 (FAIRness & Accessibility) Data Integrity and Provenance
levelStructured version control with timestamps
evidenceupdates: continuous update model with frequency 'Continuous updates through November 30, 2026', specific snapshot timestamp (August 2025) with metrics (45K+ unique admissions, 50K released, 1.6B rows, 23TB waveforms). update_details: site status tracking via GitHub interface/Google Form, Standards Project and Data Acquisition Project tracking, documentation updates in Chorus_SOP repository. retention_limit: NIH data sharing policies govern retention. Missing: formal version numbers, structured changelog
qualityGood provenance tracking with timestamped snapshots, GitHub-based tracking systems, and clear update frequency. Missing: formal version changelog or release notes documenting specific changes between snapshots
correctnessContinuous update model appropriate for ongoing multi-center data collection. August 2025 snapshot timestamp plausible within 9/2022-11/2026 timeline
consistencyGitHub tracking aligns with maintainer GitHub organization (chorus-ai). NIH data sharing policy retention consistent with federal funding requirements
retention_limit
retention_limit:
  id: chorus:retention:1
  name: Long-term dataset retention per NIH data sharing policies
  description: 'Digital data maintained according to NIH data sharing policies and institutional requirements
    at participating centers. Controlled access model ensures long-term availability for research while
    protecting patient privacy. Funded under NIH award OT2OD032701 with project end date of November 30,
    2026.

    '
  retention_details:
  - NIH data sharing policies govern long-term retention requirements
  - Institutional requirements at 14 participating data acquisition centers apply
  - Controlled access model via secure enclave for ongoing privacy protection
  - Long-term maintenance through CHoRUS Consortium and associated institutions
4/5 R20 Q13 (Technical Documentation) Version History Documentation
levelComprehensive versioning with minor gaps
evidenceupdates: Ongoing data collection and continuous expansion, frequency 'Continuous updates through November 30, 2026', current status August 2025 (45K+ unique admissions, 50K released), target 100K+ patients. update_details: site status tracking via GitHub interface or Google Form submissions, Standards Project and Data Acquisition Project tracking, documentation updates in Chorus_SOP repository, ongoing EEG extraction and imaging de-identification. retention_limit: long-term retention per NIH data sharing policies. No explicit version numbering or formal release notes
qualityGood version tracking through continuous update model with specific snapshots (August 2025) and progress metrics. Missing: formal version numbers, structured release notes, specific errata documentation
correctnessContinuous update model appropriate for ongoing multi-center data collection. NIH data sharing policy retention aligns with federal requirements
consistencyUpdate frequency aligns with project timeline (through 11/2026). Site tracking via GitHub consistent with maintainer GitHub organization (chorus-ai)
4/5 R20 Q19 (FAIRness & Accessibility) Data Integrity and Provenance
levelStructured version control with timestamps
evidenceupdates: continuous update model with frequency 'Continuous updates through November 30, 2026', specific snapshot timestamp (August 2025) with metrics (45K+ unique admissions, 50K released, 1.6B rows, 23TB waveforms). update_details: site status tracking via GitHub interface/Google Form, Standards Project and Data Acquisition Project tracking, documentation updates in Chorus_SOP repository. retention_limit: NIH data sharing policies govern retention. Missing: formal version numbers, structured changelog
qualityGood provenance tracking with timestamped snapshots, GitHub-based tracking systems, and clear update frequency. Missing: formal version changelog or release notes documenting specific changes between snapshots
correctnessContinuous update model appropriate for ongoing multi-center data collection. August 2025 snapshot timestamp plausible within 9/2022-11/2026 timeline
consistencyGitHub tracking aligns with maintainer GitHub organization (chorus-ai). NIH data sharing policy retention consistent with federal funding requirements
human_subject_research
human_subject_research:
  id: chorus:hsr:1
  name: CHoRUS Human Subjects Research
  description: 'Retrospective data collection from critically ill patients (ICU, PICU, NICU) approved
    through institutional review processes at 14 data acquisition centers. Community-facing ethics focus
    groups conducted to determine what data is appropriate for public sharing. Legal framework established
    for collecting data at scale. Patient-focused efforts determine ethical and legal approaches to manage
    privacy and bias while accounting for Social Determinants of Health. Project includes expertise from
    law, ethics, health services, biomedical science, engineering, and scientific journal publications
    disciplines. Ethics of AI component addressed through AIM-AHEAD training curriculum (safety, risk,
    and legal considerations; IRB, HIPAA/GDPR compliance for OMOP/FHIR data).

    '
  involves_human_subjects: true
  ethics_review_board:
  - Institutional review boards at 14 data acquisition centers across United States
  - Community-facing ethics focus groups determining appropriate data for public sharing
  - Legal and ethical advisory teams including law and ethics discipline experts
  - Privacy and accountability review processes per project three-pillar structure (Data, Ethics, People)
  regulatory_compliance:
  - HIPAA (Health Insurance Portability and Accountability Act) compliance for protected health information
  - 45 CFR 46 (Common Rule) for human subjects research protections
  - Institutional data privacy regulations at each of the 14 contributing sites
  - NIH Common Fund Bridge2AI program ethical and trustworthy AI requirements
  - IRB protocol drafting and HIPAA/GDPR compliance guidance provided in training curriculum
⚠ medium R10 · consistency
issuehuman_subject_research=True but informed consent not documented
fieldshuman_subject_research, informed_consent
fixAdd consent procedure documentation
⚠ medium R20 · consistency
issuehuman_subject_research.involves_human_subjects=True and ethics_review_board lists 14 IRBs, but no specific IRB approval numbers or protocols documented
fieldshuman_subject_research, ethical_reviews
fixAdd IRB protocol numbers from at least lead institution (MGH) for transparency
4/5 R20 Q8 (Metadata Quality & Content) Ethical and Privacy Declarations
levelComprehensive ethics with minor gaps
evidencehuman_subject_research.involves_human_subjects: true, ethics_review_board: Institutional review boards at 14 data acquisition centers, community-facing ethics focus groups, legal and ethical advisory teams. regulatory_compliance: HIPAA, 45 CFR 46 (Common Rule), institutional data privacy regulations, NIH ethical AI requirements. preprocessing_strategies includes de-identification: OMOP transformation, OHNLP tokenization, DICOM header de-identification, privacy scan tool. sensitive_elements: 4 categories documented (PHI-clinical EHR, waveforms/EEG, imaging, SDOH)
qualityComprehensive ethical documentation covering IRB review, regulatory compliance, de-identification methods, and privacy protections. Missing: specific IRB protocol numbers, explicit informed consent approach (retrospective waiver likely), participant compensation details, vulnerable population safeguards (though PICU/NICU implies pediatric protections)
correctnessHIPAA and 45 CFR 46 compliance appropriate for US-based multi-center health data. De-identification methods semantically appropriate for each data type
consistencyEthics review at 14 sites aligns with 14 data acquisition centers. Community ethics focus groups support public sharing claims. Minor gap: no IRB protocol numbers provided despite multi-site coordination
external_resources
external_resources:
- id: chorus:resource:1
  name: CHoRUS Project Website
  description: Official project website with dataset overview, team information, project components, and
    access instructions
  external_resources:
  - https://chorus4ai.org/
- id: chorus:resource:2
  name: CHoRUS GitHub Organization
  description: Comprehensive GitHub organization with 28 repositories including software, documentation,
    SOPs, and tooling
  external_resources:
  - https://github.com/chorus-ai
- id: chorus:resource:3
  name: Chorus_SOP Documentation Site
  description: Centralized standard operating protocol documentation with interactive workflow diagrams
    for data extraction and contribution
  external_resources:
  - https://github.com/chorus-ai/Chorus_SOP
- id: chorus:resource:4
  name: NIH RePORTER Project Details
  description: Federal grant information and project details from NIH Research Portfolio Online Reporting
    Tools for grant 1OT2OD032701-01
  external_resources:
  - https://reporter.nih.gov/project-details/10472824
- id: chorus:resource:5
  name: Bridge2AI Program
  description: Parent NIH Common Fund program supporting AI-ready biomedical datasets across four data
    generation projects
  external_resources:
  - https://bridge2ai.org/
  - https://bridge2ai.org/chorus
- id: chorus:resource:6
  name: AIM-AHEAD Bridge2AI Training Program
  description: Partnership with AIM-AHEAD for Bridge2AI Clinical Care Training Program (Cohort 1 and Cohort
    2) providing AI/ML training for underrepresented trainees
  external_resources:
  - https://aim-ahead.net/
- id: chorus:resource:7
  name: OHDSI Community and OMOP CDM
  description: Observational Health Data Sciences and Informatics community supporting OMOP Common Data
    Model used for CHoRUS structured EHR data
  external_resources:
  - https://www.ohdsi.org/
- id: chorus:resource:8
  name: Published Research (Neurocritical Care)
  description: Peer-reviewed publication documenting CHoRUS dataset methodology and design in Neurocritical
    Care journal
  external_resources:
  - https://doi.org/10.1007/s12028-024-02007
5/5 R20 Q14 (Technical Documentation) Associated Publications
levelMultiple references and dataset citation
evidenceexternal_resources: 8 resources including peer-reviewed publication (Neurocritical Care journal, DOI: 10.1007/s12028-024-02007), NIH RePORTER project details (https://reporter.nih.gov/project-details/10472824), GitHub organization (https://github.com/chorus-ai), project website (https://chorus4ai.org/), Bridge2AI program (https://bridge2ai.org/chorus), AIM-AHEAD training partnership, OHDSI community, Chorus_SOP documentation
qualityExceptional external resource documentation with formal peer-reviewed publication DOI, federal grant tracking, comprehensive GitHub repositories, and community partnerships
correctnessDOI 10.1007/s12028-024-02007 format valid for Springer journal (Neurocritical Care). NIH RePORTER URL structure correct for grant 10472824
consistencyPublication in Neurocritical Care aligns with critical care focus. GitHub organization (chorus-ai) consistent with software/tools documentation. AIM-AHEAD partnership matches training uses described
1/1 R20 Q16 (FAIRness & Accessibility) Findability (Persistent Links)
levelPass
evidencepage: https://chorus4ai.org/, id: https://chorus4ai.org/, external_resources: 8 persistent URLs including NIH RePORTER (https://reporter.nih.gov/project-details/10472824), GitHub (https://github.com/chorus-ai), Bridge2AI (https://bridge2ai.org/chorus), publication DOI (https://doi.org/10.1007/s12028-024-02007)
qualityExcellent findability with multiple persistent URLs across project website, federal grant database, GitHub organization, and peer-reviewed publication
correctnessAll URL formats valid. NIH RePORTER, GitHub, and DOI URLs follow standard patterns
consistencyAll URLs point to consistent CHoRUS project resources across different platforms
1/1 R20 Q20 (FAIRness & Accessibility) Interlinking Across Platforms
levelPass
evidenceexternal_resources: 8 cross-platform links (chorus4ai.org project website, GitHub chorus-ai organization with 28 repositories, NIH RePORTER federal grant database, Bridge2AI program portal, AIM-AHEAD training partnership, OHDSI community, peer-reviewed publication DOI 10.1007/s12028-024-02007)
qualityExceptional interlinking across project website, code repository (GitHub), federal grant tracking (NIH RePORTER), parent program (Bridge2AI), training partnerships (AIM-AHEAD), standards communities (OHDSI), and scholarly literature (publication DOI)
correctnessAll platform links follow standard URL patterns. NIH RePORTER, GitHub, DOI, and domain URLs structurally valid
consistencyCross-platform links consistently reference CHoRUS project across all resources. GitHub organization (chorus-ai) matches software tools documented
1/1 R20 Q6 (Metadata Quality & Content) Dataset Identification Metadata
levelPass
evidenceid: https://chorus4ai.org/, page: https://chorus4ai.org/, external_resources: includes DOI 10.1007/s12028-024-02007 (peer-reviewed publication), NIH RePORTER project link https://reporter.nih.gov/project-details/10472824
qualityMultiple persistent identifiers provided including project URL, publication DOI, and federal grant tracking
correctnessDOI format valid (10.1007/s12028-024-02007), NIH RePORTER URL structure correct
consistencyAll URLs point to consistent CHoRUS project resources

Unmatched feedback

Feedback that referenced multiple fields or no specific field.
✓ 1/1 R10 2.Dataset Access and Retrieval Regulatory Restrictions and Confidentiality Level Specified
evidenceRegulatory compliance: HIPAA, 45 CFR 46, institutional regulations, NIH ethical AI requirements
qualityComprehensive regulatory compliance documentation
✓ 1/1 R10 2.Dataset Access and Retrieval Distribution Formats and File Types Specified
evidence5 formats: OMOP CDM, WFDB, DICOM, OHNLP tokenized, EDF+/Persyst
qualityComprehensive format specification with standard formats and schemas
✓ 1/1 R10 2.Dataset Access and Retrieval Related Datasets and External Resources Linked
evidence8 external resources including GitHub org (28 repos), OHDSI, Bridge2AI, publication DOI
qualityExcellent external resource linkage
✓ 1/1 R10 3.Data Reuse and Interoperability License Terms Allow Reuse
evidenceControlled access with DUA specifying research use permissions
qualityClear license allowing reuse for approved research
✓ 1/1 R10 3.Data Reuse and Interoperability Data Formats Are Standardized
evidenceAll 5 modalities use international standards: OMOP, WFDB, DICOM, OHNLP, EDF+/Persyst
qualityExemplary use of standardized formats
✓ 1/1 R10 3.Data Reuse and Interoperability Schema or Ontology Conformance Stated
evidenceOMOP CDM, DICOM, WFDB/PhysioNet, OHNLP, EDF+/Persyst schemas with validation
qualityComprehensive schema conformance with validation
✓ 1/1 R10 3.Data Reuse and Interoperability Variable Metadata with Identifiers Defined
evidenceOMOP vocabulary and concept IDs; published schemas for all 5 modalities
qualityVariable-level metadata through OMOP CDM and published schemas
✓ 1/1 R10 3.Data Reuse and Interoperability Use Guidance Provided
evidence4 intended uses, 3 discouraged uses with detailed rationale
qualityComprehensive use guidance covering technical, ethical, temporal considerations
✓ 1/1 R10 4.Ethical Use and Privacy Safeguards IRB or Ethics Review Documented
evidenceIRBs at 14 centers, community ethics focus groups, legal/ethical advisory teams
qualityComprehensive multi-layered ethics oversight
✓ 1/1 R10 4.Ethical Use and Privacy Safeguards Privacy Protections Beyond Deidentification
evidenceSecure enclave, DUA prohibiting re-identification, community ethics focus groups, federated access
qualityMulti-layered privacy protections: technical, legal, governance, architectural
✗ 0/1 R10 4.Ethical Use and Privacy Safeguards Informed Consent Obtained from Participants
evidenceNo explicit mention of consent procedures
qualityRetrospective collection but consent procedures not documented
✓ 1/1 R10 4.Ethical Use and Privacy Safeguards Vulnerable Populations and Compensation Documented
evidencePediatric/neonatal ICU patients with community ethics focus groups and SDOH framework
qualityVulnerable populations acknowledged with appropriate protections
✓ 1/1 R10 5.Data Composition and Structure Cohort or Subpopulations Characteristics Described
evidenceICU/PICU/NICU patients, 14 hospitals, 45K+ current/100K+ target with SDOH
qualityComprehensive population description with diversity measures
✓ 1/1 R10 5.Data Composition and Structure Number of Instances or Samples Reported
evidence45K+ admissions, 50K released, 100K+ target, 1.6B OMOP rows, 23TB waveforms, 7,642 radiology
qualityHighly specific instance counts across multiple dimensions
✓ 1/1 R10 5.Data Composition and Structure Variable-Level Metadata and Tabular Flag
evidenceOMOP CDM tabular structure with 1.6B rows and OHDSI tool stack
qualityTabular structure confirmed with comprehensive variable metadata
✓ 1/1 R10 5.Data Composition and Structure Data Topics or Conditions Represented
evidenceAcute/critical illness, complication prediction, treatment response in ICU/PICU/NICU
qualityClear clinical topics and research focus
✓ 1/1 R10 5.Data Composition and Structure Data Quality Issues and Anomalies Documented
evidenceTransparent documentation of ongoing expansion, incomplete modalities, federated sampling
qualityCurrent completeness status and ongoing processes clearly stated
✗ 0/1 R10 6.Data Provenance and Version Tracking Dataset Version Number Provided
evidenceNo version field
qualityVersion numbering scheme not documented
✗ 0/1 R10 6.Data Provenance and Version Tracking Version Access Methods Documented
evidenceNo version access documentation
qualityVersion access mechanisms not described
✓ 1/1 R10 6.Data Provenance and Version Tracking Change Descriptions and Errata Provided
evidenceCurrent status (Aug 2025) vs target documented with GitHub tracking
qualityCurrent status clear but historical change log absent
✓ 1/1 R10 6.Data Provenance and Version Tracking Provenance and Source Derivation Documented
evidence4 source types from 14 centers with transformation pipelines documented
qualityComprehensive provenance documentation
✓ 1/1 R10 7.Scientific Motivation and Funding Transparency Funding Sources and Mechanisms Listed
evidenceNIH Common Fund Bridge2AI with detailed grant info including $5,880,300 total funding
qualityComprehensive funding documentation
✓ 1/1 R10 7.Scientific Motivation and Funding Transparency Grant IDs or Award Numbers Present
evidenceOT2OD032701 (1OT2OD032701-01) with NIH RePORTER link
qualityGrant number documented in multiple locations with valid NIH format
✓ 1/1 R10 8.Technical Transparency Collection Mechanisms and Settings Described
evidence4 collection mechanisms with specific systems and volumes
qualityComprehensive collection documentation
✓ 1/1 R10 8.Technical Transparency Data Acquisition Methods Listed
evidence5 acquisition methods for all modalities with volumes and schemas
qualityDetailed acquisition methods with complete modality coverage
✓ 1/1 R10 8.Technical Transparency Preprocessing, Cleaning, and Labeling Strategies
evidence5 preprocessing + 2 cleaning + 1 labeling strategies
qualityComprehensive processing documentation with specific tools and SOPs
✓ 1/1 R10 8.Technical Transparency Software and Tools Documented
evidence7 tools including OHDSI, OHNLP, chorus_waveform, privacy_scan_tool, DeGauss
qualityExtensive tool documentation with GitHub organization (28 repos)
✓ 1/1 R10 8.Technical Transparency External Standards and Resources Referenced
evidence8 resources including Chorus_SOP, GitHub, OHDSI, publication DOI 10.1007/s12028-024-02007
qualityComprehensive external resources with publication DOI
✓ 1/1 R10 9.Dataset Evaluation and Limitations Disclosure Known Limitations Documented
evidenceOngoing expansion limitations, incomplete modalities, partial availability documented
qualityTransparent limitation documentation with current/target metrics
✗ 0/1 R10 9.Dataset Evaluation and Limitations Disclosure Systematic Biases Identified and Described
evidenceNo explicit bias analysis beyond equity focus
qualityEquity efforts documented but systematic bias analysis absent
✓ 1/1 R10 9.Dataset Evaluation and Limitations Disclosure Data Anomalies and Quality Issues Noted
evidenceOngoing QA, harmonization challenges, schema validation needs documented
qualityQuality issues acknowledged with harmonization complexity
✓ 1/1 R10 9.Dataset Evaluation and Limitations Disclosure Sensitive Content and Warnings Provided
evidence4 sensitive element categories with detailed sensitivity_details
qualityComprehensive sensitive content documentation
✓ 1/1 R10 9.Dataset Evaluation and Limitations Disclosure Ethical Review Details Including Conflicts
evidenceIRB at 14 centers, community ethics, legal/ethical advisory, regulatory compliance
qualityComprehensive ethical review documentation
✓ 1/1 R10 10.Cross-Platform and Community Integration Dataset Published on a Recognized Platform
evidenceSecure enclave (Azure) + NIH Bridge2AI + GitHub organization (28 repos)
qualityAppropriate platform for controlled access sensitive data
✓ 1/1 R10 10.Cross-Platform and Community Integration Citation and DOI for Cross-referencing
evidencePublication DOI 10.1007/s12028-024-02007 + grant number OT2OD032701
qualityCitable publication DOI in peer-reviewed journal
✓ 1/1 R10 10.Cross-Platform and Community Integration Community Standards or Schema Conformance
evidence5 standards: OMOP (OHDSI), DICOM, WFDB/PhysioNet, OHNLP, EDF+/Persyst
qualityExemplary community standards conformance
✓ 1/1 R10 10.Cross-Platform and Community Integration Outreach Materials and Documentation Links
evidenceChorus_SOP, AIM-AHEAD training (2 cohorts), 28 GitHub repos, websites
qualityComprehensive outreach and documentation
✓ 1/1 R10 10.Cross-Platform and Community Integration Related Datasets with Typed Relationships
evidenceBridge2AI ecosystem (4 projects), OHDSI network, federated multi-center structure
qualityIntegration with Bridge2AI and OHDSI ecosystems
⚠ low R10 · content_accuracy
issueNo systematic bias analysis despite multi-center critical care dataset
fieldsknown_biases
fixDocument potential systematic biases
⚠ low R10 · semantic_understanding
issueVersion tracking absent despite ongoing expansion
fieldsversion
fixImplement version numbering scheme

Recommendations

  1. R10 · Register dataset with DOI through DataCite
  2. R10 · Document informed consent procedures for retrospective collection
  3. R10 · Implement version numbering scheme (e.g., v1.0 = 45K, v2.0 = 100K)
  4. R10 · Add systematic bias analysis section
  5. R10 · Document version access methods once versioning implemented
  6. R10 · Clarify conflicts of interest disclosure
  7. R10 · Consider dataset-specific DOI in addition to publication DOI
  8. R10 · Document historical change log as dataset evolves
  9. R20 · Add specific IRB protocol numbers from at least the lead institution (Massachusetts General Hospital) for enhanced transparency and reproducibility
  10. R20 · Add unified deidentification method field explicitly stating approach (likely HIPAA Safe Harbor based on described transformations) to complement detailed preprocessing strategies
  11. R20 · Implement formal version numbering system and structured changelog documenting progression from initial release through 100K patient target
  12. R20 · Expand instances and subpopulations documentation with specific demographic distributions (age ranges, race/ethnicity percentages, sex/gender) to support health equity research claims
  13. R20 · Standardize grant number format across all fields to 1OT2OD032701-01 (full NIH format with project prefix)
  14. R20 · Add informed_consent field documenting retrospective waiver approach approved by 14 IRBs for consistency with other ethics fields
  15. R20 · Document participant compensation (likely none for retrospective study) and add vulnerable_populations field explicitly addressing PICU/NICU pediatric protections
  16. R20 · Consider adding RRID identifier for dataset software tools (especially OHDSI, OHNLP, DeGauss) to enhance tool citability
  17. R20 · Add errata field to document known issues or corrections as dataset continues expansion through November 2026