Rubric10-Semantic Evaluation Report

Fields: publisher

Rationale: No publisher field; distribution through CHoRUS Consortium secure enclave and maintainers field documents consortium structure, but lacks formal publisher designation

Semantic Notes: Dataset distributed by CHoRUS Consortium (20 academic centers, 14 data acquisition sites) but lacks structured publisher field; NIH-funded infrastructure

Fields: citation, doi

Rationale: No citation or doi fields for dataset; external_resources references published research (DOI: 10.1007/s12028-024-02007) about CHoRUS but not dataset DOI/citation

Semantic Notes: Related publication DOI provided but dataset lacks own DOI and citation; limits scholarly referencing and tracking

Fields: conforms_to

Rationale: No conforms_to field; extensive standards usage documented throughout (OMOP CDM, DICOM, WFDB/PhysioNet, OHNLP, EDF+, Persyst) but not in structured conforms_to field

Semantic Notes: Comprehensive standards conformance described narratively (5 major standards); lacks machine-readable conforms_to declarations

Fields: external_resources, page

Rationale: page field (https://chorus4ai.org/) plus 10 external_resources entries covering: project websites (chorus4ai.org, bridge2ai.org/chorus), GitHub organization (github.com/chorus-ai, 28 repos), Chorus_SOP documentation, developer documentation, NIH RePORTER (reporter.nih.gov/project-details/10472824), Bridge2AI program, AIM-AHEAD partnership, OHDSI community, published research, and contact emails

Semantic Notes: Extensive documentation ecosystem with landing page, GitHub infrastructure, SOPs, developer guides, training partnerships, and community resources

Fields: related_datasets

Rationale: No related_datasets field; external_resources mentions Bridge2AI program and OHDSI community but no typed dataset relationships

Semantic Notes: Part of Bridge2AI ecosystem (4 data generation projects) but lacks structured relationships to AI-READI, CM4AI, VOICE, or other critical care datasets

Fields: doi, rrid, id

Rationale: Has id field (https://chorus4ai.org/) as URL identifier; lacks DOI and RRID for persistent scholarly identification

Semantic Notes: URL is valid and resolvable project landing page but lacks persistence guarantees of DOI; no research resource identifier

Fields: title, description

Rationale: Complete title (88 chars with full acronym expansion) and comprehensive description (238 words covering scope, modalities, ethics, standards, partnerships, status)

Semantic Notes: Description exceptionally detailed with project overview, dataset characteristics, ethical framework, standardization approach, training programs, and current metrics

Fields: keywords

Rationale: 24 keywords covering project name (CHoRUS, Bridge2AI), domain (critical care, acute illness), data types (EHR, waveform, imaging, EEG, clinical notes), standards (OMOP, DICOM, OHNLP), and principles (health equity, ethical AI, federated access)

Semantic Notes: Comprehensive keyword set balancing technical terms, data modalities, standards, and conceptual themes; supports diverse search strategies

Fields: page, resources

Rationale: Has page field (https://chorus4ai.org/); no hierarchical resources field but external_resources provides 10 related resources

Semantic Notes: Landing page URL provided; lacks structured sub-resources hierarchy but extensive external_resources compensate

Fields: parent_datasets, related_datasets

Rationale: No parent_datasets or related_datasets fields present despite being part of Bridge2AI program ecosystem

Semantic Notes: Missing hierarchical relationships to Bridge2AI parent program and potential related datasets in the ecosystem

Fields: license_and_use_terms, ip_restrictions

Rationale: Detailed license_and_use_terms with 6 specific license_terms including institutional email requirement, signed agreement, controlled access, data use agreement, re-identification prohibition, and compliance requirements; no ip_restrictions field

Semantic Notes: Access policy comprehensively defined with registration, approval, and enclave access process; institutional access implied but no structured IP restrictions

Fields: regulatory_restrictions, confidentiality_level

Rationale: No regulatory_restrictions or confidentiality_level enum fields; HIPAA compliance and privacy requirements mentioned in sensitive_elements descriptions

Semantic Notes: Regulatory framework (HIPAA, IRB) discussed narratively but lacks structured regulatory_restrictions and confidentiality_level classification

Fields: download_url

Rationale: No download_url field; access through secure enclave only; distribution_formats.access_urls all point to landing page (https://chorus4ai.org/)

Semantic Notes: Federated enclave-based access model without direct download URLs; controlled access requires registration and approval

Fields: distribution_formats, format, media_type

Rationale: 5 distribution_formats entries with names, descriptions, and access_urls covering OMOP CDM, WFDB waveforms, DICOM imaging, OHNLP text, and EDF+/Persyst EEG; lacks top-level format/media_type fields

Semantic Notes: Formats comprehensively described with standards (OMOP, WFDB, DICOM, OHNLP, EDF+) but not as structured format/media_type/encoding fields; access URLs provided for each modality

Fields: related_datasets, external_resources

Rationale: 10 external_resources entries covering project websites (chorus4ai.org, bridge2ai.org), GitHub organization (chorus-ai), documentation (Chorus_SOP), NIH RePORTER, AIM-AHEAD partnership, OHDSI community, published research (DOI: 10.1007/s12028-024-02007), and contact emails; no related_datasets

Semantic Notes: Rich external resources including infrastructure, documentation, training partnerships, and publications; missing typed dataset relationships

Fields: license_and_use_terms

Rationale: license_and_use_terms with 6 specific terms defining controlled access, permitted uses (research), and restrictions (re-identification prohibited); enables reuse within approved framework

Semantic Notes: License clearly permits research reuse with controlled access requirements; appropriate for protected health information

Fields: format, encoding

Rationale: No top-level format or encoding fields; formats described in distribution_formats.description only (OMOP, WFDB, DICOM, OHNLP, EDF+)

Semantic Notes: Uses international standards (OMOP CDM, DICOM, WFDB, OHNLP, EDF+) comprehensively but lacks structured format/encoding fields for machine-readable interoperability

Fields: conforms_to, conforms_to_schema

Rationale: No conforms_to or conforms_to_schema fields; standards mentioned narratively in descriptions (OMOP CDM, DICOM schema, WFDB/PhysioNet schema, OHNLP schema, EDF+ schema)

Semantic Notes: Multiple schema conformance statements in descriptions but lacks structured conforms_to fields for formal declaration

Fields: variables

Rationale: No variables field with variable-level metadata; only modality-level descriptions in subsets and acquisition_methods

Semantic Notes: Missing granular variable-level metadata despite using structured schemas (OMOP CDM has defined tables/columns)

Fields: intended_uses, prohibited_uses, discouraged_uses

Rationale: 5 intended_uses covering AI/ML development, external validation, health equity research, clinical care improvement, and education; 3 discouraged_uses covering ongoing collection, clinical use without validation, and re-identification; no prohibited_uses field

Semantic Notes: Comprehensive use guidance with detailed rationale for intended and discouraged uses; lacks explicit prohibited_uses but license terms prohibit re-identification

Fields: ethical_reviews, human_subject_research

Rationale: Comprehensive human_subject_research field with involves_human_subjects=true, ethics_review_board listing (IRBs at 14 centers, community ethics focus groups, legal/ethical advisory teams, privacy review processes); no structured ethical_reviews list

Semantic Notes: Exceptionally detailed ethics documentation including multi-site IRB approvals, community engagement, and interdisciplinary advisory teams

Fields: is_deidentified

Rationale: No structured is_deidentified field; de-identification described narratively in preprocessing_strategies (preproc-004: medical imaging, preproc-005: re-identification limitation) and sensitive_elements

Semantic Notes: De-identification methods extensively described (DICOM header removal, OHNLP tokenization, privacy-preserving transformations) but lacks structured is_deidentified field with method details

Fields: participant_privacy

Rationale: No participant_privacy field; privacy protections described in preprocessing_strategies (preproc-002: OHNLP tokenization, preproc-005: privacy-preserving transformations), license_and_use_terms (re-identification prohibited), and human_subject_research (privacy review processes)

Semantic Notes: Multiple privacy protection layers described (de-identification, tokenization, controlled access, enclave model, community ethics input) but not in structured participant_privacy field

Fields: informed_consent

Rationale: No informed_consent field; human_subject_research mentions retrospective data collection and ethical frameworks but does not explicitly document consent procedures

Semantic Notes: Retrospective data collection likely uses IRB-approved waivers or altered consent; not explicitly documented in structured field

Fields: vulnerable_populations, participant_compensation

Rationale: No vulnerable_populations or participant_compensation fields; addressing_gaps mentions Social Determinants of Health and diversity focus but lacks structured documentation

Semantic Notes: Health equity and diversity emphasized in purposes and addressing_gaps; vulnerable populations and compensation not explicitly documented

Fields: subpopulations

Rationale: 2 subpopulations entries: critically ill patients by hospital (14 hospitals) and patients with complete multi-modal data; describes geographic and institutional diversity plus data completeness stratification

Semantic Notes: Subpopulations defined by site distribution and data modality completeness; could be expanded with demographic/clinical characteristics

Fields: instances

Rationale: 1 instances entry describing critically ill patients with current count (23,400 admissions as of November 2024), target (100,000+), data collection approach (retrospective from ICUs), and timeframe

Semantic Notes: Instance count clearly documented with current status and targets; instance_type describes critically ill patients in ICU settings

Fields: variables, is_tabular

Rationale: No variables or is_tabular fields; data modalities described in subsets and acquisition_methods but lacks variable-level metadata

Semantic Notes: OMOP CDM subset is inherently tabular but is_tabular flag absent; no structured variable-level metadata despite standardized schemas

Fields: instances

Rationale: instances.description documents critically ill patients requiring acute/critical care with multi-modal data types; addressing_gaps mentions comprehensive patient conditions and clinical treatment strategies

Semantic Notes: Data topics (acute illness, critical care, treatment response) and conditions (critically ill patients) clearly described; lacks specific condition enumeration

Fields: anomalies, sampling_strategies

Rationale: No anomalies field; comprehensive sampling_strategies with is_sample=true, is_random=false, is_representative=true, and 5 detailed strategies (federated collection, balanced sampling, contextual factors, ethics input, legal framework)

Semantic Notes: Sampling approach well-documented but data quality issues and anomalies not explicitly documented; discouraged_uses mentions ongoing quality assurance

Fields: version

Rationale: No version field; updates.description mentions ongoing collection and expansion but no version numbering scheme

Semantic Notes: Dataset versioning not implemented; continuous updates model without discrete version releases

Fields: version_access

Rationale: No version_access field documenting how to access previous versions or version history

Semantic Notes: Version access methods not documented; unclear if historical versions are preserved

Fields: errata, updates

Rationale: No errata field; updates field with detailed update_details (7 items) describing ongoing collection, current status (23,400 admissions), targets (100,000+), site updates via GitHub/Google Forms, GitHub project tracking, documentation updates, software development, and semantic mapping expansion

Semantic Notes: Update process and current status documented but no historical errata or change log; GitHub project tracking suggests version control for code

Fields: updates

Rationale: updates.frequency='Continuous updates through November 2026' with detailed update_details documenting ongoing retrospective collection, site status updates, GitHub tracking, and continuous software development

Semantic Notes: Update frequency and timeline clearly stated (continuous through November 2026); process documented via GitHub and Google Forms

Fields: was_derived_from, release_notes

Rationale: No was_derived_from or release_notes fields; collection_mechanisms describes primary data collection from EHR/PACS/monitor systems but not derivation relationships

Semantic Notes: Source systems documented (EHR, PACS, bedside monitors) but lacks formal provenance chain or release notes

Fields: purposes

Rationale: 4 purposes entries with IDs, names, and detailed descriptions covering: improve recovery from acute illness, create AI-ready dataset, establish data standards/tools, and promote diversity/health equity; each purpose 60-100 words

Semantic Notes: Exceptionally detailed motivation with four complementary purposes addressing clinical, technical, and equity dimensions

Fields: tasks

Rationale: 5 tasks entries with IDs, names, and descriptions covering: characterize illness patterns, predict complications, measure treatment response, enable external validation, and label prediction targets

Semantic Notes: Research tasks clearly defined with specific ML/AI applications aligned with purposes; covers both model development and validation use cases

Fields: funders

Rationale: 1 funder entry (NIH Common Fund Bridge2AI) with comprehensive description including grant number (1OT2OD032701-01), administering office, opportunity number (OTA-21-008), study section (DCMM), fiscal year (2022), funding amount ($5,880,300 direct costs), and project dates (Sept 2022 - Nov 2026 with no-cost extension)

Semantic Notes: Exceptional funding transparency with all relevant grant metadata; single primary funder comprehensively documented

Fields: funders

Rationale: Grant ID 1OT2OD032701-01 clearly stated in funders.description with opportunity number OTA-21-008 and study section DCMM

Semantic Notes: Grant identifiers in structured narrative; could benefit from structured grant_id field but information is complete

Fields: creators, funders

Rationale: 19 creators entries with IDs, names, and descriptions including roles (Contact PI, Principal Investigators, Lecturers, Workshop Leads) and affiliations (MGH, UF, UT Health, Tufts, etc.); funders field acknowledges NIH Common Fund

Semantic Notes: Comprehensive creator documentation covering leadership, faculty, and training roles; institutional affiliations provided; funder acknowledged

Fields: collection_mechanisms

Rationale: 5 collection_mechanisms entries with IDs, names, and descriptions covering: retrospective EHR extraction (demographics, medications, procedures, nursing flowsheets, diagnoses, notes), waveform telemetry capture (continuous bedside monitoring), medical imaging acquisition (PACS/DICOM), EEG recording extraction (EDF+/Persyst), and standardized data transformation (OMOP/OHNLP/DICOM/WFDB)

Semantic Notes: Comprehensive collection mechanisms covering all data modalities with technical details on sources, formats, and standardization

Fields: acquisition_methods

Rationale: 8 acquisition_methods entries with IDs, names, and descriptions covering: structured EHR (OMOP), medication administration records, provider documentation, high-frequency nursing flowsheets, tokenized clinical notes (OHNLP), medical imaging (DICOM), waveform telemetry (WFDB), and EEG waveforms; each includes access type (controlled) and schema metadata status

Semantic Notes: Detailed acquisition methods aligned with collection mechanisms; includes data type, access model, and schema conformance for each modality

Fields: preprocessing_strategies, cleaning_strategies, labeling_strategies

Rationale: 5 preprocessing_strategies (OMOP transformation, OHNLP tokenization, waveform standardization, DICOM de-identification, re-identification limitation) with detailed preprocessing_details (5-6 steps each); 2 cleaning_strategies (multi-center harmonization, metadata schema validation) with cleaning_details; 1 labeling_strategies (visualization/annotation environment) with labeling_details

Semantic Notes: Comprehensive preprocessing pipeline with 5 strategies covering standardization, privacy protection, and quality assurance; cleaning focuses on harmonization and validation; labeling supports prediction task annotation

Fields: software_and_tools

Rationale: No software_and_tools field; tools mentioned in descriptions (OHDSI, OHNLP, gateway/middleware systems) and external_resources (GitHub organization with 28 repositories), but not in structured field

Semantic Notes: Software tools extensively documented in external_resources (chorus-ai GitHub org, OHDSI tools, OHNLP toolkit) but lacks structured software_and_tools field with versions/citations

Fields: external_resources, conforms_to

Rationale: 10 external_resources entries covering project websites, GitHub (28 repos), Chorus_SOP documentation, OHDSI community, AIM-AHEAD partnership, published research (DOI: 10.1007/s12028-024-02007), and contact information; no conforms_to field but standards extensively referenced (OMOP, DICOM, WFDB, OHNLP, EDF+)

Semantic Notes: Rich external resources including technical documentation, standards communities, and publications; standards referenced throughout but lacks structured conforms_to field

Fields: known_limitations

Rationale: No known_limitations field; discouraged_uses mentions 'completeness limitations and ongoing expansion' and 'awareness of completeness limitations' but not in structured known_limitations

Semantic Notes: Dataset completeness limitations implied by ongoing collection (23,400/100,000+ target) and discouraged_uses but lacks explicit known_limitations documentation

Fields: known_biases

Rationale: No known_biases field; addressing_gaps mentions 'manage privacy and bias' and sampling_strategies mentions 'balanced and diverse cohort' but specific biases not documented

Semantic Notes: Bias mitigation strategies described (federated sampling, diversity focus, Social Determinants of Health) but identified biases not explicitly documented

Fields: anomalies

Rationale: No anomalies field; discouraged_uses mentions 'ongoing quality assurance processes' but specific quality issues or anomalies not documented

Semantic Notes: Quality assurance processes mentioned but data anomalies not explicitly documented; may emerge as collection continues

Fields: sensitive_elements, content_warnings

Rationale: 4 sensitive_elements entries (sensitive_elements_present=true for all) with detailed sensitivity_details covering: clinical/medical data (PHI, HIPAA), physiological monitoring data (health conditions, treatment responses), medical imaging (DICOM, embedded patient info), and Social Determinants of Health (geographic/contextual factors); no content_warnings field

Semantic Notes: Exceptionally detailed sensitivity documentation with 4 categories and 4-7 bullet points each; covers data types, privacy risks, and mitigation approaches; lacks separate content_warnings but sensitive_elements comprehensive

Fields: ethical_reviews

Rationale: No structured ethical_reviews list field; human_subject_research.ethics_review_board documents 4 oversight mechanisms: IRBs at 14 centers, community-facing ethics focus groups, legal/ethical advisory teams, and privacy/accountability review processes; no conflicts of interest documented

Semantic Notes: Comprehensive ethics review documentation in human_subject_research with multi-layered oversight; conflicts of interest not explicitly addressed