# Rubric10 Evaluation Plan

**Generated:** 2025-12-07 20:36:44

## Scope

**Total files to evaluate:** 127
- Individual files: 111
- Concatenated files: 16

## Evaluation Settings

- **Rubric:** rubric10 (10 elements, 50 sub-elements, max 50 points)
- **Temperature:** 0.0 (fully deterministic)
- **Model:** claude-sonnet-4-5-20250929 (date-pinned)
- **Method:** Conversational LLM-as-judge within Claude Code

## Individual Files by Project


### AI_READI

**gpt5:** 8 files
  - `data/d4d_individual/gpt5/AI_READI/fairhub_row10_d4d.yaml`
  - `data/d4d_individual/gpt5/AI_READI/doi_row2_d4d.yaml`
  - `data/d4d_individual/gpt5/AI_READI/doi_row9_d4d.yaml`
  - `data/d4d_individual/gpt5/AI_READI/fairhub_row13_d4d.yaml`
  - `data/d4d_individual/gpt5/AI_READI/docs_google_com_document-d_d4d.yaml`
  - `data/d4d_individual/gpt5/AI_READI/fairhub_d4d.yaml`
  - `data/d4d_individual/gpt5/AI_READI/doi_d4d.yaml`
  - `data/d4d_individual/gpt5/AI_READI/docs_aireadi_org_docs-2_d4d.yaml`
**claudecode_agent:** 14 files
  - `data/d4d_individual/claudecode_agent/AI_READI/fairhub_row10_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/AI_READI/docs_google_com_document-d_row11_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/AI_READI/RePORT_RePORTER_AI-READI_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/AI_READI/docs_aireadi_org_docs-2_row8_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/AI_READI/fairhub_row16_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/AI_READI/RePORT_RePORTER_AI_READI_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/AI_READI/doi_row2_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/AI_READI/docs_aireadi_org_docs-2_row10_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/AI_READI/doi_row9_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/AI_READI/docs_google_com_document-d_row13_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/AI_READI/fairhub_row13_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/AI_READI/fairhub_row12_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/AI_READI/e097449_full_row2_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/AI_READI/zenodo_org_records-10642459_row11_d4d.yaml`
**claudecode_assistant:** 14 files
  - `data/d4d_individual/claudecode_assistant/AI_READI/fairhub_row10_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/AI_READI/docs_google_com_document-d_row11_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/AI_READI/docs_aireadi_org_docs-2_row8_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/AI_READI/fairhub_row16_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/AI_READI/RePORT_RePORTER_AI_READI_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/AI_READI/doi_row2_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/AI_READI/docs_aireadi_org_docs-2_row10_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/AI_READI/doi_row9_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/AI_READI/docs_google_com_document-d_row13_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/AI_READI/fairhub_row13_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/AI_READI/fairhub_row12_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/AI_READI/RePORT ⟩ RePORTER - AI-READI_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/AI_READI/zenodo_org_records-10642459_row11_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/AI_READI/e097449.full_row2_d4d.yaml`


### CHORUS

**gpt5:** 2 files
  - `data/d4d_individual/gpt5/CHORUS/aim-ahead-bridge2ai-for-clinical-care-informational-webinar_d4d.yaml`
  - `data/d4d_individual/gpt5/CHORUS/CHoRUS for Equitable AI - github_d4d.yaml`
**claudecode_agent:** 12 files
  - `data/d4d_individual/claudecode_agent/CHORUS/aim-ahead-bridge2ai-for-clinical-care-informational-webinar_row9.txt_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/CHORUS/aim-ahead-bridge2ai-for-clinical-care-informational-webinar_row8.txt_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/CHORUS/github_chorus_ai_row9.txt_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/CHORUS/CHoRUS for Equitable AI - github.txt_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/CHORUS/CHoRUS for Equitable AI.txt_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/CHORUS/aim-ahead-bridge2ai-for-clinical-care-informational-webinar_row7_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/CHORUS/CHoRUS for Equitable AI_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/CHORUS/aim-ahead-bridge2ai-for-clinical-care-informational-webinar_row7.txt_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/CHORUS/aim-ahead-bridge2ai-for-clinical-care-informational-webinar_row9_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/CHORUS/aim-ahead-bridge2ai-for-clinical-care-informational-webinar_row8_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/CHORUS/github_chorus_ai_row9_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/CHORUS/CHoRUS for Equitable AI - github_d4d.yaml`
**claudecode_assistant:** 12 files
  - `data/d4d_individual/claudecode_assistant/CHORUS/CHoRUS_for_Equitable_AI_github_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/CHORUS/CHoRUS_for_Equitable_AI_-_github.txt_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/CHORUS/aim-ahead-bridge2ai-for-clinical-care-informational-webinar_row9.txt_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/CHORUS/aim-ahead-bridge2ai-for-clinical-care-informational-webinar_row8.txt_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/CHORUS/github_chorus_ai_row9.txt_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/CHORUS/CHoRUS_for_Equitable_AI_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/CHORUS/aim-ahead-bridge2ai-for-clinical-care-informational-webinar_row7_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/CHORUS/aim-ahead-bridge2ai-for-clinical-care-informational-webinar_row7.txt_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/CHORUS/aim-ahead-bridge2ai-for-clinical-care-informational-webinar_row9_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/CHORUS/aim-ahead-bridge2ai-for-clinical-care-informational-webinar_row8_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/CHORUS/github_chorus_ai_row9_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/CHORUS/CHoRUS_for_Equitable_AI.txt_d4d.yaml`


### CM4AI

**gpt5:** 4 files
  - `data/d4d_individual/gpt5/CM4AI/dataverse_10.18130_V3_B35XWX_d4d.yaml`
  - `data/d4d_individual/gpt5/CM4AI/dataverse_10.18130_V3_F3TD5R_d4d.yaml`
  - `data/d4d_individual/gpt5/CM4AI/doi_row3_d4d.yaml`
  - `data/d4d_individual/gpt5/CM4AI/doi_d4d.yaml`
**claudecode_agent:** 12 files
  - `data/d4d_individual/claudecode_agent/CM4AI/creativecommons_org_licenses-by-nc-sa_row15.txt_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/CM4AI/2024.05.21.589311v1.full_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/CM4AI/RePORT_RePORTER_CM4AI_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/CM4AI/2024.05.21.589311v1.full.txt_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/CM4AI/dataverse_10.18130_V3_F3TD5R_row19.txt_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/CM4AI/doi_row3.json_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/CM4AI/creativecommons_org_licenses_by_nc_sa_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/CM4AI/dataverse_B35XWX_march2025_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/CM4AI/doi_row3_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/CM4AI/RePORT ⟩ RePORTER - CM4AI.txt_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/CM4AI/dataverse_F3TD5R_june2025_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/CM4AI/dataverse_10.18130_V3_B35XWX_row16.txt_d4d.yaml`
**claudecode_assistant:** 12 files
  - `data/d4d_individual/claudecode_assistant/CM4AI/creativecommons_org_licenses-by-nc-sa_row15_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/CM4AI/creativecommons_org_licenses-by-nc-sa_row15.txt_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/CM4AI/2024.05.21.589311v1.full_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/CM4AI/RePORT_RePORTER_CM4AI_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/CM4AI/2024.05.21.589311v1.full.txt_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/CM4AI/dataverse_10.18130_V3_F3TD5R_row19.txt_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/CM4AI/dataverse_10.18130_V3_B35XWX_row16_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/CM4AI/doi_row3.json_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/CM4AI/doi_row3_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/CM4AI/RePORT ⟩ RePORTER - CM4AI.txt_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/CM4AI/dataverse_10.18130_V3_F3TD5R_row19_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/CM4AI/dataverse_10.18130_V3_B35XWX_row16.txt_d4d.yaml`


### VOICE

**gpt5:** 3 files
  - `data/d4d_individual/gpt5/VOICE/healthnexus_d4d.yaml`
  - `data/d4d_individual/gpt5/VOICE/healthnexus_row13_d4d.yaml`
  - `data/d4d_individual/gpt5/VOICE/physionet_b2ai-voice_1.1_d4d.yaml`
**claudecode_agent:** 9 files
  - `data/d4d_individual/claudecode_agent/VOICE/physionet_b2ai-voice_1.1_row14_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/VOICE/B2AI-Voice_DTUA_2025_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/VOICE/B2AI-Voice DTUA 2025 2025-09-04_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/VOICE/RePORT ⟩ RePORTER - VOICE_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/VOICE/github_eipm_bridge2ai-docs_README_row22_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/VOICE/physionet_b2ai-voice_1.1_row17_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/VOICE/docs_google_com_document-d_row13_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/VOICE/healthnexus_row13_d4d.yaml`
  - `data/d4d_individual/claudecode_agent/VOICE/github_eipm_bridge2ai-docs_row22_d4d.yaml`
**claudecode_assistant:** 9 files
  - `data/d4d_individual/claudecode_assistant/VOICE/physionet_b2ai-voice_1.1_row14_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/VOICE/B2AI-Voice_DTUA_2025_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/VOICE/github_eipm_bridge2ai-docs_README_row22_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/VOICE/B2AI-Voice_DTUA_2025_2025-09-04_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/VOICE/RePORT_RePORTER_VOICE_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/VOICE/physionet_b2ai-voice_1.1_row17_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/VOICE/docs_google_com_document-d_row13_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/VOICE/healthnexus_row13_d4d.yaml`
  - `data/d4d_individual/claudecode_assistant/VOICE/github_eipm_bridge2ai-docs_row22_d4d.yaml`

## Concatenated Files by Project


### AI_READI

**gpt5:** `data/d4d_concatenated/gpt5/AI_READI_d4d.yaml`
**claudecode:** `data/d4d_concatenated/claudecode/AI_READI_d4d.yaml`
**claudecode_agent:** `data/d4d_concatenated/claudecode_agent/AI_READI_d4d.yaml`
**claudecode_assistant:** `data/d4d_concatenated/claudecode_assistant/AI_READI_d4d.yaml`


### CHORUS

**gpt5:** `data/d4d_concatenated/gpt5/CHORUS_d4d.yaml`
**claudecode:** `data/d4d_concatenated/claudecode/CHORUS_d4d.yaml`
**claudecode_agent:** `data/d4d_concatenated/claudecode_agent/CHORUS_d4d.yaml`
**claudecode_assistant:** `data/d4d_concatenated/claudecode_assistant/CHORUS_d4d.yaml`


### CM4AI

**gpt5:** `data/d4d_concatenated/gpt5/CM4AI_d4d.yaml`
**claudecode:** `data/d4d_concatenated/claudecode/CM4AI_d4d.yaml`
**claudecode_agent:** `data/d4d_concatenated/claudecode_agent/CM4AI_d4d.yaml`
**claudecode_assistant:** `data/d4d_concatenated/claudecode_assistant/CM4AI_d4d.yaml`


### VOICE

**gpt5:** `data/d4d_concatenated/gpt5/VOICE_d4d.yaml`
**claudecode:** `data/d4d_concatenated/claudecode/VOICE_d4d.yaml`
**claudecode_agent:** `data/d4d_concatenated/claudecode_agent/VOICE_d4d.yaml`
**claudecode_assistant:** `data/d4d_concatenated/claudecode_assistant/VOICE_d4d.yaml`

## Evaluation Procedure

For each file, the rubric10 agent will:

1. **Read D4D YAML** using Read tool
2. **Apply rubric10 criteria** across 10 elements:
   - Element 1: Dataset Discovery and Identification (5 sub-elements)
   - Element 2: Dataset Access and Retrieval (5 sub-elements)
   - Element 3: Data Reuse and Interoperability (5 sub-elements)
   - Element 4: Ethical Use and Privacy Safeguards (5 sub-elements)
   - Element 5: Data Composition and Structure (5 sub-elements)
   - Element 6: Data Provenance and Version Tracking (5 sub-elements)
   - Element 7: Scientific Motivation and Funding Transparency (5 sub-elements)
   - Element 8: Technical Transparency (5 sub-elements)
   - Element 9: Dataset Evaluation and Limitations Disclosure (5 sub-elements)
   - Element 10: Cross-Platform and Community Integration (5 sub-elements)

3. **Score each sub-element** (0 or 1):
   - 1: Field present AND meaningful
   - 0: Field absent, generic, incomplete, or vague

4. **Provide evidence** from D4D file for each score

5. **Generate JSON output** with:
   - Overall score (total_points/50)
   - Element-by-element breakdown
   - Strengths, weaknesses, recommendations

6. **Save results** to:
   - Individual: `data/evaluation_llm/rubric10/individual/{project}_{method}_{filename}_evaluation.json`
   - Concatenated: `data/evaluation_llm/rubric10/concatenated/{project}_{method}_evaluation.json`

## Expected Output

After all evaluations complete:

1. **Individual evaluation JSONs:** ~{inventory['metadata']['total_individual_files']} files
2. **Concatenated evaluation JSONs:** {inventory['metadata']['total_concatenated_files']} files
3. **Summary CSV:** `data/evaluation_llm/rubric10/all_scores.csv`
4. **Summary report:** `data/evaluation_llm/rubric10/summary_report.md`

## Output Format

Each evaluation JSON contains:

```json
{{
  "rubric": "rubric10",
  "d4d_file": "path/to/file.yaml",
  "project": "PROJECT_NAME",
  "method": "METHOD_NAME",
  "type": "individual" or "concatenated",
  "overall_score": {{
    "total_points": 38.5,
    "max_points": 50,
    "percentage": 77.0
  }},
  "elements": [
    {{
      "id": 1,
      "name": "Dataset Discovery and Identification",
      "element_score": 5,
      "element_max": 5,
      "sub_elements": [...]
    }},
    ...
  ],
  "assessment": {{
    "strengths": [...],
    "weaknesses": [...],
    "recommendations": [...]
  }}
}}
```

## Summary Table Format

After evaluation, create table:

| Project | Type | Method | Score | Elements Passing | Top Element | Weakest Element |
|---------|------|--------|-------|------------------|-------------|-----------------|
| AI_READI | concat | curated | 42/50 (84%) | 8/10 | Discovery (5/5) | Provenance (2/5) |
| AI_READI | concat | gpt5 | 18/50 (36%) | 3/10 | Access (4/5) | Integration (0/5) |
| ... | ... | ... | ... | ... | ... | ... |

## Commands to Run

This plan can be executed conversationally in Claude Code with:

```
Using the d4d-rubric10 agent, evaluate all files listed in
data/evaluation_llm/rubric10/file_inventory.json and save results
according to this evaluation plan.
```
