# Proposed Data Organization Structure

## Problem

Current data directories are inconsistent and unclear:
- Multiple extraction directories with overlapping purposes
- Inconsistent naming (alldocs vs d4d_alldocs)
- No clear distinction between GPT-5 and Claude Code generated files
- No designated place for preprocessed/transformed source files
- Old data scattered across multiple directories

## Proposed Clean Structure

### Overview
```
data/
  raw/                            # RAW DOWNLOADS (moved from downloads_by_column/)
    AI_READI/, CHORUS/, CM4AI/, VOICE/

  preprocessed/                   # Transformed source files
    individual/                   # PDF→text, HTML→text conversions
      AI_READI/, CHORUS/, CM4AI/, VOICE/
    concatenated/                 # All docs per project concatenated
      {PROJECT}_concatenated.txt

  d4d_individual/                 # D4D YAMLs from individual docs
    gpt5/AI_READI/, CHORUS/, CM4AI/, VOICE/
    claudecode/AI_READI/, CHORUS/, CM4AI/, VOICE/

  d4d_concatenated/               # D4D YAMLs from concatenated docs
    gpt5/{PROJECT}_d4d.yaml
    claudecode/{PROJECT}_d4d.yaml

  d4d_html/                       # HTML renderings
    individual/gpt5/, individual/claudecode/
    concatenated/gpt5/, concatenated/claudecode/

  ATTIC/                          # Archived old data
```

### Full Detail

data/
  preprocessed/                   # TRANSFORMED SOURCE FILES (PDF->text, HTML->text)
    individual/                   # Individual files ready for D4D agent
      AI_READI/
      CHORUS/
      CM4AI/
      VOICE/

    concatenated/                 # Concatenated files combining all docs per project
      AI_READI_concatenated.txt
      CHORUS_concatenated.txt
      CM4AI_concatenated.txt
      VOICE_concatenated.txt

  d4d_individual/                 # D4D YAMLs from INDIVIDUAL documents
    gpt5/                         # Generated by GPT-5 (validated_d4d_wrapper.py)
      AI_READI/
        file1_d4d.yaml
        file2_d4d.yaml
      CHORUS/
      CM4AI/
      VOICE/

    claudecode/                   # Generated by Claude Code
      AI_READI/
      CHORUS/
      CM4AI/
      VOICE/

  d4d_concatenated/               # D4D YAMLs from CONCATENATED docs
    gpt5/                         # Generated by GPT-5 (process_concatenated_d4d.py)
      AI_READI_d4d.yaml
      CHORUS_d4d.yaml
      CM4AI_d4d.yaml
      VOICE_d4d.yaml

    claudecode/                   # Generated by Claude Code
      AI_READI_d4d.yaml
      CHORUS_d4d.yaml
      CM4AI_d4d.yaml
      VOICE_d4d.yaml

    curated/                      # Manually-created/curated comprehensive D4D files
      AI_READI_curated.yaml
      CM4AI_curated.yaml
      VOICE_curated.yaml

  d4d_html/                       # HTML RENDERINGS of D4D YAMLs
    individual/
      gpt5/
        AI_READI/
        CHORUS/
        CM4AI/
        VOICE/
      claudecode/
        AI_READI/
        CHORUS/
        CM4AI/
        VOICE/

    concatenated/
      gpt5/
        AI_READI.html
        CHORUS.html
        CM4AI.html
        VOICE.html
      claudecode/
        AI_READI.html
        CHORUS.html
        CM4AI.html
        VOICE.html
      curated/                    # Manually-created/curated HTML
        AI_READI_human_readable.html
        AI_READI_linkml.html
        CM4AI_human_readable.html
        CM4AI_linkml.html
        VOICE_human_readable.html
        VOICE_linkml.html

    common.css                    # Shared CSS

  ATTIC/                          # OLD/DEPRECATED DATA (archived)
    extracted_by_column/          # Old location (to be moved)
    sheets_concatenated/          # Old location (to be moved)
    GC_data_sheets/
    claude_max_extractions/
    extracted_by_column_previous_run/
    extracted_by_column_tab_combined/
    extracted_claude_code/
    extracted_coding_agent/
    extracted_d4d_agent/
    validated_extracted/
    sheets/
    sheets_d4dassistant/
    RO-Crate/
```

## Key Principles

1. **Source Separation**: Raw downloads vs preprocessed vs D4D outputs
2. **Generator Labeling**: Clear distinction between GPT-5 and Claude Code (claudecode)
3. **Processing Stage**: Individual docs vs synthesized/concatenated
4. **Project Organization**: Flat project subdirectories (AI_READI, CHORUS, CM4AI, VOICE) without extra nesting
5. **Output Types**: Separate locations for YAML vs HTML outputs
6. **Archive Policy**: All old/deprecated data in ATTIC

## File Naming Conventions

### Individual D4D YAMLs
- **Format**: `{original_filename}_d4d.yaml`
- **Example**: `dataverse_10.18130_V3_B35XWX_d4d.yaml`
- **Location**: `data/d4d_individual/{gpt5|claudecode}/{PROJECT}/`

### Concatenated D4D YAMLs
- **Format**: `{PROJECT}_d4d.yaml`
- **Example**: `CM4AI_d4d.yaml`, `CHORUS_d4d.yaml`
- **Location**: `data/d4d_concatenated/{gpt5|claudecode}/`

### Preprocessed Concatenated Files
- **Format**: `{PROJECT}_concatenated.txt`
- **Example**: `AI_READI_concatenated.txt`
- **Location**: `data/preprocessed/concatenated/`

### HTML Files
- **Individual Format**: `{original_filename}_d4d.html`
- **Concatenated Format**: `{PROJECT}_d4d.html`

## Migration Plan (Completed Nov 14, 2025)

1. ✓ Create new directory structure
2. ✓ Move current working files to new locations:
   - `data/extracted_by_column/AI_READI/` → `data/d4d_individual/gpt5/AI_READI/`
   - `data/extracted_by_column/CHORUS/` → `data/d4d_individual/gpt5/CHORUS/`
   - `data/extracted_by_column/CM4AI/` → `data/d4d_individual/gpt5/CM4AI/`
   - `data/extracted_by_column/VOICE/` → `data/d4d_individual/gpt5/VOICE/`
   - `data/sheets_concatenated/*_concatenated.txt` → `data/preprocessed/concatenated/`
   - `data/sheets_concatenated/*_alldocs.yaml` → `data/d4d_concatenated/gpt5/`
   - `src/html/output/` → `data/d4d_html/`
   - `downloads_by_column/` → `data/raw/`
   - Preprocessed files (.txt, .json) → `data/preprocessed/individual/{PROJECT}/`
3. ✓ Move all old data to `data/ATTIC/`
4. ✓ Preserve curated files from ATTIC:
   - Curated D4D YAMLs → `data/d4d_concatenated/curated/`
   - Curated HTML files → `data/d4d_html/concatenated/curated/`
5. Update scripts to use new paths:
   - Update `validated_d4d_wrapper.py` output path
   - Update `process_concatenated_d4d.py` input/output paths
   - Update `concatenate_documents.py` output path
   - Update Makefile targets
6. Update CLAUDE.md documentation

## Benefits

- **Clear Data Flow**: Source → Preprocessed (individual & concatenated) → Individual D4D → Concatenated D4D → HTML
- **Easy Comparison**: GPT-5 vs Claude Code outputs in parallel directories
- **Logical Grouping**: Preprocessed files (txt/text) separate from D4D outputs (yaml)
- **Maintainability**: Consistent structure makes automation easier
- **Archival**: Old data clearly separated but preserved in ATTIC
- **Documentation**: Self-documenting through directory names
