// SPDX-License-Identifier: CC-BY-SA-4.0
// SPDX-FileCopyrightText: 2024-2025 Ehsaneddin Asgari and Contributors
= LOL

image:https://img.shields.io/badge/License-PMPL--1.0-blue.svg[License: PMPL-1.0,link="https://github.com/hyperpolymath/palimpsest-license"]
:toc: preamble
:toclevels: 3
:icons: font
:source-highlighter: rouge
:experimental:

image:https://img.shields.io/badge/RSR-Gold%20(100%25)-ffd700[RSR Compliance, link=https://github.com/Hyperpolymath/rhodium-standard-repositories]
image:https://img.shields.io/badge/License-PMPL--1.0-blue.svg[License: PMPL-1.0,link="https://github.com/hyperpolymath/palimpsest-license"]
image:https://img.shields.io/badge/ReScript-11.x-e84f4f[ReScript]
image:https://img.shields.io/badge/Podman-Ready-purple[Podman]

**Super-parallel corpus crawler for multilingual NLP and Computational Linguistics research**

A type-safe ReScript implementation for building massive parallel corpora across 1500+ languages from Bible translation sources.

== Features

* *1500+ Languages* - Crawl parallel texts from multiple Bible corpus sources
* *Type-Safe* - Built with ReScript for compile-time correctness
* *Proof-Verified* - Echidna integration for mathematical verification
* *RSR Gold Compliant* - Follows Rhodium Standard Repository specifications
* *Semantic Grounding* - OpenCyc integration for common-sense reasoning
* *Container-Ready* - Podman-native (no Docker required)

== Quick Start

=== Prerequisites

* Node.js 20+
* Just command runner
* Nix (recommended) or npm
* Podman (for containers)

=== Installation

[source,bash]
----
# Clone the repository
git clone https://github.com/Hyperpolymath/1000Langs.git
cd 1000Langs

# Using Nix (recommended)
nix develop

# Or using npm directly
npm install
----

=== Build & Run

[source,bash]
----
# Build the project
just build

# Run tests
just test

# Run the crawler
just crawl-all
----

== Project Structure

[source]
----
1000Langs/
├── src/
│   ├── Lang1000.res          # Main entry point
│   ├── crawlers/             # Web crawler implementations
│   │   ├── Crawler.res       # Base crawler module
│   │   ├── BibleCloud.res    # bible.cloud crawler
│   │   ├── BibleCom.res      # bible.com crawler
│   │   └── PngScriptures.res # pngscriptures.org crawler
│   ├── api/                  # API client wrappers
│   │   └── DigitalBiblePlatform.res
│   ├── corpus/               # Corpus management
│   │   └── Alignment.res     # Parallel text alignment
│   ├── utils/                # Utility modules
│   │   ├── Iso639.res        # Language code handling
│   │   └── Statistics.res    # Statistical functions
│   ├── proofs/               # Mathematical proofs
│   └── cyc/                  # OpenCyc integration
├── test/                     # Test suites
├── proofs/                   # Echidna proof files
├── config/                   # Nickel configuration
├── meta/                     # Reference data
├── .well-known/              # Discovery files
├── Justfile                  # Task automation
├── flake.nix                 # Nix development environment
├── Containerfile             # Podman container definition
└── rescript.json             # ReScript configuration
----

== Supported Sources

[cols="1,2,1,1"]
|===
|Source |URL |Type |Languages

|Bible Cloud
|https://bible.cloud
|API
|1500+

|Bible.com
|https://bible.com
|Scraper
|2000+

|PNG Scriptures
|https://pngscriptures.org
|Download
|800+

|eBible
|https://ebible.org
|Download
|1000+

|Find.Bible
|https://find.bible
|API
|1200+
|===

== Configuration

Configuration is managed through Nickel for type-safe, validated settings:

[source,bash]
----
# Validate configuration
just nickel-check

# Export to JSON
just nickel-export

# Show resolved config
just nickel-show
----

See `config/main.ncl` for all configuration options.

== Testing

[source,bash]
----
# Run all tests
just test

# Run with coverage
just test-coverage

# Run proof verification
just prove
----

== Proof Verification

This project integrates with https://github.com/hyperpolymath/echidna[Echidna] for mathematical proof verification:

* Data integrity proofs
* Alignment correctness verification
* Statistical property validation
* Type safety guarantees

[source,bash]
----
# Run all proofs
just prove

# Check specific proof
just prove-check alignment_correctness
----

== Container Deployment

Uses Podman (not Docker) for container operations:

[source,bash]
----
# Build container
just container-build

# Run container
just container-run

# Deploy with volume mounts
just container-dev
----

== RSR Compliance

This project targets *Gold (100%)* compliance with the https://github.com/Hyperpolymath/rhodium-standard-repositories[Rhodium Standard Repository] specification:

[source,bash]
----
# Run compliance audit
just rsr-audit

# Generate HTML report
just rsr-audit-html
----

== Contributing

See link:CONTRIBUTING.adoc[CONTRIBUTING.adoc] for guidelines.

This project uses the *Tri-Perimeter Contribution Framework (TPCF)*:

* *Perimeter 1 (Core)*: Maintainers only
* *Perimeter 2 (Expert)*: Trusted contributors
* *Perimeter 3 (Community)*: Open contributions

== License

Dual licensed under:

* Palimpsest-MPL-1.0 License
* Palimpsest License v0.8

See link:LICENSE.txt[LICENSE.txt] for details.

Commercial use with attribution is permitted. Proprietary AI training without attribution is prohibited.

== Acknowledgments

* Original Python implementation by Ehsaneddin Asgari (LMU Munich)
* Bible corpus data from various open Bible translation projects
* https://github.com/hyperpolymath/echidna[Echidna] for proof verification
* https://github.com/Hyperpolymath/rhodium-standard-repositories[RSR] for compliance framework

== See Also

* link:CONTRIBUTING.adoc[Contributing Guide]
* link:CODE_OF_CONDUCT.adoc[Code of Conduct]
* link:SECURITY.md[Security Policy]
* link:CHANGELOG.adoc[Changelog]
* link:FUTURE.adoc[Future Roadmap]
