# Data Ingestion Pipeline

This pipeline automates the ingestion of data into Vertex AI Vector Search 2.0, streamlining the process of building Retrieval Augmented Generation (RAG) applications.

It orchestrates the complete workflow: loading data, chunking it into manageable segments, and importing the processed data into your Vector Search 2.0 Collection. Embeddings are auto-generated by the Collection's configured embedding model.

You can trigger the pipeline for an initial data load or schedule it to run periodically, ensuring your search index remains current.

## Prerequisites

- `terraform` installed (for the infra step).
- A copy of `.env.example` saved as `.env` (in the sample root) with your project, region, and collection id filled in.

```bash
gcloud config set project YOUR_PROJECT_ID
```

## Getting Started

Run both steps from the sample root (`core/python/rag-vector-search`).

1.  **Provision the collection:** Create the Vector Search 2.0 Collection (and pipeline bucket) and enable required APIs.

    ```bash
    make setup-infra
    ```

2.  **Run data ingestion:** Process data and ingest it into the collection locally via KFP's `SubprocessRunner`.

    ```bash
    make data-ingestion
    ```

`make data-ingestion` loads `.env` and runs `submit_pipeline.py --local`, passing `--project`, `--region`, and `--collection-id` from `PROJECT_ID`, `REGION`, and `VECTOR_SEARCH_COLLECTION_ID`.

## Advanced: remote execution

To submit to Vertex AI Pipelines instead of running locally, call `submit_pipeline.py` directly without `--local` and provide the remote parameters:

```bash
cd data_ingestion
uv run python data_ingestion_pipeline/submit_pipeline.py \
    --project "$PROJECT_ID" --region "$REGION" \
    --collection-id "$VECTOR_SEARCH_COLLECTION_ID" \
    --service-account "$SERVICE_ACCOUNT" \
    --pipeline-root "$PIPELINE_ROOT" \
    --pipeline-name "$PIPELINE_NAME"
```

The script also supports `--cron-schedule` and `--schedule-only` for periodic execution. Run `uv run python data_ingestion_pipeline/submit_pipeline.py --help` for the full list of flags.

## Testing Your RAG Application

Once the data ingestion pipeline completes successfully, you can test your RAG application with Vector Search 2.0.
