# Data Ingestion Pipeline

This pipeline automates the ingestion of data into Vertex AI Vector Search 2.0, streamlining the process of building Retrieval Augmented Generation (RAG) applications.

It orchestrates the complete workflow: loading data, chunking it into manageable segments, and importing the processed data into your Vector Search 2.0 Collection. Embeddings are auto-generated by the Collection's configured embedding model.

You can trigger the pipeline for an initial data load or schedule it to run periodically, ensuring your search index remains current.

## Prerequisites

Before running any commands, ensure you have set your Google Cloud Project ID as an environment variable. This variable will be used by the subsequent `make` commands.

```bash
export PROJECT_ID="YOUR_PROJECT_ID"
```
Replace `"YOUR_PROJECT_ID"` with your actual Google Cloud Project ID.

Now, you can set up the development environment:

1.  **Set up Datastore:** Use the following command from the root of the repository to provision the Vector Search 2.0 Collection, GCS bucket, and service account.

    ```bash
    make setup-datastore
    ```
    This command requires `terraform` to be installed and configured.

## Running the Data Ingestion Pipeline

After setting up the infrastructure using `make setup-datastore`, you can run the data ingestion pipeline.

**Steps:**

**a. Submit the Pipeline to Vertex AI Pipelines:**
Run the following command from the root of the repository. Ensure the `PROJECT_ID` environment variable is still set in your current shell session (as configured in Prerequisites).

```bash
make data-ingestion-remote
```

This compiles the KFP DAG, submits it to Vertex AI Pipelines, and waits for completion. The underlying `submit_pipeline.py` script accepts `--service-account`, `--pipeline-root`, `--pipeline-name`, and scheduling flags like `--schedule-only` and `--cron-schedule` for periodic execution.

**b. Monitoring Pipeline Progress:**

The pipeline's configuration and execution status link is printed to the console upon submission. For detailed monitoring, use the Vertex AI Pipelines dashboard in the Google Cloud Console.

## Testing Your RAG Application

Once the data ingestion pipeline completes successfully, you can test your RAG application with Vector Search 2.0.
