# Data Ingestion

Converts raw files (PDFs, images, audio, video, CSV) into a unified JSONL stream for downstream chunking and embedding.

## Quick Start

```bash
# 1. Activate the venv (Python 3.11 required for audio/video)
source venv/bin/activate

# 2. Run ingestion on a data folder
python3.11 01_data_ingestion/ingest_multimodal.py \
  --input 00_Data_nathan \
  --output-dir 01_data_ingestion/output/nathan_data
```

That's it. Output goes to `--output-dir` as three files:

| File | Contents |
|------|----------|
| `cleaned_documents.jsonl` | One JSON record per extracted segment |
| `ingestion_stats.json` | Summary: files processed, succeeded, failed, records emitted |
| `ingestion_errors.jsonl` | One JSON line per error (empty if no errors) |

## Supported File Types

| Modality | Extensions | Extraction Method |
|----------|-----------|-------------------|
| Text | PDF, DOCX, PPTX, TXT, MD, HTML | LangChain + Unstructured |
| Images | JPG, PNG, TIF, BMP | Tesseract OCR (or PaddleOCR) |
| Audio | MP3, WAV, FLAC, M4A | faster-whisper transcription |
| Video | MP4, WEBM, MOV, AVI | ffmpeg audio extraction + faster-whisper |
| Tables | CSV | Row-wise text rendering |

## Long Video Caching

Videos are split into time-based chunks (default 10 min, configurable via `--chunk-duration`). A checkpoint file tracks progress:

```
output_dir/.checkpoint_<doc_id>.json
→ {"completed_chunks": [0, 1, 2, 3], "total_segments": 85}
```

**If the process crashes or you Ctrl+C**, just re-run the exact same command. Completed chunks are skipped and processing resumes where it left off. On successful completion the checkpoint file is automatically deleted.

For very long videos (hours), use shorter chunks for more frequent checkpoints:

```bash
python3.11 01_data_ingestion/ingest_multimodal.py \
  --input 00_Data_nathan \
  --output-dir 01_data_ingestion/output/nathan_data \
  --chunk-duration 300
```

## CLI Options

```
--input              Input directory (default: 00_Data)
--output-dir         Output directory (default: 01_data_ingestion/output)
--output-name        Output JSONL filename (default: cleaned_documents.jsonl)
--errors-name        Error log filename (default: ingestion_errors.jsonl)
--max-files          Limit number of files to process
--pdf-strategy       PDF extraction: auto | hi_res | fast | ocr_only
--ocr-engine         OCR engine: tesseract | paddle
--ocr-language       OCR language code (default: eng)
--whisper-model      ASR model: tiny | base | small | medium | large-v3
--whisper-device     Device: cpu | cuda | auto
--chunk-duration     Video chunk length in seconds (default: 600)
--extract-keyframes  Extract scene-change keyframes and run OCR on them
--scene-threshold    Scene detection sensitivity 0.0-1.0 (default: 0.3)
```

## Output Schema

Each line in `cleaned_documents.jsonl`:

```json
{
  "doc_id": "handbook-2024-pdf__a1b2c3d4",
  "page": 4,
  "section": "page_4",
  "section_index": 7,
  "content": "Employees are entitled to parental leave...",
  "source": {
    "path": "policies/handbook_2024.pdf",
    "filename": "handbook_2024.pdf",
    "filetype": "pdf",
    "modality": "text",
    "loader": "UnstructuredPDFLoader",
    "record_id": "handbook-2024-pdf__a1b2c3d4:000007",
    "prev_record_id": "...:000006",
    "next_record_id": "...:000008"
  }
}
```

Audio/video records add `timestamp_start`, `timestamp_end`, `timestamp`, `language`. Image records add `ocr_engine`. CSV records add `row_index`.

## Setup

### System Dependencies

**macOS:**
```bash
brew install tesseract poppler ffmpeg python@3.11
```

**Windows (Chocolatey):**
```powershell
choco install tesseract poppler ffmpeg
```

### Python Environment

```bash
python3.11 -m venv venv
source venv/bin/activate
pip install -r 01_data_ingestion/requirements.txt
```

Python 3.11 is required — `faster-whisper` depends on `onnxruntime` which needs Python < 3.13.

## Testing

Download test datasets and run the full test suite:

```bash
python3.11 01_data_ingestion/download_test_datasets.py \
  --out 00_Data_test --text --image --audio --video

python3.11 01_data_ingestion/test_ingestion.py \
  --data-dir 00_Data_test --modality all --verbose
```

Validate output schema:

```bash
python3.11 01_data_ingestion/validate_ingestion.py \
  --input 01_data_ingestion/output/cleaned_documents.jsonl
```

## Scripts

| Script | Purpose |
|--------|---------|
| `ingest_multimodal.py` | Core ingestion engine |
| `ingest_folder.py` | Batch wrapper with `--modality` and `--filetype` filtering |
| `test_ingestion.py` | Automated test suite (all modalities + schema + caching) |
| `validate_ingestion.py` | JSONL schema validator |
| `download_test_datasets.py` | Downloads test data (WikiText, CORD-v2, LibriSpeech, YouTube UGC) |