Data Ingestion

Converts raw files (PDFs, images, audio, video, CSV) into a unified JSONL stream for downstream chunking and embedding.

Quick Start

# 1. Activate the venv (Python 3.11 required for audio/video)
source venv/bin/activate

# 2. Run ingestion on a data folder
python3.11 01_data_ingestion/ingest_multimodal.py \
  --input 00_Data_nathan \
  --output-dir 01_data_ingestion/output/nathan_data

That's it. Output goes to --output-dir as three files:

Supported File Types

Long Video Caching

Videos are split into time-based chunks (default 10 min, configurable via --chunk-duration). A checkpoint file tracks progress:

output_dir/.checkpoint_<doc_id>.json
→ {"completed_chunks": [0, 1, 2, 3], "total_segments": 85}

If the process crashes or you Ctrl+C, just re-run the exact same command. Completed chunks are skipped and processing resumes where it left off. On successful completion the checkpoint file is automatically deleted.

For very long videos (hours), use shorter chunks for more frequent checkpoints:

python3.11 01_data_ingestion/ingest_multimodal.py \
  --input 00_Data_nathan \
  --output-dir 01_data_ingestion/output/nathan_data \
  --chunk-duration 300

CLI Options

--input              Input directory (default: 00_Data)
--output-dir         Output directory (default: 01_data_ingestion/output)
--output-name        Output JSONL filename (default: cleaned_documents.jsonl)
--errors-name        Error log filename (default: ingestion_errors.jsonl)
--max-files          Limit number of files to process
--pdf-strategy       PDF extraction: auto | hi_res | fast | ocr_only
--ocr-engine         OCR engine: tesseract | paddle
--ocr-language       OCR language code (default: eng)
--whisper-model      ASR model: tiny | base | small | medium | large-v3
--whisper-device     Device: cpu | cuda | auto
--chunk-duration     Video chunk length in seconds (default: 600)
--extract-keyframes  Extract scene-change keyframes and run OCR on them
--scene-threshold    Scene detection sensitivity 0.0-1.0 (default: 0.3)

Output Schema

Each line in cleaned_documents.jsonl:

{
  "doc_id": "handbook-2024-pdf__a1b2c3d4",
  "page": 4,
  "section": "page_4",
  "section_index": 7,
  "content": "Employees are entitled to parental leave...",
  "source": {
    "path": "policies/handbook_2024.pdf",
    "filename": "handbook_2024.pdf",
    "filetype": "pdf",
    "modality": "text",
    "loader": "UnstructuredPDFLoader",
    "record_id": "handbook-2024-pdf__a1b2c3d4:000007",
    "prev_record_id": "...:000006",
    "next_record_id": "...:000008"
  }
}

Audio/video records add timestamp_start, timestamp_end, timestamp, language. Image records add ocr_engine. CSV records add row_index.

Setup

System Dependencies

macOS:

brew install tesseract poppler ffmpeg python@3.11

Windows (Chocolatey):

choco install tesseract poppler ffmpeg

Python Environment

python3.11 -m venv venv
source venv/bin/activate
pip install -r 01_data_ingestion/requirements.txt

Python 3.11 is required — faster-whisper depends on onnxruntime which needs Python < 3.13.

Testing

Download test datasets and run the full test suite:

python3.11 01_data_ingestion/download_test_datasets.py \
  --out 00_Data_test --text --image --audio --video

python3.11 01_data_ingestion/test_ingestion.py \
  --data-dir 00_Data_test --modality all --verbose

Validate output schema:

python3.11 01_data_ingestion/validate_ingestion.py \
  --input 01_data_ingestion/output/cleaned_documents.jsonl

Scripts