# 02_chunking

Semantic chunking pipeline that converts ingested documents into meaningful segments with explicit relationship tracking.

## Installation

```bash
pip install -r 02_chunking/requirements.txt
```

## Usage

```bash
python 02_chunking/chunk_documents.py \
  --input 01_data_ingestion/output/test_text/cleaned_documents.jsonl \
  --output-dir 02_chunking/output/test_text \
  --chunk-size 2000 \
  --chunk-overlap 200
```

### Arguments

| Argument | Default | Description |
|----------|---------|-------------|
| `--input` | required | Path to input JSONL from data ingestion |
| `--output-dir` | required | Output directory for chunked JSONL and stats |
| `--chunk-size` | 2000 | Maximum characters per chunk |
| `--chunk-overlap` | 200 | Overlap characters between adjacent chunks |

### Output Files

- `chunked_documents.jsonl` - One chunk per line
- `chunking_stats.json` - Processing statistics

## Chunking Strategy

### Recursive Character Text Splitting

Uses LangChain's `RecursiveCharacterTextSplitter` which attempts to split text at semantically meaningful boundaries. The splitter tries separators in order of preference:

1. `\n\n` - Paragraph breaks (strongest boundary)
2. `\n` - Line breaks
3. `. ` - Sentence endings
4. `? ` - Question endings
5. `! ` - Exclamation endings
6. `; ` - Semicolons
7. `, ` - Commas
8. ` ` - Word boundaries
9. `` - Character-level (fallback)

This hierarchy ensures chunks break at natural language boundaries when possible, keeping related content together.

### Overlap

Adjacent chunks share `chunk_overlap` characters to preserve context across boundaries. This helps downstream retrieval and generation by ensuring concepts that span chunk boundaries aren't lost.

```
Chunk 1: [===================]
Chunk 2:              [===================]
                      ^-- overlap --^
```

### Modality-Specific Handling

| Modality | Strategy |
|----------|----------|
| **Text** (txt, pdf, docx, etc.) | Full recursive chunking with overlap |
| **Audio** | Preserved as-is (already segmented by whisper) |
| **Video** | Preserved as-is (already segmented by whisper) |
| **Image** | Preserved as-is (usually small OCR text) |

Audio and video records retain their original timestamps (`timestamp_start`, `timestamp_end`).

## Relationship Tracking

Each chunk includes explicit relationships to support downstream retrieval and evaluation:

```json
{
  "related_chunks": {
    "prev": "doc_chunk_000016",
    "next": "doc_chunk_000018",
    "same_section": ["doc_chunk_000015", "doc_chunk_000019", ...],
    "same_document": ["doc_chunk_000000", "doc_chunk_000500", ...]
  }
}
```

### Relationship Types

| Type | Description |
|------|-------------|
| `prev` | Immediately preceding chunk (null if first) |
| `next` | Immediately following chunk (null if last) |
| `same_section` | Chunks sharing the same section label (within 5-chunk window) |
| `same_document` | Sample of other chunks from the same document |

### Why Relationships Matter

When a retrieved chunk is relevant but incomplete, the answer may exist in a related chunk. Explicit relationships enable:

- **Retrieval expansion**: Fetch adjacent chunks when a single chunk is insufficient
- **Evaluation debugging**: Distinguish retrieval failure from chunk separation issues
- **Context reconstruction**: Reassemble document flow using prev/next chains

## Output Schema

```json
{
  "chunk_id": "wikitext-103-test-txt__f1eea7b4_chunk_000017",
  "doc_id": "wikitext-103-test-txt__f1eea7b4",
  "document_name": "test.txt",
  "section": "element_0",
  "page": null,
  "timestamp_start": null,
  "timestamp_end": null,
  "related_chunks": {
    "prev": "..._chunk_000016",
    "next": "..._chunk_000018",
    "same_section": ["..._chunk_000015", "..._chunk_000019"],
    "same_document": ["..._chunk_000000", "..._chunk_000412"]
  },
  "content": "The actual chunk text content...",
  "source": {
    "original_record_id": "wikitext-103-test-txt__f1eea7b4:000000",
    "path": "wikitext-103/test.txt",
    "filetype": "txt",
    "modality": "text",
    "char_offset_start": 8192,
    "char_offset_end": 10192
  }
}
```

### Field Reference

| Field | Description |
|-------|-------------|
| `chunk_id` | Unique identifier: `{doc_id}_chunk_{index:06d}` |
| `doc_id` | Parent document identifier from ingestion |
| `document_name` | Original filename |
| `section` | Section label from ingestion |
| `page` | Page number (if applicable) |
| `timestamp_start` | Start time in seconds (audio/video only) |
| `timestamp_end` | End time in seconds (audio/video only) |
| `related_chunks` | Relationship references (see above) |
| `content` | The chunk text |
| `source.original_record_id` | Record ID from ingestion stage |
| `source.path` | Relative file path |
| `source.filetype` | File extension or type |
| `source.modality` | text, audio, video, or image |
| `source.char_offset_start` | Character offset in original content |
| `source.char_offset_end` | Character offset end in original content |

## Example: Processing All Test Datasets

```bash
# Text
python 02_chunking/chunk_documents.py \
  --input 01_data_ingestion/output/test_text/cleaned_documents.jsonl \
  --output-dir 02_chunking/output/test_text

# Audio
python 02_chunking/chunk_documents.py \
  --input 01_data_ingestion/output/test_audio/cleaned_documents.jsonl \
  --output-dir 02_chunking/output/test_audio

# Image
python 02_chunking/chunk_documents.py \
  --input 01_data_ingestion/output/test_image/cleaned_documents.jsonl \
  --output-dir 02_chunking/output/test_image

# Video
python 02_chunking/chunk_documents.py \
  --input 01_data_ingestion/output/test_video/cleaned_documents.jsonl \
  --output-dir 02_chunking/output/test_video
```