02_chunking

Semantic chunking pipeline that converts ingested documents into meaningful segments with explicit relationship tracking.

Installation

pip install -r 02_chunking/requirements.txt

Usage

python 02_chunking/chunk_documents.py \
  --input 01_data_ingestion/output/test_text/cleaned_documents.jsonl \
  --output-dir 02_chunking/output/test_text \
  --chunk-size 2000 \
  --chunk-overlap 200

Arguments

Argument	Default	Description
`--input`	required	Path to input JSONL from data ingestion
`--output-dir`	required	Output directory for chunked JSONL and stats
`--chunk-size`	2000	Maximum characters per chunk
`--chunk-overlap`	200	Overlap characters between adjacent chunks

Output Files

chunked_documents.jsonl - One chunk per line
chunking_stats.json - Processing statistics

Chunking Strategy

Recursive Character Text Splitting

Uses LangChain's RecursiveCharacterTextSplitter which attempts to split text at semantically meaningful boundaries. The splitter tries separators in order of preference:

\n\n - Paragraph breaks (strongest boundary)
\n - Line breaks
. - Sentence endings
? - Question endings
! - Exclamation endings
; - Semicolons
, - Commas
- Word boundaries
`` - Character-level (fallback)

This hierarchy ensures chunks break at natural language boundaries when possible, keeping related content together.

Overlap

Adjacent chunks share chunk_overlap characters to preserve context across boundaries. This helps downstream retrieval and generation by ensuring concepts that span chunk boundaries aren't lost.

Chunk 1: [===================]
Chunk 2:              [===================]
                      ^-- overlap --^

Modality-Specific Handling

Modality	Strategy
Text (txt, pdf, docx, etc.)	Full recursive chunking with overlap
Audio	Preserved as-is (already segmented by whisper)
Video	Preserved as-is (already segmented by whisper)
Image	Preserved as-is (usually small OCR text)

Audio and video records retain their original timestamps (timestamp_start, timestamp_end).

Relationship Tracking

Each chunk includes explicit relationships to support downstream retrieval and evaluation:

{
  "related_chunks": {
    "prev": "doc_chunk_000016",
    "next": "doc_chunk_000018",
    "same_section": ["doc_chunk_000015", "doc_chunk_000019", ...],
    "same_document": ["doc_chunk_000000", "doc_chunk_000500", ...]
  }
}

Relationship Types

Type	Description
`prev`	Immediately preceding chunk (null if first)
`next`	Immediately following chunk (null if last)
`same_section`	Chunks sharing the same section label (within 5-chunk window)
`same_document`	Sample of other chunks from the same document

Why Relationships Matter

When a retrieved chunk is relevant but incomplete, the answer may exist in a related chunk. Explicit relationships enable:

Retrieval expansion: Fetch adjacent chunks when a single chunk is insufficient
Evaluation debugging: Distinguish retrieval failure from chunk separation issues
Context reconstruction: Reassemble document flow using prev/next chains

Output Schema

{
  "chunk_id": "wikitext-103-test-txt__f1eea7b4_chunk_000017",
  "doc_id": "wikitext-103-test-txt__f1eea7b4",
  "document_name": "test.txt",
  "section": "element_0",
  "page": null,
  "timestamp_start": null,
  "timestamp_end": null,
  "related_chunks": {
    "prev": "..._chunk_000016",
    "next": "..._chunk_000018",
    "same_section": ["..._chunk_000015", "..._chunk_000019"],
    "same_document": ["..._chunk_000000", "..._chunk_000412"]
  },
  "content": "The actual chunk text content...",
  "source": {
    "original_record_id": "wikitext-103-test-txt__f1eea7b4:000000",
    "path": "wikitext-103/test.txt",
    "filetype": "txt",
    "modality": "text",
    "char_offset_start": 8192,
    "char_offset_end": 10192
  }
}

Field Reference

Field	Description
`chunk_id`	Unique identifier: `{doc_id}_chunk_{index:06d}`
`doc_id`	Parent document identifier from ingestion
`document_name`	Original filename
`section`	Section label from ingestion
`page`	Page number (if applicable)
`timestamp_start`	Start time in seconds (audio/video only)
`timestamp_end`	End time in seconds (audio/video only)
`related_chunks`	Relationship references (see above)
`content`	The chunk text
`source.original_record_id`	Record ID from ingestion stage
`source.path`	Relative file path
`source.filetype`	File extension or type
`source.modality`	text, audio, video, or image
`source.char_offset_start`	Character offset in original content
`source.char_offset_end`	Character offset end in original content

Example: Processing All Test Datasets

# Text
python 02_chunking/chunk_documents.py \
  --input 01_data_ingestion/output/test_text/cleaned_documents.jsonl \
  --output-dir 02_chunking/output/test_text

# Audio
python 02_chunking/chunk_documents.py \
  --input 01_data_ingestion/output/test_audio/cleaned_documents.jsonl \
  --output-dir 02_chunking/output/test_audio

# Image
python 02_chunking/chunk_documents.py \
  --input 01_data_ingestion/output/test_image/cleaned_documents.jsonl \
  --output-dir 02_chunking/output/test_image

# Video
python 02_chunking/chunk_documents.py \
  --input 01_data_ingestion/output/test_video/cleaned_documents.jsonl \
  --output-dir 02_chunking/output/test_video