from gitsource import GithubRepositoryDataReader
reader = GithubRepositoryDataReader(
repo_owner="DataTalksClub",
repo_name="llm-zoomcamp",
commit_id="8c1834d",
allowed_extensions={"md"},
filename_filter=lambda path: "/lessons/" in path,
)
documents = [file.parse() for file in reader.read()]
from embedder import Embedder
embed = Embedder()
q1 = "How does approximate nearest neighbor search work?"
v1 = embed.encode(q1)
v1[0]
[0;93m2026-06-23 15:50:53.258506126 [W:onnxruntime:Default, device_discovery.cc:133 GetPciBusId] Skipping pci_bus_id for PCI path at "/sys/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/5620e0c7-8062-4dce-aeb7-520c7ef76171" because filename "5620e0c7-8062-4dce-aeb7-520c7ef76171" did not match expected pattern of [0-9a-f]+:[0-9a-f]+:[0-9a-f]+[.][0-9a-f]+[m
np.float64(-0.02058203437252893)
documents[0]
{'content': '# Introduction\n\nVideo: [Watch this lesson](https://www.youtube.com/watch?v=rQYyFxf1FWw&list=PL3MmuxUbc_hLZFNgSad56pDBKK8KO0XIv)\n\nIn this module, we\'ll build a working Retrieval-Augmented\nGeneration (RAG) system from scratch, step by step.\n\nWe write everything in plain Python. We build a small search index by\nhand and call the LLM ourselves. I want you to see every piece first.\nThat way you know what a framework does for you before you reach for\none.\n\nPlaces where you can find me:\n\n- [My substack](https://alexeyondata.substack.com/)\n- [LinkedIn](https://www.linkedin.com/in/agrigorev/)\n- [X](https://x.com/Al_Grigor)\n\n## LLMs\n\nAn LLM (Large Language Model) is a neural network trained on massive\namounts of text. Given a prompt, it generates a continuation - a\nplausible next piece of text.\n\nThink of your phone. When you type "how are" in WhatsApp, it suggests\n"you" as the next word. "How are you" is the most common continuation.\nYour phone uses a simple language model for that. It predicts the next\nword based on what you typed so far.\n\nA large language model does the same thing, but at a much larger scale.\nIt has billions of parameters and is trained on most of the text on the\ninternet. When it predicts the next word, it feels like you\'re talking\nto an intelligent being. It understands what you ask and gives\nmeaningful answers.\n\nIn this course, we treat LLMs as black boxes. We won\'t look inside or\ncover the theory, and we won\'t host a model ourselves. We use an LLM\nprovider and call it over an API. For us, an LLM is a box: text goes in,\ntext comes out.\n\nBut LLMs have limitations:\n\n- Knowledge cutoff: they only know what was in their training data.\n If you ask about something that happened after training, they won\'t\n know - or worse, they\'ll make something up.\n- No access to your data: they can\'t see your documents, databases,\n or internal systems unless you provide that information.\n- Hallucinations: they sometimes produce confident-sounding answers\n that are simply wrong.\n\n## The project\n\nRAG solves these problems by giving the LLM relevant documents at\nquestion time. We don\'t hope the model memorized the answer. We\nretrieve the right information and hand it to the LLM, and the model\ngenerates a grounded response. This lets us inject knowledge the model\nnever saw during training. That\'s why RAG is still the most common way\npeople use LLMs in the industry.\n\nTo make this concrete, we build a FAQ agent for our course. A student\nasks something like "when does the course start?" and the agent answers\nfrom the FAQ data we prepared.\n\nThis module has two parts.\n\nIn Part 1 (the next 9 lessons) we will:\n\n- Understand what RAG is and how it works\n- Build a search engine over a real FAQ dataset\n- Write a prompt that combines the user\'s question with search results\n- Wire it all together into a working RAG pipeline\n- Split ingestion and query into separate processes\n\nIn Part 2, we make the pipeline agentic. The LLM decides when and\nwhat to search, instead of running the same fixed flow every time.\n\nThe final code from this module is available in the\n[code/](../code/) directory.\n\n[← Back to module](../) | [Environment →](02-environment.md)',
'filename': '01-agentic-rag/lessons/01-intro.md'}
lst = list(filter(lambda x: x['filename'] == '02-vector-search/lessons/07-sqlitesearch-vector.md', documents))
v2 = embed.encode(lst[0]['content'])
v2.shape
(384,)
v2.dot(v1)
np.float64(0.36107027225589694)
from gitsource import chunk_documents
chunks = chunk_documents(documents, size=2000, step=1000)
len(chunks)
295
texts = [doc['content'] for doc in chunks]
texts[0]
'# Introduction\n\nVideo: [Watch this lesson](https://www.youtube.com/watch?v=rQYyFxf1FWw&list=PL3MmuxUbc_hLZFNgSad56pDBKK8KO0XIv)\n\nIn this module, we\'ll build a working Retrieval-Augmented\nGeneration (RAG) system from scratch, step by step.\n\nWe write everything in plain Python. We build a small search index by\nhand and call the LLM ourselves. I want you to see every piece first.\nThat way you know what a framework does for you before you reach for\none.\n\nPlaces where you can find me:\n\n- [My substack](https://alexeyondata.substack.com/)\n- [LinkedIn](https://www.linkedin.com/in/agrigorev/)\n- [X](https://x.com/Al_Grigor)\n\n## LLMs\n\nAn LLM (Large Language Model) is a neural network trained on massive\namounts of text. Given a prompt, it generates a continuation - a\nplausible next piece of text.\n\nThink of your phone. When you type "how are" in WhatsApp, it suggests\n"you" as the next word. "How are you" is the most common continuation.\nYour phone uses a simple language model for that. It predicts the next\nword based on what you typed so far.\n\nA large language model does the same thing, but at a much larger scale.\nIt has billions of parameters and is trained on most of the text on the\ninternet. When it predicts the next word, it feels like you\'re talking\nto an intelligent being. It understands what you ask and gives\nmeaningful answers.\n\nIn this course, we treat LLMs as black boxes. We won\'t look inside or\ncover the theory, and we won\'t host a model ourselves. We use an LLM\nprovider and call it over an API. For us, an LLM is a box: text goes in,\ntext comes out.\n\nBut LLMs have limitations:\n\n- Knowledge cutoff: they only know what was in their training data.\n If you ask about something that happened after training, they won\'t\n know - or worse, they\'ll make something up.\n- No access to your data: they can\'t see your documents, databases,\n or internal systems unless you provide that information.\n- Hallucinations: they sometimes produce confident-sounding answers\n that are simply '
from tqdm.auto import tqdm
import numpy as np
batch_size = 50
X = []
for i in tqdm(range(0, len(texts), batch_size)):
batch = texts[i:i + batch_size]
batch_vectors = embed.encode_batch(batch)
X.extend(batch_vectors)
X = np.array(X)
0%| | 0/6 [00:00<?, ?it/s]
scores = X.dot(v1)
idx = np.argmax(scores)
chunks[idx]
{'start': 1000,
'content': 'rch. We score\nthe query against every document and pick the top ones. It always finds\nthe true top matches, but it pays for that by touching everything.\n\nApproximate nearest neighbor (ANN) search takes a shortcut. Instead of\ncomparing against everything, it first narrows down to a region of\nlikely matches. Then it scores only within that region. It may miss the\nabsolute best match, but the results are still good and it\'s much\nfaster.\n\n```text\nNN (exact): compare query against ALL documents -> top 5\nANN (approx): narrow down to a region -> compare within region -> top 5\n```\n\n## sqlitesearch\n\nsqlitesearch is the persistent sibling of minsearch, and it solves both\nproblems at once.\n\nWe already used it in module 1 for persistent text search. It also does\nvector search through its `VectorSearchIndex` class. It stores vectors\nin SQLite, a real on-disk database, and uses ANN strategies for\nretrieval. Because the data lives on disk, one process can write the\nvectors and another can read them back.\n\nIf you didn\'t install it in the previous module, add it to your project:\n\n```bash\nuv add sqlitesearch\n```\n\n## Creating the index\n\nInitialize it:\n\n```python\nfrom sqlitesearch import VectorSearchIndex\n\nvs_index = VectorSearchIndex(\n keyword_fields=["course"],\n mode="ivf",\n db_path="faq_vectors2.db"\n)\n```\n\nsqlitesearch supports three ANN modes:\n\n- `lsh` (default): up to 100K vectors, random hyperplane projections\n- `ivf`: 10K-500K vectors, K-means clustering\n- `hnsw`: 10K-1M+ vectors, proximity graph (highest recall)\n\nFor our small dataset, `lsh` is fine. All modes use two-phase search:\napproximate candidate retrieval, then exact cosine similarity\nreranking.\n\n## Indexing the data\n\nFit the index with our vectors and documents:\n\n```python\nvs_index.fit(vectors, documents)\n```\n\nThe index is saved to `faq_vectors2.db`. Unlike minsearch, this file\npersists on disk. You can search immediately after indexing, or reopen\nthe index later without re-indexing.\n\n## Searching\n\nSearch ',
'filename': '02-vector-search/lessons/07-sqlitesearch-vector.md'}
from minsearch import VectorSearch
min_vector_index = VectorSearch(
keyword_fields=['filename']
)
min_vector_index.fit(X, chunks)
<minsearch.vector.VectorSearch at 0x7191b5042c30>
query = 'What metric do we use to evaluate a search engine?'
query_vector = embed.encode(query)
results = min_vector_index.search(query_vector, num_results=5)
results[0]
{'start': 0,
'content': "# Search Evaluation Metrics\n\nVideo: [Watch this lesson](https://www.youtube.com/watch?v=TuirMy3Pdbk&list=PL3MmuxUbc_hLZFNgSad56pDBKK8KO0XIv)\n\nIn the previous lesson, we computed relevance lists for search results.\nWe can turn those lists into metrics.\n\n## Hit Rate\n\nHit Rate (also called Recall@k) measures the fraction of queries where\nthe correct document appears anywhere in the results:\n\n```python\nexample = [\n [1, 0, 0, 0, 0],\n [0, 1, 0, 0, 0],\n [1, 0, 0, 0, 0],\n [0, 0, 0, 0, 0],\n [0, 1, 0, 0, 0],\n [1, 0, 0, 0, 0],\n [1, 0, 0, 0, 0],\n [1, 0, 0, 0, 0],\n [1, 0, 0, 0, 0],\n [0, 0, 1, 0, 0],\n [1, 0, 0, 0, 0],\n [1, 0, 0, 0, 0],\n [1, 0, 0, 0, 0],\n [1, 0, 0, 0, 0],\n [1, 0, 0, 0, 0],\n]\n```\n\nEach line is one query. If a line contains `1`, search found the\ncorrect document somewhere in the top 5 results. If the line contains\nonly zeros, search did not find the correct document.\n\nIn our setup, each query has one correct document, so Hit Rate and\nRecall@k mean the same thing.\n\nLet's calculate it:\n\n```python\ncnt = 0\n\nfor line in example:\n if 1 in line:\n cnt = cnt + 1\n\ncnt\n```\n\nThere are 14 hits. The example has 15 queries.\n\nThe Hit Rate is:\n\n```python\ncnt / len(example)\n# 0.933\n```\n\nThis means that search found the correct document for 93.3% of the\nqueries in this example.\n\nPut the same logic into a function:\n\n```python\ndef hit_rate(relevance):\n cnt = 0\n\n for line in relevance:\n if 1 in line:\n cnt = cnt + 1\n\n return cnt / len(relevance)\n```\n\nCheck it on the same example:\n\n```python\nhit_rate(example)\n# 0.933\n```\n\n## Mean Reciprocal Rank (MRR)\n\nHit Rate tells us if we found the right document, but not where it was.\n\nMRR also considers the position.\n\nFor each query, the score is based on the rank of the first correct\ndocument:\n\n- position 1: score is 1.0\n- position 2: score is 0.5\n- position 3: score is 0.333\n- not found: score is 0\n\nIn the example, most hits are at the first position. Some hits are\nlo",
'filename': '04-evaluation/lessons/05-search-metrics.md'}
from minsearch import Index
min_index = Index(
keyword_fields = ['filename'],
text_fields=['content']
)
min_index.fit(chunks)
<minsearch.minsearch.Index at 0x7191b514c710>
query = 'How do I store vectors in PostgreSQL?'
results = min_index.search(query, num_results=5)
results
[{'start': 4000,
'content': 'get 0.01.\n\nThe first score for `q1` vs `d` (0.32) is higher, so that query is more\nsimilar to the document about registration. The second score for `q2`\nvs `d` sits near 0, because installing Docker has nothing to do with\nregistration. A score near 0 means the two vectors are about as\ndifferent as they can be.\n\nThat\'s the whole idea behind vector search: similar texts get similar\nvectors, and a dot product tells us how similar.\n\n## Cosine similarity\n\nThe `all-MiniLM-L6-v2` model outputs normalized vectors - vectors with\nunit length. When both vectors are normalized, the dot product equals\ncosine similarity. That\'s why the model documentation says it "uses\ncosine similarity."\n\nCosine similarity measures the angle between two vectors, ignoring\ntheir length:\n\n- 1.0 = same direction (similar)\n- 0.0 = perpendicular (unrelated)\n- -1.0 = opposite direction (opposite meaning)\n\nFormally, if `theta` is the angle between two vectors, cosine similarity\nis `cos(theta)`:\n\n- `cos(0) = 1` - vectors point in the same direction\n- `cos(90) = 0` - vectors are perpendicular\n- `cos(180) = -1` - vectors point in opposite directions\n\nBecause our vectors are normalized, the dot product gives us cosine\nsimilarity directly. This is why we can use `v1.dot(dv)` to compare\ntexts.\n\nIn practice, we rarely get cosine similarity below 0. The embedding\nmodel maps text to a region of the vector space where most vectors\nhave positive components. There\'s no concept of "opposite meaning"\nthat maps to a vector pointing the other way.\n\n[← What is Vector Search](01-intro.md) | [Embedding Our Dataset →](03-embeddings-dataset.md)',
'filename': '02-vector-search/lessons/02-embeddings.md'},
{'start': 1000,
'content': 'arch](../../02-vector-search/lessons/04-vector-search.md).\n\n## How RAG Works in Kestra\n\nRAG has two phases. In the demo flows below they run back-to-back, but in production you\'d typically schedule them separately — ingest on a cadence, query on demand.\n\n```mermaid\ngraph LR\n subgraph Ingest ["Ingest (run once / on schedule)"]\n A[Fetch Docs] --> B[Create Embeddings]\n B --> C[Store in KV Store]\n end\n subgraph Query ["Query (run on demand)"]\n D[User Question] --> E[Find Similar Content]\n E --> F[Add Context to Prompt]\n F --> G[LLM Answer]\n end\n C --> E\n```\n\nIngest phase (run once, or on a schedule when your data changes):\n\n1. Fetch documents: load documentation, release notes, or other data sources\n2. Create embeddings: convert text into vectors using an embedding model\n3. Store embeddings: save vectors in Kestra\'s KV Store\n\n> Note: The flows store embeddings in Kestra\'s KV Store for simplicity. This is convenient for learning and small-scale demos, but it is not a replacement for a proper vector database. For any serious workload, e.g. larger document sets, low-latency retrieval, or production use, you should use a dedicated vector store. See [Module 2: Vector Search](../../02-vector-search/lessons/04-vector-search.md) for a deeper look at vector search in practice.\n\nQuery phase (runs every time a question is asked):\n\n4. Retrieve context: find the embeddings most similar to the user\'s question\n5. Augment the prompt: add the retrieved content to the LLM prompt\n6. Generate response: the LLM answers using real, grounded context\n\n## Example: Kestra Release Features\n\n### Step 1: Without RAG\n\nFlow: [`1_chat_without_rag.yaml`](../flows/1_chat_without_rag.yaml)\n\nThis flow asks Gemini: "Which features were released in Kestra 1.1?"\n\nWithout RAG, the model might hallucinate features that don\'t exist, provide outdated information, or give vague generic answers.\n\nImport and run this flow, then check the output — the response won\'t ',
'filename': '03-orchestration/lessons/05-rag.md'},
{'start': 0,
'content': '# Vector Search\n\nVideo: [Watch this lesson](https://www.youtube.com/watch?v=qyZgxTmC2cY&list=PL3MmuxUbc_hLZFNgSad56pDBKK8KO0XIv)\n\nIn module 1 we used keyword search with minsearch and sqlitesearch.\nIt matches exact words. If you search for "Docker", the document has\nto contain "Docker" to come back.\n\nBut look at these two questions:\n\n- "Can I still join the course after the start date?"\n- "Is it possible to enroll late?"\n\nThey mean the same thing, yet they share almost no words. A keyword\nengine struggles to match them. We need something that works on\nmeaning, not on the exact words.\n\nThat something is vector search. Instead of matching words, it matches\nideas.\n\n## The vector search process\n\nWe run vector search in two stages.\n\n1. Offline (indexing): we convert all documents into vectors (arrays\n of numbers) and store them in an index.\n2. Online (querying): we convert the user\'s query into a vector with\n the same model, then find the closest document vectors by similarity.\n\nAn embedding model produces these vectors. It\'s a neural network\ntrained to capture meaning, so texts that mean similar things land on\nsimilar vectors. We measure how close two vectors are with a distance\nmetric. The most common one is cosine similarity.\n\nCosine similarity measures the angle between two vectors:\n\n- Vectors pointing in the same direction: similarity close to 1\n (similar)\n- Vectors at right angles: similarity close to 0 (unrelated)\n- Vectors pointing in opposite directions: similarity close to -1\n (opposite meaning)\n\nThe larger the cosine similarity, the more similar the two texts are\nin meaning.\n\n## Keyword search vs vector search\n\nHere\'s how the two approaches differ:\n\n- Keyword search matches exact words. Vector search matches meaning.\n- Keyword search suits specific terms, IDs, and names. Vector search\n suits paraphrased questions and natural language.\n- Keyword search example: "pandas dataframe". Vector search example:\n "How do I work with tabular data?"\n- Keyword sear',
'filename': '02-vector-search/lessons/01-intro.md'},
{'start': 0,
'content': '# Retrieval Augmented Generation\n\nVideo: [RAG Workflows](https://youtu.be/FhGZV173xrk)\n\nAI Copilot solves the context problem for flow generation. But what about workflows that need to answer questions from your own data? That\'s where RAG comes in.\n\n> Note: Flows 1 and 2 use `{{ secret(\'GEMINI_API_KEY\') }}`. Flow 3 uses `{{ secret(\'OPENAI_API_KEY\') }}` and `{{ secret(\'TAVILY_API_KEY\') }}`. Make sure you\'ve completed the [setup instructions](03-setup.md) to configure the relevant secrets before running them.\n\n## What is RAG?\n\nRAG (Retrieval Augmented Generation) is a technique that retrieves relevant information from your data sources, augments the AI prompt with that context, and generates a response grounded in real data. This solves the hallucination problem by ensuring the AI has access to current, accurate information at query time.\n\nFor a deeper dive into RAG concepts, see [Module 1: Intro to RAG](../../01-agentic-rag/lessons/03-rag.md). For vector search, see [Module 2: Vector Search](../../02-vector-search/lessons/04-vector-search.md).\n\n## How RAG Works in Kestra\n\nRAG has two phases. In the demo flows below they run back-to-back, but in production you\'d typically schedule them separately — ingest on a cadence, query on demand.\n\n```mermaid\ngraph LR\n subgraph Ingest ["Ingest (run once / on schedule)"]\n A[Fetch Docs] --> B[Create Embeddings]\n B --> C[Store in KV Store]\n end\n subgraph Query ["Query (run on demand)"]\n D[User Question] --> E[Find Similar Content]\n E --> F[Add Context to Prompt]\n F --> G[LLM Answer]\n end\n C --> E\n```\n\nIngest phase (run once, or on a schedule when your data changes):\n\n1. Fetch documents: load documentation, release notes, or other data sources\n2. Create embeddings: convert text into vectors using an embedding model\n3. Store embeddings: save vectors in Kestra\'s KV Store\n\n> Note: The flows store embeddings in Kestra\'s KV Store for simplicity. This is convenient for learning and small-sc',
'filename': '03-orchestration/lessons/05-rag.md'},
{'start': 1000,
'content': 'dding model produces these vectors. It\'s a neural network\ntrained to capture meaning, so texts that mean similar things land on\nsimilar vectors. We measure how close two vectors are with a distance\nmetric. The most common one is cosine similarity.\n\nCosine similarity measures the angle between two vectors:\n\n- Vectors pointing in the same direction: similarity close to 1\n (similar)\n- Vectors at right angles: similarity close to 0 (unrelated)\n- Vectors pointing in opposite directions: similarity close to -1\n (opposite meaning)\n\nThe larger the cosine similarity, the more similar the two texts are\nin meaning.\n\n## Keyword search vs vector search\n\nHere\'s how the two approaches differ:\n\n- Keyword search matches exact words. Vector search matches meaning.\n- Keyword search suits specific terms, IDs, and names. Vector search\n suits paraphrased questions and natural language.\n- Keyword search example: "pandas dataframe". Vector search example:\n "How do I work with tabular data?"\n- Keyword search uses an inverted index (BM25, TF-IDF). Vector search\n uses a vector index based on cosine similarity.\n- Keyword search misses synonyms and paraphrases. Vector search misses\n exact term matches.\n\nVector search is usually better, but it adds a lot of operational\ncomplexity, and you\'ll feel that throughout this module. So my advice\nis to never start with vector search. Start with text search, and reach\nfor vectors once you can show they\'re worth the extra cost.\n\nIn practice the two work best together. Hybrid search combines them,\nand we cover it in the\n[Best Practices module](../../06-best-practices/lessons/02-hybrid-search.md).\n\n## Building vector search\n\nWe\'ll take the same FAQ dataset from module 1 and build vector search\nwith three tools:\n\n1. minsearch - in-memory vector search (simplest, good for\n experiments)\n2. sqlitesearch - persistent vector search backed by SQLite\n (production-friendly, same API as minsearch)\n3. PGVector - vector search in PostgreSQL (scalable, runs in\n',
'filename': '02-vector-search/lessons/01-intro.md'}]
query_vector = embed.encode(query)
results_vector = min_vector_index.search(query_vector, num_results=5)
results_vector
[{'start': 0,
'content': '# Vector Search with PGVector\n\nVideo: [Watch this lesson](https://www.youtube.com/watch?v=0P54MFyz-mc&list=PL3MmuxUbc_hLZFNgSad56pDBKK8KO0XIv)\n\nMany real databases can do vector search. Elasticsearch has it, and\nthere are dedicated stores like Qdrant and Chroma. We\'ll go with\nPostgres. Most of us already run it at work, and the data engineering\ncourse uses it too. The concept is the same as with sqlitesearch, only\nthe database under the hood changes.\n\n[pgvector](https://github.com/pgvector/pgvector) is the PostgreSQL\nextension that makes this work. Install it and Postgres can do vector\nsimilarity search. On top of that you get the usual production features,\nlike concurrent access, transactions, and large datasets.\n\nWe\'ll run Postgres with pgvector in Docker.\n\n## Starting Postgres with pgvector\n\nPull the image and start a container:\n\n```bash\ndocker run -it \\\n --name pgvector \\\n -e POSTGRES_USER=user \\\n -e POSTGRES_PASSWORD=pswd \\\n -e POSTGRES_DB=faq \\\n -v pgvector_data:/var/lib/postgresql/data \\\n -p 5432:5432 \\\n pgvector/pgvector:pg17\n```\n\nThis image has the pgvector extension pre-installed. The `-v` flag\ncreates a named volume so data persists across container restarts.\n\n## Installing the Python client\n\nInstall the driver:\n\n```bash\nuv add psycopg[binary]\n```\n\nWe\'ll use `psycopg` (v3) to connect and run queries. Note: this is\ndifferent from `psycopg2` - psycopg v3 supports `conn.execute()`\ndirectly without creating a cursor.\n\n## Preparing the data\n\nWe need the FAQ documents and their embeddings.\n\nHere\'s what we did in previous units as one script:\n\n```python\nfrom tqdm.auto import tqdm\n\nfrom ingest import load_faq_data\nfrom sentence_transformers import SentenceTransformer\n\nmodel = SentenceTransformer("all-MiniLM-L6-v2")\n\ndocuments = load_faq_data()\ntexts = [doc["question"] + " " + doc["answer"] for doc in documents]\n\nbatch_size = 50\nvectors = []\n\nfor i in tqdm(range(0, len(texts), batch_size)):\n batch = texts[i:i + batch_size]\n batch_vectors',
'filename': '02-vector-search/lessons/08-pgvector.md'},
{'start': 3000,
'content': 'PGVector:\n\n```python\ndef vec_to_str(vector):\n return "[" + ",".join(str(x) for x in vector) + "]"\n\nfor doc, vec in tqdm(zip(documents, vectors), total=len(documents)):\n conn.execute(\n """\n INSERT INTO documents (course, section, question, answer, embedding)\n VALUES (%s, %s, %s, %s, %s::vector)\n """,\n (doc["course"], doc["section"], doc["question"], doc["answer"],\n vec_to_str(vec))\n )\n\nconn.commit()\n```\n\nWe loop over the documents and insert each one with its embedding. We\nhand Postgres the vector as text, so the `::vector` cast tells it to\nparse that string back into a vector. We call `conn.commit()` to persist\nthe changes.\n\n## Searching with cosine similarity\n\nSearch with a query:\n\n```python\nquery = "I just discovered the course. Can I still join it?"\nquery_vector = model.encode(query)\nquery_str = vec_to_str(query_vector)\n```\n\nSearch for the most similar documents:\n\n```python\nresults = conn.execute(\n """\n SELECT course, question, answer,\n 1 - (embedding <=> %s::vector) AS similarity\n FROM documents\n ORDER BY embedding <=> %s::vector\n LIMIT 5\n """,\n (query_str, query_str)\n).fetchall()\n\nfor row in results:\n print(f"[{row[0]}] {row[1]} (similarity: {row[3]:.4f})")\n```\n\nThe `<=>` operator computes cosine distance (1 - cosine similarity).\nWe order by ascending distance, so the closest vectors come first.\n\n## Filtering by course\n\nBecause this is plain SQL, filtering by course is one extra `WHERE`\nclause:\n\n```python\nresults = conn.execute(\n """\n SELECT course, question, answer,\n 1 - (embedding <=> %s::vector) AS similarity\n FROM documents\n WHERE course = %s\n ORDER BY embedding <=> %s::vector\n LIMIT 5\n """,\n (query_str, "llm-zoomcamp", query_str)\n).fetchall()\n```\n\n## Creating an index for faster search\n\nSo far this runs brute-force search, comparing our query against every\nrow. For our small dataset that\'s fine.\n\nFor a larger one, create an HNSW index to ',
'filename': '02-vector-search/lessons/08-pgvector.md'},
{'start': 2000,
'content': 'ale demos, but it is not a replacement for a proper vector database. For any serious workload, e.g. larger document sets, low-latency retrieval, or production use, you should use a dedicated vector store. See [Module 2: Vector Search](../../02-vector-search/lessons/04-vector-search.md) for a deeper look at vector search in practice.\n\nQuery phase (runs every time a question is asked):\n\n4. Retrieve context: find the embeddings most similar to the user\'s question\n5. Augment the prompt: add the retrieved content to the LLM prompt\n6. Generate response: the LLM answers using real, grounded context\n\n## Example: Kestra Release Features\n\n### Step 1: Without RAG\n\nFlow: [`1_chat_without_rag.yaml`](../flows/1_chat_without_rag.yaml)\n\nThis flow asks Gemini: "Which features were released in Kestra 1.1?"\n\nWithout RAG, the model might hallucinate features that don\'t exist, provide outdated information, or give vague generic answers.\n\nImport and run this flow, then check the output — the response won\'t be accurate.\n\n### Step 2: With RAG\n\nFlow: [`2_chat_with_rag.yaml`](../flows/2_chat_with_rag.yaml)\n\nThis flow:\n\n1. Ingests the Kestra 1.1 release blog post from GitHub\n2. Creates embeddings using Gemini\'s embedding model\n3. Stores embeddings in Kestra\'s KV Store\n4. Asks the LLM the same question with RAG enabled\n5. Returns an accurate response with real features from that release\n\nImport and run `2_chat_with_rag.yaml` and compare the output quality against the previous flow.\n\n## Extending RAG with web search\n\nThe examples above use static RAG — documents are ingested once and stored in the KV Store. Kestra also supports web search as a retriever, which fetches live results at query time and passes them as context to the LLM.\n\nFlow: [`3_rag_with_websearch.yaml`](../flows/3_rag_with_websearch.yaml)\n\n> Note: This flow uses OpenAI as its AI provider. To run it, you\'ll need an OpenAI API key:\n>\n> 1. Visit [platform.openai.com](https://platform.openai.com/home) and sign in or create an accoun',
'filename': '03-orchestration/lessons/05-rag.md'},
{'start': 1000,
'content': 'ar/lib/postgresql/data \\\n -p 5432:5432 \\\n pgvector/pgvector:pg17\n```\n\nThis image has the pgvector extension pre-installed. The `-v` flag\ncreates a named volume so data persists across container restarts.\n\n## Installing the Python client\n\nInstall the driver:\n\n```bash\nuv add psycopg[binary]\n```\n\nWe\'ll use `psycopg` (v3) to connect and run queries. Note: this is\ndifferent from `psycopg2` - psycopg v3 supports `conn.execute()`\ndirectly without creating a cursor.\n\n## Preparing the data\n\nWe need the FAQ documents and their embeddings.\n\nHere\'s what we did in previous units as one script:\n\n```python\nfrom tqdm.auto import tqdm\n\nfrom ingest import load_faq_data\nfrom sentence_transformers import SentenceTransformer\n\nmodel = SentenceTransformer("all-MiniLM-L6-v2")\n\ndocuments = load_faq_data()\ntexts = [doc["question"] + " " + doc["answer"] for doc in documents]\n\nbatch_size = 50\nvectors = []\n\nfor i in tqdm(range(0, len(texts), batch_size)):\n batch = texts[i:i + batch_size]\n batch_vectors = model.encode(batch)\n vectors.extend(batch_vectors)\n```\n\n\nNow we connect to Postgres:\n\n```python\nimport psycopg\n\nconn = psycopg.connect(\n "postgresql://user:pswd@localhost:5432/faq"\n)\nconn.execute("CREATE EXTENSION IF NOT EXISTS vector")\n```\n\nThe second line activates pgvector. The Docker image we started isn\'t\nplain Postgres, it ships the extension inside, and this turns it on. It\nadds the `vector` column type and the similarity search operators.\n\n## Creating a table\n\nCreate a table for storing documents with their embeddings:\n\n```python\nconn.execute("""\n DROP TABLE IF EXISTS documents\n""")\n\nconn.execute("""\n CREATE TABLE documents (\n id SERIAL PRIMARY KEY,\n course TEXT,\n section TEXT,\n question TEXT,\n answer TEXT,\n embedding vector(384)\n )\n""")\n```\n\nThe `vector(384)` column stores our 384-dimensional embeddings from\n`all-MiniLM-L6-v2`.\n\n## Inserting documents with embeddings\n\nLet\'s insert the documents and their vectors into ',
'filename': '02-vector-search/lessons/08-pgvector.md'},
{'start': 2000,
'content': ' = model.encode(batch)\n vectors.extend(batch_vectors)\n```\n\n\nNow we connect to Postgres:\n\n```python\nimport psycopg\n\nconn = psycopg.connect(\n "postgresql://user:pswd@localhost:5432/faq"\n)\nconn.execute("CREATE EXTENSION IF NOT EXISTS vector")\n```\n\nThe second line activates pgvector. The Docker image we started isn\'t\nplain Postgres, it ships the extension inside, and this turns it on. It\nadds the `vector` column type and the similarity search operators.\n\n## Creating a table\n\nCreate a table for storing documents with their embeddings:\n\n```python\nconn.execute("""\n DROP TABLE IF EXISTS documents\n""")\n\nconn.execute("""\n CREATE TABLE documents (\n id SERIAL PRIMARY KEY,\n course TEXT,\n section TEXT,\n question TEXT,\n answer TEXT,\n embedding vector(384)\n )\n""")\n```\n\nThe `vector(384)` column stores our 384-dimensional embeddings from\n`all-MiniLM-L6-v2`.\n\n## Inserting documents with embeddings\n\nLet\'s insert the documents and their vectors into PGVector:\n\n```python\ndef vec_to_str(vector):\n return "[" + ",".join(str(x) for x in vector) + "]"\n\nfor doc, vec in tqdm(zip(documents, vectors), total=len(documents)):\n conn.execute(\n """\n INSERT INTO documents (course, section, question, answer, embedding)\n VALUES (%s, %s, %s, %s, %s::vector)\n """,\n (doc["course"], doc["section"], doc["question"], doc["answer"],\n vec_to_str(vec))\n )\n\nconn.commit()\n```\n\nWe loop over the documents and insert each one with its embedding. We\nhand Postgres the vector as text, so the `::vector` cast tells it to\nparse that string back into a vector. We call `conn.commit()` to persist\nthe changes.\n\n## Searching with cosine similarity\n\nSearch with a query:\n\n```python\nquery = "I just discovered the course. Can I still join it?"\nquery_vector = model.encode(query)\nquery_str = vec_to_str(query_vector)\n```\n\nSearch for the most similar documents:\n\n```python\nresults = conn.execute(\n """\n SELECT course, questio',
'filename': '02-vector-search/lessons/08-pgvector.md'}]
vector_set = set([doc['filename'] for doc in results_vector])
vector_set
{'02-vector-search/lessons/08-pgvector.md',
'03-orchestration/lessons/05-rag.md'}
text_set = set([doc['filename'] for doc in results])
text_set
{'02-vector-search/lessons/01-intro.md',
'02-vector-search/lessons/02-embeddings.md',
'03-orchestration/lessons/05-rag.md'}
vector_set - text_set
{'02-vector-search/lessons/08-pgvector.md'}
def rrf(result_lists, k=60, num_results=5):
scores = {}
docs = {}
for results in result_lists:
for rank, doc in enumerate(results):
key = (doc["filename"], doc["start"])
scores[key] = scores.get(key, 0) + 1 / (k + rank)
docs[key] = doc
ranked = sorted(scores, key=scores.get, reverse=True)
return [docs[key] for key in ranked[:num_results]]
query = 'How do I give the model access to tools?'
text_results = min_index.search(query, num_results=5)
text_results
[{'start': 0,
'content': '# The Agentic Loop\n\nVideo: [Watch this lesson](https://www.youtube.com/watch?v=ePlQUcTPPjw&list=PL3MmuxUbc_hLZFNgSad56pDBKK8KO0XIv)\n\nIn the previous lesson, we did function calling by hand. We sent a\nmessage and got back a function call. We ran it, sent the result back,\nand got the answer.\n\nThat works for one function call. It breaks down when the model wants\nto search several times, or when the first search misses the answer.\nWe don\'t know in advance how many calls the model will want. So we\nneed a loop that keeps calling the model and running tools until it\'s\ndone. An agent is exactly that.\n\n## Anatomy of an agent\n\nWith the LLM in the driver\'s seat, we have an agent. It\'s an AI\nassistant whose goal is to help the user.\n\nAn agent has three parts:\n\n- Instructions, the role and behavior we want. We pass this as the\n `developer` message. The better the instructions, the better the\n agent helps.\n- Tools, the functions the agent can call to carry out the task. For\n us that\'s only `search`.\n- Memory, the message history. We append every prompt, every model\n output, and every tool result. The agent reads this to know what it\n has already tried.\n\nEverything below is the code that wires these three together inside a\nloop.\n\n## A developer prompt\n\nSo far we\'ve relied on the model to figure out when to search. We make\nthat more reliable with a `developer` message that spells out how to\nbehave. This is where we give the agent its role. The same message\nalso pushes it toward multiple searches, so we get to watch the loop\nrun more than once.\n\n```python\ninstructions = """\nYou\'re a course teaching assistant.\nYou\'re given a question from a course student and your task is to answer it.\n\nIf you want to look up information, use the search function. \nUse as many keywords from the user question as possible when making first requests.\n\nMake multiple searches.\n\nTry to expand your search by using new keywords\nbased on the results you get from the search.\n\nAt the end, ask if there are o',
'filename': '01-agentic-rag/lessons/14-agentic-loop.md'},
{'start': 4000,
'content': ' function. `parameters` is a JSON schema\nfor the arguments, and we mark `query` as required so the model always\nfills it in.\n\n## Sending the question with the tool\n\nNow we send the same question as before, but this time we include the\ntool in the request:\n\n```python\nresponse = openai_client.responses.create(\n model="gpt-5.4-mini",\n input=messages,\n tools=[search_tool],\n)\n\nresponse.output\n```\n\nLook at the output. Instead of a message with the answer, the response\ncontains a `function_call` entry. The model decided it needs to search\nthe FAQ before answering. Rather than reply, it asked us to run the\nsearch function first.\n\nLook at the arguments too. The model didn\'t pass our question\nverbatim. It judged the raw question wasn\'t the best query to search\nwith. So it rewrote our enrollment question into search keywords like\n"enroll late join course".\n\n## Executing the function and sending the result back\n\nThe function call contains JSON arguments. We parse them, call our\n`search` function, and serialize the result.\n\n```python\nimport json\n\ncall = response.output[0]\nargs = json.loads(call.arguments)\n\nresults = search(**args)\nresult_json = json.dumps(results, indent=2)\n```\n\nNow we send this result back to the model. First, we add the model\'s\noutput to the conversation history - the model needs to see its own\nfunction call. Then we add the tool result.\n\n```python\nmessages.extend(response.output)\n\nmessages.append({\n "type": "function_call_output",\n "call_id": call.call_id,\n "output": result_json,\n})\n```\n\nThe `call_id` links the tool result to the specific function call the\nmodel requested. If the model makes multiple function calls in one\nturn, each one gets its own `call_id`.\n\n## Asking the model again\n\nWe call the API a second time with the expanded history:\n\n```python\nresponse = openai_client.responses.create(\n model="gpt-5.4-mini",\n input=messages,\n tools=[search_tool],\n)\n\nresponse.output_text\n```\n\nThis time the model has the original question, ',
'filename': '01-agentic-rag/lessons/13-function-calling.md'},
{'start': 5000,
'content': 'unction, and serialize the result.\n\n```python\nimport json\n\ncall = response.output[0]\nargs = json.loads(call.arguments)\n\nresults = search(**args)\nresult_json = json.dumps(results, indent=2)\n```\n\nNow we send this result back to the model. First, we add the model\'s\noutput to the conversation history - the model needs to see its own\nfunction call. Then we add the tool result.\n\n```python\nmessages.extend(response.output)\n\nmessages.append({\n "type": "function_call_output",\n "call_id": call.call_id,\n "output": result_json,\n})\n```\n\nThe `call_id` links the tool result to the specific function call the\nmodel requested. If the model makes multiple function calls in one\nturn, each one gets its own `call_id`.\n\n## Asking the model again\n\nWe call the API a second time with the expanded history:\n\n```python\nresponse = openai_client.responses.create(\n model="gpt-5.4-mini",\n input=messages,\n tools=[search_tool],\n)\n\nresponse.output_text\n```\n\nThis time the model has the original question, its own decision to\ncall `search`, and the FAQ results. It can now produce a proper\ncourse-specific answer.\n\nWe have to send the whole history because LLMs are stateless between\nAPI calls. The memory is the list you send as `input`. If you send\nonly the tool result, the model has no idea what\'s going on. So on\nthis second call we replay everything we have so far. That means the\nquestion, the decision to call `search`, and the result we got back.\n\nThat\'s the full function-calling loop for a single turn. With plain\nRAG we made one call, and here we make two. Turning RAG agentic means\nmore round-trips.\n\nPeople call this pattern "agentic RAG", "tool use", or "function\ncalling". The idea behind all of them is the same. The LLM decides\nwhich tools to call.\n\n## Token usage and cost\n\nWe just made two API calls instead of one. Each call we send to the\nmodel costs money, so it\'s worth checking how much one tool-using turn\nactually costs.\n\nThe response has a `usage` field with the token counts:\n\n',
'filename': '01-agentic-rag/lessons/13-function-calling.md'},
{'start': 1000,
'content': 'decides when to call it and what to search for.\n\nThe same typo question now goes like this:\n\n```mermaid\nflowchart TD\n U([User: How do I run Olama?])\n L1[LLM: I\'ll search for \'Olama\']\n S1[search - Olama - no useful results]\n L2[LLM: Hmm, no results. Maybe a typo for \'Ollama\'?]\n S2[search - Ollama - found results!]\n A([LLM: Here\'s how to run Ollama locally...])\n\n U --> L1 --> S1 --> L2 --> S2 --> A\n```\n\nThe LLM searched, saw the results were bad, and decided to try again\nwith a different query. It made that decision on its own. We didn\'t\nwrite any code to handle typos.\n\nThe difference is about who makes the decisions:\n\n- With RAG, the developer decides. We fix the steps up front, so\n search always runs once with the exact user query.\n- With an agent, the LLM decides. It chooses which actions to take\n and when to stop.\n\nThe mechanism that makes this possible is function calling, and that\'s\nwhat the rest of this lesson is about.\n\n## Asking without tools\n\nFirst, let\'s see what the LLM does without any tools. We ask it a\ncourse-specific question and look at the answer.\n\n```python\nmessages = [\n {"role": "user", "content": "I just discovered the course. Can I join it?"}\n]\n\nresponse = openai_client.responses.create(\n model="gpt-5.4-mini",\n input=messages,\n)\n\nresponse.output_text\n```\n\nThe model answers from its general knowledge, something like "it\ndepends on the course" or "check the course website". It doesn\'t know\nabout our FAQ, so the answer is vague and not helpful. This is exactly\nwhy we need RAG, and why we want to hand the model a tool.\n\n## Defining the tool\n\nFirst we define a top-level `search` function that queries the `index`\ndirectly. The model will reference it by this name. We keep the Python\nfunction and the tool name aligned so the dispatch is easier later.\n\n```python\ndef search(query):\n boost_dict = {"question": 3.0, "section": 0.5}\n filter_dict = {"course": "llm-zoomcamp"}\n\n return index.search(\n query,\n ',
'filename': '01-agentic-rag/lessons/13-function-calling.md'},
{'start': 3000,
'content': 'the output. The model returns the\nsame structure every time. We can access the generated questions\ndirectly instead of parsing text manually.\n\nWe want the output as a list of strings, so we define that structure\nwith a Pydantic model:\n\n```python\nfrom pydantic import BaseModel\n\nclass Questions(BaseModel):\n questions: list[str]\n```\n\nThe instructions for the LLM:\n\n```python\ndata_gen_instructions = """\nYou emulate a student who\'s taking our course.\nFormulate 5 questions this student might ask based on a FAQ record. The record\nshould contain the answer to the questions, and the questions should be complete and not too short.\nIf possible, use as fewer words as possible from the record.\n\nThe output should resemble how people ask questions\non the internet. Not too formal, not too short, not too long.\n""".strip()\n```\n\nWe ask the LLM to use different wording from the original document.\nThis makes the evaluation more realistic - real users won\'t phrase\ntheir questions the same way as the FAQ.\n\nCall the LLM for one document:\n\n```python\nfrom dotenv import load_dotenv\nfrom openai import OpenAI\n\nload_dotenv()\nopenai_client = OpenAI()\n```\n\nPrepare the document as JSON:\n\n```python\nimport json\n\nuser_prompt = json.dumps(doc)\n```\n\nCreate the messages:\n\n```python\nmessages = [\n {"role": "developer", "content": data_gen_instructions},\n {"role": "user", "content": user_prompt}\n]\n```\n\nUntil now we called `responses.create` and read `response.output_text`.\nFor structured output we switch to `responses.parse` and pass\n`text_format=Questions`, which tells the API to return our class instead\nof free text.\n\nCall the model:\n\n```python\nresponse = openai_client.responses.parse(\n model="gpt-5.4-mini",\n input=messages,\n text_format=Questions\n)\n```\n\nThe parsed object is available in `response.output_parsed`:\n\n```python\nresult = response.output_parsed\n\nprint(result)\n```\n\nWe can access the list directly:\n\n```python\nprint(result.questions)\n```\n\nYou should see 5 questions that relate to ',
'filename': '04-evaluation/lessons/02-ground-truth.md'}]
query_vector = embed.encode(query)
vector_results = min_vector_index.search(query_vector, num_results=5)
vector_results
[{'start': 2000,
'content': 'wrong.\n\n## The project\n\nRAG solves these problems by giving the LLM relevant documents at\nquestion time. We don\'t hope the model memorized the answer. We\nretrieve the right information and hand it to the LLM, and the model\ngenerates a grounded response. This lets us inject knowledge the model\nnever saw during training. That\'s why RAG is still the most common way\npeople use LLMs in the industry.\n\nTo make this concrete, we build a FAQ agent for our course. A student\nasks something like "when does the course start?" and the agent answers\nfrom the FAQ data we prepared.\n\nThis module has two parts.\n\nIn Part 1 (the next 9 lessons) we will:\n\n- Understand what RAG is and how it works\n- Build a search engine over a real FAQ dataset\n- Write a prompt that combines the user\'s question with search results\n- Wire it all together into a working RAG pipeline\n- Split ingestion and query into separate processes\n\nIn Part 2, we make the pipeline agentic. The LLM decides when and\nwhat to search, instead of running the same fixed flow every time.\n\nThe final code from this module is available in the\n[code/](../code/) directory.\n\n[← Back to module](../) | [Environment →](02-environment.md)',
'filename': '01-agentic-rag/lessons/01-intro.md'},
{'start': 1000,
'content': 'swer.\n\n## Loading the documents\n\nWe\'ll use helper files from module 01 and this module.\n\nIf you don\'t have them in your notebook directory, download them:\n\n```bash\nPREFIX=https://raw.githubusercontent.com/DataTalksClub/llm-zoomcamp/main\n\nwget ${PREFIX}/01-agentic-rag/code/ingest.py\nwget ${PREFIX}/01-agentic-rag/code/rag_helper.py\nwget ${PREFIX}/04-evaluation/code/evaluation_utils.py\n```\n\nThen load the FAQ data:\n\n```python\nfrom ingest import load_faq_data\ndocuments = load_faq_data()\n```\n\nWe\'ll generate questions only for the LLM Zoomcamp FAQ. The full FAQ\ndataset contains documents from multiple courses. Generating five\nquestions for every document would take longer and cost more.\n\n```python\ndocuments_llm = []\n\nfor doc in documents:\n if doc["course"] == "llm-zoomcamp":\n documents_llm.append(doc)\n\nlen(documents_llm)\n```\n\nWe\'ll use these documents from now on so let\'s name them as `documents`\n\n```python\ndocuments = documents_llm\n```\n\nEach document already has an `id` field:\n\n```python\ndoc = documents[0]\nprint(doc["id"])\nprint(doc["question"])\nprint(doc["answer"])\n```\n\nThe ID becomes the label in our ground truth dataset. We generate\nquestions from a document, so we know that this document holds the\nanswer. Later, search evaluation checks whether search brings back the\ndocument with this ID.\n\nThis is why every record needs a stable ID. If you can\'t uniquely\nidentify a document, you can\'t tell whether search retrieved the right\none. When you build your own evaluation set, assign an ID to each record\nin your knowledge base first.\n\n## Generating questions with structured output\n\nWe use an LLM to generate questions for each document.\n\nThis is the first time we\'re using structured output in the course.\nWith structured output, we ask the LLM to return data in a specific\nformat instead of free-form text. For example, instead of getting a\nparagraph that contains questions, we can ask for a Python object with\na `questions` field.\n\nThis is useful when code will process ',
'filename': '04-evaluation/lessons/02-ground-truth.md'},
{'start': 0,
'content': "# Other Frameworks\n\nVideo: [Watch this lesson](https://www.youtube.com/watch?v=4yiCbKX9RhI&list=PL3MmuxUbc_hLZFNgSad56pDBKK8KO0XIv)\n\nThe concepts you learned in Part 2 are the same across frameworks.\nFunction calling, the agent loop, and tool definitions all wrap the\nsame pattern. Send messages, run any function calls, and repeat until\nthe model is done.\n\nYou now understand how the loop works. So you can pick up any\nproduction framework and know what it's doing under the hood. I kept\nthis module framework agnostic on purpose, so you can explore and pick\nthe one you like.\n\nHere are some frameworks worth a look:\n\n## OpenAI Agents SDK\n\nThe official SDK from OpenAI for building agents. It uses the same\nResponses API we used throughout this module. It supports tool\ndefinition, multi-turn conversations, and handoffs between agents.\n\n```bash\nuv add openai-agents\n```\n\nGood choice if you're already using OpenAI and want something\nofficial and well-maintained.\n\n## PydanticAI\n\nA type-safe agent framework that supports multiple LLM providers.\nTools are plain Python functions with type hints, no wrappers needed.\nSwitching providers is as simple as changing the model string.\n\n```bash\nuv add pydantic-ai\n```\n\nThis is my personal favorite. The reason isn't the type safety, since\nthe other frameworks have that too. It's how it feels to use, and the\nteam behind Pydantic. Good choice if you want type safety and\nmulti-provider support.\n\n## LangChain / LangGraph\n\nA popular framework with lots of integrations. LangChain handles the\nbasics, and LangGraph adds graph-based workflows for more complex\nagent patterns.\n\nGood choice if you need lots of integrations (vector stores, document\nloaders, etc.) and a large community.\n\n## Google ADK\n\nThe Agent Development Kit from Google. If you plan to use Gemini\nmodels, this is the one I'd reach for. It exposes the same building\nblocks we've seen, like tools, instructions, and sessions. It also\nintegrates with Google Cloud.\n\nGood choice if your stack i",
'filename': '01-agentic-rag/lessons/16-other-frameworks.md'},
{'start': 2000,
'content': ' num_results=5,\n boost_dict={"question": 3.0, "section": 0.5},\n filter_dict={"course": "llm-zoomcamp"}\n )\n```\n\nThen register it without passing a schema:\n\n```python\nagent_tools = Tools()\nagent_tools.add_tool(search)\n```\n\nYou can look at what ToyAIKit produced.\n\n```python\nagent_tools.get_tools()\n```\n\nThe output is the same JSON schema we hand-wrote in the function\ncalling lesson. ToyAIKit generated it from the docstring and the type\nhint.\n\nEvery modern agent framework does this same trick. It reads a typed\nPython function with a docstring and builds the schema from it. The\nOpenAI Agents SDK, PydanticAI, LangChain and Google ADK all work this\nway. You write the tool and the framework figures out how to describe\nit.\n\n## The chat interface and runner\n\nCreate the chat interface and a callback, then build the runner:\n\n```python\nchat_interface = IPythonChatInterface()\ncallback = DisplayingRunnerCallback(chat_interface)\n\nrunner = OpenAIResponsesRunner(\n tools=agent_tools,\n developer_prompt=instructions,\n chat_interface=chat_interface,\n llm_client=OpenAIClient(model="gpt-5.4-mini")\n)\n```\n\nThe `chat_interface` handles display in the notebook. The `callback`\nrenders model messages and tool calls as they happen. The runner runs\nthe agent loop, the same `while True` we wrote by hand. It sends\nmessages, executes function calls, adds tool outputs back, and repeats\nuntil the model is done.\n\nWe pick `gpt-5.4-mini` here on purpose. Without it, ToyAIKit falls\nback to a smaller, faster default that doesn\'t follow the instructions\nas reliably.\n\n## Running one prompt\n\nRun a single prompt:\n\n```python\nresult = runner.loop(\n prompt="How do I run Olama locally?",\n callback=callback,\n)\n```\n\nWe used the typo "Olama" on purpose. The agent searches and gets poor\nresults, then retries with "Ollama". The recovery is the same as the\nhandwritten loop. The notebook output is nicer to watch. Each tool\ncall and message renders inline, so you can look at every search\nr',
'filename': '01-agentic-rag/lessons/15-frameworks.md'},
{'start': 4000,
'content': ' function. `parameters` is a JSON schema\nfor the arguments, and we mark `query` as required so the model always\nfills it in.\n\n## Sending the question with the tool\n\nNow we send the same question as before, but this time we include the\ntool in the request:\n\n```python\nresponse = openai_client.responses.create(\n model="gpt-5.4-mini",\n input=messages,\n tools=[search_tool],\n)\n\nresponse.output\n```\n\nLook at the output. Instead of a message with the answer, the response\ncontains a `function_call` entry. The model decided it needs to search\nthe FAQ before answering. Rather than reply, it asked us to run the\nsearch function first.\n\nLook at the arguments too. The model didn\'t pass our question\nverbatim. It judged the raw question wasn\'t the best query to search\nwith. So it rewrote our enrollment question into search keywords like\n"enroll late join course".\n\n## Executing the function and sending the result back\n\nThe function call contains JSON arguments. We parse them, call our\n`search` function, and serialize the result.\n\n```python\nimport json\n\ncall = response.output[0]\nargs = json.loads(call.arguments)\n\nresults = search(**args)\nresult_json = json.dumps(results, indent=2)\n```\n\nNow we send this result back to the model. First, we add the model\'s\noutput to the conversation history - the model needs to see its own\nfunction call. Then we add the tool result.\n\n```python\nmessages.extend(response.output)\n\nmessages.append({\n "type": "function_call_output",\n "call_id": call.call_id,\n "output": result_json,\n})\n```\n\nThe `call_id` links the tool result to the specific function call the\nmodel requested. If the model makes multiple function calls in one\nturn, each one gets its own `call_id`.\n\n## Asking the model again\n\nWe call the API a second time with the expanded history:\n\n```python\nresponse = openai_client.responses.create(\n model="gpt-5.4-mini",\n input=messages,\n tools=[search_tool],\n)\n\nresponse.output_text\n```\n\nThis time the model has the original question, ',
'filename': '01-agentic-rag/lessons/13-function-calling.md'}]
results = rrf([vector_results, text_results])
results[0]
{'start': 4000,
'content': ' function. `parameters` is a JSON schema\nfor the arguments, and we mark `query` as required so the model always\nfills it in.\n\n## Sending the question with the tool\n\nNow we send the same question as before, but this time we include the\ntool in the request:\n\n```python\nresponse = openai_client.responses.create(\n model="gpt-5.4-mini",\n input=messages,\n tools=[search_tool],\n)\n\nresponse.output\n```\n\nLook at the output. Instead of a message with the answer, the response\ncontains a `function_call` entry. The model decided it needs to search\nthe FAQ before answering. Rather than reply, it asked us to run the\nsearch function first.\n\nLook at the arguments too. The model didn\'t pass our question\nverbatim. It judged the raw question wasn\'t the best query to search\nwith. So it rewrote our enrollment question into search keywords like\n"enroll late join course".\n\n## Executing the function and sending the result back\n\nThe function call contains JSON arguments. We parse them, call our\n`search` function, and serialize the result.\n\n```python\nimport json\n\ncall = response.output[0]\nargs = json.loads(call.arguments)\n\nresults = search(**args)\nresult_json = json.dumps(results, indent=2)\n```\n\nNow we send this result back to the model. First, we add the model\'s\noutput to the conversation history - the model needs to see its own\nfunction call. Then we add the tool result.\n\n```python\nmessages.extend(response.output)\n\nmessages.append({\n "type": "function_call_output",\n "call_id": call.call_id,\n "output": result_json,\n})\n```\n\nThe `call_id` links the tool result to the specific function call the\nmodel requested. If the model makes multiple function calls in one\nturn, each one gets its own `call_id`.\n\n## Asking the model again\n\nWe call the API a second time with the expanded history:\n\n```python\nresponse = openai_client.responses.create(\n model="gpt-5.4-mini",\n input=messages,\n tools=[search_tool],\n)\n\nresponse.output_text\n```\n\nThis time the model has the original question, ',
'filename': '01-agentic-rag/lessons/13-function-calling.md'}