A walkthrough of how the First Aid QA Assistant works, with real input/output examples, a tour of the UI, and instructions for using the Python API directly.
Your question
│
▼ 1. Query rewriting
│ The LLM reformulates informal phrasing into precise, keyword-rich text
│ to improve search recall.
│
▼ 2. Hybrid search (BM25 + HNSW vector, fused with RRF)
│ Both a text index and a vector index are searched in parallel.
│ Results are merged using Reciprocal Rank Fusion so the best
│ matching Q&A pairs from 2,775 entries surface regardless of
│ whether the query matches by keyword or by meaning.
│
▼ 3. Prompt building
│ The top 10 retrieved Q&A pairs are assembled into a context block.
│ The LLM is instructed to answer only from that context.
│
▼ 4. LLM answer (Mistral ministral-3b)
│ A concise, grounded answer is generated.
│
▼ 5. Automatic quality check (LLM-as-judge)
│ A second LLM call scores the answer as RELEVANT / PARTLY_RELEVANT /
│ NON_RELEVANT and writes the verdict to PostgreSQL.
│
▼ 6. Persistence
Conversation saved to PostgreSQL.
OTel spans (latency, token counts) saved to SQLite.
User +1 / -1 feedback saved to PostgreSQL.
Start the app:
uv run streamlit run src/app.py
Open http://localhost:8501.
The input field accepts any natural-language question. You do not need to phrase it formally — the query rewriter handles that automatically.
Enter your question:
┌─────────────────────────────────────────────────────────────────┐
│ my friend got stung by a stingray, should I use cold water? │
└─────────────────────────────────────────────────────────────────┘
[ Ask ]
After clicking Ask the panel shows:
✅ Completed!
No, cold water should not be used to treat a stingray sting. Applying cold water
can exacerbate the pain and discomfort. The best treatment is to wash the affected
area with warm water (as hot as the person can tolerate) for at least 20 minutes.
Remove any visible barbs carefully with tweezers. Seek medical attention if the
sting is severe or if symptoms worsen.
Response time: 1.84s
Prompt tokens: 1,203
Completion tokens: 118
Cost: $0.0001
Relevance: RELEVANT
Explanation: The answer directly addresses the question about stingray sting
treatment and correctly advises against cold water.
[ +1 ] [ -1 ]
Click +1 if the answer was helpful, -1 if not. The score is stored in the feedback table alongside the auto-judge verdict.
| Raw input | I just got hit by a car and my friend is lying there bleeding badly — do I need to drag them out of the way or just wait for the ambulance? |
| Rewritten query | Should you move an injured bleeding person at a road accident scene or wait for emergency services to arrive? |
| Answer | You should only move an injured person if there is immediate danger such as fire, oncoming traffic, or toxic fumes. Otherwise, leave them where they are, apply direct pressure to the bleeding wound with a clean cloth, and wait for professional medical help. Moving a casualty unnecessarily risks worsening a potential spinal injury. |
| Relevance | RELEVANT |
| Raw input | someone touched a live wire, they're unconscious, what do I do |
| Rewritten query | First aid steps for electric shock victim who is unconscious — safety, CPR, emergency |
| Answer | Do not touch the person until you are certain the power source is off or they are no longer in contact with it. Turn off the power at the mains if possible. Once safe, check for breathing and pulse. If absent, begin CPR immediately and call emergency services. Do not leave the casualty alone. |
| Relevance | RELEVANT |
| Raw input | my mate whacked his shoulder bad — crutch on the good side? |
| Rewritten query | Correct crutch placement for shoulder injury — which side should crutch be used on? |
| Answer | When using a crutch for a lower-limb injury, place it on the uninjured side to offload weight from the injured limb. For a shoulder injury specifically, a sling is typically more appropriate than a crutch. If a crutch is needed for a co-existing leg injury, it should still go on the stronger, uninjured side. |
| Relevance | PARTLY_RELEVANT |
| Raw input | what is the weather in London today |
| Answer | I don't know. |
| Relevance | NON_RELEVANT |
The LLM is instructed to answer only from the retrieved first-aid context. When no relevant context is found it says "I don't know." rather than hallucinating.
| Raw input | how do I know if someone has diabetic ketoacidosis |
| Rewritten query | Signs and symptoms of diabetic ketoacidosis (DKA) first aid recognition |
| Answer | Signs of diabetic ketoacidosis include excessive thirst, frequent urination, nausea and vomiting, abdominal pain, weakness, a fruity or acetone smell on the breath, and in severe cases confusion or unconsciousness. If DKA is suspected, call emergency services immediately and do not give the person food or drink. |
| Relevance | RELEVANT |
| Use case | Example question |
|---|---|
| Accident scene decisions | "Should I remove the helmet of an unconscious motorcyclist?" |
| Bleeding and wounds | "How do I apply a tourniquet correctly?" |
| Burns | "Hot water burn on a child — what do I do first?" |
| Bites and stings | "What's the first thing to do after a redback spider bite?" |
| Fractures and splinting | "How do I splint a broken arm without a proper splint?" |
| Cardiac events | "Someone collapsed, I think it's a heart attack — CPR steps?" |
| Choking | "Baby is choking, back blows not working, what next?" |
| Eye injuries | "Flash burn to the eye — should I flush it with water?" |
| Spinal injuries | "Motorbike crash, unconscious, how do I move them safely?" |
| Environmental | "Heatstroke vs heat exhaustion — how do I tell the difference?" |
You can use the assistant directly in a script without the Streamlit UI:
from dotenv import load_dotenv
load_dotenv()
from assistant import create_assistant
assistant = create_assistant()
# Ask a question
answer = assistant.rag("What should I do if someone is choking?")
print(answer)
# Inspect the last call's metrics
record = assistant.last_call
print(f"Response time: {record.response_time:.2f}s")
print(f"Prompt tokens: {record.prompt_tokens}")
print(f"Completion tokens: {record.completion_tokens}")
print(f"Cost: ${record.cost:.5f}")
# See the rewritten query
print(f"Rewritten query: {assistant.last_rewritten_question}")
Run it:
uv run python src/assistant.py "What should I do if someone is choking?"
Every RAG call emits three OpenTelemetry spans (rag → search → llm) stored in sqlite/otel/traces.db. Query them directly:
sqlite3 sqlite/otel/traces.db \
"SELECT name, round((end_time - start_time)/1e9, 3) AS duration_s,
input_tokens, output_tokens
FROM spans ORDER BY start_time DESC LIMIT 9;"
name duration_s input_tokens output_tokens
------ ---------- ------------ -------------
rag 1.843 (null) (null)
search 0.012 (null) (null)
llm 1.831 1203 118
rag 2.105 (null) (null)
search 0.009 (null) (null)
llm 2.096 1189 134
See notebooks/010-monitoring.ipynb for a full monitoring dashboard built on these traces.
-- Last 5 conversations with their auto-judge verdict
SELECT
c.id,
left(c.question, 60) AS question,
c.response_time,
c.cost,
f.relevance,
f.score
FROM conversations c
LEFT JOIN feedback f
ON f.conversation_id = c.id AND f.source = 'judge'
ORDER BY c.timestamp DESC
LIMIT 5;
id question response_time cost relevance
-- ---------------------------------------------------- ------------- -------- -----------
12 my friend got stung by a stingray, should I use ... 1.84 0.000013 RELEVANT
11 someone touched a live wire, they're unconscious... 2.11 0.000015 RELEVANT
10 my mate whacked his shoulder bad — crutch on th... 1.93 0.000012 PARTLY_RELEVANT
The notebooks/ directory contains end-to-end evaluation experiments:
| What to evaluate | Notebook |
|---|---|
| BM25 retrieval hit-rate and MRR | 02-sqlite-text-search-evals.ipynb |
| Vector search hit-rate and MRR | 06-sqlite-vector-num-of-results-evals.ipynb |
| Hybrid search (RRF fusion) | 08-sqlite-hybrid-search-evals.ipynb |
| Answer quality (LLM-as-judge) | 09-llm-as-judge.ipynb |
| Live monitoring dashboard | 010-monitoring.ipynb |
Run any notebook from the notebooks/ directory:
cd notebooks
uv run jupyter notebook 08-sqlite-hybrid-search-evals.ipynb