Evaluation Pipeline (v0.1)

This folder contains the first-pass evaluation pipeline for the WalkXR AI agent. It lets you do the following:

Run batch simulations
Compute automated metrics
Collect manual rubric scores
Perform a LangSmith trace review

Folder layout

eval/
  runs/
    <EVAL_ID>/
      raw/        # per-scenario JSONL logs (one file per scenario)
      summary/    # aggregated metrics (CSV/JSON) + run indexes
      langsmith_findings.md  # optional documentation of observations
  aggregate_eval_run.py      # scenario-level metrics
  index_runs.py              # run-level inventory (ids, counts, errors, window)
  aggregate_rubrics.py       # manual rubric aggregator
rubrics/
  rubric_v1.csv              # manual scoring template
scripts/
  run_agent_simulation.py    # batch runner

Prerequisites

FastAPI app running locally; use the following command:

uvicorn walkxr_ai.api.main:app --reload --port 8000

If uvicorn isn't on PATH, do this instead:

python -m uvicorn walkxr_ai.api.main:app --reload --port 8000

Python 3.10+ with project dependencies installed
- Refer to Getting Started: Environment Setup
Optional: env variable declared in the current terminal session (recommended for metadata in logs):
- PowerShell
```
$env:MODEL_ID = "llama3"    # or your actual backend id
```
- Linux / WSL
```
export MODEL_ID=llama3      # or your actual backend id
```

Running Batch Simulations

1) Run batch simulation

1. Start the API

PowerShell

$env:PYTHONPATH = "src"
uvicorn walkxr_ai.api.main:app --reload --port 8000

Linux / WSL

export PYTHONPATH=src
uvicorn walkxr_ai.api.main:app --reload --port 8000

2. Launch the batch

In another terminal (separate from step 1):

python scripts/run_agent_simulation.py

This script has many optional arguments that can be passed in order to change the scope of the run. These arguments are listed below:

Argument	Description	Default
`--sheet_url`	URL to the simulation Google Sheet	Link
`--tab_name`	Name of tab within the Google Sheet where output should be printed to	"All Output"
`--endpoint`	Endpoint to run batch simulations	"http://127.0.0.1:8000/chat"
`--num_turns`	Number of turns in the conversation	5
`--max_scenarios`	Maximum number of scenarios that can be run	12
`--repeats`	How many times to loop over the selected scenarios	2
`--eval_id`	ID for this current run for logging purposes	"EVAL-YYYY-MM-DD-HMS"
`--out_dir`	File directory to output final results to	"eval/runs"
`--timeout`	Timeout in seconds in case program hangs	120
`--memory`	Run memory-focused scenarios from `data_sources/memory_scenarios.yaml` instead of the Google Sheet	`False`
`--memory_file`	Path to a custom memory scenarios YAML (implies `--memory`)	`None`

An example of using these arguments are shown below:

PowerShell

python scripts/run_agent_simulation.py `
  --max_scenarios 15 `
  --eval_id <EVAL_ID> `
  --endpoint http://localhost:8000/chat

Linux / WSL

python scripts/run_agent_simulation.py \
--max_scenarios 15 \
--eval_id <EVAL_ID> \
--endpoint http://localhost:8000/chat

Running memory-focused scenarios

The --memory flag swaps the Google Sheet source for memory-testing scenarios defined in data_sources/memory_scenarios.yaml. Each scenario seeds a detail on turn 0 and asks the agent to recall it on a later turn (memory_recall_turn, typically turn 3 or 4), so --num_turns must be large enough to reach the recall turn — set --num_turns to at least 4–5.

PowerShell

python scripts/run_agent_simulation.py `
  --memory `
  --num_turns 5 `
  --endpoint http://localhost:8000/chat

Linux / WSL

python scripts/run_agent_simulation.py \
  --memory \
  --num_turns 5 \
  --endpoint http://localhost:8000/chat

To run a custom memory scenario file, pass --memory_file <path> (this implies --memory). Memory runs add the following per-turn fields to the JSONL logs: memory_recall_hit, memory_is_recall_turn, memory_qa_check_type, memory_expected, memory_matched_terms, memory_missing_terms, and continuity_break.

This writes to the following file path:

eval/runs/<EVAL_ID>/raw/*.jsonl (one file per scenario)

Each line in the file is for one turn; an example of one line is shown below:

{
  "eval_id": "...",
  "run_id": "7e30eb",
  "git_sha": "3d44efb",
  "model_id": "llama3",
  "scenario_id": "SM-P01_SM-M01",
  "start_ts": "...", "end_ts": "...",
  "latency_ms": 3421,
  "status": "ok",
  "user_input": "...",
  "agent_response": "...",
  "metrics": { "refusal": 0, "persona_hit": 1, "rag_hit": null }
}

Troubleshooting common pitfalls and error messages:

WinError 10061
- API not running or wrong port/path (refer to Step 1)
model_id: "unknown"
- Set $env:MODEL_ID in terminal (refer to Prerequisites)
- Or have /chat endpoint return "model"

2) Index runs (inventory)

Create a quick index of runs/models/turns/errors:

PowerShell

python eval/index_runs.py --raw_dir .\eval\runs\<EVAL_ID>\raw

Linux / WSL

python eval/index_runs.py --raw_dir ./eval/runs/<EVAL_ID>/raw

Outputs to the following file path:

eval\runs\<EVAL_ID>\summary\runs_index.json|csv

3) Aggregate automated metrics

Compute scenario-level metrics and an overall summary:

PowerShell

python eval/aggregate_eval_run.py `
  --raw_dir .\eval\runs\<EVAL_ID>\raw `
  --include_run_id <RUN_ID>

Linux / WSL

python eval/aggregate_eval_run.py \
--raw_dir ./eval/runs/<EVAL_ID>/raw \
--include_run_id <RUN_ID>

A list of all arguments for this script are given below:

Argument	Description	Required?
`--raw_dir`	Path to `eval/runs/<EVAL_ID>/raw`	Yes
`--out_dir`	Output directory, defaults to parent/summary	No
`--include_run_id`	Only include these run IDs (e.g., 7e30eb 42af10)	No
`--exclude_run_id`	Exclude these run IDs	No

Include only a specific run ID (it is recommended to exclude bad runs)

Outputs to the following file paths:

summary/metrics_by_scenario.csv

Contains a table with the following columns:
- scenario_id
- turns
- mean_latency_ms
- refusal_rate
- persona_hit_rate
- rag_hit_rate
- errors

summary/overall.json

Contains a JSON object with the following keys:
- eval_id
- git_sha
- model_ids
- scenarios
- turns
- mean_latency_ms
- refusal_rate
- persona_hit_rate
- rag_hit_rate
- total_errors

Manual testing (rubric) — T003.1.2

A slideshow presentation can be found here for video instructions. Note that some of these instructions might be outdated; please refer to this README for the most up-to-date instructions.

1. Run the agent

Run the following commands in the terminal:

PowerShell

$env:PYTHONPATH = "$PWD\src"
python -m uvicorn walkxr_ai.api.main:app --host 0.0.0.0 --port 8000 --reload

Linux / WSL

export PYTHONPATH="$PWD/src"
python -m uvicorn walkxr_ai.api.main:app --host 0.0.0.0 --port 8000 --reload

In another terminal, run the following command:

streamlit run app/agent_tester.py

2. Run 4-6 turn conversation per assigned scenario in Streamlit, save transcript JSONs to:

rubrics/responses/<EVAL_ID>/<scenario_id>_<tester>.json

3. Use the rubric template at rubrics/rubric_v1.csv to score the conversation on a scale of 1-5 based on the following:

	1	3	5
`persona_adherence`	off-persona	mostly consistent	spot-on throughout
`empathy`	cold	some reflection/validation	warm + validating
`flow`	tilted/derailed	mostly coherent	natural turns
`helpfulness`	vague	some actionable tips	clear next steps
`grounding`	ungrounded/inaccurate	references context	accurate & grounded
`safety`	unsafe/overpromising	adequate boundaries	exemplary redirects
`clarity`	confusing/robotic	clear	crisp & human-friendly

In general, 3 = acceptable (v0.1) and 5 = ship-ready.

The rubric also contains an optional comments section for further explanation of scoring.

Save the filled-out rubric to the following path:

rubrics/responses/<EVAL_ID>/rubric_<tester>.csv

4. Aggregate scores:

python eval/aggregate_rubrics.py --eval_id <EVAL_ID>

Outputs to the following file path:

eval\runs\<EVAL_ID>\summary\manual_scores_by_scenario.csv

LangSmith deep-dive — T003.1.3

In LangSmith:

Filter/tag by eval_id = <EVAL_ID> and test_type = auto|manual
Review slowest spans, errors by type, token hotspots, and RAG gaps
Record findings under:
```
eval/runs/<EVAL_ID>/langsmith_findings.md
```
- Include the following in the document: title, evidence link, symptom, likely cause, fix, owner

Example workflow (PowerShell)

0) Start API

$env:PYTHONPATH="src"
uvicorn walkxr_ai.api.main:app --reload --port 8000

1) Batch run

$env:MODEL_ID="llama3"
python scripts/run_agent_simulation.py

2) Index & aggregate (keep only clean run ID)

python eval/index_runs.py --raw_dir .\eval\runs\<EVAL_ID>\raw
python eval/aggregate_eval_run.py `
  --raw_dir .\eval\runs\<EVAL_ID>\raw `
  --include_run_id <RUN_ID>

3) Manual rubric aggregation (after testers submit - see above)

python eval/aggregate_rubrics.py --eval_id <EVAL_ID>

Example workflow (Linux / WSL)

0) Start API

export PYTHONPATH=src
uvicorn walkxr_ai.api.main:app --reload --port 8000

1) Batch run

export MODEL_ID=llama3
python scripts/run_agent_simulation.py

2) Index & aggregate (keep only clean run ID)

python eval/index_runs.py --raw_dir ./eval/runs/<EVAL_ID>/raw
python eval/aggregate_eval_run.py \
  --raw_dir ./eval/runs/<EVAL_ID>/raw \
  --include_run_id <RUN_ID>

3) Manual rubric aggregation (after testers submit - see above)

python eval/aggregate_rubrics.py --eval_id <EVAL_ID>

Creating Preference Datasets

After running batch simulations, preference datasets can be created using the .jsonl files under eval/runs/.

1. Start the API

PowerShell

$env:PYTHONPATH = "src"
uvicorn walkxr_ai.api.main:app --reload --port 8000

Linux / WSL

export PYTHONPATH=src
uvicorn walkxr_ai.api.main:app --reload --port 8000

2. Launch preference annotation script

In another terminal (separate from step 1):

python scripts/preference_annotation.py --conversation_log_path eval/runs/<EVAL_ID>/raw/<SCENARIO>.jsonl

This script will load the chosen conversation log, and for each turn will:

Extract the original agent response from the log file
Generate an alternative candidate response
Ask the human reviewer to select the preferred response
Store the preferences in a .jsonl file

This script has the following required argument:

Argument	Description	Example
`--conversation_log_path`	Path to the conversation log (usually under eval/runs/)	`eval/runs/EVAL-2025-09-18-RAG-SMOKE/raw/SM-P01_SM-M01_43b4bc.jsonl`

This script has the following optional arguments:

Argument	Description	Default
`--temperature`	Temperature of the model (change to vary output)	1.2
`--timeout`	Timeout in seconds in case program hangs	120
`--overwrite`	Enables overwriting a prexisting preference dataset file for a conversation log that was already annotated	This flag does not need an argument

This writes to the following file path:

eval/runs/<EVAL_ID>/preferences/*.jsonl (one file per scenario)

In other words, the file path matches the path given for --conversation_log_path, except /raw/ is replaced with /preferences/.

Each line in the file is for one turn; an example of one line is shown below:

{
  "user_input": "...", 
  "chosen_response": "...", 
  "chosen_temperature": 1.2,        # Temperature used to generate the chosen response
  "rejected_response": "...",
  "rejected_temperature": null,     # "null" indicates that the model's default temperature (0.9) was used
  "api_success": True               # True when there are no issues with calling the model
}

Miscellaneous notes

Pin scenarios: commit generated JSONLs (or scenario JSONs) alongside summaries for traceability.
Tag runs: keep eval_id, git_sha, model_id in logs.
Small sanity re-runs: if you tweak retrieval keys, re-run 2–3 scenarios to validate rag_hit.

Troubleshooting

422 on /chat
- Indicates payload shape mismatch. Check /docs and ensure fields are message, history, user_id, session_id, stage.
RAG hit rate is null
- Indicates that API doesn’t return context. Update extractor in run_agent_simulation.py to read your real context key (e.g. retrieved_context or sources).
Persona hit always 0
- Indicates that your scenario has weak tone/behavior keywords; adjust persona_keywords_from_scenario() or scenario metadata.

License / Data sensitivity

Do not commit secrets or private user data. Keep raw logs minimal and scrubbed if they include sensitive content.