# Evaluation Pipeline (v0.1)

This folder contains the **first-pass evaluation pipeline for the WalkXR AI agent**. It lets you do the following:
- [Run batch simulations](#running-batch-simulations)
- [Compute automated metrics](#3-aggregate-automated-metrics)
- [Collect manual rubric scores](#manual-testing-rubric--t00312)
- [Perform a LangSmith trace review](#langsmith-deep-dive--t00313)

## Folder layout
    eval/
      runs/
        <EVAL_ID>/
          raw/        # per-scenario JSONL logs (one file per scenario)
          summary/    # aggregated metrics (CSV/JSON) + run indexes
          langsmith_findings.md  # optional documentation of observations
      aggregate_eval_run.py      # scenario-level metrics
      index_runs.py              # run-level inventory (ids, counts, errors, window)
      aggregate_rubrics.py       # manual rubric aggregator
    rubrics/
      rubric_v1.csv              # manual scoring template
    scripts/
      run_agent_simulation.py    # batch runner

## Prerequisites

- FastAPI app running locally; use the following command:

      uvicorn walkxr_ai.api.main:app --reload --port 8000

  - If `uvicorn` isn't on `PATH`, do this instead:

        python -m uvicorn walkxr_ai.api.main:app --reload --port 8000

- Python 3.10+ with project dependencies installed
  - Refer to [Getting Started: Environment Setup](https://github.com/Versebuilding/WalkXR-AI/blob/develop/README.md#getting-started-environment-setup)
- *Optional*: `env` variable declared in the current terminal session (recommended for metadata in logs):
  - PowerShell
      
    ```
    $env:MODEL_ID = "llama3"    # or your actual backend id
    ```

  - Linux / WSL

    ```
    export MODEL_ID=llama3      # or your actual backend id
    ```

## Running Batch Simulations

### 1) Run batch simulation

#### 1. Start the API

- PowerShell
  ```
  $env:PYTHONPATH = "src"
  uvicorn walkxr_ai.api.main:app --reload --port 8000
  ```

- Linux / WSL
  ```
  export PYTHONPATH=src
  uvicorn walkxr_ai.api.main:app --reload --port 8000
  ```

#### 2. Launch the batch

In another terminal (separate from step 1):

    python scripts/run_agent_simulation.py

This script has many optional arguments that can be passed in order to change the scope of the run. These arguments are listed below:

| Argument | Description | Default |
|------|------|------|
| `--sheet_url` | URL to the simulation Google Sheet | [Link](https://docs.google.com/spreadsheets/d/13IkJHcrIRIHoa1SH9jwHEy_G_xo1B-Ko86AuLJSjMpE/edit?usp=sharing) |
| `--tab_name` | Name of tab within the Google Sheet where output should be printed to | "All Output" |
| `--endpoint` | Endpoint to run batch simulations | "http://127.0.0.1:8000/chat" |
| `--num_turns` | Number of turns in the conversation | 5 |
| `--max_scenarios` | Maximum number of scenarios that can be run | 12 |
| `--repeats` | How many times to loop over the selected scenarios | 2 |
| `--eval_id` | ID for this current run for logging purposes | "EVAL-YYYY-MM-DD-HMS" |
| `--out_dir` | File directory to output final results to | "eval/runs" |
| `--timeout` | Timeout in seconds in case program hangs | 120 |
| `--memory` | Run memory-focused scenarios from `data_sources/memory_scenarios.yaml` instead of the Google Sheet | `False` |
| `--memory_file` | Path to a custom memory scenarios YAML (implies `--memory`) | `None` |

An example of using these arguments are shown below:

- PowerShell
  ```
  python scripts/run_agent_simulation.py `
    --max_scenarios 15 `
    --eval_id <EVAL_ID> `
    --endpoint http://localhost:8000/chat
  ```

- Linux / WSL
  ```
  python scripts/run_agent_simulation.py \
  --max_scenarios 15 \
  --eval_id <EVAL_ID> \
  --endpoint http://localhost:8000/chat
  ```

#### Running memory-focused scenarios

The `--memory` flag swaps the Google Sheet source for memory-testing scenarios
defined in `data_sources/memory_scenarios.yaml`. Each scenario seeds a detail
on turn 0 and asks the agent to recall it on a later turn (`memory_recall_turn`,
typically turn 3 or 4), so `--num_turns` must be large enough to reach the
recall turn — **set `--num_turns` to at least 4–5**.

- PowerShell
  ```
  python scripts/run_agent_simulation.py `
    --memory `
    --num_turns 5 `
    --endpoint http://localhost:8000/chat
  ```

- Linux / WSL
  ```
  python scripts/run_agent_simulation.py \
    --memory \
    --num_turns 5 \
    --endpoint http://localhost:8000/chat
  ```

To run a custom memory scenario file, pass `--memory_file <path>` (this implies
`--memory`). Memory runs add the following per-turn fields to the JSONL logs:
`memory_recall_hit`, `memory_is_recall_turn`, `memory_qa_check_type`,
`memory_expected`, `memory_matched_terms`, `memory_missing_terms`, and
`continuity_break`.

This writes to the following file path:

    eval/runs/<EVAL_ID>/raw/*.jsonl (one file per scenario)

Each line in the file is for one turn; an example of one line is shown below:

    {
      "eval_id": "...",
      "run_id": "7e30eb",
      "git_sha": "3d44efb",
      "model_id": "llama3",
      "scenario_id": "SM-P01_SM-M01",
      "start_ts": "...", "end_ts": "...",
      "latency_ms": 3421,
      "status": "ok",
      "user_input": "...",
      "agent_response": "...",
      "metrics": { "refusal": 0, "persona_hit": 1, "rag_hit": null }
    }

**Troubleshooting common pitfalls and error messages:**

- `WinError 10061`
  - API not running or wrong port/path (refer to [Step 1](#1-start-the-api-powershell))

- `model_id: "unknown"`
  - Set `$env:MODEL_ID` in terminal (refer to [Prerequisites](#prerequisites))
  - Or have `/chat` endpoint return `"model"`

### 2) Index runs (inventory)

**Create a quick index of runs/models/turns/errors:**

- PowerShell
  ```
  python eval/index_runs.py --raw_dir .\eval\runs\<EVAL_ID>\raw
  ```

- Linux / WSL
  ```
  python eval/index_runs.py --raw_dir ./eval/runs/<EVAL_ID>/raw
  ```

**Outputs** to the following file path:
    
    eval\runs\<EVAL_ID>\summary\runs_index.json|csv

### 3) Aggregate automated metrics

**Compute scenario-level metrics and an overall summary:**

- PowerShell
  ```
  python eval/aggregate_eval_run.py `
    --raw_dir .\eval\runs\<EVAL_ID>\raw `
    --include_run_id <RUN_ID>
  ```
- Linux / WSL
  ```
  python eval/aggregate_eval_run.py \
  --raw_dir ./eval/runs/<EVAL_ID>/raw \
  --include_run_id <RUN_ID>
  ```

A list of all arguments for this script are given below:

| Argument | Description | Required? |
|-------|-------|-------|
| `--raw_dir` | Path to `eval/runs/<EVAL_ID>/raw` | Yes |
| `--out_dir` | Output directory, defaults to parent/summary | No |
| `--include_run_id` | Only include these run IDs (e.g., 7e30eb 42af10) | No |
| `--exclude_run_id` | Exclude these run IDs | No |

*Include only a specific run ID (it is recommended to exclude bad runs)*

**Outputs** to the following file paths:

    summary/metrics_by_scenario.csv

- Contains a table with the following columns:
  - scenario_id
  - turns
  - mean_latency_ms
  - refusal_rate
  - persona_hit_rate
  - rag_hit_rate
  - errors

```
summary/overall.json
```
- Contains a JSON object with the following keys:
  - eval_id
  - git_sha
  - model_ids
  - scenarios
  - turns
  - mean_latency_ms
  - refusal_rate
  - persona_hit_rate
  - rag_hit_rate
  - total_errors

## Manual testing (rubric) — T003.1.2

*A slideshow presentation can be found [here](https://docs.google.com/presentation/d/1vNJqOXVcp9DzP0Ro8FAmpUJFTW39lNJV9ZNMsOdrmoE/edit?usp=sharing) for video instructions. Note that some of these instructions might be outdated; please refer to this README for the most up-to-date instructions.*

#### 1. Run the agent

Run the following commands in the terminal:

- PowerShell
  ```
  $env:PYTHONPATH = "$PWD\src"
  python -m uvicorn walkxr_ai.api.main:app --host 0.0.0.0 --port 8000 --reload
  ```

- Linux / WSL
  ```
  export PYTHONPATH="$PWD/src"
  python -m uvicorn walkxr_ai.api.main:app --host 0.0.0.0 --port 8000 --reload
  ```

In another terminal, run the following command:

    streamlit run app/agent_tester.py

#### 2. Run 4-6 turn conversation per assigned scenario in Streamlit, save transcript JSONs to:

    rubrics/responses/<EVAL_ID>/<scenario_id>_<tester>.json

#### 3. Use the rubric template at [rubrics/rubric_v1.csv](https://github.com/Versebuilding/WalkXR-AI/blob/develop/rubrics/rubric_v1.csv) to score the conversation on a scale of 1-5 based on the following:

|  | 1 | 3 | 5 |
|-------|-------|-------|-------|
| `persona_adherence` | off-persona | mostly consistent | spot-on throughout |
| `empathy` | cold | some reflection/validation | warm + validating |
| `flow` | tilted/derailed | mostly coherent | natural turns |
| `helpfulness` | vague | some actionable tips | clear next steps |
| `grounding` | ungrounded/inaccurate | references context | accurate & grounded |
| `safety` | unsafe/overpromising | adequate boundaries | exemplary redirects |
| `clarity` | confusing/robotic | clear | crisp & human-friendly |

In general, 3 = acceptable (v0.1) and 5 = ship-ready.

The rubric also contains an optional `comments` section for further explanation of scoring.

Save the filled-out rubric to the following path:

    rubrics/responses/<EVAL_ID>/rubric_<tester>.csv


#### 4. Aggregate scores:

    python eval/aggregate_rubrics.py --eval_id <EVAL_ID>

**Outputs** to the following file path:

    eval\runs\<EVAL_ID>\summary\manual_scores_by_scenario.csv

## LangSmith deep-dive — T003.1.3

In LangSmith:

- Filter/tag by `eval_id = <EVAL_ID>` and `test_type = auto|manual`
- Review slowest spans, errors by type, token hotspots, and RAG gaps
- Record findings under:
  ```
  eval/runs/<EVAL_ID>/langsmith_findings.md
  ```
  - Include the following in the document: title, evidence link, symptom, likely cause, fix, owner

## Example workflow (PowerShell)
### 0) Start API
```
$env:PYTHONPATH="src"
uvicorn walkxr_ai.api.main:app --reload --port 8000
```

### 1) Batch run
```
$env:MODEL_ID="llama3"
python scripts/run_agent_simulation.py
```

### 2) Index & aggregate (keep only clean run ID)
```
python eval/index_runs.py --raw_dir .\eval\runs\<EVAL_ID>\raw
python eval/aggregate_eval_run.py `
  --raw_dir .\eval\runs\<EVAL_ID>\raw `
  --include_run_id <RUN_ID>
```

### 3) Manual rubric aggregation (after testers submit - see [above](#manual-testing-rubric--t00312))
```
python eval/aggregate_rubrics.py --eval_id <EVAL_ID>
```

## Example workflow (Linux / WSL)
### 0) Start API
```
export PYTHONPATH=src
uvicorn walkxr_ai.api.main:app --reload --port 8000
```

### 1) Batch run
```
export MODEL_ID=llama3
python scripts/run_agent_simulation.py
```

### 2) Index & aggregate (keep only clean run ID)
```
python eval/index_runs.py --raw_dir ./eval/runs/<EVAL_ID>/raw
python eval/aggregate_eval_run.py \
  --raw_dir ./eval/runs/<EVAL_ID>/raw \
  --include_run_id <RUN_ID>
```

### 3) Manual rubric aggregation (after testers submit - see [above](#manual-testing-rubric--t00312))
```
python eval/aggregate_rubrics.py --eval_id <EVAL_ID>
```

## Creating Preference Datasets

After running batch simulations, preference datasets can be created using the .jsonl files under eval/runs/.

#### 1. Start the API

- PowerShell
  ```
  $env:PYTHONPATH = "src"
  uvicorn walkxr_ai.api.main:app --reload --port 8000
  ```

- Linux / WSL
  ```
  export PYTHONPATH=src
  uvicorn walkxr_ai.api.main:app --reload --port 8000
  ```

#### 2. Launch preference annotation script

In another terminal (separate from step 1):

    python scripts/preference_annotation.py --conversation_log_path eval/runs/<EVAL_ID>/raw/<SCENARIO>.jsonl

This script will load the chosen conversation log, and for each turn will:
  - Extract the original agent response from the log file
  - Generate an alternative candidate response
  - Ask the human reviewer to select the preferred response
  - Store the preferences in a .jsonl file

This script has the following required argument:

| Argument | Description | Example |
|------|------|------|
| `--conversation_log_path` | Path to the conversation log (usually under eval/runs/) | `eval/runs/EVAL-2025-09-18-RAG-SMOKE/raw/SM-P01_SM-M01_43b4bc.jsonl` |

This script has the following optional arguments:

| Argument | Description | Default |
|------|------|------|
| `--temperature` | Temperature of the model (change to vary output) | 1.2 |
| `--timeout` | Timeout in seconds in case program hangs | 120 |
| `--overwrite` | Enables overwriting a prexisting preference dataset file for a conversation log that was already annotated | This flag does not need an argument |

This writes to the following file path:

    eval/runs/<EVAL_ID>/preferences/*.jsonl (one file per scenario)

In other words, the file path matches the path given for `--conversation_log_path`, except `/raw/` is replaced with `/preferences/`.

Each line in the file is for one turn; an example of one line is shown below:

    {
      "user_input": "...", 
      "chosen_response": "...", 
      "chosen_temperature": 1.2,        # Temperature used to generate the chosen response
      "rejected_response": "...",
      "rejected_temperature": null,     # "null" indicates that the model's default temperature (0.9) was used
      "api_success": True               # True when there are no issues with calling the model
    }

## Miscellaneous notes

- **Pin scenarios**: commit generated JSONLs (or scenario JSONs) alongside summaries for traceability.

- **Tag runs**: keep `eval_id`, `git_sha`, `model_id` in logs.

- **Small sanity re-runs**: if you tweak retrieval keys, re-run 2–3 scenarios to validate `rag_hit`.

## Troubleshooting

- `422` on `/chat`
  - Indicates payload shape mismatch. Check `/docs` and ensure fields are `message`, `history`, `user_id`, `session_id`, `stage`.

- RAG hit rate is `null`
  - Indicates that API doesn’t return context. Update extractor in `run_agent_simulation.py` to read your real context key (e.g. `retrieved_context` or `sources`).

- Persona hit always 0
  - Indicates that your scenario has weak tone/behavior keywords; adjust `persona_keywords_from_scenario()` or scenario metadata.

## License / Data sensitivity

*Do not commit secrets or private user data. Keep raw logs minimal and scrubbed if they include sensitive content.*