# WalkXR-AI Small Moments Agent  
## v0.1 Evaluation & v0.2 Roadmap

**Author:** Jason Chen  
**Date:** 2025-12-04  
**Tasks:** T003.1 → T003.2  

---

## 1. Overview

This document summarizes findings from the v0.1 Small Moments Roleplay Agent evaluation and defines the requirements for v0.2.

The evaluation consisted of:

- Automated simulation runs using `run_agent_simulation.py`
- Manual testing via the Streamlit Agent Tester
- LangSmith trace inspection for reasoning issues, structured output errors, RAG failures, and latency hotspots

v0.1 runs end-to-end successfully and produces empathetic responses, but several limitations affect reliability and efficiency.  
This roadmap reflects only observed behavior and verified issues.

---

## 2. Strengths of v0.1

- End-to-end loop works: scenario loading, retrieval, agent generation, structured output.
- Consistent emotional tone with generally appropriate phrasing.
- Low crash rate after retry logic was added.
- Structured output is valid JSON most of the time.
- LangSmith traces provide full transparency for debugging and later evaluation (T003.2+).

---

## 3. Error Modes Observed

### 3.1 Reasoning and Content Issues

- Overly long internal reflection steps.
- Emotional responses are correct but formulaic, lacking specificity.
- Occasional scenario drift during longer conversations or when retrieval returns weakly relevant content.

### 3.2 Structured Output Errors

Approx. 10–15% of turns required repair due to:

- Missing required fields  
- Trailing commas or invalid JSON syntax  
- Hallucinated extra keys  
- Empty or generic "options" lists despite obvious branching opportunities

### 3.3 Retrieval Problems

- Retrieval sometimes yields chunks with low semantic relevance.
- Retrieved chunks are often too long, increasing prompt size without improving response quality.
- Even when useful, retrieved content is rarely referenced directly by the agent.

### 3.4 Latency Findings

- Latency spikes occur predominantly in first-turn reasoning and multi-step reflection loops.
- Average latency per turn: ~1.2–1.8 seconds  
- Worst cases exceed 4 seconds

---

## 4. Efficiency Issues Identified

### 4.1 Redundant Steps

- Repeating the "think → restate → validate → respond" chain.
- System prompt includes unused blocks that add unnecessary tokens.

### 4.2 Context Window Usage

- Full chunk injection increases token count.
- Agent sometimes summarizes irrelevant retrieval results instead of focusing on the most useful content.

---

## 5. v0.2 Requirements

### 5.1 Prompting and Reasoning

**R1. Add a concise reflection limit**  
Require a single short reasoning step to avoid multi-paragraph chains.

**R2. Improve scenario anchoring**  
Include a compact, consistent scenario summary in every prompt to prevent drift.

**R3. Increase emotional specificity**  
Integrate a broader empathy lexicon and encourage references to retrieved content when appropriate.

---

### 5.2 Structured Output Reliability

**R4. Introduce a schema validator**  
Validate required fields, data types, and enforce constraints (e.g., non-empty options).

**R5. Expand positive and negative output examples**  
Examples should cover branching, terminal states, and emotionally heavy cases.

---

### 5.3 Retrieval and RAG Improvements

**R6. Add semantic filtering prior to lexical overlap checks**  
Use similarity scoring to limit retrieval to the top 1–2 relevant chunks.

**R7. Add a retrieval transparency field (internal only)**  
Log how retrieved content influenced the response for debugging and future evaluation.

---

### 5.4 Latency and Efficiency

**R8. Remove unused system prompt blocks**  
Trim sections that are never referenced by reasoning traces.

**R9. Reduce maximum reflection depth**  
Target a hard cap of three internal reasoning steps.

**R10. Trim retrieval chunks to excerpts**  
Pass only the minimal portion of retrieved text needed for context.

---

### 5.5 Safety and User Experience

**R11. Add escalation logic for vulnerable user statements**  
Acknowledge distress and offer grounding actions without making therapeutic claims.

**R12. Improve cross-turn consistency ("check_back")**  
Agent responses must explicitly reference relevant parts of the user’s latest message.

---

## 6. Required v0.2 Deliverables

| Deliverable | Description |
|------------|-------------|
| D1 | Revised system prompt with reflection limits and scenario anchoring |
| D2 | Updated RAG pipeline with semantic filtering and trimmed chunks |
| D3 | Structured output validator and improved output examples |
| D4 | Latency benchmarks before and after optimization |
| D5 | Error taxonomy table from v0.1 to support future reward modeling |

---

## 7. Conclusion

v0.1 is stable and functional but inconsistent in reasoning depth, structured output accuracy, retrieval quality, and latency.  
The requirements defined above provide a clear and achievable path toward a more reliable, efficient, and emotionally grounded v0.2 agent.