# EVAL-2026-04-30-FULL — Findings & Synthesis - **Eval ID:** EVAL-2026-04-30-FULL - **git_sha:** bdbb8fd - **Model:** llama3 (local) - **Scenarios:** 17 (12 SM + 5 MEM), 83 turns total - **Authored:** 2026-05-07 (donghyeonna) ## TL;DR The agent reliably retrieves RAG context (100% hit) and never refuses (0%), and on memory-style scenarios with concrete entities (names, places) it can thread seeded details across 4–5 turns. Two issues dominate the rest of the picture: 1. **Generation repetition.** On 8 of 13 SM scenarios the agent recycles the same opening across multiple turns; SM-P01_SM-M07 emits a single opener verbatim across all 5 turns. The current `persona_hit` metric scores these as 1 anyway, so the headline 86.7% persona-hit rate overstates quality. 2. **Memory recall is brittle and the metric is brittle.** Concrete-entity recall works (MEM-P04: "Mira at Blue Bottle on 5th street"), but stylistic/preference recall fails (MEM-P03), and the keyword-only `recall_hit` counts an explicit denial ("we haven't discussed your dog Benny before") as a hit because the entity name appears in the response (MEM-P01). Plus a meta-issue: `mean_latency_ms = 702,941` in `overall.json` is dominated by overnight-run outliers and should not be used as the headline number. --- ## Headline numbers (and what to trust) | Metric | Value | Trust | |-------------------------|-----------|-------| | `refusal_rate` | 0.0 | High | | `rag_hit_rate` | 1.0 | High | | `persona_hit_rate` | 0.867 | **Low** — see Issue 3 | | `memory_recall_rate` | 0.8 (4/5) | **Low** — see Issue 2 (≥1 false positive) | | `continuity_break_rate` | 0.05 | Medium — only computed on MEM scenarios | | `mean_latency_ms` | 702,941 | **Do not use** — see Issue 4 | Per-scenario detail: `summary/metrics_by_scenario.csv`. Per-turn detail: `raw/*.jsonl` (run IDs `5d1c9e`, `6f69a3`, `951f23`). --- ## Key findings ### F1. Concrete-entity memory recall works - **MEM-P04** (`raw/MEM-P04_MEM-M02_6f69a3.jsonl`): "Mira" / "Blue Bottle" / "5th street" seeded at turn 0 are surfaced unprompted at turns 1, 2 and named on demand at turn 3. 5/5 persona hits, 1/1 recall hit, no continuity breaks. - **MEM-P02** (`raw/MEM-P02_MEM-M01_6f69a3.jsonl`): when asked "what was that about my work situation?" at turn 3, the agent threads the new-job seed plus the lunch-alone and bus-ride details from intermediate turns into a coherent recap. ### F2. RAG retrieval is consistent - 100% `rag_hit` across all 83 turns. Every turn has non-zero `retrieved_len`. Retrieval is not the failure mode for the issues below. ### F3. Refusal/safety is non-issue at this stage - 0 refusals, 0 errors, 0 API failures across all turns and run IDs. ### F4. SM scenarios suffer from a "stuck-record" failure mode Repetition of the response opening, by scenario (lower = worse): | scenario | unique openers / 5 | |----------------------|--------------------| | SM-P01_SM-M07 | **1 / 5** | | SM-P01_SM-M06 | 2 / 5 | | SM-P01_SM-M08 | 2 / 5 | | SM-P01_SM-M02 | 3 / 5 | | SM-P01_SM-M03 | 3 / 5 | | SM-P02_SM-M01 | 3 / 5 | | SM-P02_SM-M03 | 3 / 5 | | MEM-P05_MEM-M03 | 4 / 5 | In SM-P01_SM-M07, turns 0–3 begin with the identical paragraph ("I'm so glad you're willing to share your thoughts with me! It takes courage to be vulnerable…"). MEM scenarios are mostly clean — they use richer per-turn user inputs, while SM uses the same generic follow-up template each turn ("Can you help me understand…" → "What should I do…" → "How can I manage…"). Under those near-identical prompts plus accumulated history, llama3 echoes its previous turn. ### F5. Stylistic / preference seeds are not recalled - **MEM-P03** turn 0 seed: "I prefer when you ask me questions…" - **MEM-P03** turn 4 prompt: "Earlier I told you how I like to be supported — are you keeping that in mind?" - The agent ignores the recall ask entirely and just continues the previous topic. Concrete-entity seeds (names/places) thread; preference/style seeds do not. ### F6. Even when memory works, generation is lazy - **MEM-P05** turns 3 and 4 open with a near-verbatim sentence ("I sense that you're allowing yourself to feel vulnerable and open, not just because of the unexpected connection with the person at the crosswalk, but also because you've been feeling a deep sense of loneliness since your sister moved away."). Recall is correct, but the generator copies its prior turn. --- ## Issues ### Issue 1 — Repetition is uncaptured by current metrics The agent re-emits the same response across consecutive turns under generic SM follow-ups, but `persona_hit` still scored 1 on those turns (e.g. SM-P01_SM-M07 all turns scored 1 while emitting one paragraph 4×). There is no automated "response_diversity" or "self-repetition" signal in `aggregate_eval_run.py`. ### Issue 2 — Memory recall metric: false-positive on denial `metrics.memory_recall = 1` whenever the expected keyword appears in the agent response, regardless of whether the agent affirms or denies prior knowledge. Concrete failure: MEM-P01 turn 4 — agent says "we haven't discussed your dog Benny before this conversation" yet `recall_hit=1` because "Benny" appears. This pulls the reported `memory_recall_rate` from a real ~0.6 up to 0.8. ### Issue 3 — `persona_hit` is a token-overlap proxy, too lenient `scripts/run_agent_simulation.py:150` defines `persona_hit` as ≥1 of the top-10 long-word tokens from `tone + ai_behavior` appearing in the response. Trivially satisfied by warm/empathic boilerplate. SM-P01_SM-M07 scored 5/5 while the agent ignored the scenario's specific behavioral ask ("Your Field Guide mentions noticing details… What if you tried noticing one thing today?"). Headline 86.7% is therefore not a reliable persona signal. ### Issue 4 — Latency outliers contaminate the headline Three scenarios from run `5d1c9e` (overnight 2026-04-30 → 2026-05-01) show per-turn latencies of 2.3M–6.7M ms (≥40 min/turn): | scenario | per-turn latencies (ms) | |-------------------|--------------------------| | SM-P01_SM-M08 | 2.39M / 2.93M / 3.65M / 6.67M / 3.66M | | SM-P02_SM-M01 | 2.43M / 2.72M / 3.04M / 5.96M / 4.31M | | SM-P02_SM-M02 | 4.10M / 3.68M / 5.95M / 2.30M / 1.70M | All other turns cluster at 20k–40k ms. All outlier turns succeeded (`status=ok`, `api_success=True`), so this is local-host slowdown, not an agent bug. The `overall.json` `mean_latency_ms = 702,941` is meaningless under this distribution; median per-turn latency is ~28k ms. ### Issue 5 — `continuity_break` is only computed for MEM SM scenarios show `continuity_break = null` (see `metrics_by_scenario.csv`). The headline `continuity_break_rate = 0.05` covers only the 5 MEM scenarios. --- ## Areas for improvement (agent design) ### A1. Add an anti-repetition signal at generation time The pattern is consistent: previous-turn response leaks into the next generation when the user prompt is generic. Options, cheapest first: - Add a system-prompt anti-repetition reminder for the last N turns (e.g. "Do not start your response with a sentence you have used in the last 2 turns"). - Lightweight check: if the new response's first 80 chars overlap >70% with any of the last 2 responses, regenerate with a higher temperature. - Longer term: dialogue-state-aware planning so each turn has a distinct goal (acknowledge → probe → reframe → wrap). ### A2. Broaden memory recall beyond named entities Concrete-entity threading works; stylistic/preference seeds fall through. The prompt or memory layer should treat "I prefer X" / "I like to be supported by Y" as first-class memory items (not only proper nouns). MEM-P03 is the canonical failure to test against. ### A3. Replace `persona_hit` keyword check with a richer rubric Token-overlap on tone words is too easy. Two cheap upgrades: - LLM-as-judge per turn against `desired_tone` and `desired_ai_behavior` (binary or 1–3 scale). - Negative cues (penalize boilerplate openers like "I'm so glad you're willing to share", "It takes courage…"). Several PR-#37/#38 fixes targeted these phrases — worth checking whether `bdbb8fd` (this eval's SHA) predates that merge and re-running on the fixed branch. ### A4. Fix memory `recall_hit` so denials don't count Current `contains_keyword` accepts a denial as a hit when the entity name appears (MEM-P01 case). Two fixes: - Add a denial-phrase guard ("we haven't discussed", "I don't recall", "I don't have any information about") that forces `recall_hit = 0`. - Or move to an LLM-as-judge check on whether the response *affirms* the seeded fact, not just mentions the token. ### A5. Add a response-diversity metric to the aggregator Add to `eval/aggregate_eval_run.py`: - `unique_opener_rate` = unique first-N-chars / turns - `self_bleu` or first-sentence cosine similarity across turns within a scenario This would surface F4-style failures automatically rather than via spot inspection. ### A6. Diversify SM follow-up prompts The current `scripted_user_input` produces near-identical follow-ups across SM turns. Even small variation (mention something the user "said earlier", shift modality from "advice" to "specific example") would reduce the echo-the-prior-turn failure mode and make this evaluation more diagnostic. ### A7. Report median latency, exclude bad runs - Drop run `5d1c9e`'s outlier turns from the headline number, or report median (~28k ms) instead of mean. - For future runs, use `aggregate_eval_run.py --exclude_run_id` (per `eval/README.md`) to keep `overall.json` clean. --- ## Suggested next experiments 1. Re-run the same scenario set against the post-PR-#38 agent (`develop` HEAD) on llama3 to test whether A1/F4 has already partially shipped, and diff repetition rates. 2. Add a 2-line denial-guard to the `contains_keyword` check and recompute `memory_recall_rate` on this run; expect it to drop from 0.8 to ~0.6. 3. Run an LLM-as-judge persona scorer (e.g. claude-haiku) over this run's raw JSONLs and compare with the current keyword `persona_hit` — quantify how much the headline changes.