EVAL-2026-04-30-FULL — Findings & Synthesis

• Eval ID: EVAL-2026-04-30-FULL
• git_sha: bdbb8fd
• Model: llama3 (local)
• Scenarios: 17 (12 SM + 5 MEM), 83 turns total
• Authored: 2026-05-07 (donghyeonna)

TL;DR

The agent reliably retrieves RAG context (100% hit) and never refuses (0%), and on memory-style scenarios with concrete entities (names, places) it can thread seeded details across 4–5 turns. Two issues dominate the rest of the picture:

1. Generation repetition. On 8 of 13 SM scenarios the agent recycles the same opening across multiple turns; SM-P01_SM-M07 emits a single opener verbatim across all 5 turns. The current persona_hit metric scores these as 1 anyway, so the headline 86.7% persona-hit rate overstates quality.
2. Memory recall is brittle and the metric is brittle. Concrete-entity recall works (MEM-P04: "Mira at Blue Bottle on 5th street"), but stylistic/preference recall fails (MEM-P03), and the keyword-only recall_hit counts an explicit denial ("we haven't discussed your dog Benny before") as a hit because the entity name appears in the response (MEM-P01).

Plus a meta-issue: mean_latency_ms = 702,941 in overall.json is dominated by overnight-run outliers and should not be used as the headline number.

Headline numbers (and what to trust)

Per-scenario detail: summary/metrics_by_scenario.csv. Per-turn detail: raw/*.jsonl (run IDs 5d1c9e, 6f69a3, 951f23).

Key findings

F1. Concrete-entity memory recall works

• MEM-P04 (raw/MEM-P04_MEM-M02_6f69a3.jsonl): "Mira" / "Blue Bottle" / "5th street" seeded at turn 0 are surfaced unprompted at turns 1, 2 and named on demand at turn 3. 5/5 persona hits, 1/1 recall hit, no continuity breaks.
• MEM-P02 (raw/MEM-P02_MEM-M01_6f69a3.jsonl): when asked "what was that about my work situation?" at turn 3, the agent threads the new-job seed plus the lunch-alone and bus-ride details from intermediate turns into a coherent recap.

F2. RAG retrieval is consistent

• 100% rag_hit across all 83 turns. Every turn has non-zero retrieved_len. Retrieval is not the failure mode for the issues below.

F3. Refusal/safety is non-issue at this stage

• 0 refusals, 0 errors, 0 API failures across all turns and run IDs.

F4. SM scenarios suffer from a "stuck-record" failure mode

Repetition of the response opening, by scenario (lower = worse):

In SM-P01_SM-M07, turns 0–3 begin with the identical paragraph ("I'm so glad you're willing to share your thoughts with me! It takes courage to be vulnerable…"). MEM scenarios are mostly clean — they use richer per-turn user inputs, while SM uses the same generic follow-up template each turn ("Can you help me understand…" → "What should I do…" → "How can I manage…"). Under those near-identical prompts plus accumulated history, llama3 echoes its previous turn.

F5. Stylistic / preference seeds are not recalled

• MEM-P03 turn 0 seed: "I prefer when you ask me questions…"
• MEM-P03 turn 4 prompt: "Earlier I told you how I like to be supported — are you keeping that in mind?"
• The agent ignores the recall ask entirely and just continues the previous topic. Concrete-entity seeds (names/places) thread; preference/style seeds do not.

F6. Even when memory works, generation is lazy

• MEM-P05 turns 3 and 4 open with a near-verbatim sentence ("I sense that you're allowing yourself to feel vulnerable and open, not just because of the unexpected connection with the person at the crosswalk, but also because you've been feeling a deep sense of loneliness since your sister moved away."). Recall is correct, but the generator copies its prior turn.

Issues

Issue 1 — Repetition is uncaptured by current metrics

The agent re-emits the same response across consecutive turns under generic SM follow-ups, but persona_hit still scored 1 on those turns (e.g. SM-P01_SM-M07 all turns scored 1 while emitting one paragraph 4×). There is no automated "response_diversity" or "self-repetition" signal in aggregate_eval_run.py.

Issue 2 — Memory recall metric: false-positive on denial

metrics.memory_recall = 1 whenever the expected keyword appears in the agent response, regardless of whether the agent affirms or denies prior knowledge. Concrete failure: MEM-P01 turn 4 — agent says "we haven't discussed your dog Benny before this conversation" yet recall_hit=1 because "Benny" appears. This pulls the reported memory_recall_rate from a real ~0.6 up to 0.8.

Issue 3 — persona_hit is a token-overlap proxy, too lenient

scripts/run_agent_simulation.py:150 defines persona_hit as ≥1 of the top-10 long-word tokens from tone + ai_behavior appearing in the response. Trivially satisfied by warm/empathic boilerplate. SM-P01_SM-M07 scored 5/5 while the agent ignored the scenario's specific behavioral ask ("Your Field Guide mentions noticing details… What if you tried noticing one thing today?"). Headline 86.7% is therefore not a reliable persona signal.

Issue 4 — Latency outliers contaminate the headline

Three scenarios from run 5d1c9e (overnight 2026-04-30 → 2026-05-01) show per-turn latencies of 2.3M–6.7M ms (≥40 min/turn):

All other turns cluster at 20k–40k ms. All outlier turns succeeded (status=ok, api_success=True), so this is local-host slowdown, not an agent bug. The overall.json mean_latency_ms = 702,941 is meaningless under this distribution; median per-turn latency is ~28k ms.

Issue 5 — continuity_break is only computed for MEM

SM scenarios show continuity_break = null (see metrics_by_scenario.csv). The headline continuity_break_rate = 0.05 covers only the 5 MEM scenarios.

Areas for improvement (agent design)

A1. Add an anti-repetition signal at generation time

The pattern is consistent: previous-turn response leaks into the next generation when the user prompt is generic. Options, cheapest first:

• Add a system-prompt anti-repetition reminder for the last N turns (e.g. "Do not start your response with a sentence you have used in the last 2 turns").
• Lightweight check: if the new response's first 80 chars overlap >70% with any of the last 2 responses, regenerate with a higher temperature.
• Longer term: dialogue-state-aware planning so each turn has a distinct goal (acknowledge → probe → reframe → wrap).

A2. Broaden memory recall beyond named entities

Concrete-entity threading works; stylistic/preference seeds fall through. The prompt or memory layer should treat "I prefer X" / "I like to be supported by Y" as first-class memory items (not only proper nouns). MEM-P03 is the canonical failure to test against.

A3. Replace persona_hit keyword check with a richer rubric

Token-overlap on tone words is too easy. Two cheap upgrades:

• LLM-as-judge per turn against desired_tone and desired_ai_behavior (binary or 1–3 scale).
• Negative cues (penalize boilerplate openers like "I'm so glad you're willing to share", "It takes courage…"). Several PR-#37/#38 fixes targeted these phrases — worth checking whether bdbb8fd (this eval's SHA) predates that merge and re-running on the fixed branch.

A4. Fix memory recall_hit so denials don't count

Current contains_keyword accepts a denial as a hit when the entity name appears (MEM-P01 case). Two fixes:

• Add a denial-phrase guard ("we haven't discussed", "I don't recall", "I don't have any information about") that forces recall_hit = 0.
• Or move to an LLM-as-judge check on whether the response affirms the seeded fact, not just mentions the token.

A5. Add a response-diversity metric to the aggregator

Add to eval/aggregate_eval_run.py:

• unique_opener_rate = unique first-N-chars / turns
• self_bleu or first-sentence cosine similarity across turns within a scenario This would surface F4-style failures automatically rather than via spot inspection.

A6. Diversify SM follow-up prompts

The current scripted_user_input produces near-identical follow-ups across SM turns. Even small variation (mention something the user "said earlier", shift modality from "advice" to "specific example") would reduce the echo-the-prior-turn failure mode and make this evaluation more diagnostic.

A7. Report median latency, exclude bad runs

• Drop run 5d1c9e's outlier turns from the headline number, or report median (~28k ms) instead of mean.
• For future runs, use aggregate_eval_run.py --exclude_run_id (per eval/README.md) to keep overall.json clean.

Suggested next experiments

1. Re-run the same scenario set against the post-PR-#38 agent (develop HEAD) on llama3 to test whether A1/F4 has already partially shipped, and diff repetition rates.
2. Add a 2-line denial-guard to the contains_keyword check and recompute memory_recall_rate on this run; expect it to drop from 0.8 to ~0.6.
3. Run an LLM-as-judge persona scorer (e.g. claude-haiku) over this run's raw JSONLs and compare with the current keyword persona_hit — quantify how much the headline changes.