jpmschweitzerandClaude Opus 4.5
49f0da8068
Build and Push / build (release) Successful in 1m14s
Two-Phase Tatlock Execution: - orchestrate_tool_calls() for Phase 1 coordination - synthesize_from_results() for Phase 2 butler-toned synthesis - Guarantees butler personality in all responses Automatic Think Slugs: - Deterministic butler-perspective messages during expert delegation - ActionType enum: RETRIEVE, RESEARCH, CREATE, CONTROL, RECORD - HOUSEHOLD_THINK_MESSAGES mapping for all experts - Streaming delegation wrappers with automatic think messages Steward Query Enrichment: - Auto-fill user context (location, timezone) when not specified - _build_enriched_query() with regex word boundary matching - enriched_query field in StewardRecommendation schema Documentation: - ORCHESTRATION_SCENARIOS.md rewritten with Mermaid diagrams - New Housekeeper and Biographer scenarios - TESTING_IMPROVEMENTS.md for future LLM testing patterns 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
106 lines
2.6 KiB
Markdown
106 lines
2.6 KiB
Markdown
# Testing Improvements for LLM Outputs
|
|
|
|
## Problem
|
|
|
|
LLM outputs are non-deterministic. Tests checking for exact string matches fail when the LLM writes "thirty-seven" instead of "37".
|
|
|
|
## Proposed Solutions
|
|
|
|
### 1. LLM-as-Judge Pattern
|
|
|
|
Use a smaller/faster model to evaluate semantic correctness:
|
|
|
|
```python
|
|
async def llm_judge(output: str, criteria: str) -> bool:
|
|
"""Use LLM to evaluate if output meets criteria."""
|
|
prompt = f"""
|
|
Evaluate if this output is correct:
|
|
Output: {output}
|
|
Criteria: {criteria}
|
|
Answer only YES or NO.
|
|
"""
|
|
result = await judge_model.run(prompt)
|
|
return "YES" in result.output.upper()
|
|
|
|
# Usage in test:
|
|
assert await llm_judge(
|
|
response,
|
|
"The answer correctly states that sqrt(144) + 25 = 37"
|
|
)
|
|
```
|
|
|
|
### 2. Fuzzy/Regex Matching
|
|
|
|
For numeric answers, accept multiple representations:
|
|
|
|
```python
|
|
import re
|
|
|
|
def contains_number(text: str, number: int) -> bool:
|
|
"""Check if text contains number in any form."""
|
|
patterns = [
|
|
rf'\b{number}\b', # Digit form
|
|
number_to_words(number), # Word form
|
|
]
|
|
return any(re.search(p, text, re.I) for p in patterns)
|
|
|
|
# Usage:
|
|
assert contains_number(response, 37) # Matches "37" or "thirty-seven"
|
|
```
|
|
|
|
### 3. DeepEval Framework
|
|
|
|
```python
|
|
from deepeval.metrics import AnswerRelevancyMetric
|
|
from deepeval.test_case import LLMTestCase
|
|
|
|
def test_calculation():
|
|
test_case = LLMTestCase(
|
|
input="What is sqrt(144) + 25?",
|
|
actual_output=response,
|
|
expected_output="37"
|
|
)
|
|
metric = AnswerRelevancyMetric(threshold=0.7)
|
|
assert metric.measure(test_case)
|
|
```
|
|
|
|
### 4. pytest-evals Plugin
|
|
|
|
Minimal pytest plugin for LLM testing with metrics collection.
|
|
|
|
```bash
|
|
pip install pytest-evals
|
|
```
|
|
|
|
### 5. Multiple Runs with Threshold
|
|
|
|
Run flaky tests multiple times and require majority pass:
|
|
|
|
```python
|
|
@pytest.mark.flaky(reruns=3, reruns_delay=1)
|
|
def test_llm_response():
|
|
...
|
|
```
|
|
|
|
Or custom:
|
|
|
|
```python
|
|
@pytest.mark.parametrize("run", range(3))
|
|
def test_llm_response(run):
|
|
...
|
|
# Aggregate results across runs
|
|
```
|
|
|
|
## Resources
|
|
|
|
- [DeepEval](https://github.com/confident-ai/deepeval) - LLM evaluation framework
|
|
- [pytest-evals](https://github.com/AlmogBaku/pytest-evals) - pytest plugin for LLM evals
|
|
- [LLM Testing Guide 2025](https://www.confident-ai.com/blog/llm-testing-in-2024-top-methods-and-strategies)
|
|
- [Testing LLM Applications - Langfuse](https://langfuse.com/blog/2025-10-21-testing-llm-applications)
|
|
|
|
## Implementation Priority
|
|
|
|
1. Add fuzzy number matching helper (quick win)
|
|
2. Evaluate DeepEval for complex output testing
|
|
3. Consider LLM-as-judge for semantic correctness
|