jpmschweitzer/tatlock

Fork 0

Files

T

jpmschweitzerandClaude Opus 4.5 49f0da8068

Build and Push / build (release) Successful in 1m14s

Details

feat: two-phase execution, think slugs, query enrichment (v1.6.0)

Two-Phase Tatlock Execution:
- orchestrate_tool_calls() for Phase 1 coordination
- synthesize_from_results() for Phase 2 butler-toned synthesis
- Guarantees butler personality in all responses

Automatic Think Slugs:
- Deterministic butler-perspective messages during expert delegation
- ActionType enum: RETRIEVE, RESEARCH, CREATE, CONTROL, RECORD
- HOUSEHOLD_THINK_MESSAGES mapping for all experts
- Streaming delegation wrappers with automatic think messages

Steward Query Enrichment:
- Auto-fill user context (location, timezone) when not specified
- _build_enriched_query() with regex word boundary matching
- enriched_query field in StewardRecommendation schema

Documentation:
- ORCHESTRATION_SCENARIOS.md rewritten with Mermaid diagrams
- New Housekeeper and Biographer scenarios
- TESTING_IMPROVEMENTS.md for future LLM testing patterns

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

2025-12-15 14:00:32 +01:00

2.6 KiB

Raw Blame History

Testing Improvements for LLM Outputs

Problem

LLM outputs are non-deterministic. Tests checking for exact string matches fail when the LLM writes "thirty-seven" instead of "37".

Proposed Solutions

1. LLM-as-Judge Pattern

Use a smaller/faster model to evaluate semantic correctness:

async def llm_judge(output: str, criteria: str) -> bool:
    """Use LLM to evaluate if output meets criteria."""
    prompt = f"""
    Evaluate if this output is correct:
    Output: {output}
    Criteria: {criteria}
    Answer only YES or NO.
    """
    result = await judge_model.run(prompt)
    return "YES" in result.output.upper()

# Usage in test:
assert await llm_judge(
    response,
    "The answer correctly states that sqrt(144) + 25 = 37"
)

2. Fuzzy/Regex Matching

For numeric answers, accept multiple representations:

import re

def contains_number(text: str, number: int) -> bool:
    """Check if text contains number in any form."""
    patterns = [
        rf'\b{number}\b',  # Digit form
        number_to_words(number),  # Word form
    ]
    return any(re.search(p, text, re.I) for p in patterns)

# Usage:
assert contains_number(response, 37)  # Matches "37" or "thirty-seven"

3. DeepEval Framework

from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

def test_calculation():
    test_case = LLMTestCase(
        input="What is sqrt(144) + 25?",
        actual_output=response,
        expected_output="37"
    )
    metric = AnswerRelevancyMetric(threshold=0.7)
    assert metric.measure(test_case)

4. pytest-evals Plugin

Minimal pytest plugin for LLM testing with metrics collection.

pip install pytest-evals

5. Multiple Runs with Threshold

Run flaky tests multiple times and require majority pass:

@pytest.mark.flaky(reruns=3, reruns_delay=1)
def test_llm_response():
    ...

Or custom:

@pytest.mark.parametrize("run", range(3))
def test_llm_response(run):
    ...
    # Aggregate results across runs

Resources

DeepEval - LLM evaluation framework
pytest-evals - pytest plugin for LLM evals
LLM Testing Guide 2025
Testing LLM Applications - Langfuse

Implementation Priority

Add fuzzy number matching helper (quick win)
Evaluate DeepEval for complex output testing
Consider LLM-as-judge for semantic correctness

2.6 KiB Raw Blame History