Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
84467c121a | ||
|
|
2290320e9c | ||
|
|
bf13f9f0de | ||
|
|
4f42bc047a | ||
|
|
738ff10b93 | ||
|
|
a90536314e | ||
|
|
19e32cfbd6 | ||
|
|
99569e786e | ||
|
|
2cf3252a19 | ||
|
|
cdd5a55613 | ||
|
|
287d66fff7 | ||
|
|
65debb6e44 | ||
|
|
0d7514b90e | ||
|
|
99683357d2 |
@@ -25,7 +25,7 @@ jobs:
|
||||
- name: Login to Gitea Registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: git.schweitz.internal
|
||||
registry: git.schweitz.net
|
||||
username: ${{ secrets.REGISTRY_USER }}
|
||||
password: ${{ secrets.REGISTRY_PASSWORD }}
|
||||
|
||||
@@ -37,8 +37,8 @@ jobs:
|
||||
provenance: false
|
||||
sbom: false
|
||||
tags: |
|
||||
git.schweitz.internal/jpmschweitzer/tatlock:latest
|
||||
git.schweitz.internal/jpmschweitzer/tatlock:${{ github.ref_name }}
|
||||
git.schweitz.net/jpmschweitzer/tatlock:latest
|
||||
git.schweitz.net/jpmschweitzer/tatlock:${{ github.ref_name }}
|
||||
|
||||
- name: Trigger Watchtower update
|
||||
if: success()
|
||||
|
||||
@@ -17,8 +17,8 @@ This document contains instructions and documentation references for AI assistan
|
||||
|
||||
### 🧪 Local Development Setup
|
||||
* **Always test locally first** before committing and deploying. The build-deploy loop is slow.
|
||||
* **Start the local server** with `./wakeup.sh` - logs are written to `logs/server.log` for easy tailing
|
||||
* **Auto-reload**: The wakeup script runs uvicorn in reload mode - code changes are picked up automatically without restart (except for requirements.txt changes)
|
||||
* **Start the local server** with `make run` - logs are written to `build/logs/server.log` for easy tailing
|
||||
* **Auto-reload**: `make run` runs uvicorn in reload mode - code changes are picked up automatically without restart (except for dependency changes)
|
||||
* **Test REST endpoints** against `http://localhost:8777` using curl or similar tools
|
||||
* **Only deploy** when a phase or feature is complete and tested locally
|
||||
* **Environment**: Copy `.env.example` to `.env` and configure for your local setup (Ollama, Redis, Qdrant hosts)
|
||||
|
||||
@@ -7,6 +7,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [2.4.3] - 2026-08-08
|
||||
|
||||
### Fixed
|
||||
|
||||
- Steward routing no longer triggers on words inside its own explanation. Capability
|
||||
extraction reads the declared `DELEGATE:` line instead of substring-matching
|
||||
capability domains across the whole response, where ordinary English routed
|
||||
requests — "description" contains the housekeeper domain "script", "acknowledge"
|
||||
contains "knowledge" and "know". A spurious capability meant a real agent call,
|
||||
including web searches, on queries that needed none.
|
||||
|
||||
## [2.4.2] - 2026-07-19
|
||||
|
||||
### Fixed
|
||||
|
||||
- Container crash-loop on fresh builds: cap `opentelemetry-api` below 1.44,
|
||||
which removed the private `_events` module that pydantic-ai 1.27 imports
|
||||
|
||||
## [2.4.1] - 2026-07-19
|
||||
|
||||
### Changed
|
||||
|
||||
- **Container-name network defaults** - `SEARXNG_HOST`, `LIBRARY_DESK_HOST`, and `CORE_API_HOST` now default to docker container names on the docker-dataplane network (`http://searxng:8080`, `http://library-desk:8089`, `http://core-api:8083`) instead of host `localhost` ports, ahead of the loopback port rebinding; this also fixes `CORE_API_HOST` pointing at port 8090 (the Scheduler's host port) rather than Core-API's 8083. `scripts/test_housekeeper.sh` now reaches Core-API via `localhost:8083` instead of the LAN IP. Local development against host-published ports still works via `.env` overrides
|
||||
|
||||
## [2.4.0] - 2026-07-14
|
||||
|
||||
### Removed
|
||||
|
||||
@@ -27,7 +27,7 @@ Dependencies are in `pyproject.toml` (`[project.dependencies]` and `[project.opt
|
||||
|
||||
**Claude Sonnet 5+ rejects sampling parameters.** `temperature`/`top_p`/`top_k` return a 400. Use `get_sampling_settings()` from the model selector instead of passing `ModelSettings(temperature=...)` directly to agents that can run on the Claude fallback. The contract test suite pins this (`make test-contracts`).
|
||||
|
||||
**Integration test timeouts.** Set to 120s to match `OLLAMA_TIMEOUT` config (300s for the pure-Ollama fallback test, which cannot be rescued by Claude). The full local Steward → orchestrate → synthesize flow takes ~2 minutes on gemma4. Steward analysis alone needs ~35s warm — `STEWARD_TIMEOUT` defaults to 60s.
|
||||
**Integration test timeouts.** Set to 120s to match `OLLAMA_TIMEOUT` config (300s for the pure-Ollama fallback test, which cannot be rescued by Claude). Current GPU-resident numbers (measured 2026-08-07, gemma4:e2b at ~95 tok/s): full Steward → orchestrate → synthesize flow ~10–13s for simple turns; librarian-routed queries ~20-25s (not re-measured). A single turn costs **3 sequential Ollama calls and ~710 generated tokens** even for "what is 61 plus 12?" — most of it the model's own reasoning, paid three times. Cold model load is ~36s, avoided while the model is pinned with `keep_alive: -1`; the `OLLAMA_KEEP_ALIVE=2h` default otherwise reintroduces it. The old "~35s steward / ~2 min flow" and "11–25s flow" figures are superseded — do not plan against them. `STEWARD_TIMEOUT` defaults to 60s.
|
||||
|
||||
**`get_benchmark_store` does not exist.** The benchmarking module (`src/core/benchmarks.py`) was never implemented. `scripts/benchmark_analysis.py` also references it and is broken. Do not add mocks for it in tests.
|
||||
|
||||
|
||||
@@ -28,7 +28,7 @@ src/mcp/
|
||||
|
||||
```yaml
|
||||
tatlock-mcp:
|
||||
image: git.schweitz.internal/jpmschweitzer/tatlock:latest
|
||||
image: git.schweitz.net/jpmschweitzer/tatlock:latest
|
||||
command: ["python", "-m", "src.mcp.server"]
|
||||
ports:
|
||||
- "8002:8002"
|
||||
|
||||
@@ -0,0 +1,110 @@
|
||||
# Steward Routing & Thinking — Findings
|
||||
|
||||
**Outcome: no change shipped.** The Steward stays on `gemma4:e2b` with model
|
||||
thinking left at its default (on). Every alternative was measured and every one
|
||||
loses. This document exists so the experiment is not repeated on the same
|
||||
premise.
|
||||
|
||||
Run 2026-08-08 with `scripts/benchmark_routing.py` and
|
||||
`scripts/fixtures/routing_fixtures.py` (40 labelled queries, one repeat per
|
||||
cell, temperature 0.3 as production sends).
|
||||
|
||||
---
|
||||
|
||||
## The premise was wrong
|
||||
|
||||
The experiment was designed around an observation that the Steward pays ~300
|
||||
tokens per turn for reasoning that is generated and thrown away: it calls
|
||||
`/api/generate`, gemma4 reasons by default, and **no `thinking` field comes back
|
||||
in the response**. Disabling thinking therefore looked close to free.
|
||||
|
||||
It is not. The reasoning is not discarded — it is emitted inline in `response`,
|
||||
and it is what produces a correct `DELEGATE:` line. Those tokens are the work,
|
||||
not waste. Suppressing them costs 12.5 points of routing accuracy.
|
||||
|
||||
## Results
|
||||
|
||||
| config | exact | under | over | tokens | latency | resident | predicted | co-resident with nomic |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| **e2b, thinking** *(production)* | **97.5%** | 2.5% | 0% | 361 | 5179 ms | 1778 MB | 7.8 GiB | yes |
|
||||
| e2b, `think: false` | 85.0% | 12.5% | 5.0% | 48 | 1435 ms | 1778 MB | 7.8 GiB | yes |
|
||||
| e4b, thinking | 100% | 0% | 0% | 192 | 4726 ms | 3089 MB | 10.6 GiB | **no** |
|
||||
| e4b, `think: false` | 97.5% | 2.5% | 0% | 52 | 2269 ms | 3089 MB | 10.6 GiB | **no** |
|
||||
|
||||
`think: true` was also measured and landed within one fixture of the default on
|
||||
both models, so production's implicit thinking is the same thing as asking for
|
||||
it explicitly. Format compliance was 100% in every cell — a `DELEGATE:` line is
|
||||
always emitted.
|
||||
|
||||
With 40 fixtures and one repeat, each result is worth 2.5 points, so the
|
||||
97.5-vs-100 gaps are single fixtures and inside the noise. The latency and token
|
||||
medians (40 calls each) and the e2b `think: false` degradation (6 failures with a
|
||||
consistent mechanism) are the parts worth trusting.
|
||||
|
||||
## Why each alternative loses
|
||||
|
||||
**`think: false` on e2b** — 85% exact, and the failures are not random. All three
|
||||
multi-capability fixtures under-route, each missing a second capability. Without
|
||||
reasoning the model names one capability and stops decomposing. It is not
|
||||
degraded across the board; it specifically stops handling compound requests,
|
||||
which is where a user would most notice the Butler quietly doing half the job.
|
||||
|
||||
**e4b, either setting** — disqualified by memory, not by quality. Ollama predicts
|
||||
**10.6 GiB** for it at 16k context. Maximum available on this card is ~7.9 GiB
|
||||
(10.4 free − 2.0 GPU overhead − 0.46 minimum), so e4b *always* exceeds the budget
|
||||
and evicts every co-resident before loading. Observed directly: loading it threw
|
||||
out both `gemma4:e2b` and `nomic-embed-text`. Losing nomic means Tatlock memory
|
||||
and library-desk thrash on every embedding call. Note this is not caused by the
|
||||
2 GiB reservation — without it, available would be ~9.7 GiB, still under 10.6.
|
||||
|
||||
**Lower `OLLAMA_CONTEXT_LENGTH`** — the obvious way to free headroom, and it does
|
||||
not work. Dropping 16384 → 2048, an 8× reduction, moved the prediction only from
|
||||
7.8 to 6.7 GiB. The prediction is dominated by weights and batch size, not KV
|
||||
cache. It would also truncate the Librarian's retrieved passages and webber's
|
||||
code context for a 14% saving that funds nothing.
|
||||
|
||||
**Per-request `num_ctx`** — worse. A single request with a different `num_ctx`
|
||||
reloads the shared runner, which **drops the `keep_alive: -1` pin** (expiry fell
|
||||
from year-2318 to a 2-hour default) and evicts nomic. Three services share this
|
||||
Ollama, so mixed context sizes are a thrash generator, and it fails silently.
|
||||
|
||||
**`OLLAMA_NUM_PARALLEL > 1`** — never viable here. e2b already predicts 7.8 GiB
|
||||
against ~7.9 available, so there is no room for a second slot at any context
|
||||
length. It is also set to 1 deliberately, to avoid batch overflow panics.
|
||||
|
||||
## What the two axes actually control
|
||||
|
||||
They do not interact, which is the useful part:
|
||||
|
||||
- **Model choice** governs VRAM and co-residency. e2b 1778 MB, e4b 3089 MB.
|
||||
- **Think setting** governs tokens, latency and routing quality — and costs
|
||||
**nothing** in VRAM. Verified: e2b is resident at 1778 MB with `think` unset,
|
||||
true and false alike, because the KV cache is allocated for the full context at
|
||||
load time and `think` is a per-request generation parameter.
|
||||
|
||||
So the only real question is whether 313 tokens and 3.7 seconds are worth 12.5
|
||||
points of compound-query routing. On a turn that is already three sequential
|
||||
Ollama calls, they are.
|
||||
|
||||
## Prerequisite: the extraction fix
|
||||
|
||||
These numbers are only meaningful because `_extract_capabilities` was fixed first
|
||||
(commit `a905363`). It previously substring-matched capability *domains* across
|
||||
the Steward's entire response, so ordinary English in the `REASON:` line selected
|
||||
agents — "description" contains the housekeeper domain "script", "acknowledge"
|
||||
contains "knowledge" and "know".
|
||||
|
||||
That made **prose length a routing input**. Benchmarking against it would have
|
||||
shown `think: false` improving routing purely because shorter output produces
|
||||
fewer accidental substring hits — a thinking policy derived from a parsing
|
||||
artefact. The `adversarial` fixture group is regression coverage for exactly this.
|
||||
|
||||
## If this is revisited
|
||||
|
||||
The constraint is the single 11 GB card, not the model. A second inference host
|
||||
(*forge*) removes it entirely, and e4b's 100% routing becomes reachable without
|
||||
evicting anything. Re-run then; on this card the answer is settled.
|
||||
|
||||
`scripts/benchmark_routing.py` takes `--models`, `--think` and `--repeats`, and
|
||||
restores GPU residency on exit — including on SIGTERM, which the first version
|
||||
did not.
|
||||
+4
-1
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "tatlock"
|
||||
version = "2.4.0"
|
||||
version = "2.4.3"
|
||||
description = "OpenAI-compatible API with Ollama backend"
|
||||
requires-python = ">=3.12"
|
||||
dependencies = [
|
||||
@@ -13,6 +13,9 @@ dependencies = [
|
||||
"pydantic>=2.11,<2.13",
|
||||
"pydantic-settings>=2.12,<2.13",
|
||||
"pydantic-ai-slim[openai,anthropic]>=1.27,<1.28",
|
||||
# pydantic-ai 1.27 imports the private opentelemetry._events module,
|
||||
# removed in opentelemetry-api 1.44 — cap until pydantic-ai is bumped
|
||||
"opentelemetry-api>=1.30,<1.44",
|
||||
"anthropic>=0.77,<1.0",
|
||||
"httpx>=0.28,<0.29",
|
||||
"sse-starlette>=3.0,<3.1",
|
||||
|
||||
@@ -0,0 +1,235 @@
|
||||
"""
|
||||
Benchmark Steward routing quality against model and thinking settings.
|
||||
|
||||
Talks to Ollama directly. No Tatlock server, no agents, no tools, nothing is
|
||||
executed — the mutating fixtures ("turn on the lights", "update the wiki") only
|
||||
ever produce a routing decision. That makes this cheap and repeatable, and it
|
||||
isolates the question: does the Steward still pick the right capabilities when
|
||||
the model reasons less?
|
||||
|
||||
The request body is byte-identical to StewardAgent._call_ollama, plus the
|
||||
`think` flag under test, so a cell labelled `unset` is exactly what production
|
||||
sends today.
|
||||
|
||||
Three thinking settings, because "on vs off" hides the interesting case:
|
||||
|
||||
unset what production sends now. gemma4 reasons by default, and the
|
||||
response carries no `thinking` field, so those tokens are generated
|
||||
and discarded.
|
||||
true reasoning requested explicitly and returned in `thinking`.
|
||||
false reasoning suppressed.
|
||||
|
||||
Scoring is deliberately asymmetric. A missing capability under-routes and the
|
||||
Butler answers without a tool it needed; a spurious one over-routes, and that is
|
||||
a real agent call — a stray librarian is a multi-second web search on a query
|
||||
that asked for arithmetic. Over-routing is the predicted failure when thinking
|
||||
is off, so `forbid` violations are reported separately rather than folded into
|
||||
one accuracy number.
|
||||
|
||||
Usage:
|
||||
.venv/bin/python scripts/benchmark_routing.py
|
||||
.venv/bin/python scripts/benchmark_routing.py --models gemma4:e2b
|
||||
.venv/bin/python scripts/benchmark_routing.py --think false --repeats 3
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import statistics
|
||||
import sys
|
||||
import time
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
from scripts.fixtures.routing_fixtures import FIXTURES # noqa: E402
|
||||
from scripts.ollama_residency import ( # noqa: E402
|
||||
install_sigterm_handler,
|
||||
residency_guard,
|
||||
)
|
||||
from src.agents.steward.agent import build_steward_prompt # noqa: E402
|
||||
from src.agents.steward.service import _DELEGATE_LINE_RE, _extract_capabilities # noqa: E402
|
||||
from src.core.startup import register_household_members # noqa: E402
|
||||
|
||||
OLLAMA_URL = "http://localhost:11434"
|
||||
DEFAULT_MODELS = ["gemma4:e2b", "gemma4:e4b"]
|
||||
DEFAULT_THINK = ["unset", "true", "false"]
|
||||
RESULTS_DIR = PROJECT_ROOT / "logs"
|
||||
|
||||
|
||||
def build_body(model: str, prompt: str, think: str) -> dict[str, Any]:
|
||||
"""Mirror StewardAgent._call_ollama exactly, then add the flag under test."""
|
||||
body: dict[str, Any] = {
|
||||
"model": model,
|
||||
"prompt": prompt,
|
||||
"stream": False,
|
||||
"options": {
|
||||
"temperature": 0.3, # Lower = more consistent
|
||||
"top_p": 0.9,
|
||||
},
|
||||
}
|
||||
if think != "unset":
|
||||
body["think"] = think == "true"
|
||||
return body
|
||||
|
||||
|
||||
def call(client: httpx.Client, body: dict[str, Any]) -> dict[str, Any] | None:
|
||||
try:
|
||||
response = client.post(f"{OLLAMA_URL}/api/generate", json=body)
|
||||
response.raise_for_status()
|
||||
return response.json()
|
||||
except Exception as exc: # noqa: BLE001 - a failed cell must not abort the run
|
||||
print(f" ! {exc}", file=sys.stderr)
|
||||
return None
|
||||
|
||||
|
||||
def score(fixture: dict, found: list[str]) -> dict[str, Any]:
|
||||
expected = set(fixture["expect"])
|
||||
forbidden = set(fixture["forbid"])
|
||||
got = set(found)
|
||||
missing = sorted(expected - got)
|
||||
spurious = sorted(got & forbidden)
|
||||
return {
|
||||
"found": found,
|
||||
"missing": missing,
|
||||
"spurious": spurious,
|
||||
# Exact only when everything expected arrived and nothing forbidden did.
|
||||
"exact": not missing and not spurious,
|
||||
"under_routed": bool(missing),
|
||||
"over_routed": bool(spurious),
|
||||
}
|
||||
|
||||
|
||||
def run_cell(client: httpx.Client, model: str, think: str, repeats: int) -> list[dict[str, Any]]:
|
||||
rows: list[dict[str, Any]] = []
|
||||
for fixture in FIXTURES:
|
||||
prompt = build_steward_prompt(fixture["query"], [])
|
||||
body = build_body(model, prompt, think)
|
||||
for rep in range(repeats):
|
||||
started = time.perf_counter()
|
||||
data = call(client, body)
|
||||
elapsed_ms = (time.perf_counter() - started) * 1000
|
||||
if data is None:
|
||||
rows.append({
|
||||
"id": fixture["id"], "group": fixture["group"], "rep": rep,
|
||||
"error": True, "exact": False, "under_routed": False, "over_routed": False,
|
||||
})
|
||||
continue
|
||||
|
||||
text = data.get("response", "") or ""
|
||||
found = _extract_capabilities(text)
|
||||
rows.append({
|
||||
"id": fixture["id"],
|
||||
"group": fixture["group"],
|
||||
"rep": rep,
|
||||
"error": False,
|
||||
"latency_ms": round(elapsed_ms, 1),
|
||||
"eval_tokens": data.get("eval_count"),
|
||||
"prompt_tokens": data.get("prompt_eval_count"),
|
||||
# Did the model obey the documented output shape at all?
|
||||
"has_delegate_line": bool(_DELEGATE_LINE_RE.search(text)),
|
||||
# Whether reasoning came back, as opposed to being generated and dropped.
|
||||
"thinking_returned": bool(data.get("thinking")),
|
||||
"response_chars": len(text),
|
||||
**score(fixture, found),
|
||||
})
|
||||
return rows
|
||||
|
||||
|
||||
def summarise(rows: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
ok = [r for r in rows if not r["error"]]
|
||||
if not ok:
|
||||
return {"n": 0, "errors": len(rows)}
|
||||
latencies = [r["latency_ms"] for r in ok]
|
||||
tokens = [r["eval_tokens"] for r in ok if r["eval_tokens"] is not None]
|
||||
return {
|
||||
"n": len(ok),
|
||||
"errors": len(rows) - len(ok),
|
||||
"exact_pct": round(100 * sum(r["exact"] for r in ok) / len(ok), 1),
|
||||
"under_routed_pct": round(100 * sum(r["under_routed"] for r in ok) / len(ok), 1),
|
||||
"over_routed_pct": round(100 * sum(r["over_routed"] for r in ok) / len(ok), 1),
|
||||
"format_ok_pct": round(100 * sum(r["has_delegate_line"] for r in ok) / len(ok), 1),
|
||||
"thinking_returned_pct": round(100 * sum(r["thinking_returned"] for r in ok) / len(ok), 1),
|
||||
"latency_ms_median": round(statistics.median(latencies), 1),
|
||||
"latency_ms_mean": round(statistics.fmean(latencies), 1),
|
||||
"eval_tokens_median": round(statistics.median(tokens), 1) if tokens else None,
|
||||
"eval_tokens_total": sum(tokens) if tokens else None,
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
install_sigterm_handler()
|
||||
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--models", default=",".join(DEFAULT_MODELS))
|
||||
parser.add_argument("--think", default=",".join(DEFAULT_THINK),
|
||||
help="comma-separated subset of unset,true,false")
|
||||
parser.add_argument("--repeats", type=int, default=1)
|
||||
parser.add_argument("--timeout", type=float, default=180.0)
|
||||
args = parser.parse_args()
|
||||
|
||||
models = [m.strip() for m in args.models.split(",") if m.strip()]
|
||||
think_modes = [t.strip() for t in args.think.split(",") if t.strip()]
|
||||
|
||||
# build_steward_prompt reads the registry, and the registry is populated at
|
||||
# application startup. Without this the prompt lists no capabilities and every
|
||||
# cell scores zero for reasons that have nothing to do with the model.
|
||||
register_household_members()
|
||||
|
||||
print(f"{len(FIXTURES)} fixtures x {len(models)} models x {len(think_modes)} think "
|
||||
f"x {args.repeats} repeats = {len(FIXTURES) * len(models) * len(think_modes) * args.repeats} calls\n")
|
||||
|
||||
cells: dict[str, Any] = {}
|
||||
# The guard restores production's pinned models however this exits — a
|
||||
# finished run, a failed cell, Ctrl-C or SIGTERM.
|
||||
with residency_guard(models_used=models), httpx.Client(timeout=args.timeout) as client:
|
||||
for model in models:
|
||||
# Absorb the cold load (~36s) outside the measurements.
|
||||
print(f"warming {model} ...", flush=True)
|
||||
call(client, build_body(model, "hi", "false"))
|
||||
for think in think_modes:
|
||||
key = f"{model}|think={think}"
|
||||
print(f" {key} ...", end=" ", flush=True)
|
||||
started = time.perf_counter()
|
||||
rows = run_cell(client, model, think, args.repeats)
|
||||
summary = summarise(rows)
|
||||
cells[key] = {"summary": summary, "rows": rows}
|
||||
print(f"exact={summary.get('exact_pct')}% "
|
||||
f"over={summary.get('over_routed_pct')}% "
|
||||
f"median={summary.get('latency_ms_median')}ms "
|
||||
f"({time.perf_counter() - started:.0f}s)")
|
||||
|
||||
RESULTS_DIR.mkdir(parents=True, exist_ok=True)
|
||||
stamp = datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ")
|
||||
out = RESULTS_DIR / f"routing-bench-{stamp}.json"
|
||||
out.write_text(json.dumps({
|
||||
"generated_at": datetime.now(UTC).isoformat(),
|
||||
"fixtures": len(FIXTURES),
|
||||
"repeats": args.repeats,
|
||||
"cells": cells,
|
||||
}, indent=2))
|
||||
|
||||
print(f"\n{'cell':28} {'exact':>7} {'under':>7} {'over':>7} {'fmt':>6} {'tok':>7} {'ms':>8}")
|
||||
print("-" * 76)
|
||||
for key, cell in cells.items():
|
||||
s = cell["summary"]
|
||||
print(f"{key:28} {s.get('exact_pct'):>6}% {s.get('under_routed_pct'):>6}% "
|
||||
f"{s.get('over_routed_pct'):>6}% {s.get('format_ok_pct'):>5}% "
|
||||
f"{str(s.get('eval_tokens_median')):>7} {s.get('latency_ms_median'):>8}")
|
||||
print(f"\nwritten to {out}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
sys.exit(main())
|
||||
except KeyboardInterrupt:
|
||||
# The residency guard has already run by the time this is caught;
|
||||
# a traceback here would just bury its output.
|
||||
print("\ninterrupted", file=sys.stderr)
|
||||
sys.exit(130)
|
||||
@@ -268,7 +268,7 @@ async def run_benchmarks(iterations: int = 10, verbose: bool = False):
|
||||
print(f" Max: {overall_max:.3f}s (target: ≤5.0s)")
|
||||
print(f" Avg: {overall_avg:.3f}s (target: ≤1.67s)")
|
||||
print(f"\n Recommendations:")
|
||||
print(f" - Switch to a faster model (current: mistral-nemo)")
|
||||
print(f" - Switch to a faster model (current: gemma4:e2b)")
|
||||
print(f" - Reduce system prompt complexity")
|
||||
print(f" - Limit tool calls (currently limited to 3)")
|
||||
print(f" - Consider caching household registry responses")
|
||||
|
||||
@@ -17,12 +17,16 @@ import asyncio
|
||||
import json
|
||||
import re
|
||||
import statistics
|
||||
import sys
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
from scripts.ollama_residency import install_sigterm_handler, residency_guard
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Configuration
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -525,18 +529,31 @@ async def main():
|
||||
original_env = ENV_PATH.read_text()
|
||||
|
||||
all_stats = []
|
||||
# Both restores must survive a crash or an interrupt. The .env one especially:
|
||||
# this script rewrites OLLAMA_DEFAULT_MODEL and lets uvicorn reload onto it,
|
||||
# so bailing out mid-run used to leave the *running server* pointed at the
|
||||
# benchmark model — and DEFAULT_MODELS starts at mistral-nemo-large, the 9.2G
|
||||
# model implicated in the 2026-08-07 VRAM outage.
|
||||
install_sigterm_handler()
|
||||
try:
|
||||
with residency_guard(models_used=models):
|
||||
async with httpx.AsyncClient() as client:
|
||||
for model in models:
|
||||
stats = await benchmark_model(client, model, args.iterations)
|
||||
all_stats.append(stats)
|
||||
|
||||
# Restore original .env
|
||||
finally:
|
||||
ENV_PATH.write_text(original_env)
|
||||
print(f"\n .env restored to original")
|
||||
print("\n .env restored to original")
|
||||
|
||||
print_comparison(all_stats)
|
||||
save_results(all_stats, Path(args.output))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
asyncio.run(main())
|
||||
except KeyboardInterrupt:
|
||||
# .env and GPU residency are both restored by now; do not bury that
|
||||
# output under a traceback.
|
||||
print("\ninterrupted", file=sys.stderr)
|
||||
raise SystemExit(130) from None
|
||||
|
||||
@@ -0,0 +1,159 @@
|
||||
"""
|
||||
Labelled queries for the Steward routing benchmark.
|
||||
|
||||
Each fixture carries both `expect` and `forbid`:
|
||||
|
||||
expect capabilities that must appear. Missing one is under-routing — the
|
||||
Butler answers without a tool it needed.
|
||||
forbid capabilities that must not appear. Over-routing is not cosmetic: a
|
||||
spurious librarian is a real multi-second web call, and a spurious
|
||||
housekeeper can actuate hardware.
|
||||
|
||||
`forbid` matters more than `expect` here, because over-recommendation is the
|
||||
predicted failure when model thinking is disabled and the Steward has less room
|
||||
to discriminate.
|
||||
|
||||
The `adversarial` group deserves explanation. Until 2026-08-08 the extractor
|
||||
substring-matched capability *domains* across the Steward's whole response, so
|
||||
ordinary English in its REASON line selected agents: "description" contains the
|
||||
housekeeper domain "script", "acknowledge" contains "knowledge" and "know",
|
||||
"economy" contains the biographer domain "my". Those queries invite exactly that
|
||||
vocabulary. They now serve as an end-to-end regression: routing must depend on
|
||||
what the Steward *decided*, not on the words it happened to use while explaining.
|
||||
|
||||
Expectations follow the routing rules stated in the Steward prompt itself
|
||||
(src/agents/steward/agent.py), not on what a capability could plausibly cover.
|
||||
"""
|
||||
|
||||
CORE = "tatlock_core"
|
||||
LIB = "librarian"
|
||||
BIO = "biographer"
|
||||
HOUSE = "housekeeper"
|
||||
ALL = [CORE, LIB, BIO, HOUSE]
|
||||
|
||||
|
||||
def _others(*keep: str) -> list[str]:
|
||||
return [c for c in ALL if c not in keep]
|
||||
|
||||
|
||||
FIXTURES: list[dict] = [
|
||||
# --- arithmetic and computation -> tatlock_core --------------------------
|
||||
{"id": "math_add", "group": "math", "query": "What is 61 plus 12?",
|
||||
"expect": [CORE], "forbid": _others(CORE)},
|
||||
{"id": "math_percent", "group": "math", "query": "What is 15% of 240?",
|
||||
"expect": [CORE], "forbid": _others(CORE)},
|
||||
{"id": "math_compound", "group": "math", "query": "If I save 200 a month for 3 years, how much is that?",
|
||||
"expect": [CORE], "forbid": _others(CORE)},
|
||||
{"id": "math_sqrt", "group": "math", "query": "What is the square root of 1764?",
|
||||
"expect": [CORE], "forbid": _others(CORE)},
|
||||
|
||||
# --- date and time -> tatlock_core ---------------------------------------
|
||||
{"id": "time_now", "group": "datetime", "query": "What time is it?",
|
||||
"expect": [CORE], "forbid": _others(CORE)},
|
||||
{"id": "time_date", "group": "datetime", "query": "What is today's date?",
|
||||
"expect": [CORE], "forbid": _others(CORE)},
|
||||
{"id": "time_delta", "group": "datetime", "query": "How many days until Christmas?",
|
||||
"expect": [CORE], "forbid": _others(CORE)},
|
||||
|
||||
# --- personal memory -> biographer ---------------------------------------
|
||||
{"id": "bio_location", "group": "biographer", "query": "Where do I live?",
|
||||
"expect": [BIO], "forbid": [LIB, HOUSE]},
|
||||
{"id": "bio_name", "group": "biographer", "query": "What's my name?",
|
||||
"expect": [BIO], "forbid": [LIB, HOUSE]},
|
||||
{"id": "bio_car", "group": "biographer", "query": "What car do I drive?",
|
||||
"expect": [BIO], "forbid": [LIB, HOUSE]},
|
||||
{"id": "bio_store", "group": "biographer", "query": "Remember that I prefer my coffee black.",
|
||||
"expect": [BIO], "forbid": [LIB, HOUSE]},
|
||||
{"id": "bio_list", "group": "biographer", "query": "What do you know about me?",
|
||||
"expect": [BIO], "forbid": [LIB, HOUSE]},
|
||||
{"id": "bio_forget", "group": "biographer", "query": "Forget my old address.",
|
||||
"expect": [BIO], "forbid": [LIB, HOUSE]},
|
||||
|
||||
# --- research and current information -> librarian ------------------------
|
||||
{"id": "lib_weather", "group": "librarian", "query": "What's the weather in Rotterdam tomorrow?",
|
||||
"expect": [LIB], "forbid": [HOUSE]},
|
||||
{"id": "lib_news", "group": "librarian", "query": "What's in the news today?",
|
||||
"expect": [LIB], "forbid": [HOUSE, BIO]},
|
||||
{"id": "lib_url", "group": "librarian", "query": "Read https://example.com/article and summarise it.",
|
||||
"expect": [LIB], "forbid": [HOUSE, BIO]},
|
||||
{"id": "lib_research", "group": "librarian", "query": "Research how tidal power stations work.",
|
||||
"expect": [LIB], "forbid": [HOUSE, BIO]},
|
||||
{"id": "lib_wiki_create", "group": "librarian", "query": "Create a wiki page about our network topology.",
|
||||
"expect": [LIB], "forbid": [HOUSE, BIO]},
|
||||
|
||||
# --- home automation -> housekeeper --------------------------------------
|
||||
{"id": "house_lights_on", "group": "housekeeper", "query": "Turn on the kitchen lights.",
|
||||
"expect": [HOUSE], "forbid": [LIB, BIO, CORE]},
|
||||
{"id": "house_lights_off", "group": "housekeeper", "query": "Switch off all the lights downstairs.",
|
||||
"expect": [HOUSE], "forbid": [LIB, BIO, CORE]},
|
||||
{"id": "house_thermostat", "group": "housekeeper", "query": "Set the thermostat to 20 degrees.",
|
||||
"expect": [HOUSE], "forbid": [LIB, BIO]},
|
||||
{"id": "house_blinds", "group": "housekeeper", "query": "Close the blinds in the living room.",
|
||||
"expect": [HOUSE], "forbid": [LIB, BIO, CORE]},
|
||||
|
||||
# --- conversational -> nothing at all -------------------------------------
|
||||
# The expensive failure mode: a greeting that triggers a web search.
|
||||
{"id": "chat_greeting", "group": "conversational", "query": "Hello!",
|
||||
"expect": [], "forbid": ALL},
|
||||
{"id": "chat_thanks", "group": "conversational", "query": "Thanks, that's helpful.",
|
||||
"expect": [], "forbid": ALL},
|
||||
{"id": "chat_joke", "group": "conversational", "query": "Tell me a joke.",
|
||||
"expect": [], "forbid": ALL},
|
||||
{"id": "chat_howareyou", "group": "conversational", "query": "How are you doing today?",
|
||||
"expect": [], "forbid": ALL},
|
||||
{"id": "chat_prior_turn", "group": "conversational", "query": "What did I just say?",
|
||||
"expect": [], "forbid": ALL},
|
||||
|
||||
# --- genuinely multi-capability -------------------------------------------
|
||||
{"id": "multi_weather_home", "group": "multi",
|
||||
"query": "What's the weather here, and remember that I like it warm?",
|
||||
"expect": [LIB, BIO], "forbid": []},
|
||||
{"id": "multi_recall_search", "group": "multi",
|
||||
"query": "Look up the best route from my home address to Utrecht.",
|
||||
"expect": [BIO, LIB], "forbid": []},
|
||||
{"id": "multi_math_memory", "group": "multi",
|
||||
"query": "Remember that my budget is 500 euro, then work out 12% of it.",
|
||||
"expect": [BIO, CORE], "forbid": [LIB, HOUSE]},
|
||||
|
||||
# --- adversarial: vocabulary that used to select agents by substring ------
|
||||
# "temperature" is a housekeeper domain, but this is a unit conversion.
|
||||
{"id": "adv_temperature", "group": "adversarial", "query": "Convert 98.6 Fahrenheit to Celsius.",
|
||||
"expect": [CORE], "forbid": [HOUSE, LIB, BIO]},
|
||||
# "description" contains "script"; "discover" contains "cover".
|
||||
{"id": "adv_description", "group": "adversarial",
|
||||
"query": "Give me a short description of what 17 times 23 comes to.",
|
||||
"expect": [CORE], "forbid": [HOUSE, LIB]},
|
||||
# "acknowledge" contains "knowledge" and "know".
|
||||
{"id": "adv_acknowledge", "group": "adversarial",
|
||||
"query": "Just acknowledge this and add 5 and 6 for me.",
|
||||
"expect": [CORE], "forbid": [LIB, BIO]},
|
||||
# "my" appears inside "economy".
|
||||
{"id": "adv_economy", "group": "adversarial",
|
||||
"query": "How many zeros are in one trillion?",
|
||||
"expect": [CORE], "forbid": [BIO, HOUSE]},
|
||||
# "fan" inside "fantastic"; also a climate word without a home-control intent.
|
||||
{"id": "adv_fantastic", "group": "adversarial",
|
||||
"query": "That's fantastic. What is 8 squared?",
|
||||
"expect": [CORE], "forbid": [HOUSE, LIB]},
|
||||
# "home" without any actuation intent.
|
||||
{"id": "adv_home_word", "group": "adversarial", "query": "What time do I usually get home?",
|
||||
"expect": [BIO], "forbid": [HOUSE]},
|
||||
# "search" as ordinary English, not a web-search request.
|
||||
{"id": "adv_search_word", "group": "adversarial",
|
||||
"query": "No need to search anything, just tell me what 9 times 9 is.",
|
||||
"expect": [CORE], "forbid": [LIB]},
|
||||
# "create"/"write" are librarian domains but this is conversational.
|
||||
{"id": "adv_write_word", "group": "adversarial", "query": "Can you write that more simply?",
|
||||
"expect": [], "forbid": [LIB, HOUSE]},
|
||||
|
||||
# --- mutating intents: routing only, nothing is ever executed -------------
|
||||
{"id": "mutate_wiki_update", "group": "mutating", "query": "Update the dossier page with today's findings.",
|
||||
"expect": [LIB], "forbid": [HOUSE, CORE]},
|
||||
{"id": "mutate_scene", "group": "mutating", "query": "Run the movie night scene.",
|
||||
"expect": [HOUSE], "forbid": [LIB, BIO, CORE]},
|
||||
]
|
||||
|
||||
|
||||
GROUPS = sorted({f["group"] for f in FIXTURES})
|
||||
|
||||
assert len({f["id"] for f in FIXTURES}) == len(FIXTURES), "duplicate fixture id"
|
||||
@@ -0,0 +1,121 @@
|
||||
"""
|
||||
Guard production's GPU residency across a benchmark run.
|
||||
|
||||
Benchmarks swap models on the card production is serving from. Ollama evicts to
|
||||
make room, so a run leaves its own models resident and the production one gone:
|
||||
the next voice turn pays a ~36s cold load, and the pin that prevented it is
|
||||
silently lost. That happened on 2026-08-08 — a routing benchmark evicted
|
||||
gemma4:e2b and left gemma4:e4b behind, and only the monitoring noticing
|
||||
`unexpected_models` caught it.
|
||||
|
||||
Snapshot before, restore after, and wire the restore to SIGTERM as well as the
|
||||
normal path. Python runs `finally` for SIGINT, which arrives as
|
||||
KeyboardInterrupt, but the default SIGTERM action terminates outright — so
|
||||
`timeout`, a systemd stop or a plain `kill` would skip the guard entirely.
|
||||
|
||||
from scripts.ollama_residency import residency_guard, install_sigterm_handler
|
||||
|
||||
install_sigterm_handler()
|
||||
with residency_guard(models_used=["gemma4:e4b"]):
|
||||
...
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import signal
|
||||
from collections.abc import Iterator
|
||||
from contextlib import contextmanager
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
OLLAMA_URL = "http://localhost:11434"
|
||||
|
||||
# keep_alive:-1 yields a year-2318 expiry, so "pinned" is simply "expires more
|
||||
# than a day out". Matches check-ai-pipeline.sh in system-admin-toj.
|
||||
PINNED_THRESHOLD_SECONDS = 86400
|
||||
|
||||
|
||||
def install_sigterm_handler() -> None:
|
||||
"""Make SIGTERM raise, so `finally` blocks and context managers still run."""
|
||||
def _raise(signum, _frame):
|
||||
raise KeyboardInterrupt(f"signal {signum}")
|
||||
|
||||
signal.signal(signal.SIGTERM, _raise)
|
||||
|
||||
|
||||
def snapshot_residency(client: httpx.Client | None = None) -> dict[str, bool]:
|
||||
"""Resident models mapped to whether each is pinned."""
|
||||
owns = client is None
|
||||
client = client or httpx.Client(timeout=30)
|
||||
try:
|
||||
data = client.get(f"{OLLAMA_URL}/api/ps", timeout=10).json()
|
||||
except Exception: # noqa: BLE001 - a missing snapshot must not abort the run
|
||||
return {}
|
||||
finally:
|
||||
if owns:
|
||||
client.close()
|
||||
|
||||
resident: dict[str, bool] = {}
|
||||
now = datetime.now(UTC)
|
||||
for model in data.get("models", []):
|
||||
pinned = False
|
||||
try:
|
||||
expires = datetime.fromisoformat(model.get("expires_at", "").replace("Z", "+00:00"))
|
||||
pinned = (expires - now).total_seconds() > PINNED_THRESHOLD_SECONDS
|
||||
except ValueError:
|
||||
pass
|
||||
resident[model["name"]] = pinned
|
||||
return resident
|
||||
|
||||
|
||||
def set_keep_alive(model: str, keep_alive: Any, client: httpx.Client | None = None) -> bool:
|
||||
"""Load, unload or pin a model. Embedding models reject /api/generate."""
|
||||
owns = client is None
|
||||
client = client or httpx.Client(timeout=180)
|
||||
payload = {"model": model, "keep_alive": keep_alive}
|
||||
try:
|
||||
for endpoint in ("generate", "embed"):
|
||||
try:
|
||||
response = client.post(f"{OLLAMA_URL}/api/{endpoint}", json=payload, timeout=180)
|
||||
except Exception: # noqa: BLE001
|
||||
return False
|
||||
if response.status_code == 200:
|
||||
return True
|
||||
if response.status_code == 400 and "does not support generate" in response.text:
|
||||
continue # embedding-only model; try /api/embed
|
||||
return False
|
||||
return False
|
||||
finally:
|
||||
if owns:
|
||||
client.close()
|
||||
|
||||
|
||||
def restore_residency(before: dict[str, bool], used: list[str]) -> None:
|
||||
"""Evict what the benchmark loaded, then re-pin what was pinned before."""
|
||||
base = {name.split(":")[0] for name in before}
|
||||
with httpx.Client(timeout=180) as client:
|
||||
for model in used:
|
||||
if model not in before and model.split(":")[0] not in base:
|
||||
print(f" residency: unloading benchmark model {model}")
|
||||
set_keep_alive(model, 0, client)
|
||||
for name, pinned in before.items():
|
||||
if not pinned:
|
||||
continue
|
||||
ok = set_keep_alive(name, -1, client)
|
||||
print(f" residency: re-pinned {name}" if ok
|
||||
else f" residency: FAILED to re-pin {name} -- run warmup-ollama.sh")
|
||||
|
||||
|
||||
@contextmanager
|
||||
def residency_guard(models_used: list[str]) -> Iterator[dict[str, bool]]:
|
||||
"""Snapshot residency on entry, restore it on exit however that happens."""
|
||||
before = snapshot_residency()
|
||||
pinned = [n for n, p in before.items() if p]
|
||||
print(f" residency: resident before {sorted(before)}"
|
||||
f"{f' (pinned: {pinned})' if pinned else ''}")
|
||||
try:
|
||||
yield before
|
||||
finally:
|
||||
print(" residency: restoring ...")
|
||||
restore_residency(before, models_used)
|
||||
@@ -3,7 +3,7 @@
|
||||
# Verifies room groups are controlled by checking actual state changes
|
||||
|
||||
API_URL="http://localhost:8777/v1/chat/completions"
|
||||
CORE_API="http://192.168.86.149:8083"
|
||||
CORE_API="http://localhost:8083"
|
||||
RESULTS_FILE="/tmp/housekeeper_test_results.txt"
|
||||
|
||||
GREEN='\033[0;32m'
|
||||
|
||||
@@ -19,35 +19,88 @@ from .schemas import ConversationContext, StewardRecommendation
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
_DELEGATE_LINE_RE = re.compile(r"^[ \t]*DELEGATE:[ \t]*(.+)$", re.IGNORECASE | re.MULTILINE)
|
||||
|
||||
|
||||
def _mentions(needle: str, haystack: str) -> bool:
|
||||
"""Whole-word containment. Substring matching is what made this go wrong."""
|
||||
return re.search(rf"(?<!\w){re.escape(needle)}(?!\w)", haystack) is not None
|
||||
|
||||
|
||||
def _extract_capabilities(text: str) -> list[str]:
|
||||
"""
|
||||
Extract capability names from Steward's text response.
|
||||
Extract capability names from the Steward's declared delegation.
|
||||
|
||||
Uses keyword matching to find mentioned capabilities.
|
||||
The prompt instructs the Steward to answer in a fixed shape::
|
||||
|
||||
DELEGATE: <capability> to <action> <task>
|
||||
REASON: ...
|
||||
COMPLEXITY: ...
|
||||
CONTEXT: ...
|
||||
|
||||
Only the DELEGATE line states intent; the rest is free prose. An earlier
|
||||
version substring-matched capability *domains* across the whole response,
|
||||
which routed on ordinary English: "description" contains "script" and
|
||||
"discover" contains "cover" (both housekeeper domains), "acknowledge"
|
||||
contains "knowledge" and "know" (librarian, biographer), and "economy"
|
||||
contains "my" (biographer). Any REASON line could therefore summon agents
|
||||
the Steward never asked for, and a spurious librarian is a real
|
||||
multi-second web call.
|
||||
|
||||
It also made prose length a routing input, so anything that shortened the
|
||||
Steward's output — such as disabling model thinking — would look like it had
|
||||
improved routing.
|
||||
|
||||
Resolution is layered, most explicit first:
|
||||
1. a DELEGATE line beginning with a capability name — the documented shape
|
||||
2. a capability named anywhere on a DELEGATE line
|
||||
3. a capability *domain* on a DELEGATE line, for a loosely worded answer
|
||||
4. no DELEGATE line: capability names only, never domains
|
||||
|
||||
Args:
|
||||
text: Steward's plain text analysis
|
||||
|
||||
Returns:
|
||||
List of capability names (e.g., ['tatlock_core'])
|
||||
List of capability names (e.g. ['tatlock_core']), de-duplicated.
|
||||
"""
|
||||
text_lower = text.lower()
|
||||
registry = get_household_registry()
|
||||
capabilities = registry.get_all_capabilities()
|
||||
delegate_lines = [line.strip().lower() for line in _DELEGATE_LINE_RE.findall(text or "")]
|
||||
|
||||
found_caps = []
|
||||
found_caps: list[str] = []
|
||||
|
||||
def _add(name: str) -> None:
|
||||
if name not in found_caps:
|
||||
found_caps.append(name)
|
||||
|
||||
if not delegate_lines:
|
||||
# Either the Steward judged no capability necessary — the prompt's
|
||||
# conversational path, whose correct answer is [] — or it ignored the
|
||||
# format. Names only: domain words are ordinary English and would fire
|
||||
# on any prose, which is the bug described above.
|
||||
haystack = (text or "").lower()
|
||||
for cap in capabilities:
|
||||
# Check if capability name is mentioned
|
||||
if cap.name.lower() in text_lower:
|
||||
found_caps.append(cap.name)
|
||||
if _mentions(cap.name.lower(), haystack):
|
||||
_add(cap.name)
|
||||
return found_caps
|
||||
|
||||
for line in delegate_lines:
|
||||
leading = next((c for c in capabilities if line.startswith(c.name.lower())), None)
|
||||
if leading is not None:
|
||||
_add(leading.name)
|
||||
continue
|
||||
|
||||
# Check if any domains are mentioned
|
||||
for domain in cap.domains:
|
||||
if domain.lower() in text_lower:
|
||||
found_caps.append(cap.name)
|
||||
break
|
||||
named = [c for c in capabilities if _mentions(c.name.lower(), line)]
|
||||
if named:
|
||||
for cap in named:
|
||||
_add(cap.name)
|
||||
continue
|
||||
|
||||
# Last resort. Scoped to this line, so the REASON and CONTEXT prose that
|
||||
# caused the original misrouting can no longer reach it.
|
||||
for cap in capabilities:
|
||||
if any(_mentions(domain.lower(), line) for domain in cap.domains):
|
||||
_add(cap.name)
|
||||
|
||||
return found_caps
|
||||
|
||||
|
||||
+6
-6
@@ -110,8 +110,8 @@ class Config(BaseSettings):
|
||||
|
||||
# SearXNG Configuration
|
||||
SEARXNG_HOST: HttpUrl = Field(
|
||||
default="http://localhost:8087",
|
||||
description="SearXNG server URL"
|
||||
default="http://searxng:8080",
|
||||
description="SearXNG server URL (container name; internal port 8080)"
|
||||
)
|
||||
SEARXNG_TIMEOUT: int = Field(
|
||||
default=30,
|
||||
@@ -138,8 +138,8 @@ class Config(BaseSettings):
|
||||
description="Total time budget for a librarian delegation in seconds"
|
||||
)
|
||||
LIBRARY_DESK_HOST: HttpUrl = Field(
|
||||
default="http://localhost:8089",
|
||||
description="Library-Desk API URL"
|
||||
default="http://library-desk:8089",
|
||||
description="Library-Desk API URL (container name; internal port 8089)"
|
||||
)
|
||||
LIBRARY_DESK_API_KEY: str = Field(
|
||||
default="",
|
||||
@@ -152,8 +152,8 @@ class Config(BaseSettings):
|
||||
|
||||
# Core-API Configuration (The Housekeeper backend)
|
||||
CORE_API_HOST: HttpUrl = Field(
|
||||
default="http://localhost:8090",
|
||||
description="Core-API URL for Home Assistant integration"
|
||||
default="http://core-api:8083",
|
||||
description="Core-API URL for Home Assistant integration (container name; internal port 8083)"
|
||||
)
|
||||
CORE_API_KEY: str = Field(
|
||||
default="",
|
||||
|
||||
@@ -8,7 +8,12 @@ from unittest.mock import AsyncMock, MagicMock, patch
|
||||
import pytest
|
||||
|
||||
from src.agents.steward.schemas import ConversationContext, StewardRecommendation
|
||||
from src.agents.steward.service import analyze_request, format_steward_note, _build_enriched_query
|
||||
from src.agents.steward.service import (
|
||||
_build_enriched_query,
|
||||
_extract_capabilities,
|
||||
analyze_request,
|
||||
format_steward_note,
|
||||
)
|
||||
from src.core.startup import register_household_members
|
||||
|
||||
|
||||
@@ -283,3 +288,89 @@ class TestBuildEnrichedQuery:
|
||||
result = _build_enriched_query(query, memory_context)
|
||||
|
||||
assert result == query
|
||||
|
||||
|
||||
class TestExtractCapabilities:
|
||||
"""Capability extraction reads the declared DELEGATE line, not free prose.
|
||||
|
||||
The prompt tells the Steward to state its choice on a DELEGATE line and to
|
||||
explain itself on REASON/COMPLEXITY/CONTEXT lines. An earlier version
|
||||
substring-matched capability domains across the entire response, so ordinary
|
||||
English in the explanation routed requests: "description" contains the
|
||||
housekeeper domain "script", "acknowledge" contains "know". These tests pin
|
||||
that the explanation can no longer influence routing.
|
||||
"""
|
||||
|
||||
# (prose, why it used to misroute)
|
||||
SUBSTRING_TRAPS = [
|
||||
("The user wants a description of the algorithm.", "script -> housekeeper"),
|
||||
("I should discover what the answer is.", "cover -> housekeeper"),
|
||||
("That sounds fantastic, let me compute it.", "fan -> housekeeper"),
|
||||
("I acknowledge the request to add two numbers.", "knowledge/know -> librarian, biographer"),
|
||||
("The user asks about the economy myth.", "my -> biographer"),
|
||||
("Convert 98.6 Fahrenheit to Celsius.", "temperature is a housekeeper domain"),
|
||||
]
|
||||
|
||||
@pytest.mark.parametrize("prose,reason", SUBSTRING_TRAPS)
|
||||
def test_reason_prose_cannot_add_capabilities(self, prose, reason):
|
||||
"""Explanatory prose must not summon agents the Steward did not request."""
|
||||
text = f"DELEGATE: tatlock_core to calculate\nREASON: {prose}\nCOMPLEXITY: simple"
|
||||
|
||||
assert _extract_capabilities(text) == ["tatlock_core"], f"regression: {reason}"
|
||||
|
||||
def test_delegate_line_task_text_does_not_leak(self):
|
||||
"""A domain word inside the task description must not add a capability.
|
||||
|
||||
"home" is a housekeeper domain, but this is plainly a memory recall.
|
||||
"""
|
||||
text = "DELEGATE: biographer to recall the user's home address\nREASON: personal data"
|
||||
|
||||
assert _extract_capabilities(text) == ["biographer"]
|
||||
|
||||
def test_multiple_delegate_lines(self):
|
||||
"""Each DELEGATE line contributes its capability, in order, deduplicated."""
|
||||
text = (
|
||||
"DELEGATE: biographer to recall the user's location\n"
|
||||
"DELEGATE: librarian to search_web for the forecast\n"
|
||||
"DELEGATE: biographer to recall preferences\n"
|
||||
)
|
||||
|
||||
assert _extract_capabilities(text) == ["biographer", "librarian"]
|
||||
|
||||
def test_capability_named_later_on_the_line(self):
|
||||
"""A loosely worded DELEGATE line still resolves by name."""
|
||||
text = "DELEGATE: ask the librarian to search the web"
|
||||
|
||||
assert _extract_capabilities(text) == ["librarian"]
|
||||
|
||||
def test_domain_fallback_within_delegate_line(self):
|
||||
"""With no capability named, domains on the DELEGATE line still resolve."""
|
||||
text = "DELEGATE: turn on the lights in the kitchen"
|
||||
|
||||
assert _extract_capabilities(text) == ["housekeeper"]
|
||||
|
||||
def test_conversational_response_selects_nothing(self):
|
||||
"""No DELEGATE line means no capability, which is the prompt's chat path."""
|
||||
text = "This is a simple greeting. No capabilities are needed. COMPLEXITY: simple"
|
||||
|
||||
assert _extract_capabilities(text) == []
|
||||
|
||||
def test_malformed_response_still_routes_by_name(self):
|
||||
"""If the format is ignored, a named capability is still honoured."""
|
||||
text = "I think the librarian should handle this research request."
|
||||
|
||||
assert _extract_capabilities(text) == ["librarian"]
|
||||
|
||||
def test_malformed_response_does_not_route_on_domains(self):
|
||||
"""...but bare prose must not route on domain words alone."""
|
||||
text = "The user wants a description of home automation, and I acknowledge it."
|
||||
|
||||
assert _extract_capabilities(text) == []
|
||||
|
||||
def test_case_insensitive_delegate_marker(self):
|
||||
text = "delegate: Librarian to search_web"
|
||||
|
||||
assert _extract_capabilities(text) == ["librarian"]
|
||||
|
||||
def test_empty_input(self):
|
||||
assert _extract_capabilities("") == []
|
||||
|
||||
+5
-5
@@ -4,8 +4,8 @@ These tests make real HTTP requests to the running Tatlock API server to verify
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. **Server must be running** on `http://localhost:8777` (use `./wakeup.sh`)
|
||||
2. **Ollama must be running** with `mistral-nemo:latest` model
|
||||
1. **Server must be running** on `http://localhost:8777` (use `make run`)
|
||||
2. **Ollama must be running** with the `gemma4:e2b` model
|
||||
3. **Redis must be running** (for benchmarking)
|
||||
4. **Qdrant must be running** on `http://localhost:6333` (for memory tests)
|
||||
|
||||
@@ -15,9 +15,9 @@ These tests make real HTTP requests to the running Tatlock API server to verify
|
||||
|
||||
```bash
|
||||
# Terminal 1: Start the server (auto-reload enabled)
|
||||
./wakeup.sh
|
||||
make run
|
||||
|
||||
# Logs are written to logs/server.log - tail them in another terminal:
|
||||
# Logs are written to build/logs/server.log - tail them in another terminal:
|
||||
tail -f logs/server.log
|
||||
```
|
||||
|
||||
@@ -132,7 +132,7 @@ memory = await qdrant.find_memory_by_key("memories_llm_tester", "favorite_color"
|
||||
|
||||
Make sure the server is running:
|
||||
```bash
|
||||
./wakeup.sh
|
||||
make run
|
||||
curl http://localhost:8777/health # Should return 200
|
||||
```
|
||||
|
||||
|
||||
@@ -11,9 +11,9 @@ These tests hit the actual running server and verify data persistence.
|
||||
They use the `llm_tester` user for isolation from production data.
|
||||
|
||||
Requirements:
|
||||
- Server running on localhost:8777 (use ./wakeup.sh)
|
||||
- Server running on localhost:8777 (use `make run`)
|
||||
- Qdrant running on localhost:6333
|
||||
- Ollama running with mistral-nemo model
|
||||
- Ollama running with the gemma4:e2b model
|
||||
|
||||
Note: LLM outputs are non-deterministic. Tests use flexible assertions
|
||||
that check for behavioral patterns rather than exact text matches.
|
||||
|
||||
Reference in New Issue
Block a user