Compare commits

...
33 Commits
Author SHA1 Message Date
jpmschweitzerandClaude Fable 5 cdd5a55613 chore: release v2.4.2
Build and Push / release (push) Successful in 3s
Build and Push / build (push) Successful in 1m3s
Fixes the v2.4.1 crash-loop: fresh image builds resolved
opentelemetry-api 1.44.0, which removed the private _events module
that pydantic-ai 1.27 imports at startup.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 13:10:57 +02:00
jpmschweitzerandClaude Fable 5 287d66fff7 chore: release v2.4.1
Build and Push / release (push) Successful in 3s
Build and Push / build (push) Successful in 2m44s
Move the container-name network-defaults change from [Unreleased] into
the 2.4.1 section and bump pyproject.toml. Patch release: the change
corrects service-host defaults (SEARXNG_HOST, LIBRARY_DESK_HOST,
CORE_API_HOST) for the docker-dataplane deployment, including the
wrong CORE_API_HOST port.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 12:32:10 +02:00
jpmschweitzerandClaude Fable 5 65debb6e44 fix(config): default service hosts to docker container names
The homelab is retiring *.schweitz.internal and will rebind host
ports to loopback; container-to-container traffic must use container
names on docker-dataplane.

- SEARXNG_HOST: http://localhost:8087 -> http://searxng:8080
  (SearXNG's internal port is 8080; 8087 was the host-published port)
- LIBRARY_DESK_HOST: http://localhost:8089 -> http://library-desk:8089
- CORE_API_HOST: http://localhost:8090 -> http://core-api:8083
  (8090 is the Scheduler's host port; Core-API serves 8083 internally,
  confirmed by the housekeeper client and test suite hitting :8083)
- scripts/test_housekeeper.sh: reach Core-API via localhost:8083
  instead of the LAN IP, which will refuse after loopback rebinding

Local development against host-published ports keeps working via .env
overrides (.env.example unchanged; localhost stays valid on the host).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 12:21:40 +02:00
jpmschweitzerandClaude Fable 5 0d7514b90e chore(ci): push images via git.schweitz.net registry
The .internal registry domain is being retired; git.schweitz.net now
serves the registry without SSO on /v2/.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 11:10:04 +02:00
jpmschweitzerandClaude Fable 5 99683357d2 docs: replace CPU-era latency figures with measured GPU numbers
The ~35s steward / ~2 min flow figures dated from the driver-mismatch era
and were being inherited by downstream consumers (desklock architecture
doc) as planning baselines. Current measured: steward ~6s warm, full flow
11-25s.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 17:17:13 +02:00
jpmschweitzerandClaude Fable 5 f0a08ede64 chore: release v2.4.0
Build and Push / release (push) Successful in 3s
Build and Push / build (push) Successful in 1m55s
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 15:50:02 +02:00
jpmschweitzerandClaude Fable 5 a8bc282576 fix(librarian): expose page_id in wiki search results
search_wiki printed ordinally numbered results with no page ID while
get_wiki_page demands 'the page ID from search results' - the model
passed the list position (page 1) and 404'd. Results now carry
page_id and drop the ordinals.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 15:37:00 +02:00
jpmschweitzerandClaude Fable 5 9184b48673 fix(ollama): sanitize null content on every message shape
gemma thinking-only assistant turns carry content: null with no
tool_calls, slipping past the tool-call-only sanitizer and 400ing the
whole agent run ('invalid message content type: <nil>'). Null content is
now blanked for any role.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 15:37:00 +02:00
jpmschweitzerandClaude Fable 5 fccbe65ecf fix(librarian): use terse tool-phase prompt so gemma4 calls tools
The scholarly persona prompt reproduced the exact pathology
TATLOCK_ORCHESTRATION_PROMPT fixed for the butler: gemma4 answered in
character ('please provide your request') without calling a single tool.
The research phase now uses a tool-discipline prompt; Tatlock's synthesis
supplies the voice. Anti-fabrication rules kept verbatim.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 15:37:00 +02:00
jpmschweitzerandClaude Fable 5 2f6e444147 docs: drop deleted run_librarian_stream from unreleased changelog entry
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 12:45:48 +02:00
jpmschweitzerandClaude Fable 5 5376a34645 refactor(agents): reduce protocol.py to the live AgentError
Post-coordination-removal sweep: the coordination wire protocol
(AgentRequest, AgentResponse, DelegationIntent, CoordinationResult,
DelegationReason, TaskComplexity, ToolCallRecord, AgentTimeoutError,
AgentUnavailableError, DelegationError) had zero importers left in
src/ - only its own test module. AgentError stays (raised by
run_librarian, mapped to user-safe failures by delegation.py).
Also drops the stale coordination.py line from the README tree.

Import-cycle sanity: python -c 'import src.main' passes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
2026-07-14 12:03:31 +02:00
jpmschweitzerandClaude Fable 5 8cf3609948 fix(librarian): quiet healthy-search coverage notes, tag clearing, strict result pairing
Phase A review minors:

- Coverage note: source_status (when present) is now used exclusively;
  the source_counts-absence fallback only considers the optional legs
  the request explicitly enabled (web/documents/volatile). library-desk
  computes source_counts from the final top-N fused results only, so
  absence of the always-on vector/graph legs is normal ranking behavior
  - the old heuristic warned on virtually every healthy search
- update_wiki_page: the empty-list tags sentinel (leave unchanged) made
  clearing all tags impossible; pass exactly ["__CLEAR__"] to send an
  empty tag list, documented in the docstring for the local model
- Text-delegation parallel fallback: zip(..., strict=True) with an
  explicit count-mismatch guard so results can never be silently
  attributed to the wrong agent

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
2026-07-14 11:59:15 +02:00
jpmschweitzerandClaude Fable 5 31b948a748 refactor(agents): delete dead coordination/streaming delegation stack
One delegation implementation remains (src/agents/delegation.py).
Removed, after verifying zero live importers post-Phase-A/B:

- src/agents/coordination.py: CoordinationEngine, duplicate
  delegate_to_librarian, AGENT_EXECUTORS/AGENT_STREAM_EXECUTORS
  (only importer was its own test module)
- run_librarian_stream: documented-broken path (Ollama streaming +
  tool call bug, PydanticAI #1292/#2256), only called by the deleted
  coordination engine
- stream_delegate_to_* wrappers + STREAMING_DELEGATION_WRAPPERS and
  the never-parsed __DELEGATION_RESULT__ marker in delegation.py
- HouseholdRegistry.get_streaming_delegation_tools() (no callers)
- tests/agents/test_coordination.py and the wrapper/stream tests

Note: the STREAMING_DELEGATION_WRAPPERS import in
src/responses/streaming.py was already removed by Phase A (7ce1c1a);
nothing to delete there.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
2026-07-14 11:52:36 +02:00
jpmschweitzerandClaude Fable 5 c00224222b fix(librarian): apply tenant guard to explicit user args in client
LibraryDeskClient._resolve_user only enforced non-empty: an explicit
user argument to any tenant-scoped method bypassed tatlock's tenant
guard entirely and went straight to library-desk, and padded values
were sent un-stripped on the wire.

Route the explicit-arg path through the same apply_tenant_guard() used
by context resolution and strip whitespace before the empty check, so
a non-production environment can never send the production tenant (or
a sanitization-collision variant) to library-desk, regardless of how
the user was supplied. Defense in depth - no in-repo caller passes an
explicit user today.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 11:29:14 +02:00
jpmschweitzerandClaude Fable 5 e8e5d367b6 fix(tenant): guard against sanitization collisions with production tenant
The request-level tenant guard compared the raw user string exactly
(user == PRODUCTION_TENANT), but all local namespaces (Qdrant
collections, Redis keys) are derived through sanitize_user_id(), which
lowercases and strips/maps punctuation. Case or punctuation variants
("JPMSchweitzer", "jpmschweitzer.", " jpmschweitzer") therefore passed
the guard yet resolved to the production namespaces, letting a dev
instance on the shared services read/write production tenant data.

- context.py: compare sanitize_user_id(user) against the sanitized
  production tenant; expose the guard as public apply_tenant_guard()
- config.py: startup refusal validator uses the same sanitized
  comparison, so a colliding DEFAULT_USER refuses startup loudly
  instead of relying on the allowlist fallback
- tests: variant matrix at both config and request-context level,
  plus a non-colliding passthrough case

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 11:23:36 +02:00
jpmschweitzerandClaude Fable 5 b47c5b9281 test: hard-fail the suite when the tenant resolves to production
Session-scoped autouse guard in tests/conftest.py refuses to run any
test (pytest.exit, returncode 1) when the effective tenant resolves
to the production tenant jpmschweitzer - the same guard library-desk
applies on its side. _initialize_app now depends on the guard so the
refusal happens before any initialization.

Suite-level assertions pin that the live session runs under the
llm_tester namespaces: Qdrant memories_llm_tester collection and
Redis session:llm_tester:* keys. The biographer/memory unit tests
already run fully mocked (no shared-service writes); the e2e
isolation tests already used llm_tester - their constants now derive
from the shared TEST_TENANT/PRODUCTION_TENANT config constants so a
drift fails loudly instead of silently splitting.

Verified: ENVIRONMENT=production pytest run exits 1 with the TENANT
GUARD message and zero tests executed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 11:08:39 +02:00
jpmschweitzerandClaude Fable 5 b9eae38556 feat(librarian): send explicit non-empty user on every library-desk request
Library-desk is removing its server-side default user, so a request
without an explicit tenant will 422 after its next deploy:

- New client-level _resolve_user() resolves the tenant (explicit arg
  or request context) and raises ValueError on an empty/whitespace
  value BEFORE any bytes hit the wire; all 15 tenant-scoped methods
  use it
- extract_content / extract_content_batch now accept and send the
  user (query param), matching the rest of the API surface
- search_web no longer falls back to a phantom "tatlock-librarian"
  tenant; it sends the resolved user
- health_check stays user-less (public, not tenant-scoped)

Tests: parametrized sweep pins the wire contract (user present in
params or payload) for every tenant-scoped method, for both context
and explicit users; empty-tenant calls are asserted to fail without
any HTTP call; the recorded-fixture hybrid contract test now pins
user as an explicit query param.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 11:02:46 +02:00
jpmschweitzerandClaude Fable 5 4b786a766c feat(core): enforce tenant isolation guard outside production
Non-production environments (development/testing) now force the
effective tenant to the reserved test tenant "llm_tester" (or a
test_-prefixed override) regardless of DEFAULT_USER misconfiguration:

- Config.effective_default_user only honors DEFAULT_USER outside
  production when it is llm_tester or test_-prefixed; anything else
  is forced to llm_tester (tenant_forced flags the override)
- Config refuses startup (validation error) when a non-production
  environment is explicitly configured with the production tenant
  jpmschweitzer
- get_user() applies the same guard at request-context resolution,
  so an explicit request for the production tenant in dev/test is
  forced to llm_tester with a warning log
- initialize_application() emits one loud startup log line
  (tenant_guard_active / tenant_guard_production) stating the
  effective tenant

Unit tests cover the dev/test/prod x default/explicit-user matrix.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 10:56:29 +02:00
jpmschweitzerandClaude Fable 5 7ce1c1a314 feat(responses): pass conversation context and stream thinks in real time
- delegate_to_* now receives a trimmed conversation history (last ~6
  turns, 500 chars/turn) as context on both live direct-delegation
  paths (streaming and steward non-streaming), via new
  build_delegation_context helper
- _stream_direct_delegation restructured as an async generator: the
  butler 'start' think message streams BEFORE the expert runs and the
  success/error message right after it finishes, instead of all
  messages arriving after the research completed

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 10:30:34 +02:00
jpmschweitzerandClaude Fable 5 0708c759fc fix(librarian): replace nullable tool params with sentinel defaults
Ollama's OpenAI-compatible API mishandles anyOf[X, null] parameter
schemas. update_wiki_page (content/title/tags/description) and
smart_create_wiki_page (path) now use empty-string/empty-list
sentinels translated to None inside the tool, following the
biographer pattern from 9d7ce39.

Adds a snapshot test that walks every registered librarian tool's
emitted JSON schema and fails on any anyOf[..., null].

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 10:26:55 +02:00
jpmschweitzerandClaude Fable 5 24fed8814f feat(librarian): bounded retries, timeout wiring, and client reuse
- 2-attempt short-backoff retry for GETs and the read-only
  POST /query/* and /rag/search endpoints only; wiki writes are never
  retried (duplicate-page risk)
- honor the defined-but-ignored LIBRARY_DESK_TIMEOUT config instead of
  hardcoded 60s/30s per-call values
- hold ONE shared httpx.AsyncClient per librarian run via
  library_client_session (contextvar), instead of constructing a
  client per tool call; nested sessions are no-ops and custom targets
  still get their own client
- read tools raise ModelRetry on transient HTTP errors (transport
  errors, 5xx, 429) so Agent(retries=2) engages; write tools keep
  returning safe failure messages

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 10:24:42 +02:00
jpmschweitzerandClaude Fable 5 18f2e0efbd feat(agents): enforce one librarian timeout budget
- add LIBRARIAN_TIMEOUT config (default 180s) and enforce it with
  asyncio.wait_for inside delegate_to_librarian, covering the live
  paths (steward direct delegation and SSE streaming) that had no cap
- timeouts fail honestly: success=False with a curated butler sentence,
  detail in logs
- set an explicit timeout on TatlockOllamaProvider's AsyncOpenAI client
  from OLLAMA_TIMEOUT instead of the SDK default (~600s per LLM call)
- remove the contradictory unused 60s default from
  AgentRequest.timeout_seconds; coordination falls back to the
  configured budget

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 10:19:16 +02:00
jpmschweitzerandClaude Fable 5 99e1fe33ca feat(librarian): signal degraded search coverage
- parse source_counts into HybridRAGResponse and additively parse the
  shared-contract source_status/degraded fields when present (absence
  tolerated, so deploy order between tatlock and library-desk never
  matters)
- hybrid_search appends a one-line coverage note when a leg reported
  'failed' (or degraded is set), falling back to inferring silent legs
  from source_counts on older library-desk versions

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 10:15:09 +02:00
jpmschweitzerandClaude Fable 5 f853db8ccc fix(agents): make librarian failures structured and user-safe
- run_librarian / run_librarian_stream raise AgentError instead of
  returning/yielding error text as normal output; detail stays in logs
- delegate_to_* wrappers now put a curated butler-toned sentence in
  DelegationResult.output on failure and never expose str(e), so
  streaming's error branch is reachable and honest
- _execute_single_delegation propagates success; direct delegation only
  records delegate_to_* as called when the expert actually succeeded
- librarian tools return user-safe messages instead of
  'Error searching: {e}' strings that leaked internal URLs into
  synthesis; coordination stream errors are curated as well
- ruff cleanups (TYPE_CHECKING forward refs, B904, unused locals) in
  the touched files to keep them lint-clean

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 10:12:30 +02:00
jpmschweitzerandClaude Fable 5 59f5b54ac9 fix(librarian): map live HybridRAG response fields correctly
The client parsed field names the live library-desk service never
returns, so every result rendered as "unknown (score: 0.00)":

- source_type/sources -> source + sources (icons key off sources values)
- rrf_score -> score
- context -> formatted_context
- related_dossiers are per-result; top level aggregates unique titles
- synonyms live inside the keywords dict as a {term: [synonyms]} map

Also stop sending zero limits (service 422s on limit < 1); disabled
legs now rely on the enable_* flags with limits clamped to >= 1.

Adds a recorded live response as a fixture plus contract tests that
pin the mapping (non-unknown sources, non-zero scores, icon coverage).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 10:00:10 +02:00
jpmschweitzerandClaude Fable 5 6ec77091b6 style(librarian): apply ruff autofixes to client and tools
Mechanical Optional[X] -> X | None and f-string cleanups so subsequent
librarian changes lint clean against the dirty baseline.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 09:58:43 +02:00
jpmschweitzerandClaude Fable 5 11405e0acb chore: release v2.3.0
Build and Push / release (push) Successful in 25s
Build and Push / build (push) Successful in 4m34s
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 23:54:32 +02:00
jpmschweitzerandClaude Fable 5 d2aeb8957b docs: document local-first backend, gemma4 gotchas, and contract tests
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 18:35:53 +02:00
jpmschweitzerandClaude Fable 5 d8c84d080c test: add wire-level service contract tests
tests/contracts sends the raw requests the code sends to Ollama (native API
and OpenAI-compat tool calling), Anthropic (including the pinned Sonnet 5
temperature-rejection contract), Qdrant, SearXNG, library-desk, and Redis.
Unreachable services skip; wrong response shapes fail. Run via
make test-contracts; excluded from the unit suite.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 18:35:53 +02:00
jpmschweitzerandClaude Fable 5 f03d41c698 fix: use dedicated tool-phase prompt for gemma4 orchestration
With the butler persona prompt attached, gemma4 reasons about calling the
calculator and then answers from memory with a different wrong product every
run; tool_choice=required via extra_body is advisory at best on Ollama's
OpenAI-compat layer. orchestrate_tool_calls() now uses a terse
TATLOCK_ORCHESTRATION_PROMPT; synthesize_from_results() keeps the persona,
so the user-visible voice is unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 18:35:53 +02:00
jpmschweitzerandClaude Fable 5 033a1c01e8 feat: make Ollama/gemma4 the primary backend with Claude as fallback
Rolls back the claudification backend preference: PREFER_CLOUD_BACKEND now
defaults to false, resolve_backend() picks Ollama first and uses Claude when
explicitly preferred or when the new Ollama startup health check fails. The
Steward retries mid-request failures on the other backend in both directions.

Also hardens the fallback itself: Anthropic SDK imports are lazy so a broken
anthropic package degrades to Ollama-only instead of crashing at import time
(root cause of the production outage since April), anthropic is pinned to a
pydantic-ai-1.27-compatible range, ANTHROPIC_MODEL defaults to claude-sonnet-5
(sonnet-4-20250514 retired 2026-06-15), sampling parameters are stripped from
Claude calls (Sonnet 5 rejects them), and the Steward timeout is configurable
(STEWARD_TIMEOUT, default 60s) since gemma4 needs ~35s warm for analysis.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 18:35:53 +02:00
jpmschweitzerandClaude Opus 4.6 427ad311dc feat: switch default Ollama model to gemma4:e2b
Build and Push / release (push) Successful in 21s
Build and Push / build (push) Successful in 5m27s
gemma4:e2b has native function calling with dedicated tool tokens,
achieving 100% tool selection accuracy in benchmarks vs 67% for
mistral-nemo-large, with 5-8x faster response times (2-4s vs 15-20s)
and lower VRAM usage (8GB vs 9.2GB).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 22:44:27 +02:00
jpmschweitzerandClaude Opus 4.6 4f911929f4 ci: remove test gate from release pipeline
Build and Push / release (push) Successful in 2s
Build and Push / build (push) Successful in 1m23s
Tests are run locally before tagging. Removes the slow CI test job
and its dependency gates on release and build jobs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-05 21:22:51 +01:00
50 changed files with 4078 additions and 1851 deletions
+11 -9
View File
@@ -8,17 +8,19 @@ API_HOST=0.0.0.0
API_PORT=8000 API_PORT=8000
API_PREFIX=/v1 API_PREFIX=/v1
# Anthropic Configuration (Claude - preferred backend) # Ollama Configuration (local - primary backend)
# Set ANTHROPIC_API_KEY to enable Claude as the default backend
# Without an API key, Tatlock uses Ollama exclusively
# ANTHROPIC_API_KEY=sk-ant-api03-your-key-here
ANTHROPIC_MODEL=claude-sonnet-4-20250514
PREFER_CLOUD_BACKEND=true
# Ollama Configuration (local fallback when Claude unavailable)
OLLAMA_HOST=http://localhost:11434 OLLAMA_HOST=http://localhost:11434
OLLAMA_DEFAULT_MODEL=mistral-nemo:latest OLLAMA_DEFAULT_MODEL=gemma4:e2b
OLLAMA_TIMEOUT=120 OLLAMA_TIMEOUT=120
STEWARD_TIMEOUT=60
# Anthropic Configuration (Claude - cloud fallback)
# Set ANTHROPIC_API_KEY to keep the Claude fallback available: it is used
# automatically when Ollama is down, or exclusively when PREFER_CLOUD_BACKEND=true
# Without an API key, Tatlock uses Ollama only
# ANTHROPIC_API_KEY=sk-ant-api03-your-key-here
ANTHROPIC_MODEL=claude-sonnet-5
PREFER_CLOUD_BACKEND=false
# SearXNG Configuration # SearXNG Configuration
SEARXNG_HOST=http://localhost:8087 SEARXNG_HOST=http://localhost:8087
+3 -21
View File
@@ -6,24 +6,7 @@ on:
- 'v[0-9]*' - 'v[0-9]*'
jobs: jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install dependencies
run: make setup
- name: Run unit tests
run: make test
release: release:
needs: test
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- name: Create Gitea Release - name: Create Gitea Release
@@ -35,7 +18,6 @@ jobs:
"${{ github.server_url }}/api/v1/repos/${{ github.repository }}/releases" "${{ github.server_url }}/api/v1/repos/${{ github.repository }}/releases"
build: build:
needs: test
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
@@ -43,7 +25,7 @@ jobs:
- name: Login to Gitea Registry - name: Login to Gitea Registry
uses: docker/login-action@v3 uses: docker/login-action@v3
with: with:
registry: git.schweitz.internal registry: git.schweitz.net
username: ${{ secrets.REGISTRY_USER }} username: ${{ secrets.REGISTRY_USER }}
password: ${{ secrets.REGISTRY_PASSWORD }} password: ${{ secrets.REGISTRY_PASSWORD }}
@@ -55,8 +37,8 @@ jobs:
provenance: false provenance: false
sbom: false sbom: false
tags: | tags: |
git.schweitz.internal/jpmschweitzer/tatlock:latest git.schweitz.net/jpmschweitzer/tatlock:latest
git.schweitz.internal/jpmschweitzer/tatlock:${{ github.ref_name }} git.schweitz.net/jpmschweitzer/tatlock:${{ github.ref_name }}
- name: Trigger Watchtower update - name: Trigger Watchtower update
if: success() if: success()
+71
View File
@@ -7,6 +7,77 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased] ## [Unreleased]
## [2.4.2] - 2026-07-19
### Fixed
- Container crash-loop on fresh builds: cap `opentelemetry-api` below 1.44,
which removed the private `_events` module that pydantic-ai 1.27 imports
## [2.4.1] - 2026-07-19
### Changed
- **Container-name network defaults** - `SEARXNG_HOST`, `LIBRARY_DESK_HOST`, and `CORE_API_HOST` now default to docker container names on the docker-dataplane network (`http://searxng:8080`, `http://library-desk:8089`, `http://core-api:8083`) instead of host `localhost` ports, ahead of the loopback port rebinding; this also fixes `CORE_API_HOST` pointing at port 8090 (the Scheduler's host port) rather than Core-API's 8083. `scripts/test_housekeeper.sh` now reaches Core-API via `localhost:8083` instead of the LAN IP. Local development against host-published ports still works via `.env` overrides
## [2.4.0] - 2026-07-14
### Removed
- **Dead delegation stack** - deleted the duplicate, never-wired coordination layer so exactly ONE delegation implementation remains (`src/agents/delegation.py`): `src/agents/coordination.py` (`CoordinationEngine`, its own `delegate_to_librarian`, `AGENT_EXECUTORS`/`AGENT_STREAM_EXECUTORS`), the broken-by-design `run_librarian_stream` path it used (Ollama streaming + tool call bug), the `stream_delegate_to_*` wrappers with their never-parsed `__DELEGATION_RESULT__` marker, and `HouseholdRegistry.get_streaming_delegation_tools()` (no callers)
- **Orphaned agent protocol models** - `src/agents/protocol.py` now contains only the live `AgentError`; the coordination wire protocol it carried (`AgentRequest`, `AgentResponse`, `DelegationIntent`, `CoordinationResult`, `DelegationReason`, `TaskComplexity`, `ToolCallRecord`, `AgentTimeoutError`, `AgentUnavailableError`, `DelegationError`) had no importer left outside its own tests after the coordination stack removal
### Added
- **Test-suite tenant guard** - `tests/conftest.py` hard-fails the whole pytest session (exit code 1, zero tests run) if the effective tenant resolves to the production tenant `jpmschweitzer`, mirroring the guard library-desk applies on its side. Suite-level assertions pin that the session runs under `llm_tester` namespaces (Qdrant `memories_llm_tester`, Redis `session:llm_tester:*`), and the e2e isolation constants now derive from the shared `TEST_TENANT`/`PRODUCTION_TENANT` config constants instead of string literals
- **Explicit tenant on every library-desk request** - the librarian client now resolves and sends the `user` parameter explicitly on every request (library-desk is removing its server-side default; a missing user would 422). The content extraction endpoints now carry the tenant too, `search_web` no longer falls back to a phantom `tatlock-librarian` user, and a client-level assertion rejects an empty/whitespace tenant before any bytes hit the wire. A parametrized sweep pins the wire contract for all 15 tenant-scoped client methods
- **Tenant isolation guard** - non-production environments (development/testing) now FORCE the effective tenant to the reserved test tenant `llm_tester` (only `llm_tester` itself or a `test_`-prefixed override is accepted), regardless of `DEFAULT_USER` misconfiguration, at both config resolution and request-context resolution (`get_user()`). Startup refuses (clear error) when a non-production environment is explicitly configured with the production tenant `jpmschweitzer`, and one loud startup log line states the effective/forced tenant
- **Conversation context for experts + real-time think messages** - direct delegation (streaming and non-streaming) now passes a trimmed conversation history (last 6 turns) as expert context, so follow-up questions keep their referent; `_stream_direct_delegation` is now an async generator, so butler think messages ("Allow me to consult the archives, sir.") stream BEFORE the research runs instead of after it completes
- **Bounded retries and connection reuse for library-desk** - GETs and the read-only `POST /query/*` and `POST /rag/search` endpoints retry once (2 attempts, short backoff) on transport errors and retryable 5xx; wiki writes are never retried. The client now honors `LIBRARY_DESK_TIMEOUT` instead of hardcoded 60s/30s, a librarian run holds one shared HTTP connection instead of constructing a client per tool call, and read tools raise `ModelRetry` on transient HTTP errors so the agent's retry budget engages
- **One librarian timeout budget** - new `LIBRARIAN_TIMEOUT` (default 180s) enforced with `asyncio.wait_for` inside `delegate_to_librarian`, capping the previously uncapped live paths (steward direct delegation and streaming). The Ollama provider's AsyncOpenAI client now carries an explicit `OLLAMA_TIMEOUT` instead of the SDK's ~600s default, and the contradictory unused 60s default in `AgentRequest.timeout_seconds` was removed (None defers to the configured budget)
- **Search degradation signaling** - The librarian client parses `source_counts` (plus the additive `source_status`/`degraded` fields when a newer library-desk sends them; absence is tolerated), and `hybrid_search` appends a one-line coverage note when a search is degraded or an enabled source leg contributed nothing, so outages are visible to the model and the user. When `source_status` is present it is used exclusively; without it, count-absence is only inferred for the optional legs the request explicitly enabled (web/documents/volatile) - never the always-on vector/graph legs, whose absence from the top-N counts is normal ranking behavior, so healthy searches no longer emit warnings
### Fixed
- **Clearing all wiki-page tags is possible again** - the Ollama-safe empty-list sentinel in `update_wiki_page` means "leave unchanged", which made it impossible to remove all tags; passing exactly `["__CLEAR__"]` now sends an empty tag list to library-desk (documented in the tool docstring for the local model)
- **Text-delegation fallback pairs results strictly** - the parallel branch now verifies `asyncio.gather` returned one result per parsed delegation (`zip(..., strict=True)`); a count mismatch fails loudly with a curated apology instead of silently attributing outputs to the wrong agent
- **Ollama-safe librarian tool schemas** - `update_wiki_page` and `smart_create_wiki_page` no longer use `X | None` parameters (Ollama's OpenAI-compatible API mishandles `anyOf[X, null]`); empty-string/empty-list sentinels are translated to `None` inside the tools, matching the biographer pattern. A snapshot test pins every librarian tool schema to contain no nullable `anyOf`
- **Honest expert failures** - `run_librarian` now raises a structured `AgentError` instead of returning error text as if it were research output, so delegation correctly reports `success=False` and the streaming error branch is reachable. Failures surface to the user as curated butler-toned sentences; exception detail (including internal URLs) stays in the logs only. Librarian tool errors no longer leak `str(e)` into synthesis
- **HybridRAG response mapping** - The librarian client now parses the field names library-desk actually returns (`source_type`/`sources`, `rrf_score`, `context`, per-item `related_dossiers`, synonyms nested in the `keywords` dict); previously every result rendered as "unknown (score: 0.00)". Source icons now key off the per-item `sources` list. Requests no longer send zero limits (the service rejects them with 422); legs are disabled via `enable_*` flags. Pinned by a contract test against a recorded live response (`tests/agents/librarian/fixtures/`)
## [2.3.0] - 2026-07-13
### Changed
- **Local-first backend (claudification rollback)** - Ollama/gemma4 is now the primary backend; Claude remains as fallback. `PREFER_CLOUD_BACKEND` defaults to `false`, Claude is used automatically when the Ollama startup health check fails, and the Steward retries mid-request failures on the other backend in both directions
- **Default Claude model `claude-sonnet-5`** - `claude-sonnet-4-20250514` was retired by Anthropic on 2026-06-15 and would 404, leaving the fallback dead
- **Dedicated orchestration prompt** - `orchestrate_tool_calls()` now uses a terse tool-execution prompt (`TATLOCK_ORCHESTRATION_PROMPT`); the butler persona prompt suppressed gemma4 tool calling (the model reasoned about the calculator, then answered from memory with wrong arithmetic). Synthesis keeps the persona prompt, so user-visible voice is unchanged
### Fixed
- **Startup crash with broken anthropic package** - Anthropic SDK imports in the model selector are now lazy, so an incompatible `anthropic` install degrades to Ollama-only operation instead of crashing the app at import time (root cause of the production outage since April)
- **Claude Sonnet 5 rejects sampling parameters** - removed `temperature` from the Steward's direct Claude call and made the Housekeeper's temperature setting backend-conditional via `get_sampling_settings()`
- **Pin `anthropic>=0.77,<1.0`** - the April image resolved an anthropic version incompatible with pydantic-ai 1.27
- **Steward timeout configurable** - new `STEWARD_TIMEOUT` (default 60s) replaces the hardcoded 30s, which gemma4 chronically exceeded (~35s warm analysis), causing every request to fail or fall back
### Added
- **Ollama startup health check** - verifies the server is reachable and `OLLAMA_DEFAULT_MODEL` is pulled; feeds backend resolution and `get_model_info()`
- **Contract tests** (`tests/contracts/`, `make test-contracts`) - wire-level tests that send the raw requests the code sends to Ollama (native + OpenAI-compat tool calling), Anthropic (including the pinned temperature-rejection contract), Qdrant, SearXNG, library-desk, and Redis; unreachable services skip, wrong response shapes fail
- **Backend resolution unit tests** (`tests/anthropic/`)
## [2.2.0] - 2026-04-04
### Changed
- **Switch default Ollama model to gemma4:e2b** - Replaces mistral-nemo as the local LLM backend; gemma4:e2b has native function calling support, faster tool calling (2-4s vs 15-20s), better parameter accuracy on word problems, and uses less VRAM (8GB vs 9.2GB)
### Added
- **Tool calling benchmark script** (`scripts/benchmark_tool_calling.py`) - Compares tool calling accuracy and latency across Ollama models via the Tatlock API
## [2.1.0] - 2026-02-05 ## [2.1.0] - 2026-02-05
### Fixed ### Fixed
+6 -3
View File
@@ -8,6 +8,7 @@ Claude Code-specific notes for this project. For general development instruction
make setup # Create venv and install all dependencies make setup # Create venv and install all dependencies
make test # Unit tests (no external services) make test # Unit tests (no external services)
make test-integration # Integration tests (needs Claude/Ollama) make test-integration # Integration tests (needs Claude/Ollama)
make test-contracts # Wire-level contract tests against live service boundaries
make run # Start dev server on port 8777 make run # Start dev server on port 8777
make lint # Ruff linter + formatter check make lint # Ruff linter + formatter check
make typecheck # Mypy make typecheck # Mypy
@@ -18,13 +19,15 @@ Dependencies are in `pyproject.toml` (`[project.dependencies]` and `[project.opt
## Critical Gotchas ## Critical Gotchas
**ASGITransport does NOT trigger FastAPI lifespan events.** The session-scoped `_initialize_app` fixture in `tests/conftest.py` calls `initialize_application()` explicitly via `asyncio.run()`. Without this, `check_claude_health()` never runs and `_claude_available` stays `None`, causing all tests to silently fall back to Ollama. **ASGITransport does NOT trigger FastAPI lifespan events.** The session-scoped `_initialize_app` fixture in `tests/conftest.py` calls `initialize_application()` explicitly via `asyncio.run()`. Without this, the Ollama/Claude health checks never run: `_ollama_available` stays `None` (treated as available, so requests go to Ollama) and `_claude_available` stays `None` (treated as unavailable, so the Claude fallback never engages).
**AsyncIO scope mismatch.** `asyncio_default_fixture_loop_scope = function` is set in `pyproject.toml`. Session-scoped async fixtures cause `ScopeMismatch` errors. The fix is to use a sync fixture with `asyncio.run()` for session-scoped initialization. **AsyncIO scope mismatch.** `asyncio_default_fixture_loop_scope = function` is set in `pyproject.toml`. Session-scoped async fixtures cause `ScopeMismatch` errors. The fix is to use a sync fixture with `asyncio.run()` for session-scoped initialization.
**Ollama is unreliable for tool calling.** `mistral-nemo` on Ollama often does mental math instead of calling calculator tools, and frequently gets wrong answers. Claude reliably calls tools. If integration tests give wrong math answers, check which backend is actually being used. **The butler persona prompt suppresses local-model tool calling.** With `TATLOCK_SYSTEM_PROMPT` attached, gemma4 reasons about calling the calculator, then answers from memory with wrong arithmetic (a different wrong product each run). `orchestrate_tool_calls()` therefore uses the terse `TATLOCK_ORCHESTRATION_PROMPT`; the persona is applied in `synthesize_from_results()`. Do not reattach the persona prompt to a tool-phase agent. `tool_choice: "required"` via extra_body does NOT force Ollama to call tools — it is advisory at best.
**Integration test timeouts.** Set to 120s to match `OLLAMA_TIMEOUT` config. Ollama on tower-of-joy can be slow, especially on first request. **Claude Sonnet 5+ rejects sampling parameters.** `temperature`/`top_p`/`top_k` return a 400. Use `get_sampling_settings()` from the model selector instead of passing `ModelSettings(temperature=...)` directly to agents that can run on the Claude fallback. The contract test suite pins this (`make test-contracts`).
**Integration test timeouts.** Set to 120s to match `OLLAMA_TIMEOUT` config (300s for the pure-Ollama fallback test, which cannot be rescued by Claude). Current GPU-resident numbers (2026-07-14, driver 570, gemma4:e2b at ~100 tok/s): Steward analysis ~6s warm, full Steward → orchestrate → synthesize flow 1125s, librarian-routed queries ~20-25s. The old "~35s steward / ~2 min flow" figures were measured during the CPU-only era (driver mismatch, 13 tok/s) — do not plan against them. Cold start after 2h idle adds ~8s (`OLLAMA_KEEP_ALIVE=2h`). `STEWARD_TIMEOUT` defaults to 60s.
**`get_benchmark_store` does not exist.** The benchmarking module (`src/core/benchmarks.py`) was never implemented. `scripts/benchmark_analysis.py` also references it and is broken. Do not add mocks for it in tests. **`get_benchmark_store` does not exist.** The benchmarking module (`src/core/benchmarks.py`) was never implemented. `scripts/benchmark_analysis.py` also references it and is broken. Do not add mocks for it in tests.
+5 -2
View File
@@ -1,4 +1,4 @@
.PHONY: help setup run test test-unit test-integration lint typecheck clean .PHONY: help setup run test test-unit test-integration test-contracts lint typecheck clean
VENV := .venv VENV := .venv
PYTHON := $(VENV)/bin/python PYTHON := $(VENV)/bin/python
@@ -29,13 +29,16 @@ run: ## Start the development server on port 8777
$(UVICORN) src.main:app --reload --host $(HOST) --port $(PORT) 2>&1 | tee build/logs/server.log $(UVICORN) src.main:app --reload --host $(HOST) --port $(PORT) 2>&1 | tee build/logs/server.log
test: ## Run unit tests (no external services needed) test: ## Run unit tests (no external services needed)
$(PYTEST) --ignore=tests/e2e --ignore=tests/integration $(PYTEST) --ignore=tests/e2e --ignore=tests/integration --ignore=tests/contracts
test-unit: test ## Alias for test test-unit: test ## Alias for test
test-integration: ## Run integration tests (needs Claude/Ollama) test-integration: ## Run integration tests (needs Claude/Ollama)
$(PYTEST) tests/agents/test_tatlock_agent.py -v $(PYTEST) tests/agents/test_tatlock_agent.py -v
test-contracts: ## Wire-level contract tests against live service boundaries
$(PYTEST) tests/contracts -v --no-cov
lint: ## Run ruff linter and formatter check lint: ## Run ruff linter and formatter check
$(RUFF) check src tests $(RUFF) check src tests
$(RUFF) format --check src tests $(RUFF) format --check src tests
+16 -9
View File
@@ -58,7 +58,7 @@ A privacy-first, offline-capable personal assistant system that coordinates spec
- Error triggers for testing (rate_limit, context_overflow) - Error triggers for testing (rate_limit, context_overflow)
- **Tatlock**: Real PydanticAI agent with butler personality - **Tatlock**: Real PydanticAI agent with butler personality
- **LLM Backend**: Ollama (mistral-nemo:latest by default) - **LLM Backend**: Ollama (gemma4:e2b by default, local-first) with optional Claude fallback
- **Personality**: Witty British butler, research-oriented - **Personality**: Witty British butler, research-oriented
- **Core Tools**: - **Core Tools**:
- **Calculator**: Safe mathematical expression evaluation - **Calculator**: Safe mathematical expression evaluation
@@ -74,7 +74,7 @@ A privacy-first, offline-capable personal assistant system that coordinates spec
- Python 3.12+ (Python 3.12.11 recommended) - Python 3.12+ (Python 3.12.11 recommended)
- **External Services** (must be running separately): - **External Services** (must be running separately):
- **Ollama**: LLM inference (mistral-nemo:latest, nomic-embed-text) - **Ollama**: LLM inference (gemma4:e2b, nomic-embed-text)
- **Redis**: Caching and session memory - **Redis**: Caching and session memory
- **Qdrant**: Vector storage for The Biographer's memory - **Qdrant**: Vector storage for The Biographer's memory
- **SearXNG**: Web search (optional) - **SearXNG**: Web search (optional)
@@ -264,7 +264,10 @@ Interactive documentation available at:
pytest pytest
# Run unit tests only (no external services needed) # Run unit tests only (no external services needed)
pytest --ignore=tests/e2e --ignore=tests/integration pytest --ignore=tests/e2e --ignore=tests/integration --ignore=tests/contracts
# Wire-level contract tests against live service boundaries
make test-contracts
# Run with coverage # Run with coverage
pytest --cov=src --cov-report=term-missing pytest --cov=src --cov-report=term-missing
@@ -303,12 +306,17 @@ Create a `.env` file for custom configuration:
API_HOST=0.0.0.0 API_HOST=0.0.0.0
API_PORT=8000 API_PORT=8000
# Ollama Configuration # Ollama Configuration (primary backend)
OLLAMA_HOST=http://localhost:11434 OLLAMA_HOST=http://localhost:11434
OLLAMA_DEFAULT_MODEL=mistral-nemo:latest OLLAMA_DEFAULT_MODEL=gemma4:e2b
OLLAMA_EMBEDDING_MODEL=nomic-embed-text OLLAMA_EMBEDDING_MODEL=nomic-embed-text
OLLAMA_TIMEOUT=120 OLLAMA_TIMEOUT=120
# Claude fallback (optional; used when Ollama is down or PREFER_CLOUD_BACKEND=true)
# ANTHROPIC_API_KEY=sk-ant-api03-your-key-here
ANTHROPIC_MODEL=claude-sonnet-5
PREFER_CLOUD_BACKEND=false
# Redis Configuration # Redis Configuration
REDIS_HOST=localhost REDIS_HOST=localhost
REDIS_PORT=6379 REDIS_PORT=6379
@@ -376,9 +384,8 @@ tatlock/
│ │ ├── steward/ # The Steward - request analysis │ │ ├── steward/ # The Steward - request analysis
│ │ ├── tatlock_core/ # Core butler tools │ │ ├── tatlock_core/ # Core butler tools
│ │ ├── tatlock.py # Tatlock PydanticAI agent │ │ ├── tatlock.py # Tatlock PydanticAI agent
│ │ ├── coordination.py # Multi-agent coordination
│ │ ├── delegation.py # Expert delegation wrappers │ │ ├── delegation.py # Expert delegation wrappers
│ │ └── protocol.py # Agent communication protocol │ │ └── protocol.py # Agent error protocol
│ ├── responses/ # Responses API (primary endpoint) │ ├── responses/ # Responses API (primary endpoint)
│ ├── chat/ # Chat Completions wrapper │ ├── chat/ # Chat Completions wrapper
│ ├── models/ # Models listing │ ├── models/ # Models listing
@@ -427,8 +434,8 @@ For LLM agent development guidelines and architectural decisions, see [AGENTS.md
## Version ## Version
Current version: **1.3.2** - Biographer tool type hints fix Current version: see [CHANGELOG.md](CHANGELOG.md)
--- ---
**Note**: Tatlock is a production-ready homelab butler. All household staff use PydanticAI with Ollama for local LLM inference. **Note**: Tatlock is a production-ready homelab butler. All household staff use PydanticAI with local Ollama inference (gemma4), with an optional Claude cloud fallback.
+1 -1
View File
@@ -3,7 +3,7 @@
## Overview ## Overview
Tatlock uses a bidirectional Claude architecture: Tatlock uses a bidirectional Claude architecture:
- **Scenario A**: Tatlock powered by Claude backend (with Ollama fallback) — **COMPLETE** - **Scenario A**: Tatlock powered by Claude backend (with Ollama fallback) — **COMPLETE**, then **rolled back to local-first**: Ollama/gemma4 is primary, Claude is retained as fallback (`PREFER_CLOUD_BACKEND=false`)
- **Scenario B**: Tatlock exposed as MCP server for external Claude instances — **OPEN** - **Scenario B**: Tatlock exposed as MCP server for external Claude instances — **OPEN**
- **Scenario C**: Offline operation via Ollama — **COMPLETE** - **Scenario C**: Offline operation via Ollama — **COMPLETE**
+1 -1
View File
@@ -12,7 +12,7 @@ This document tracks open/planned work. Completed phases have been removed.
- Household staff: Tatlock (Butler), Steward, Librarian, Biographer - Household staff: Tatlock (Butler), Steward, Librarian, Biographer
- Core tools: Calculator, Date/Time, Web search (SearXNG) - Core tools: Calculator, Date/Time, Web search (SearXNG)
- Memory system: Qdrant (vector), Redis (session cache), multi-tenancy via ContextVar - Memory system: Qdrant (vector), Redis (session cache), multi-tenancy via ContextVar
- Dual backend: Claude (preferred) + Ollama (fallback) - Dual backend: Ollama/gemma4 (primary) + Claude (fallback)
- 439 tests with good coverage - 439 tests with good coverage
--- ---
+6 -1
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project] [project]
name = "tatlock" name = "tatlock"
version = "2.1.0" version = "2.4.2"
description = "OpenAI-compatible API with Ollama backend" description = "OpenAI-compatible API with Ollama backend"
requires-python = ">=3.12" requires-python = ">=3.12"
dependencies = [ dependencies = [
@@ -13,6 +13,10 @@ dependencies = [
"pydantic>=2.11,<2.13", "pydantic>=2.11,<2.13",
"pydantic-settings>=2.12,<2.13", "pydantic-settings>=2.12,<2.13",
"pydantic-ai-slim[openai,anthropic]>=1.27,<1.28", "pydantic-ai-slim[openai,anthropic]>=1.27,<1.28",
# pydantic-ai 1.27 imports the private opentelemetry._events module,
# removed in opentelemetry-api 1.44 — cap until pydantic-ai is bumped
"opentelemetry-api>=1.30,<1.44",
"anthropic>=0.77,<1.0",
"httpx>=0.28,<0.29", "httpx>=0.28,<0.29",
"sse-starlette>=3.0,<3.1", "sse-starlette>=3.0,<3.1",
"python-dotenv>=1.2,<1.3", "python-dotenv>=1.2,<1.3",
@@ -46,6 +50,7 @@ markers = [
"unit: Unit tests", "unit: Unit tests",
"integration: Integration tests", "integration: Integration tests",
"slow: Slow running tests", "slow: Slow running tests",
"contract: Wire-level contract tests against live service boundaries",
] ]
addopts = [ addopts = [
"--verbose", "--verbose",
+542
View File
@@ -0,0 +1,542 @@
"""
Benchmark tool calling across different Ollama models via Tatlock API.
Sends test prompts through the full Tatlock pipeline (Steward -> Orchestration
-> Synthesis) and records tool selection accuracy, latency, and response quality.
Between models, swaps OLLAMA_DEFAULT_MODEL in .env and waits for uvicorn
auto-reload. Requires the server to be running via ./wakeup.sh.
Usage:
.venv/bin/python scripts/benchmark_tool_calling.py
.venv/bin/python scripts/benchmark_tool_calling.py --models "gemma4:e4b,gemma4:e2b"
.venv/bin/python scripts/benchmark_tool_calling.py --iterations 3
"""
import argparse
import asyncio
import json
import re
import statistics
import time
from dataclasses import dataclass, field
from pathlib import Path
import httpx
# ---------------------------------------------------------------------------
# Configuration
# ---------------------------------------------------------------------------
API_BASE = "http://localhost:8777"
CHAT_URL = f"{API_BASE}/v1/chat/completions"
HEALTH_URL = f"{API_BASE}/health"
OLLAMA_URL = "http://localhost:11434"
ENV_PATH = Path(__file__).parent.parent / ".env"
DEFAULT_MODELS = ["mistral-nemo-large:latest", "gemma4:e4b", "gemma4:e2b"]
# ---------------------------------------------------------------------------
# Test scenarios
# ---------------------------------------------------------------------------
@dataclass
class Scenario:
name: str
prompt: str
expected_tool: str | None # None = no tool expected
# Patterns to check in the response text for indirect tool-use evidence
success_patterns: list[str] = field(default_factory=list)
category: str = "basic"
SCENARIOS = [
# --- Should call calculate_math ---
Scenario(
name="Simple arithmetic",
prompt="What is 144 divided by 12?",
expected_tool="calculate_math",
success_patterns=["12"],
category="calculator",
),
Scenario(
name="Square root",
prompt="What's the square root of 256?",
expected_tool="calculate_math",
success_patterns=["16"],
category="calculator",
),
Scenario(
name="Complex math",
prompt="Calculate pi times the square of 5",
expected_tool="calculate_math",
success_patterns=["78.5"], # pi * 25 ≈ 78.54
category="calculator",
),
Scenario(
name="Word problem",
prompt="If I have 3 bags with 17 apples each and I eat 4, how many apples do I have?",
expected_tool="calculate_math",
success_patterns=["47"],
category="calculator",
),
# --- Should call get_current_time ---
Scenario(
name="Current date",
prompt="What's today's date?",
expected_tool="get_current_time",
success_patterns=["2026"], # Should contain current year
category="datetime",
),
Scenario(
name="Current time",
prompt="What time is it right now?",
expected_tool="get_current_time",
success_patterns=[":"], # Time format contains colons
category="datetime",
),
# --- Should call calculate_date_offset ---
Scenario(
name="Relative date past",
prompt="What was the date 2 weeks ago?",
expected_tool="calculate_date_offset",
success_patterns=["2026"],
category="datetime",
),
# --- Should call calculate_time_difference ---
Scenario(
name="Date difference",
prompt="How many days between January 1st 2025 and March 15th 2025?",
expected_tool="calculate_time_difference",
success_patterns=["73", "74"], # 73 or 74 days
category="datetime",
),
# --- Should NOT call any tool ---
Scenario(
name="Greeting",
prompt="Hello! How are you?",
expected_tool=None,
success_patterns=["sir"], # Butler personality
category="no_tool",
),
Scenario(
name="Knowledge question",
prompt="What is the capital of France?",
expected_tool=None,
success_patterns=["Paris"],
category="no_tool",
),
Scenario(
name="Opinion request",
prompt="What do you think about rainy days?",
expected_tool=None,
category="no_tool",
),
]
# ---------------------------------------------------------------------------
# Result tracking
# ---------------------------------------------------------------------------
@dataclass
class RunResult:
scenario: str
model: str
iteration: int
latency: float
response_text: str
has_correct_answer: bool
error: str | None = None
@dataclass
class ModelStats:
model: str
results: list[RunResult] = field(default_factory=list)
@property
def total(self) -> int:
return len(self.results)
@property
def errors(self) -> int:
return sum(1 for r in self.results if r.error)
@property
def accuracy(self) -> float:
valid = [r for r in self.results if not r.error]
if not valid:
return 0
return sum(1 for r in valid if r.has_correct_answer) / len(valid) * 100
@property
def avg_latency(self) -> float:
lats = [r.latency for r in self.results if not r.error]
return statistics.mean(lats) if lats else 0
@property
def p95_latency(self) -> float:
lats = sorted(r.latency for r in self.results if not r.error)
if not lats:
return 0
return lats[min(int(len(lats) * 0.95), len(lats) - 1)]
@property
def max_latency(self) -> float:
lats = [r.latency for r in self.results if not r.error]
return max(lats) if lats else 0
def category_accuracy(self, category: str) -> float:
cat_scenarios = {s.name for s in SCENARIOS if s.category == category}
valid = [r for r in self.results if not r.error and r.scenario in cat_scenarios]
if not valid:
return 0
return sum(1 for r in valid if r.has_correct_answer) / len(valid) * 100
# ---------------------------------------------------------------------------
# .env manipulation
# ---------------------------------------------------------------------------
def swap_model_in_env(model_name: str):
"""Swap OLLAMA_DEFAULT_MODEL in .env file."""
content = ENV_PATH.read_text()
content = re.sub(
r'^OLLAMA_DEFAULT_MODEL=.*$',
f'OLLAMA_DEFAULT_MODEL={model_name}',
content,
flags=re.MULTILINE,
)
ENV_PATH.write_text(content)
print(f" .env updated: OLLAMA_DEFAULT_MODEL={model_name}")
async def wait_for_server_reload(client: httpx.AsyncClient, timeout: float = 30):
"""Wait for uvicorn to auto-reload after .env change."""
# Give uvicorn a moment to detect the file change
await asyncio.sleep(3)
# Poll health endpoint
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
try:
r = await client.get(HEALTH_URL, timeout=5)
if r.status_code == 200:
return
except Exception:
pass
await asyncio.sleep(1)
raise TimeoutError("Server did not come back after reload")
async def warm_up_ollama_model(client: httpx.AsyncClient, model_name: str):
"""Send a throwaway request to load the model into VRAM."""
print(f" Warming up {model_name} in Ollama...", end=" ", flush=True)
try:
r = await client.post(
f"{OLLAMA_URL}/api/generate",
json={"model": model_name, "prompt": "hi", "stream": False},
timeout=120,
)
r.raise_for_status()
duration = r.json().get("total_duration", 0) / 1e9
print(f"OK ({duration:.1f}s)")
except Exception as e:
print(f"WARN: {e}")
# ---------------------------------------------------------------------------
# Core benchmark logic
# ---------------------------------------------------------------------------
async def run_scenario(
client: httpx.AsyncClient,
scenario: Scenario,
model: str,
iteration: int,
) -> RunResult:
"""Run a single scenario through the Tatlock API."""
payload = {
"model": "Tatlock",
"messages": [{"role": "user", "content": scenario.prompt}],
}
start = time.monotonic()
try:
r = await client.post(CHAT_URL, json=payload, timeout=120)
latency = time.monotonic() - start
if r.status_code != 200:
return RunResult(
scenario=scenario.name,
model=model,
iteration=iteration,
latency=latency,
response_text="",
has_correct_answer=False,
error=f"HTTP {r.status_code}: {r.text[:100]}",
)
data = r.json()
response_text = data["choices"][0]["message"]["content"]
# Check if the response contains expected patterns
has_correct = True
if scenario.success_patterns:
has_correct = any(
p.lower() in response_text.lower()
for p in scenario.success_patterns
)
return RunResult(
scenario=scenario.name,
model=model,
iteration=iteration,
latency=latency,
response_text=response_text,
has_correct_answer=has_correct,
)
except Exception as e:
latency = time.monotonic() - start
return RunResult(
scenario=scenario.name,
model=model,
iteration=iteration,
latency=latency,
response_text="",
has_correct_answer=False,
error=str(e)[:200],
)
async def benchmark_model(
client: httpx.AsyncClient,
model_name: str,
iterations: int,
) -> ModelStats:
"""Run all scenarios for a single model."""
stats = ModelStats(model=model_name)
print(f"\n{'=' * 70}")
print(f" Model: {model_name}")
print(f"{'=' * 70}")
# Swap model in .env
swap_model_in_env(model_name)
# Warm up model in Ollama BEFORE server reload picks it up
await warm_up_ollama_model(client, model_name)
# Wait for server to reload with new model
print(" Waiting for server reload...", end=" ", flush=True)
await wait_for_server_reload(client)
print("OK")
# Run a throwaway request through the full pipeline to warm up
print(" Warming up pipeline...", end=" ", flush=True)
try:
await client.post(
CHAT_URL,
json={"model": "Tatlock", "messages": [{"role": "user", "content": "hi"}]},
timeout=120,
)
print("OK")
except Exception as e:
print(f"WARN: {e}")
for iteration in range(iterations):
if iterations > 1:
print(f"\n --- Iteration {iteration + 1}/{iterations} ---")
for scenario in SCENARIOS:
result = await run_scenario(client, scenario, model_name, iteration)
stats.results.append(result)
# Display
if result.error:
print(
f" [ERR ] {scenario.name:30s} {result.latency:5.1f}s "
f"{result.error[:60]}"
)
elif result.has_correct_answer:
preview = result.response_text[:60].replace("\n", " ")
print(f" [OK ] {scenario.name:30s} {result.latency:5.1f}s {preview}")
else:
preview = result.response_text[:60].replace("\n", " ")
print(f" [MISS] {scenario.name:30s} {result.latency:5.1f}s {preview}")
return stats
def print_comparison(all_stats: list[ModelStats]):
"""Print side-by-side comparison table."""
print("\n" + "=" * 80)
print(" COMPARISON SUMMARY")
print("=" * 80)
col_width = max(len(s.model) for s in all_stats) + 2
label_width = 32
header = f"{'Metric':<{label_width}}"
for s in all_stats:
header += f" {s.model:>{col_width}}"
print(f"\n{header}")
print("-" * (label_width + (col_width + 2) * len(all_stats)))
# Answer accuracy
row = f"{'Correct answer rate':<{label_width}}"
for s in all_stats:
row += f" {s.accuracy:>{col_width - 1}.1f}%"
print(row)
# Latency
row = f"{'Avg latency':<{label_width}}"
for s in all_stats:
row += f" {s.avg_latency:>{col_width - 1}.1f}s"
print(row)
row = f"{'P95 latency':<{label_width}}"
for s in all_stats:
row += f" {s.p95_latency:>{col_width - 1}.1f}s"
print(row)
row = f"{'Max latency':<{label_width}}"
for s in all_stats:
row += f" {s.max_latency:>{col_width - 1}.1f}s"
print(row)
# Errors
row = f"{'Errors':<{label_width}}"
for s in all_stats:
row += f" {s.errors:>{col_width}}"
print(row)
# Per-category
categories = sorted(set(sc.category for sc in SCENARIOS))
print(f"\n{'Per-category accuracy':<{label_width}}")
print("-" * (label_width + (col_width + 2) * len(all_stats)))
for cat in categories:
row = f" {cat:<{label_width - 2}}"
for s in all_stats:
row += f" {s.category_accuracy(cat):>{col_width - 1}.1f}%"
print(row)
# Mismatches
print(f"\n{'Missed answers':<50}")
print("-" * 80)
any_miss = False
for scenario in SCENARIOS:
misses = []
for s in all_stats:
sc_results = [r for r in s.results if r.scenario == scenario.name]
fails = [r for r in sc_results if not r.has_correct_answer and not r.error]
if fails:
preview = fails[0].response_text[:50].replace("\n", " ")
misses.append(f"{s.model}: \"{preview}\"")
if misses:
any_miss = True
print(f" {scenario.name}")
for m in misses:
print(f" {m}")
if not any_miss:
print(" (none)")
print("\n" + "=" * 80)
def save_results(all_stats: list[ModelStats], output_path: Path):
"""Save detailed results to JSON."""
data = {}
for stats in all_stats:
data[stats.model] = {
"summary": {
"accuracy": stats.accuracy,
"avg_latency": round(stats.avg_latency, 2),
"p95_latency": round(stats.p95_latency, 2),
"max_latency": round(stats.max_latency, 2),
"errors": stats.errors,
"total_runs": stats.total,
},
"runs": [
{
"scenario": r.scenario,
"iteration": r.iteration,
"latency": round(r.latency, 3),
"has_correct_answer": r.has_correct_answer,
"response_text": r.response_text,
"error": r.error,
}
for r in stats.results
],
}
output_path.parent.mkdir(parents=True, exist_ok=True)
output_path.write_text(json.dumps(data, indent=2))
print(f"\nDetailed results saved to: {output_path}")
async def main():
parser = argparse.ArgumentParser(description="Benchmark tool calling across Ollama models via Tatlock API")
parser.add_argument(
"--iterations", type=int, default=1,
help="Iterations per model (default: 1)",
)
parser.add_argument(
"--models", type=str, default=",".join(DEFAULT_MODELS),
help=f"Comma-separated models (default: {','.join(DEFAULT_MODELS)})",
)
parser.add_argument(
"--output", type=str, default="logs/benchmark_results.json",
help="JSON output path (default: logs/benchmark_results.json)",
)
args = parser.parse_args()
models = [m.strip() for m in args.models.split(",")]
# Verify server is running
async with httpx.AsyncClient() as client:
try:
r = await client.get(HEALTH_URL, timeout=5)
r.raise_for_status()
print("Server is running.")
except Exception:
print("ERROR: Server not running. Start it with ./wakeup.sh first.")
return
print("=" * 70)
print(" Tool Calling Benchmark (via Tatlock API)")
print("=" * 70)
print(f" Models: {', '.join(models)}")
print(f" Scenarios: {len(SCENARIOS)}")
print(f" Iterations: {args.iterations}")
print(f" Total runs: {len(SCENARIOS) * args.iterations * len(models)}")
# Remember original model to restore after benchmark
original_env = ENV_PATH.read_text()
all_stats = []
async with httpx.AsyncClient() as client:
for model in models:
stats = await benchmark_model(client, model, args.iterations)
all_stats.append(stats)
# Restore original .env
ENV_PATH.write_text(original_env)
print(f"\n .env restored to original")
print_comparison(all_stats)
save_results(all_stats, Path(args.output))
if __name__ == "__main__":
asyncio.run(main())
+1 -1
View File
@@ -3,7 +3,7 @@
# Verifies room groups are controlled by checking actual state changes # Verifies room groups are controlled by checking actual state changes
API_URL="http://localhost:8777/v1/chat/completions" API_URL="http://localhost:8777/v1/chat/completions"
CORE_API="http://192.168.86.149:8083" CORE_API="http://localhost:8083"
RESULTS_FILE="/tmp/housekeeper_test_results.txt" RESULTS_FILE="/tmp/housekeeper_test_results.txt"
GREEN='\033[0;32m' GREEN='\033[0;32m'
-407
View File
@@ -1,407 +0,0 @@
"""
Multi-agent coordination engine.
Orchestrates delegation from Tatlock to expert agents (Librarian, etc.)
based on Steward recommendations. Handles:
- Routing tasks to appropriate agents
- Parallel and sequential execution
- Result aggregation
- Error handling and graceful degradation
"""
import asyncio
import time
from typing import Any, AsyncGenerator, Optional
from src.agents.librarian import run_librarian, run_librarian_stream
from src.agents.protocol import (
AgentError,
AgentRequest,
AgentResponse,
AgentTimeoutError,
AgentUnavailableError,
CoordinationResult,
DelegationIntent,
DelegationReason,
ToolCallRecord,
)
from src.core.household_registry import get_household_registry
from src.core.logging_config import get_logger
logger = get_logger(__name__)
# Agent execution functions registry
AGENT_EXECUTORS: dict[str, Any] = {
"librarian": run_librarian,
}
AGENT_STREAM_EXECUTORS: dict[str, Any] = {
"librarian": run_librarian_stream,
}
class CoordinationEngine:
"""
Coordinates multi-agent task execution.
Routes tasks from Tatlock to appropriate expert agents,
handles execution, and aggregates results.
"""
def __init__(self):
"""Initialize the coordination engine."""
self.registry = get_household_registry()
logger.info("coordination_engine_initialized")
def get_available_agents(self) -> list[str]:
"""
Get list of available expert agents.
Returns:
List of agent names that can accept delegations
"""
available = []
for name in self.registry.list_members():
member = self.registry.get_member(name)
if member and member.agent is not None:
available.append(name)
return available
def can_delegate_to(self, agent_name: str) -> bool:
"""
Check if delegation to an agent is possible.
Args:
agent_name: Name of the target agent
Returns:
True if agent is available and can accept tasks
"""
if agent_name not in AGENT_EXECUTORS:
return False
member = self.registry.get_member(agent_name)
return member is not None and member.agent is not None
async def execute_delegation(
self,
intent: DelegationIntent,
context: str = "",
message_history: Optional[list[Any]] = None,
) -> AgentResponse:
"""
Execute a single delegation to an expert agent.
Args:
intent: The delegation intent with task details
context: Additional context for the agent
message_history: Optional conversation history
Returns:
AgentResponse with results
Raises:
AgentUnavailableError: If agent is not available
AgentTimeoutError: If execution times out
AgentError: For other execution errors
"""
start_time = time.time()
agent_name = intent.target_agent
logger.info(
"delegation_started",
agent=agent_name,
task=intent.task[:100],
reason=intent.reason.value,
)
# Check if agent is available
if not self.can_delegate_to(agent_name):
raise AgentUnavailableError(
f"Agent '{agent_name}' is not available for delegation",
agent_name=agent_name,
)
# Get the executor
executor = AGENT_EXECUTORS.get(agent_name)
if not executor:
raise AgentUnavailableError(
f"No executor found for agent '{agent_name}'",
agent_name=agent_name,
)
try:
# Build the request
request = AgentRequest(
task=intent.task,
context=context,
delegation_reason=intent.reason,
)
# Execute with timeout
timeout = request.timeout_seconds or 60
result = await asyncio.wait_for(
executor(
task=request.task,
context=request.context,
message_history=message_history,
),
timeout=timeout,
)
duration_ms = int((time.time() - start_time) * 1000)
logger.info(
"delegation_completed",
agent=agent_name,
duration_ms=duration_ms,
output_length=len(result),
)
return AgentResponse(
success=True,
result=result,
reasoning=f"Delegated to {agent_name}: {intent.expected_outcome}",
duration_ms=duration_ms,
)
except asyncio.TimeoutError:
duration_ms = int((time.time() - start_time) * 1000)
logger.error(
"delegation_timeout",
agent=agent_name,
duration_ms=duration_ms,
)
raise AgentTimeoutError(
f"Agent '{agent_name}' timed out after {duration_ms}ms",
agent_name=agent_name,
)
except Exception as e:
duration_ms = int((time.time() - start_time) * 1000)
logger.error(
"delegation_error",
agent=agent_name,
error=str(e),
duration_ms=duration_ms,
exc_info=True,
)
return AgentResponse(
success=False,
result="",
error_message=str(e),
duration_ms=duration_ms,
)
async def execute_delegation_stream(
self,
intent: DelegationIntent,
context: str = "",
message_history: Optional[list[Any]] = None,
) -> AsyncGenerator[str, None]:
"""
Execute a delegation with streaming output.
Args:
intent: The delegation intent with task details
context: Additional context for the agent
message_history: Optional conversation history
Yields:
Text deltas from the agent
Raises:
AgentUnavailableError: If agent is not available
"""
agent_name = intent.target_agent
logger.info(
"delegation_stream_started",
agent=agent_name,
task=intent.task[:100],
)
# Check if agent is available
if agent_name not in AGENT_STREAM_EXECUTORS:
raise AgentUnavailableError(
f"Agent '{agent_name}' does not support streaming",
agent_name=agent_name,
)
executor = AGENT_STREAM_EXECUTORS[agent_name]
try:
async for delta in executor(
task=intent.task,
context=context,
message_history=message_history,
):
yield delta
logger.info("delegation_stream_completed", agent=agent_name)
except Exception as e:
logger.error(
"delegation_stream_error",
agent=agent_name,
error=str(e),
exc_info=True,
)
yield f"\n\n[Error from {agent_name}: {str(e)}]"
async def coordinate(
self,
intents: list[DelegationIntent],
context: str = "",
message_history: Optional[list[Any]] = None,
) -> CoordinationResult:
"""
Coordinate execution of multiple delegations.
Handles parallel execution for independent tasks and
sequential execution for dependent tasks.
Args:
intents: List of delegation intents to execute
context: Shared context for all agents
message_history: Optional conversation history
Returns:
CoordinationResult with aggregated results
"""
start_time = time.time()
agent_responses: dict[str, AgentResponse] = {}
agents_consulted: list[str] = []
logger.info(
"coordination_started",
intent_count=len(intents),
agents=[i.target_agent for i in intents],
)
# Sort by priority
sorted_intents = sorted(intents, key=lambda x: x.priority)
# Group by dependencies (simple version: sequential for now)
# TODO: Implement parallel execution for independent tasks
for intent in sorted_intents:
try:
response = await self.execute_delegation(
intent=intent,
context=context,
message_history=message_history,
)
agent_responses[intent.target_agent] = response
if response.success:
agents_consulted.append(intent.target_agent)
except AgentError as e:
agent_responses[intent.target_agent] = AgentResponse(
success=False,
result="",
error_message=str(e),
)
# Aggregate results
successful_results = [
r.result for r in agent_responses.values() if r.success and r.result
]
final_response = "\n\n---\n\n".join(successful_results) if successful_results else ""
total_duration = int((time.time() - start_time) * 1000)
logger.info(
"coordination_completed",
total_duration_ms=total_duration,
agents_consulted=agents_consulted,
success_count=len(successful_results),
)
return CoordinationResult(
final_response=final_response,
agent_responses=agent_responses,
delegation_intents=intents,
total_duration_ms=total_duration,
agents_consulted=agents_consulted,
)
# Global coordination engine instance
_coordination_engine: Optional[CoordinationEngine] = None
def get_coordination_engine() -> CoordinationEngine:
"""Get the global coordination engine instance."""
global _coordination_engine
if _coordination_engine is None:
_coordination_engine = CoordinationEngine()
return _coordination_engine
async def delegate_to_librarian(
task: str,
context: str = "",
reason: DelegationReason = DelegationReason.DOMAIN_EXPERTISE,
message_history: Optional[list[Any]] = None,
) -> AgentResponse:
"""
Convenience function to delegate a task to The Librarian.
Args:
task: Research task description
context: Additional context
reason: Why delegating to Librarian
message_history: Optional conversation history
Returns:
AgentResponse with research results
"""
engine = get_coordination_engine()
intent = DelegationIntent(
target_agent="librarian",
task=task,
reason=reason,
expected_outcome="Research findings and relevant information",
)
return await engine.execute_delegation(
intent=intent,
context=context,
message_history=message_history,
)
async def delegate_to_librarian_stream(
task: str,
context: str = "",
message_history: Optional[list[Any]] = None,
) -> AsyncGenerator[str, None]:
"""
Convenience function to delegate to Librarian with streaming.
Args:
task: Research task description
context: Additional context
message_history: Optional conversation history
Yields:
Text deltas from The Librarian
"""
engine = get_coordination_engine()
intent = DelegationIntent(
target_agent="librarian",
task=task,
reason=DelegationReason.DOMAIN_EXPERTISE,
expected_outcome="Research findings",
)
async for delta in engine.execute_delegation_stream(
intent=intent,
context=context,
message_history=message_history,
):
yield delta
+96 -111
View File
@@ -8,12 +8,13 @@ returns a structured result for synthesis.
This implements the agent-as-tool pattern recommended by PydanticAI: This implements the agent-as-tool pattern recommended by PydanticAI:
agents call other agents via tool wrappers, keeping each agent focused. agents call other agents via tool wrappers, keeping each agent focused.
""" """
import asyncio
from dataclasses import dataclass, field from dataclasses import dataclass, field
from enum import Enum from enum import Enum
from typing import AsyncGenerator, Callable, Optional, Any
from src.core.config import config
from src.core.logging_config import get_logger from src.core.logging_config import get_logger
from src.core.tracing import trace_span, SpanType from src.core.tracing import SpanType, trace_span
logger = get_logger(__name__) logger = get_logger(__name__)
@@ -126,6 +127,50 @@ def _detect_action_type(expert: str, task: str) -> ActionType:
return ActionType.RETRIEVE return ActionType.RETRIEVE
def build_delegation_context(
conversation_history: list[dict] | None,
max_turns: int = 6,
max_chars_per_turn: int = 500,
) -> str:
"""
Format the most recent conversation turns as delegation context.
Experts accept a context string but the live paths never passed the
in-scope conversation history; this trims it to the last few turns
so follow-up questions ("and what about X?") keep their referent.
Args:
conversation_history: Prior messages as {"role", "content"} dicts
max_turns: How many trailing turns to include
max_chars_per_turn: Truncation limit per turn
Returns:
str: Newline-joined "role: content" lines ("" when no history)
"""
if not conversation_history:
return ""
lines = []
for msg in conversation_history[-max_turns:]:
if not isinstance(msg, dict):
continue
role = msg.get("role", "user")
content = msg.get("content", "")
if isinstance(content, list):
# Tolerate structured content parts
content = " ".join(
part.get("text", "") if isinstance(part, dict) else str(part)
for part in content
)
content = str(content).strip()
if content:
lines.append(f"{role}: {content[:max_chars_per_turn]}")
if not lines:
return ""
return "Recent conversation:\n" + "\n".join(lines)
def get_think_message(expert: str, task: str, phase: str) -> str: def get_think_message(expert: str, task: str, phase: str) -> str:
""" """
Get the appropriate think message for an expert delegation. Get the appropriate think message for an expert delegation.
@@ -167,7 +212,7 @@ class DelegationTask:
action: str = "" action: str = ""
priority: int = 0 priority: int = 0
depends_on: list[str] = field(default_factory=list) depends_on: list[str] = field(default_factory=list)
result: Optional[str] = None result: str | None = None
task_id: str = "" task_id: str = ""
def __post_init__(self): def __post_init__(self):
@@ -186,14 +231,16 @@ class DelegationResult:
expert_name: Which expert handled the task expert_name: Which expert handled the task
task: Original task description task: Original task description
success: Whether the delegation succeeded success: Whether the delegation succeeded
output: Expert's response/findings output: Expert's response/findings. On failure this holds a
error: Error message if failed curated, user-safe butler sentence (never exception detail)
error: Short user-safe error label if failed. Exception detail
stays in the logs only
""" """
expert_name: str expert_name: str
task: str task: str
success: bool success: bool
output: str output: str
error: Optional[str] = None error: str | None = None
async def delegate_to_librarian( async def delegate_to_librarian(
@@ -250,8 +297,14 @@ async def delegate_to_librarian(
}, },
) as span: ) as span:
try: try:
# Use run() not run_stream() - avoids Ollama bug # Use run() not run_stream() - avoids Ollama bug.
output = await run_librarian(task=task, context=context) # One timeout budget for the whole delegation - covers both
# live paths (steward direct delegation and streaming), which
# previously had no cap at all (SDK default ~600s per LLM call).
output = await asyncio.wait_for(
run_librarian(task=task, context=context),
timeout=config.LIBRARIAN_TIMEOUT,
)
logger.info( logger.info(
"delegation_to_librarian_completed", "delegation_to_librarian_completed",
@@ -273,6 +326,30 @@ async def delegate_to_librarian(
output=output, output=output,
) )
except TimeoutError:
logger.error(
"delegation_to_librarian_timeout",
task=task[:50],
timeout_seconds=config.LIBRARIAN_TIMEOUT,
)
if span:
span.metadata["success"] = False
span.details["error"] = (
f"timed out after {config.LIBRARIAN_TIMEOUT}s"
)
return DelegationResult(
expert_name="librarian",
task=task,
success=False,
output=(
"I'm afraid the research took longer than expected "
"and had to be abandoned, sir."
),
error="The Librarian did not respond within the time budget.",
)
except Exception as e: except Exception as e:
logger.error( logger.error(
"delegation_to_librarian_error", "delegation_to_librarian_error",
@@ -285,12 +362,15 @@ async def delegate_to_librarian(
span.metadata["success"] = False span.metadata["success"] = False
span.details["error"] = str(e) span.details["error"] = str(e)
# Exception detail stays in the logs; the user-facing output
# is a curated butler sentence so internals never leak into
# synthesis.
return DelegationResult( return DelegationResult(
expert_name="librarian", expert_name="librarian",
task=task, task=task,
success=False, success=False,
output="", output=get_think_message("librarian", task, "error"),
error=str(e), error="The Librarian was unable to complete the task.",
) )
@@ -383,12 +463,13 @@ async def delegate_to_biographer(
span.metadata["success"] = False span.metadata["success"] = False
span.details["error"] = str(e) span.details["error"] = str(e)
# Exception detail stays in the logs only.
return DelegationResult( return DelegationResult(
expert_name="biographer", expert_name="biographer",
task=task, task=task,
success=False, success=False,
output="", output=get_think_message("biographer", task, "error"),
error=str(e), error="The Biographer was unable to complete the task.",
) )
@@ -480,112 +561,16 @@ async def delegate_to_housekeeper(
span.metadata["success"] = False span.metadata["success"] = False
span.details["error"] = str(e) span.details["error"] = str(e)
# Exception detail stays in the logs only.
return DelegationResult( return DelegationResult(
expert_name="housekeeper", expert_name="housekeeper",
task=task, task=task,
success=False, success=False,
output="", output=get_think_message("housekeeper", task, "error"),
error=str(e), error="The Housekeeper was unable to complete the task.",
) )
# =============================================================================
# Streaming Delegation Wrappers (with Think Messages)
# =============================================================================
async def stream_delegate_to_librarian(
task: str,
context: str = "",
) -> AsyncGenerator[str, None]:
"""
Stream delegation to Librarian with automatic think messages.
Yields butler-perspective think messages before and after the delegation,
allowing the UI to show progress to the user.
Args:
task: Task description
context: Additional context
Yields:
str: Think messages and final result marker
"""
# Yield start message (deterministic)
yield get_think_message("librarian", task, "start") + "\n"
# Execute delegation
result = await delegate_to_librarian(task, context)
# Yield completion message (deterministic)
if result.success:
yield get_think_message("librarian", task, "success") + "\n"
else:
yield get_think_message("librarian", task, "error") + "\n"
# Yield result marker for extraction
yield f"__DELEGATION_RESULT__:librarian:{result.output}"
async def stream_delegate_to_biographer(
task: str,
context: str = "",
) -> AsyncGenerator[str, None]:
"""
Stream delegation to Biographer with automatic think messages.
Args:
task: Task description
context: Additional context
Yields:
str: Think messages and final result marker
"""
yield get_think_message("biographer", task, "start") + "\n"
result = await delegate_to_biographer(task, context)
if result.success:
yield get_think_message("biographer", task, "success") + "\n"
else:
yield get_think_message("biographer", task, "error") + "\n"
yield f"__DELEGATION_RESULT__:biographer:{result.output}"
async def stream_delegate_to_housekeeper(
task: str,
context: str = "",
) -> AsyncGenerator[str, None]:
"""
Stream delegation to Housekeeper with automatic think messages.
Args:
task: Task description
context: Additional context
Yields:
str: Think messages and final result marker
"""
yield get_think_message("housekeeper", task, "start") + "\n"
result = await delegate_to_housekeeper(task, context)
if result.success:
yield get_think_message("housekeeper", task, "success") + "\n"
else:
yield get_think_message("housekeeper", task, "error") + "\n"
yield f"__DELEGATION_RESULT__:housekeeper:{result.output}"
# Mapping of streaming delegation wrappers
STREAMING_DELEGATION_WRAPPERS = {
"librarian": stream_delegate_to_librarian,
"biographer": stream_delegate_to_biographer,
"housekeeper": stream_delegate_to_housekeeper,
}
# Future expert delegation wrappers will be added here: # Future expert delegation wrappers will be added here:
# - delegate_to_developer(task, context) -> DelegationResult # - delegate_to_developer(task, context) -> DelegationResult
# - delegate_to_secretary(task, context) -> DelegationResult # - delegate_to_secretary(task, context) -> DelegationResult
+6 -6
View File
@@ -204,13 +204,13 @@ async def run_housekeeper(
) )
try: try:
# Use temperature 0.1 for slight exploration # Temperature 0.1 for slight exploration (skipped on Claude backend)
from pydantic_ai.settings import ModelSettings from src.anthropic.model_selector import get_sampling_settings
result = await agent.run( result = await agent.run(
prompt, prompt,
message_history=message_history, message_history=message_history,
model_settings=ModelSettings(temperature=0.1), model_settings=get_sampling_settings(0.1),
) )
logger.info( logger.info(
@@ -266,13 +266,13 @@ async def run_housekeeper_stream(
) )
try: try:
# Use temperature 0.1 for slight exploration # Temperature 0.1 for slight exploration (skipped on Claude backend)
from pydantic_ai.settings import ModelSettings from src.anthropic.model_selector import get_sampling_settings
async with agent.run_stream( async with agent.run_stream(
prompt, prompt,
message_history=message_history, message_history=message_history,
model_settings=ModelSettings(temperature=0.1), model_settings=get_sampling_settings(0.1),
) as response: ) as response:
async for delta in response.stream_text(delta=True): async for delta in response.stream_text(delta=True):
yield delta yield delta
-2
View File
@@ -10,7 +10,6 @@ Connects to the library-desk API to provide:
from src.agents.librarian.agent import ( from src.agents.librarian.agent import (
get_librarian_agent, get_librarian_agent,
run_librarian, run_librarian,
run_librarian_stream,
) )
from src.agents.librarian.capability import ( from src.agents.librarian.capability import (
LIBRARIAN_CAPABILITY, LIBRARIAN_CAPABILITY,
@@ -26,5 +25,4 @@ __all__ = [
"register_librarian", "register_librarian",
"unregister_librarian", "unregister_librarian",
"run_librarian", "run_librarian",
"run_librarian_stream",
] ]
+45 -64
View File
@@ -7,10 +7,11 @@ the library-desk API, offering:
- Wiki and document management - Wiki and document management
- Semantic search and knowledge graph exploration - Semantic search and knowledge graph exploration
""" """
from typing import Any, Optional from typing import Any
from pydantic_ai import Agent from pydantic_ai import Agent
from src.agents.librarian.client import library_client_session
from src.agents.librarian.tools import ( from src.agents.librarian.tools import (
create_wiki_page, create_wiki_page,
explore_knowledge_graph, explore_knowledge_graph,
@@ -27,7 +28,7 @@ from src.agents.librarian.tools import (
smart_create_wiki_page, smart_create_wiki_page,
update_wiki_page, update_wiki_page,
) )
from src.core.config import config from src.agents.protocol import AgentError
from src.core.logging_config import get_logger from src.core.logging_config import get_logger
logger = get_logger(__name__) logger = get_logger(__name__)
@@ -137,8 +138,32 @@ If a tool fails or you cannot access a data source:
- It is better to return no information than to return fabricated information - It is better to return no information than to return fabricated information
""" """
# Tool-phase prompt actually used by the agent. The scholarly persona prompt
# above suppresses tool calling on small local models (gemma4 answers in
# character - "please provide your request" - without ever calling a tool),
# the same pathology TATLOCK_ORCHESTRATION_PROMPT fixed for the butler.
# Tatlock's synthesis phase supplies the user-facing voice, so the research
# phase only needs tool discipline. Kept: the anti-fabrication rule.
LIBRARIAN_TASK_PROMPT = """You are The Librarian, the research executor of the \
Tatlock household. Your only job is to gather accurate findings by calling the \
provided tools.
- ALWAYS use tools - never answer a research task from memory alone.
- Research or wiki questions: call hybrid_search first; then search_wiki and \
get_wiki_page to read specific pages BEFORE summarizing them.
- Current or external information (weather, news, live facts): call search_web; \
call read_url when given a specific URL.
- Wiki writing: smart_create_wiki_page when asked for a page about a topic; \
create_wiki_page only for user-provided verbatim content; update_wiki_page for \
edits (search_wiki, then get_wiki_page, then update).
- Reply with a concise factual summary of what the tools returned, citing page \
titles and URLs. A later step writes the polished answer, so no personality.
- NEVER fabricate. If a tool fails or returns nothing, state exactly what you \
could not retrieve and stop."""
# Lazy initialization to avoid connection issues during imports # Lazy initialization to avoid connection issues during imports
_librarian_agent: Optional[Agent[None, str]] = None _librarian_agent: Agent[None, str] | None = None
def _create_librarian_agent() -> Agent[None, str]: def _create_librarian_agent() -> Agent[None, str]:
@@ -150,7 +175,7 @@ def _create_librarian_agent() -> Agent[None, str]:
agent: Agent[None, str] = Agent( agent: Agent[None, str] = Agent(
model=model, model=model,
system_prompt=LIBRARIAN_SYSTEM_PROMPT, system_prompt=LIBRARIAN_TASK_PROMPT,
retries=2, retries=2,
) )
@@ -204,7 +229,7 @@ def get_librarian_agent() -> Agent[None, str]:
async def run_librarian( async def run_librarian(
task: str, task: str,
context: str = "", context: str = "",
message_history: Optional[list[Any]] = None, message_history: list[Any] | None = None,
) -> str: ) -> str:
""" """
Execute a research task with The Librarian. Execute a research task with The Librarian.
@@ -220,6 +245,10 @@ async def run_librarian(
Returns: Returns:
Research results and findings Research results and findings
Raises:
AgentError: If the research task fails. Exception detail is
logged here; callers map the failure to a user-safe message.
Example: Example:
result = await run_librarian( result = await run_librarian(
task="Find information about Docker networking", task="Find information about Docker networking",
@@ -241,10 +270,12 @@ async def run_librarian(
) )
try: try:
result = await agent.run( # One shared library-desk connection for all tool calls in this run
prompt, async with library_client_session():
message_history=message_history, result = await agent.run(
) prompt,
message_history=message_history,
)
logger.info( logger.info(
"librarian_task_completed", "librarian_task_completed",
@@ -255,64 +286,14 @@ async def run_librarian(
return result.output return result.output
except Exception as e: except Exception as e:
# Full detail stays in the logs; callers receive a structured
# failure instead of error text masquerading as research output.
logger.error( logger.error(
"librarian_task_error", "librarian_task_error",
task=task[:50], task=task[:50],
error=str(e), error=str(e),
exc_info=True, exc_info=True,
) )
return f"The Librarian encountered an error: {str(e)}" raise AgentError(
"Research task failed", agent_name="librarian"
) from e
async def run_librarian_stream(
task: str,
context: str = "",
message_history: Optional[list[Any]] = None,
):
"""
Execute a research task with streaming output.
Yields text deltas as The Librarian generates the response.
Args:
task: The research task or question
context: Additional context from conversation
message_history: Optional conversation history
Yields:
str: Text deltas from the response
Example:
async for delta in run_librarian_stream("Find Docker docs"):
print(delta, end="", flush=True)
"""
agent = get_librarian_agent()
# Build prompt with context if provided
prompt = task
if context:
prompt = f"Context: {context}\n\nTask: {task}"
logger.info(
"librarian_stream_started",
task=task[:100],
)
try:
async with agent.run_stream(
prompt,
message_history=message_history,
) as response:
async for delta in response.stream_text(delta=True):
yield delta
logger.info("librarian_stream_completed", task=task[:50])
except Exception as e:
logger.error(
"librarian_stream_error",
task=task[:50],
error=str(e),
exc_info=True,
)
yield f"\n\nThe Librarian encountered an error: {str(e)}"
+289 -88
View File
@@ -7,17 +7,32 @@ Provides async methods for all relevant library-desk endpoints:
- Vector search - Vector search
- Knowledge graph queries - Knowledge graph queries
""" """
from typing import Any, Optional import asyncio
from collections.abc import AsyncIterator, Awaitable, Callable
from contextlib import asynccontextmanager
from contextvars import ContextVar
from typing import Any
import httpx import httpx
from pydantic import BaseModel, Field from pydantic import BaseModel, Field
from src.core.config import config from src.core.config import config
from src.core.context import get_user from src.core.context import apply_tenant_guard, get_user
from src.core.logging_config import get_logger from src.core.logging_config import get_logger
logger = get_logger(__name__) logger = get_logger(__name__)
# Retry policy for idempotent/read-only requests (GETs, POST /query/*,
# POST /rag/search). Writes are never retried.
_RETRY_ATTEMPTS = 2
_RETRY_BACKOFF_SECONDS = 0.5
_RETRYABLE_STATUS_CODES = {502, 503, 504}
# One shared HTTP connection per librarian run (see library_client_session)
_shared_http_client: ContextVar[httpx.AsyncClient | None] = ContextVar(
"library_desk_http_client", default=None
)
# ============================================================================ # ============================================================================
# Response Models # Response Models
@@ -28,11 +43,11 @@ class WikiPage(BaseModel):
id: int id: int
path: str path: str
title: str title: str
description: Optional[str] = None description: str | None = None
content: Optional[str] = None content: str | None = None
tags: list[str] = Field(default_factory=list) tags: list[str] = Field(default_factory=list)
created_at: Optional[str] = None created_at: str | None = None
updated_at: Optional[str] = None updated_at: str | None = None
class WikiSearchResult(BaseModel): class WikiSearchResult(BaseModel):
@@ -40,8 +55,8 @@ class WikiSearchResult(BaseModel):
id: int id: int
path: str path: str
title: str title: str
description: Optional[str] = None description: str | None = None
locale: Optional[str] = None locale: str | None = None
class VectorSearchResult(BaseModel): class VectorSearchResult(BaseModel):
@@ -56,12 +71,14 @@ class VectorSearchResult(BaseModel):
class HybridSearchResult(BaseModel): class HybridSearchResult(BaseModel):
"""Result from HybridRAG search.""" """Result from HybridRAG search."""
source: str # "vector", "graph", "web" source: str # source_type: "wiki", "web", "volatile", "document"
sources: list[str] = Field(default_factory=list) # legs that found it: "vector", "graph", "web", ...
title: str title: str
content: str content: str
url: Optional[str] = None url: str | None = None
score: float score: float # rrf_score from the live service
page_id: Optional[int] = None page_id: int | None = None
related_dossiers: list[dict[str, Any]] = Field(default_factory=list)
metadata: dict[str, Any] = Field(default_factory=dict) metadata: dict[str, Any] = Field(default_factory=dict)
@@ -72,8 +89,15 @@ class HybridRAGResponse(BaseModel):
synonyms: list[str] = Field(default_factory=list) synonyms: list[str] = Field(default_factory=list)
related_dossiers: list[str] = Field(default_factory=list) related_dossiers: list[str] = Field(default_factory=list)
formatted_context: str = "" formatted_context: str = ""
search_id: Optional[str] = None search_id: str | None = None
source_counts: dict[str, int] = Field(default_factory=dict)
timing: dict[str, float] = Field(default_factory=dict) timing: dict[str, float] = Field(default_factory=dict)
# Additive degradation contract - only newer library-desk versions
# send these; absence means "no status reported", not "healthy".
# Maps each leg (vector/graph/web/volatile/documents) to
# "ok" | "failed" | "disabled".
source_status: dict[str, str] = Field(default_factory=dict)
degraded: bool = False
class GraphNode(BaseModel): class GraphNode(BaseModel):
@@ -105,7 +129,7 @@ class WebSearchResult(BaseModel):
content: str = "" # Full extracted text via Trafilatura content: str = "" # Full extracted text via Trafilatura
snippet: str = "" # Original search engine snippet snippet: str = "" # Original search engine snippet
source: str = "" # Domain name source: str = "" # Domain name
published_date: Optional[str] = None published_date: str | None = None
class WebSearchResponse(BaseModel): class WebSearchResponse(BaseModel):
@@ -121,13 +145,13 @@ class WebSearchResponse(BaseModel):
class ContentExtractionResult(BaseModel): class ContentExtractionResult(BaseModel):
"""Result from content extraction.""" """Result from content extraction."""
url: str url: str
title: Optional[str] = None title: str | None = None
content: str = "" content: str = ""
author: Optional[str] = None author: str | None = None
date: Optional[str] = None date: str | None = None
language: Optional[str] = None language: str | None = None
success: bool = True success: bool = True
error: Optional[str] = None error: str | None = None
class BatchExtractionResponse(BaseModel): class BatchExtractionResponse(BaseModel):
@@ -151,7 +175,7 @@ class SmartCreateResponse(BaseModel):
page: WikiPage page: WikiPage
research_summary: ResearchSummary = Field(default_factory=ResearchSummary) research_summary: ResearchSummary = Field(default_factory=ResearchSummary)
sources_used: int = 0 sources_used: int = 0
search_id: Optional[str] = None search_id: str | None = None
entity_linking: EntityLinking = Field(default_factory=EntityLinking) entity_linking: EntityLinking = Field(default_factory=EntityLinking)
@@ -170,9 +194,9 @@ class LibraryDeskClient:
def __init__( def __init__(
self, self,
base_url: Optional[str] = None, base_url: str | None = None,
api_key: Optional[str] = None, api_key: str | None = None,
timeout: int = 60, timeout: int | None = None,
): ):
""" """
Initialize the client. Initialize the client.
@@ -181,30 +205,55 @@ class LibraryDeskClient:
base_url: Library-desk API URL (defaults to config) base_url: Library-desk API URL (defaults to config)
api_key: API key for authentication (defaults to config) api_key: API key for authentication (defaults to config)
timeout: Request timeout in seconds timeout: Request timeout in seconds
(defaults to config.LIBRARY_DESK_TIMEOUT)
""" """
self.base_url = base_url or str(config.LIBRARY_DESK_HOST) self.base_url = base_url or str(config.LIBRARY_DESK_HOST)
self.api_key = api_key or config.LIBRARY_DESK_API_KEY self.api_key = api_key or config.LIBRARY_DESK_API_KEY
self.timeout = timeout self.timeout = timeout if timeout is not None else config.LIBRARY_DESK_TIMEOUT
self._client: Optional[httpx.AsyncClient] = None self._client: httpx.AsyncClient | None = None
self._owns_client = False
async def __aenter__(self) -> "LibraryDeskClient": def _build_http_client(self) -> httpx.AsyncClient:
"""Create HTTP client on context entry.""" """Build a configured httpx client."""
headers = {} headers = {}
if self.api_key: if self.api_key:
headers["Authorization"] = f"Bearer {self.api_key}" headers["Authorization"] = f"Bearer {self.api_key}"
self._client = httpx.AsyncClient( return httpx.AsyncClient(
base_url=self.base_url, base_url=self.base_url,
headers=headers, headers=headers,
timeout=self.timeout, timeout=self.timeout,
) )
def _uses_default_target(self) -> bool:
"""Whether this client targets the configured library-desk instance."""
return (
self.base_url == str(config.LIBRARY_DESK_HOST)
and self.api_key == config.LIBRARY_DESK_API_KEY
)
async def __aenter__(self) -> "LibraryDeskClient":
"""
Acquire an HTTP client on context entry.
Reuses the run-level shared connection (see library_client_session)
when one is active, instead of constructing a new client per call.
"""
shared = _shared_http_client.get()
if shared is not None and not shared.is_closed and self._uses_default_target():
self._client = shared
self._owns_client = False
else:
self._client = self._build_http_client()
self._owns_client = True
return self return self
async def __aexit__(self, exc_type: Any, exc_val: Any, exc_tb: Any) -> None: async def __aexit__(self, exc_type: Any, exc_val: Any, exc_tb: Any) -> None:
"""Close HTTP client on context exit.""" """Close HTTP client on context exit (only if we own it)."""
if self._client: if self._client and self._owns_client:
await self._client.aclose() await self._client.aclose()
self._client = None self._client = None
self._owns_client = False
def _ensure_client(self) -> httpx.AsyncClient: def _ensure_client(self) -> httpx.AsyncClient:
"""Ensure client is initialized.""" """Ensure client is initialized."""
@@ -214,6 +263,69 @@ class LibraryDeskClient:
) )
return self._client return self._client
def _resolve_user(self, user: str | None) -> str:
"""
Resolve the effective tenant for a request and require it non-empty.
Library-desk is removing its server-side default user, so every
request must carry an explicit tenant (a missing user will 422).
An empty tenant is a programming or configuration error - fail
loudly here, before any bytes hit the wire.
Explicit user arguments are stripped and routed through the same
tenant guard as context resolution (get_user() already applies
it), so a dev environment can never send the production tenant -
or a sanitization-collision variant of it - to library-desk.
"""
effective = (user if user is not None else get_user()).strip()
if not effective:
raise ValueError(
"library-desk request requires a non-empty user (tenant); "
"got an empty value from the caller or request context"
)
return apply_tenant_guard(effective)
async def _request_with_retry(
self,
send: Callable[[], Awaitable[httpx.Response]],
description: str,
) -> httpx.Response:
"""
Send an idempotent/read-only request with a bounded retry.
Retries once (2 attempts total) with a short backoff on transport
errors and retryable 5xx statuses. Only used for GETs and the
read-only POST /query/* and /rag/search endpoints - never for
wiki writes.
"""
for attempt in range(1, _RETRY_ATTEMPTS + 1):
try:
response = await send()
except httpx.TransportError as e:
if attempt >= _RETRY_ATTEMPTS:
raise
logger.warning(
"library_desk_retry",
request=description,
error=str(e),
attempt=attempt,
)
else:
if (
response.status_code not in _RETRYABLE_STATUS_CODES
or attempt >= _RETRY_ATTEMPTS
):
return response
logger.warning(
"library_desk_retry",
request=description,
status_code=response.status_code,
attempt=attempt,
)
await asyncio.sleep(_RETRY_BACKOFF_SECONDS * attempt)
raise RuntimeError("unreachable") # pragma: no cover
# ======================================================================== # ========================================================================
# HybridRAG # HybridRAG
# ======================================================================== # ========================================================================
@@ -247,17 +359,19 @@ class LibraryDeskClient:
Returns: Returns:
HybridRAGResponse with ranked results and context HybridRAGResponse with ranked results and context
""" """
user = user or get_user() user = self._resolve_user(user)
client = self._ensure_client() client = self._ensure_client()
# The live service requires all limits >= 1 (422 otherwise);
# legs are disabled via the enable_* flags, not a zero limit.
payload = { payload = {
"query": query, "query": query,
"config": { "config": {
"vector_limit": vector_limit, "vector_limit": max(vector_limit, 1),
"graph_limit": graph_limit, "graph_limit": max(graph_limit, 1),
"web_limit": web_limit, "web_limit": max(web_limit, 1),
"document_limit": document_limit, "document_limit": max(document_limit, 1),
"volatile_limit": volatile_limit, "volatile_limit": max(volatile_limit, 1),
"enable_documents": document_limit > 0, "enable_documents": document_limit > 0,
"enable_volatile": volatile_limit > 0, "enable_volatile": volatile_limit > 0,
"enable_web": web_limit > 0, "enable_web": web_limit > 0,
@@ -268,43 +382,69 @@ class LibraryDeskClient:
logger.info("library_desk_hybrid_search", query=query, user=user) logger.info("library_desk_hybrid_search", query=query, user=user)
response = await client.post( response = await self._request_with_retry(
"/query/hybrid", lambda: client.post(
json=payload, "/query/hybrid",
params={"user": user}, json=payload,
params={"user": user},
),
"POST /query/hybrid",
) )
response.raise_for_status() response.raise_for_status()
data = response.json() data = response.json()
# Parse results # Parse results (live field names: source_type, sources, rrf_score,
# related_dossiers; older names kept as fallbacks)
results = [] results = []
for r in data.get("results", []): for r in data.get("results", []):
results.append(HybridSearchResult( results.append(HybridSearchResult(
source=r.get("source", "unknown"), source=r.get("source_type") or r.get("source", "unknown"),
sources=r.get("sources", []),
title=r.get("title", ""), title=r.get("title", ""),
content=r.get("content", ""), content=r.get("content", ""),
url=r.get("url"), url=r.get("url"),
score=r.get("score", 0.0), score=r.get("rrf_score", r.get("score", 0.0)),
page_id=r.get("page_id"), page_id=r.get("page_id"),
related_dossiers=r.get("related_dossiers", []),
metadata=r.get("metadata", {}), metadata=r.get("metadata", {}),
)) ))
# Handle keywords being either a list or a dict with core_keywords # Handle keywords being either a list or a dict with core_keywords;
# the live service nests synonyms inside the keywords dict as a
# {term: [synonyms]} map.
raw_keywords = data.get("keywords", []) raw_keywords = data.get("keywords", [])
raw_synonyms: Any = data.get("synonyms", [])
if isinstance(raw_keywords, dict): if isinstance(raw_keywords, dict):
keywords = raw_keywords.get("core_keywords", []) keywords = raw_keywords.get("core_keywords", [])
raw_synonyms = raw_keywords.get("synonyms", {})
else: else:
keywords = raw_keywords keywords = raw_keywords
if isinstance(raw_synonyms, dict):
synonyms = [s for values in raw_synonyms.values() for s in values]
else:
synonyms = raw_synonyms
# Aggregate per-result related dossiers into unique top-level titles
related_dossiers: list[str] = []
for result in results:
for dossier in result.related_dossiers:
title = dossier.get("title", "")
if title and title not in related_dossiers:
related_dossiers.append(title)
return HybridRAGResponse( return HybridRAGResponse(
results=results, results=results,
keywords=keywords, keywords=keywords,
synonyms=data.get("synonyms", []), synonyms=synonyms,
related_dossiers=data.get("related_dossiers", []), related_dossiers=related_dossiers,
formatted_context=data.get("formatted_context", ""), formatted_context=data.get("context", data.get("formatted_context", "")),
search_id=data.get("search_id"), search_id=data.get("search_id"),
source_counts=data.get("source_counts", {}),
timing=data.get("timing", {}), timing=data.get("timing", {}),
# Additive fields - tolerate absence on older library-desk
source_status=data.get("source_status") or {},
degraded=bool(data.get("degraded", False)),
) )
# ======================================================================== # ========================================================================
@@ -328,14 +468,17 @@ class LibraryDeskClient:
Returns: Returns:
List of matching wiki pages List of matching wiki pages
""" """
user = user or get_user() user = self._resolve_user(user)
client = self._ensure_client() client = self._ensure_client()
logger.debug("library_desk_wiki_search", query=query, user=user) logger.debug("library_desk_wiki_search", query=query, user=user)
response = await client.get( response = await self._request_with_retry(
"/wiki/search", lambda: client.get(
params={"q": query, "user": user, "limit": limit}, "/wiki/search",
params={"q": query, "user": user, "limit": limit},
),
"GET /wiki/search",
) )
response.raise_for_status() response.raise_for_status()
@@ -357,12 +500,15 @@ class LibraryDeskClient:
Returns: Returns:
WikiPage with full content WikiPage with full content
""" """
user = user or get_user() user = self._resolve_user(user)
client = self._ensure_client() client = self._ensure_client()
response = await client.get( response = await self._request_with_retry(
f"/wiki/pages/{page_id}", lambda: client.get(
params={"user": user}, f"/wiki/pages/{page_id}",
params={"user": user},
),
f"GET /wiki/pages/{page_id}",
) )
response.raise_for_status() response.raise_for_status()
@@ -371,7 +517,7 @@ class LibraryDeskClient:
async def list_wiki_pages( async def list_wiki_pages(
self, self,
user: str | None = None, user: str | None = None,
tag: Optional[str] = None, tag: str | None = None,
limit: int = 50, limit: int = 50,
) -> list[WikiPage]: ) -> list[WikiPage]:
""" """
@@ -385,14 +531,17 @@ class LibraryDeskClient:
Returns: Returns:
List of wiki pages List of wiki pages
""" """
user = user or get_user() user = self._resolve_user(user)
client = self._ensure_client() client = self._ensure_client()
params: dict[str, Any] = {"user": user, "limit": limit} params: dict[str, Any] = {"user": user, "limit": limit}
if tag: if tag:
params["tag"] = tag params["tag"] = tag
response = await client.get("/wiki/pages", params=params) response = await self._request_with_retry(
lambda: client.get("/wiki/pages", params=params),
"GET /wiki/pages",
)
response.raise_for_status() response.raise_for_status()
data = response.json() data = response.json()
@@ -405,7 +554,7 @@ class LibraryDeskClient:
content: str, content: str,
user: str | None = None, user: str | None = None,
description: str = "", description: str = "",
tags: Optional[list[str]] = None, tags: list[str] | None = None,
) -> WikiPage: ) -> WikiPage:
""" """
Create a new wiki page. Create a new wiki page.
@@ -421,7 +570,7 @@ class LibraryDeskClient:
Returns: Returns:
Created WikiPage Created WikiPage
""" """
user = user or get_user() user = self._resolve_user(user)
client = self._ensure_client() client = self._ensure_client()
payload = { payload = {
@@ -444,10 +593,10 @@ class LibraryDeskClient:
self, self,
page_id: int, page_id: int,
user: str | None = None, user: str | None = None,
content: Optional[str] = None, content: str | None = None,
title: Optional[str] = None, title: str | None = None,
tags: Optional[list[str]] = None, tags: list[str] | None = None,
description: Optional[str] = None, description: str | None = None,
) -> WikiPage: ) -> WikiPage:
""" """
Update an existing wiki page. Update an existing wiki page.
@@ -466,7 +615,7 @@ class LibraryDeskClient:
Returns: Returns:
Updated WikiPage Updated WikiPage
""" """
user = user or get_user() user = self._resolve_user(user)
client = self._ensure_client() client = self._ensure_client()
# Build update payload with only provided fields # Build update payload with only provided fields
@@ -500,7 +649,7 @@ class LibraryDeskClient:
topic: str, topic: str,
tags: list[str], tags: list[str],
user: str | None = None, user: str | None = None,
path: Optional[str] = None, path: str | None = None,
include_web_research: bool = True, include_web_research: bool = True,
include_wiki_search: bool = True, include_wiki_search: bool = True,
) -> SmartCreateResponse: ) -> SmartCreateResponse:
@@ -524,7 +673,7 @@ class LibraryDeskClient:
Returns: Returns:
SmartCreateResponse with page and research metadata SmartCreateResponse with page and research metadata
""" """
user = user or get_user() user = self._resolve_user(user)
client = self._ensure_client() client = self._ensure_client()
payload: dict[str, Any] = { payload: dict[str, Any] = {
@@ -575,12 +724,15 @@ class LibraryDeskClient:
Returns: Returns:
List of dossiers with page counts List of dossiers with page counts
""" """
user = user or get_user() user = self._resolve_user(user)
client = self._ensure_client() client = self._ensure_client()
response = await client.get( response = await self._request_with_retry(
"/wiki/dossiers", lambda: client.get(
params={"user": user}, "/wiki/dossiers",
params={"user": user},
),
"GET /wiki/dossiers",
) )
response.raise_for_status() response.raise_for_status()
@@ -610,7 +762,7 @@ class LibraryDeskClient:
Returns: Returns:
List of matching document chunks with scores List of matching document chunks with scores
""" """
user = user or get_user() user = self._resolve_user(user)
client = self._ensure_client() client = self._ensure_client()
payload = { payload = {
@@ -636,7 +788,7 @@ class LibraryDeskClient:
self, self,
cypher_query: str, cypher_query: str,
user: str | None = None, user: str | None = None,
parameters: Optional[dict[str, Any]] = None, parameters: dict[str, Any] | None = None,
) -> list[dict[str, Any]]: ) -> list[dict[str, Any]]:
""" """
Execute a Cypher query on the knowledge graph. Execute a Cypher query on the knowledge graph.
@@ -651,7 +803,7 @@ class LibraryDeskClient:
Returns: Returns:
List of result records List of result records
""" """
user = user or get_user() user = self._resolve_user(user)
client = self._ensure_client() client = self._ensure_client()
payload = { payload = {
@@ -670,7 +822,7 @@ class LibraryDeskClient:
async def list_graph_nodes( async def list_graph_nodes(
self, self,
user: str | None = None, user: str | None = None,
node_type: Optional[str] = None, node_type: str | None = None,
limit: int = 100, limit: int = 100,
) -> list[GraphNode]: ) -> list[GraphNode]:
""" """
@@ -684,14 +836,17 @@ class LibraryDeskClient:
Returns: Returns:
List of graph nodes List of graph nodes
""" """
user = user or get_user() user = self._resolve_user(user)
client = self._ensure_client() client = self._ensure_client()
params: dict[str, Any] = {"user": user, "limit": limit} params: dict[str, Any] = {"user": user, "limit": limit}
if node_type: if node_type:
params["node_type"] = node_type params["node_type"] = node_type
response = await client.get("/graph/nodes", params=params) response = await self._request_with_retry(
lambda: client.get("/graph/nodes", params=params),
"GET /graph/nodes",
)
response.raise_for_status() response.raise_for_status()
data = response.json() data = response.json()
@@ -712,12 +867,15 @@ class LibraryDeskClient:
Returns: Returns:
Node with relationships and connected nodes Node with relationships and connected nodes
""" """
user = user or get_user() user = self._resolve_user(user)
client = self._ensure_client() client = self._ensure_client()
response = await client.get( response = await self._request_with_retry(
f"/graph/nodes/{node_id}", lambda: client.get(
params={"user": user}, f"/graph/nodes/{node_id}",
params={"user": user},
),
f"GET /graph/nodes/{node_id}",
) )
response.raise_for_status() response.raise_for_status()
@@ -736,7 +894,10 @@ class LibraryDeskClient:
""" """
try: try:
client = self._ensure_client() client = self._ensure_client()
response = await client.get("/health") response = await self._request_with_retry(
lambda: client.get("/health"),
"GET /health",
)
return response.status_code == 200 return response.status_code == 200
except Exception as e: except Exception as e:
logger.warning("library_desk_health_check_failed", error=str(e)) logger.warning("library_desk_health_check_failed", error=str(e))
@@ -768,19 +929,22 @@ class LibraryDeskClient:
Returns: Returns:
WebSearchResponse with results and pre-formatted sources WebSearchResponse with results and pre-formatted sources
""" """
user = user or get_user() user = self._resolve_user(user)
client = self._ensure_client() client = self._ensure_client()
payload = { payload = {
"query": query, "query": query,
"search_type": search_type, "search_type": search_type,
"limit": limit, "limit": limit,
"user": user or "tatlock-librarian", "user": user,
} }
logger.info("library_desk_web_search", query=query, limit=limit) logger.info("library_desk_web_search", query=query, limit=limit)
response = await client.post("/rag/search", json=payload, timeout=30.0) response = await self._request_with_retry(
lambda: client.post("/rag/search", json=payload),
"POST /rag/search",
)
response.raise_for_status() response.raise_for_status()
data = response.json() data = response.json()
@@ -813,6 +977,7 @@ class LibraryDeskClient:
async def extract_content( async def extract_content(
self, self,
url: str, url: str,
user: str | None = None,
include_metadata: bool = True, include_metadata: bool = True,
max_length: int = 5000, max_length: int = 5000,
) -> ContentExtractionResult: ) -> ContentExtractionResult:
@@ -826,12 +991,14 @@ class LibraryDeskClient:
Args: Args:
url: URL to extract content from url: URL to extract content from
user: User identifier (defaults to request context)
include_metadata: Whether to extract author, date, etc. include_metadata: Whether to extract author, date, etc.
max_length: Maximum content length max_length: Maximum content length
Returns: Returns:
ContentExtractionResult (check .success and .error fields) ContentExtractionResult (check .success and .error fields)
""" """
user = self._resolve_user(user)
client = self._ensure_client() client = self._ensure_client()
payload = { payload = {
@@ -842,7 +1009,11 @@ class LibraryDeskClient:
logger.debug("library_desk_extract_content", url=url) logger.debug("library_desk_extract_content", url=url)
response = await client.post("/content/extract", json=payload, timeout=30.0) response = await client.post(
"/content/extract",
json=payload,
params={"user": user},
)
response.raise_for_status() response.raise_for_status()
data = response.json() data = response.json()
@@ -862,6 +1033,7 @@ class LibraryDeskClient:
async def extract_content_batch( async def extract_content_batch(
self, self,
urls: list[str], urls: list[str],
user: str | None = None,
include_metadata: bool = True, include_metadata: bool = True,
max_length: int = 2000, max_length: int = 2000,
) -> BatchExtractionResponse: ) -> BatchExtractionResponse:
@@ -875,12 +1047,14 @@ class LibraryDeskClient:
Args: Args:
urls: List of URLs to extract (max 20) urls: List of URLs to extract (max 20)
user: User identifier (defaults to request context)
include_metadata: Whether to extract author, date, etc. include_metadata: Whether to extract author, date, etc.
max_length: Maximum content length per URL max_length: Maximum content length per URL
Returns: Returns:
BatchExtractionResponse with results and stats BatchExtractionResponse with results and stats
""" """
user = self._resolve_user(user)
client = self._ensure_client() client = self._ensure_client()
payload = { payload = {
@@ -894,7 +1068,7 @@ class LibraryDeskClient:
response = await client.post( response = await client.post(
"/content/extract/batch", "/content/extract/batch",
json=payload, json=payload,
timeout=60.0, # Longer timeout for batch params={"user": user},
) )
response.raise_for_status() response.raise_for_status()
@@ -933,3 +1107,30 @@ async def get_library_client() -> LibraryDeskClient:
results = await client.hybrid_search("query") results = await client.hybrid_search("query")
""" """
return LibraryDeskClient() return LibraryDeskClient()
@asynccontextmanager
async def library_client_session() -> AsyncIterator[None]:
"""
Hold ONE shared HTTP connection for the duration of a librarian run.
While the session is active, every LibraryDeskClient targeting the
configured library-desk instance reuses the shared httpx client
instead of constructing (and tearing down) a connection per tool
call. Nested sessions are no-ops.
Usage:
async with library_client_session():
... # librarian tools reuse one connection
"""
if _shared_http_client.get() is not None:
yield
return
http_client = LibraryDeskClient()._build_http_client()
token = _shared_http_client.set(http_client)
try:
yield
finally:
_shared_http_client.reset(token)
await http_client.aclose()
+181 -46
View File
@@ -4,12 +4,109 @@ Librarian tools for PydanticAI agent.
These tools wrap the library-desk API and are registered with These tools wrap the library-desk API and are registered with
The Librarian agent for research and knowledge management tasks. The Librarian agent for research and knowledge management tasks.
""" """
from src.agents.librarian.client import LibraryDeskClient import httpx
from pydantic_ai import ModelRetry
from src.agents.librarian.client import HybridRAGResponse, LibraryDeskClient
from src.core.logging_config import get_logger from src.core.logging_config import get_logger
logger = get_logger(__name__) logger = get_logger(__name__)
def _retry_if_transient(e: Exception, what: str) -> None:
"""
Convert transient HTTP errors into ModelRetry so the agent's
retry budget (Agent(retries=2)) engages instead of the tool
swallowing the failure.
Only read tools call this - writes are never retried to avoid
duplicate wiki pages.
"""
retryable = isinstance(e, httpx.TransportError)
if isinstance(e, httpx.HTTPStatusError):
status = e.response.status_code
retryable = status >= 500 or status == 429
if retryable:
raise ModelRetry(
f"{what} is temporarily unavailable; please retry."
) from e
# Icons keyed by the values library-desk emits in each result's `sources`
# list (search legs) and `source_type` (result origin).
SOURCE_ICONS = {
"vector": "📄",
"graph": "🔗",
"web": "🌐",
"document": "📑",
"documents": "📑",
"volatile": "",
"wiki": "📄",
}
def _coverage_note(
response: HybridRAGResponse,
include_web: bool,
include_documents: bool,
include_volatile: bool,
) -> str:
"""
Build a one-line coverage note when the search was degraded or an
enabled source leg contributed nothing, so outages stay visible to
the model and the user instead of silently narrowing results.
When the additive source_status/degraded contract is present it is
authoritative and used EXCLUSIVELY - no count heuristics. Without
it, absence from source_counts is only inferred for the optional
legs this request explicitly enabled (web/documents/volatile);
the always-on wiki legs (vector/graph) are never inferred, because
source_counts only tallies the sources of the final top-N fused
results, so their absence is normal ranking behavior, not an outage.
"""
if response.source_status:
failed = sorted(
leg
for leg, status in response.source_status.items()
if status == "failed"
)
if failed:
return (
"⚠️ *Coverage note: results are partial - "
f"these sources failed: {', '.join(failed)}.*"
)
if response.degraded:
return (
"⚠️ *Coverage note: results are partial - "
"one or more sources failed during this search.*"
)
return ""
if not response.source_counts:
# Older library-desk without per-source reporting - nothing to infer
return ""
# Only legs the request explicitly enabled; never vector/graph (their
# absence from the top-N counts is healthy, see docstring)
expected = set()
if include_web:
expected.add("web")
if include_documents:
expected.add("documents")
if include_volatile:
expected.add("volatile")
# Normalize count keys to leg names (document/documents)
aliases = {"document": "documents"}
reported = {aliases.get(key, key) for key in response.source_counts}
missing = sorted(expected - reported)
if missing:
return (
"⚠️ *Coverage note: no results came from: "
f"{', '.join(missing)} (source unavailable or nothing found).*"
)
return ""
# ============================================================================ # ============================================================================
# HybridRAG Search # HybridRAG Search
# ============================================================================ # ============================================================================
@@ -76,13 +173,10 @@ async def hybrid_search(
# Add results # Add results
for i, result in enumerate(response.results, 1): for i, result in enumerate(response.results, 1):
source_icon = { source_keys = result.sources or [result.source]
"vector": "📄", source_icon = "".join(
"graph": "🔗", dict.fromkeys(SOURCE_ICONS.get(key, "") for key in source_keys)
"web": "🌐", )
"document": "📑",
"volatile": "",
}.get(result.source, "")
output_parts.append( output_parts.append(
f"{i}. {source_icon} **{result.title}** (score: {result.score:.2f})" f"{i}. {source_icon} **{result.title}** (score: {result.score:.2f})"
@@ -92,17 +186,30 @@ async def hybrid_search(
output_parts.append(f" {result.content[:300]}...") output_parts.append(f" {result.content[:300]}...")
output_parts.append("") output_parts.append("")
# Surface degraded coverage so outages are visible downstream
coverage_note = _coverage_note(
response,
include_web=include_web,
include_documents=include_documents,
include_volatile=include_volatile,
)
if coverage_note:
output_parts.append(coverage_note)
logger.info( logger.info(
"librarian_hybrid_search", "librarian_hybrid_search",
query=query, query=query,
result_count=len(response.results), result_count=len(response.results),
degraded=response.degraded,
source_counts=response.source_counts,
) )
return "\n".join(output_parts) return "\n".join(output_parts)
except Exception as e: except Exception as e:
logger.error("librarian_hybrid_search_error", error=str(e), query=query) logger.error("librarian_hybrid_search_error", error=str(e), query=query)
return f"Error searching: {str(e)}" _retry_if_transient(e, "The knowledge archive")
return "I was unable to search the knowledge archives; the search service did not respond properly."
# ============================================================================ # ============================================================================
@@ -139,18 +246,22 @@ async def search_wiki(
output_parts = [f"## Wiki Search: {query}\n"] output_parts = [f"## Wiki Search: {query}\n"]
for i, page in enumerate(results, 1): # No ordinal numbering: small models pass the list position to
output_parts.append(f"{i}. **{page.title}**") # get_wiki_page instead of the page ID unless the ID is the only
output_parts.append(f" Path: {page.path}") # number in sight.
for page in results:
output_parts.append(f"- **{page.title}** (page_id: {page.id})")
output_parts.append(f" Path: {page.path}")
if page.description: if page.description:
output_parts.append(f" {page.description}") output_parts.append(f" {page.description}")
output_parts.append("") output_parts.append("")
return "\n".join(output_parts) return "\n".join(output_parts)
except Exception as e: except Exception as e:
logger.error("librarian_wiki_search_error", error=str(e)) logger.error("librarian_wiki_search_error", error=str(e))
return f"Error searching wiki: {str(e)}" _retry_if_transient(e, "The wiki search")
return "I was unable to search the wiki at this time."
async def get_wiki_page( async def get_wiki_page(
@@ -193,7 +304,8 @@ async def get_wiki_page(
except Exception as e: except Exception as e:
logger.error("librarian_get_page_error", error=str(e), page_id=page_id) logger.error("librarian_get_page_error", error=str(e), page_id=page_id)
return f"Error getting page {page_id}: {str(e)}" _retry_if_transient(e, "The wiki")
return f"I was unable to retrieve wiki page {page_id}."
async def list_dossiers() -> str: async def list_dossiers() -> str:
@@ -227,7 +339,8 @@ async def list_dossiers() -> str:
except Exception as e: except Exception as e:
logger.error("librarian_list_dossiers_error", error=str(e)) logger.error("librarian_list_dossiers_error", error=str(e))
return f"Error listing dossiers: {str(e)}" _retry_if_transient(e, "The dossier index")
return "I was unable to retrieve the list of dossiers."
async def get_dossier_pages( async def get_dossier_pages(
@@ -268,7 +381,8 @@ async def get_dossier_pages(
except Exception as e: except Exception as e:
logger.error("librarian_get_dossier_error", error=str(e)) logger.error("librarian_get_dossier_error", error=str(e))
return f"Error getting dossier: {str(e)}" _retry_if_transient(e, "The dossier index")
return f"I was unable to retrieve the dossier '{dossier_name}'."
# ============================================================================ # ============================================================================
@@ -317,7 +431,8 @@ async def semantic_search(
except Exception as e: except Exception as e:
logger.error("librarian_semantic_search_error", error=str(e)) logger.error("librarian_semantic_search_error", error=str(e))
return f"Error in semantic search: {str(e)}" _retry_if_transient(e, "The semantic search")
return "I was unable to complete the semantic search."
# ============================================================================ # ============================================================================
@@ -370,7 +485,8 @@ async def explore_knowledge_graph(
except Exception as e: except Exception as e:
logger.error("librarian_explore_graph_error", error=str(e)) logger.error("librarian_explore_graph_error", error=str(e))
return f"Error exploring knowledge graph: {str(e)}" _retry_if_transient(e, "The knowledge graph")
return "I was unable to explore the knowledge graph."
async def find_related_entities( async def find_related_entities(
@@ -441,7 +557,8 @@ async def find_related_entities(
except Exception as e: except Exception as e:
logger.error("librarian_find_related_error", error=str(e)) logger.error("librarian_find_related_error", error=str(e))
return f"Error finding related entities: {str(e)}" _retry_if_transient(e, "The knowledge graph")
return f"I was unable to look up entities related to '{entity_name}'."
# ============================================================================ # ============================================================================
@@ -524,7 +641,8 @@ async def search_web(
except Exception as e: except Exception as e:
logger.error("librarian_web_search_error", error=str(e), query=query) logger.error("librarian_web_search_error", error=str(e), query=query)
return f"Error searching web: {str(e)}" _retry_if_transient(e, "The web search")
return "I was unable to search the web at this time."
async def read_url( async def read_url(
@@ -600,7 +718,8 @@ async def read_url(
except Exception as e: except Exception as e:
logger.error("librarian_read_url_error", error=str(e), url=url) logger.error("librarian_read_url_error", error=str(e), url=url)
return f"Error reading URL: {str(e)}" _retry_if_transient(e, "Content extraction")
return f"I was unable to read the page at {url}."
async def read_urls_batch( async def read_urls_batch(
@@ -635,7 +754,7 @@ async def read_urls_batch(
) )
output_parts = [ output_parts = [
f"## Batch Content Extraction", "## Batch Content Extraction",
f"*Extracted {response.successful}/{response.total_urls} URLs in {response.extraction_time_ms}ms*\n", f"*Extracted {response.successful}/{response.total_urls} URLs in {response.extraction_time_ms}ms*\n",
] ]
@@ -674,19 +793,23 @@ async def read_urls_batch(
except Exception as e: except Exception as e:
logger.error("librarian_read_urls_batch_error", error=str(e)) logger.error("librarian_read_urls_batch_error", error=str(e))
return f"Error reading URLs: {str(e)}" _retry_if_transient(e, "Content extraction")
return "I was unable to read the requested pages."
# ============================================================================ # ============================================================================
# Wiki Write Operations # Wiki Write Operations
# ============================================================================ # ============================================================================
CLEAR_TAGS_SENTINEL = "__CLEAR__"
async def update_wiki_page( async def update_wiki_page(
page_id: int, page_id: int,
content: str | None = None, content: str = "",
title: str | None = None, title: str = "",
tags: list[str] | None = None, tags: list[str] = [], # noqa: B006 - sentinel, never mutated
description: str | None = None, description: str = "",
) -> str: ) -> str:
""" """
Update an existing wiki page. Update an existing wiki page.
@@ -700,12 +823,17 @@ async def update_wiki_page(
- Updating tags to organize pages into dossiers - Updating tags to organize pages into dossiers
- Fixing descriptions or titles - Fixing descriptions or titles
Note: empty values are sentinels for "leave unchanged" (Ollama's
OpenAI-compatible API mishandles anyOf[X, null] parameter schemas).
Args: Args:
page_id: ID of the page to update (get from search_wiki results) page_id: ID of the page to update (get from search_wiki results)
content: New markdown content (optional - only if changing content) content: New markdown content (empty = leave unchanged)
title: New title (optional - only if renaming) title: New title (empty = leave unchanged)
tags: New tag list (optional - replaces existing tags) tags: New tag list, replaces existing tags (empty = leave unchanged).
description: New description (optional) To remove ALL tags from a page, pass exactly ["__CLEAR__"]
(an empty list means "leave unchanged", not "clear")
description: New description (empty = leave unchanged)
Returns: Returns:
Confirmation with updated page details Confirmation with updated page details
@@ -713,27 +841,34 @@ async def update_wiki_page(
Examples: Examples:
update_wiki_page(42, content="# Updated Content\\n\\nNew information here") update_wiki_page(42, content="# Updated Content\\n\\nNew information here")
update_wiki_page(42, tags=["projects", "devops"]) # Add to dossiers update_wiki_page(42, tags=["projects", "devops"]) # Add to dossiers
update_wiki_page(42, tags=["__CLEAR__"]) # Remove all tags
update_wiki_page(42, description="Updated description") update_wiki_page(42, description="Updated description")
""" """
# Empty list = leave unchanged; the explicit clear sentinel sends an
# empty tag list to the service, which replaces (clears) all tags.
clear_tags = tags == [CLEAR_TAGS_SENTINEL]
try: try:
async with LibraryDeskClient() as client: async with LibraryDeskClient() as client:
page = await client.update_wiki_page( page = await client.update_wiki_page(
page_id=page_id, page_id=page_id,
content=content, content=content if content else None,
title=title, title=title if title else None,
tags=tags, tags=[] if clear_tags else (tags if tags else None),
description=description, description=description if description else None,
) )
# Build update summary # Build update summary
updated_fields = [] updated_fields = []
if content is not None: if content:
updated_fields.append("content") updated_fields.append("content")
if title is not None: if title:
updated_fields.append("title") updated_fields.append("title")
if tags is not None: if clear_tags:
updated_fields.append("tags (cleared)")
elif tags:
updated_fields.append("tags") updated_fields.append("tags")
if description is not None: if description:
updated_fields.append("description") updated_fields.append("description")
output_parts = [ output_parts = [
@@ -757,7 +892,7 @@ async def update_wiki_page(
except Exception as e: except Exception as e:
logger.error("librarian_update_page_error", error=str(e), page_id=page_id) logger.error("librarian_update_page_error", error=str(e), page_id=page_id)
return f"Error updating page {page_id}: {str(e)}" return f"I was unable to update wiki page {page_id}."
async def create_wiki_page( async def create_wiki_page(
@@ -832,13 +967,13 @@ async def create_wiki_page(
except Exception as e: except Exception as e:
logger.error("librarian_create_page_error", error=str(e), title=title) logger.error("librarian_create_page_error", error=str(e), title=title)
return f"Error creating page: {str(e)}" return f"I was unable to create the page '{title}'."
async def smart_create_wiki_page( async def smart_create_wiki_page(
topic: str, topic: str,
tags: list[str], tags: list[str],
path: str | None = None, path: str = "",
include_web_research: bool = True, include_web_research: bool = True,
include_wiki_search: bool = True, include_wiki_search: bool = True,
) -> str: ) -> str:
@@ -860,7 +995,7 @@ async def smart_create_wiki_page(
Args: Args:
topic: The topic to research and create a page about topic: The topic to research and create a page about
tags: List of tags/dossiers for categorization tags: List of tags/dossiers for categorization
path: Optional custom path (auto-generated from topic if not provided) path: Optional custom path (empty = auto-generated from topic)
include_web_research: Whether to search the web (default: True) include_web_research: Whether to search the web (default: True)
include_wiki_search: Whether to search existing wiki (default: True) include_wiki_search: Whether to search existing wiki (default: True)
@@ -876,7 +1011,7 @@ async def smart_create_wiki_page(
response = await client.smart_create_wiki_page( response = await client.smart_create_wiki_page(
topic=topic, topic=topic,
tags=tags, tags=tags,
path=path, path=path if path else None,
include_web_research=include_web_research, include_web_research=include_web_research,
include_wiki_search=include_wiki_search, include_wiki_search=include_wiki_search,
) )
@@ -921,7 +1056,7 @@ async def smart_create_wiki_page(
except Exception as e: except Exception as e:
logger.error("librarian_smart_create_error", error=str(e), topic=topic) logger.error("librarian_smart_create_error", error=str(e), topic=topic)
return f"Error creating page about '{topic}': {str(e)}" return f"I was unable to create a page about '{topic}'."
# ============================================================================ # ============================================================================
+5 -189
View File
@@ -1,180 +1,11 @@
""" """
Agent communication protocol for multi-agent coordination. Agent error protocol.
Defines standardized request/response formats for communication between: Structured exceptions raised by expert agents (e.g. The Librarian) so
- Steward (request analysis) → Tatlock (coordination) callers - the delegation wrappers in src/agents/delegation.py - can
- Tatlock (coordination) → Expert agents (Librarian, Developer, etc.) report success=False and map failures to curated user-safe messages
while exception detail stays in the logs.
""" """
from enum import Enum
from typing import Any, Optional
from pydantic import BaseModel, Field
class DelegationReason(str, Enum):
"""Why a task is being delegated to an expert agent."""
DOMAIN_EXPERTISE = "domain_expertise" # Expert has specialized knowledge
TOOL_ACCESS = "tool_access" # Expert has required tools
RESOURCE_EFFICIENCY = "resource_efficiency" # Better handled by specialist
USER_PREFERENCE = "user_preference" # User requested specific agent
class TaskComplexity(str, Enum):
"""Complexity estimate for task execution."""
SIMPLE = "simple" # Single tool call, fast
MODERATE = "moderate" # Multiple steps, moderate time
COMPLEX = "complex" # Multi-agent, significant processing
class AgentRequest(BaseModel):
"""
Request to an expert agent.
Contains everything the agent needs to execute a task,
including context from the conversation and delegation intent.
"""
task: str = Field(
...,
description="Clear description of what the agent should do"
)
context: str = Field(
default="",
description="Relevant context from conversation history"
)
constraints: list[str] = Field(
default_factory=list,
description="Any constraints or requirements for the task"
)
delegation_reason: DelegationReason = Field(
default=DelegationReason.DOMAIN_EXPERTISE,
description="Why this task was delegated to this agent"
)
user_id: str = Field(
default="default",
description="User identifier for multi-tenant operations"
)
max_tokens: Optional[int] = Field(
default=None,
description="Optional token limit for response"
)
timeout_seconds: Optional[int] = Field(
default=60,
description="Maximum time for task completion"
)
class ToolCallRecord(BaseModel):
"""Record of a tool call made during execution."""
tool_name: str
arguments: dict[str, Any]
result: str
duration_ms: int
class AgentResponse(BaseModel):
"""
Response from an expert agent.
Contains the result, reasoning, and metadata about execution.
"""
success: bool = Field(
...,
description="Whether the task completed successfully"
)
result: str = Field(
...,
description="The main output/answer from the agent"
)
reasoning: str = Field(
default="",
description="Agent's reasoning process (for transparency)"
)
tool_calls: list[ToolCallRecord] = Field(
default_factory=list,
description="Tools called during execution"
)
confidence: float = Field(
default=1.0,
ge=0.0,
le=1.0,
description="Agent's confidence in the result (0.0-1.0)"
)
sources: list[str] = Field(
default_factory=list,
description="Sources or references used"
)
error_message: Optional[str] = Field(
default=None,
description="Error details if success=False"
)
duration_ms: int = Field(
default=0,
description="Total execution time in milliseconds"
)
class DelegationIntent(BaseModel):
"""
Intent to delegate a task to an expert agent.
Created by Tatlock when deciding to delegate, based on
Steward's recommendations.
"""
target_agent: str = Field(
...,
description="Name of the expert agent to delegate to"
)
task: str = Field(
...,
description="Task description for the agent"
)
reason: DelegationReason = Field(
default=DelegationReason.DOMAIN_EXPERTISE,
description="Why delegating to this agent"
)
expected_outcome: str = Field(
default="",
description="What we expect the agent to provide"
)
priority: int = Field(
default=1,
ge=1,
le=10,
description="Priority (1=highest, 10=lowest)"
)
depends_on: list[str] = Field(
default_factory=list,
description="Other delegation IDs this depends on (for sequencing)"
)
class CoordinationResult(BaseModel):
"""
Result of multi-agent coordination.
Aggregates results from multiple expert agents into
a single coherent response.
"""
final_response: str = Field(
...,
description="Synthesized response from all agents"
)
agent_responses: dict[str, AgentResponse] = Field(
default_factory=dict,
description="Individual responses keyed by agent name"
)
delegation_intents: list[DelegationIntent] = Field(
default_factory=list,
description="All delegations that were executed"
)
total_duration_ms: int = Field(
default=0,
description="Total coordination time"
)
agents_consulted: list[str] = Field(
default_factory=list,
description="Names of agents that contributed"
)
class AgentError(Exception): class AgentError(Exception):
@@ -184,18 +15,3 @@ class AgentError(Exception):
self.message = message self.message = message
self.agent_name = agent_name self.agent_name = agent_name
super().__init__(f"[{agent_name}] {message}") super().__init__(f"[{agent_name}] {message}")
class AgentTimeoutError(AgentError):
"""Agent execution timed out."""
pass
class AgentUnavailableError(AgentError):
"""Agent is not available or registered."""
pass
class DelegationError(AgentError):
"""Error during task delegation."""
pass
+28 -14
View File
@@ -11,7 +11,7 @@ Uses plain text output (not JSON) for reliability. Supports both Claude
import httpx import httpx
from typing import Optional from typing import Optional
from src.anthropic.model_selector import is_claude_available, get_model_info from src.anthropic.model_selector import get_model_info, is_claude_available, resolve_backend
from src.core.config import config from src.core.config import config
from src.core.household_registry import get_household_registry from src.core.household_registry import get_household_registry
from src.core.logging_config import get_logger from src.core.logging_config import get_logger
@@ -113,18 +113,19 @@ class StewardAgent:
def __init__(self): def __init__(self):
"""Initialize Steward with backend selection based on availability.""" """Initialize Steward with backend selection based on availability."""
# Ollama config (fallback) # Ollama config (primary)
self.ollama_host = str(config.OLLAMA_HOST).rstrip('/') self.ollama_host = str(config.OLLAMA_HOST).rstrip('/')
self.ollama_model = config.OLLAMA_DEFAULT_MODEL self.ollama_model = config.OLLAMA_DEFAULT_MODEL
# Claude config (preferred) # Claude config (fallback)
self.claude_model = config.ANTHROPIC_MODEL self.claude_model = config.ANTHROPIC_MODEL
self._anthropic_client = None self._anthropic_client = None
# Determine which backend to use # Determine which backend to use (Ollama-first, Claude when
self._use_claude = config.PREFER_CLOUD_BACKEND and is_claude_available() # preferred via config or when Ollama is down)
self._use_claude = resolve_backend() == "claude"
self.timeout = 30.0 # 30 second timeout for analysis self.timeout = float(config.STEWARD_TIMEOUT)
model_info = get_model_info() model_info = get_model_info()
logger.info( logger.info(
@@ -145,12 +146,12 @@ class StewardAgent:
"""Call Claude API directly for plain text generation.""" """Call Claude API directly for plain text generation."""
client = self._get_anthropic_client() client = self._get_anthropic_client()
# No temperature: rejected by Claude Sonnet 5+ (sampling params deprecated)
response = await client.messages.create( response = await client.messages.create(
model=self.claude_model, model=self.claude_model,
max_tokens=1024, max_tokens=1024,
system=system_prompt, system=system_prompt,
messages=[{"role": "user", "content": user_message}], messages=[{"role": "user", "content": user_message}],
temperature=0.3, # Lower = more consistent
) )
return response.content[0].text.strip() return response.content[0].text.strip()
@@ -227,20 +228,33 @@ class StewardAgent:
return analysis_text return analysis_text
except Exception as e: except Exception as e:
# If Claude fails, try Ollama as fallback # Mid-request fallback: retry on the other backend when possible
if self._use_claude: if self._use_claude:
logger.warning( logger.warning(
"steward_claude_fallback", "steward_claude_fallback",
error=str(e), error=str(e),
) )
analysis_text = await self._call_ollama(prompt) analysis_text = await self._call_ollama(prompt)
logger.debug( fallback_backend = "ollama_fallback"
"steward_analysis_received", elif is_claude_available():
backend="ollama_fallback", logger.warning(
text_preview=analysis_text[:150], "steward_ollama_fallback",
error=str(e),
) )
return analysis_text analysis_text = await self._call_claude(
raise system_prompt="You are the Steward of the household, advising the Butler (Tatlock) on which capabilities to use. Be concise and specific.",
user_message=prompt,
)
fallback_backend = "claude_fallback"
else:
raise
logger.debug(
"steward_analysis_received",
backend=fallback_backend,
text_preview=analysis_text[:150],
)
return analysis_text
# Global Steward instance # Global Steward instance
+18 -2
View File
@@ -129,6 +129,22 @@ or
""" """
# Tool-phase prompt for orchestrate_tool_calls(). The butler personality prompt
# suppresses tool calling on small local models (gemma4 reasons about the tool,
# then answers from memory with wrong arithmetic), so the orchestration phase
# uses a terse operator prompt; synthesize_from_results() applies the persona.
TATLOCK_ORCHESTRATION_PROMPT = """You are the tool-execution phase of Tatlock, \
a butler assistant. Your only job is to gather accurate results by calling the \
provided tools.
- ALWAYS use tools for the task - never answer from memory and never do mental math.
- Mathematics: call the calculate tool, even for trivial arithmetic.
- Dates and times: call the date/time tools, never guess.
- When the instructions say DELEGATE to an agent, call the matching delegate_to_* tool.
- After the tool results arrive, reply with a one-line factual summary of the results. \
A later step writes the polished reply, so do not add personality."""
class TatlockAgent(AgentInterface): class TatlockAgent(AgentInterface):
""" """
Tatlock - The Butler agent using PydanticAI with Ollama. Tatlock - The Butler agent using PydanticAI with Ollama.
@@ -654,10 +670,10 @@ class TatlockAgent(AgentInterface):
# Create a fresh agent instance with scoped tools only # Create a fresh agent instance with scoped tools only
model = get_model() model = get_model()
# Create agent with scoped tools # Create agent with scoped tools, using the tool-phase prompt
scoped_agent = Agent( scoped_agent = Agent(
model, model,
system_prompt=TATLOCK_SYSTEM_PROMPT, system_prompt=TATLOCK_ORCHESTRATION_PROMPT,
tools=scoped_tools, tools=scoped_tools,
) )
+8 -1
View File
@@ -1,19 +1,26 @@
""" """
Anthropic/Claude integration module. Anthropic/Claude integration module.
Provides model selection with automatic fallback between Claude and Ollama. Provides model selection with Ollama as primary backend and Claude
as the cloud fallback.
""" """
from src.anthropic.model_selector import ( from src.anthropic.model_selector import (
check_claude_health, check_claude_health,
check_ollama_health,
get_model, get_model,
get_tool_choice_settings, get_tool_choice_settings,
is_claude_available, is_claude_available,
is_ollama_available,
resolve_backend,
) )
__all__ = [ __all__ = [
"check_claude_health", "check_claude_health",
"check_ollama_health",
"get_model", "get_model",
"get_tool_choice_settings", "get_tool_choice_settings",
"is_claude_available", "is_claude_available",
"is_ollama_available",
"resolve_backend",
] ]
+155 -31
View File
@@ -1,23 +1,84 @@
""" """
Model selector for Claude/Ollama backend switching. Model selector for Ollama/Claude backend switching.
Provides automatic model selection with Claude as preferred backend Provides automatic model selection with Ollama as the primary local backend
and Ollama as offline fallback. and Claude as the cloud fallback. Claude is used when PREFER_CLOUD_BACKEND
is enabled, or automatically when Ollama is unavailable at startup.
The Anthropic SDK is imported lazily so a missing or broken `anthropic`
package degrades to Ollama-only operation instead of crashing the app.
""" """
from typing import Union from __future__ import annotations
from pydantic_ai.models.anthropic import AnthropicModel from typing import TYPE_CHECKING
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.anthropic import AnthropicProvider import httpx
from src.core.config import config from src.core.config import config
from src.core.logging_config import get_logger from src.core.logging_config import get_logger
if TYPE_CHECKING:
from pydantic_ai.models.anthropic import AnthropicModel
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.settings import ModelSettings
logger = get_logger(__name__) logger = get_logger(__name__)
# Cached health check result (set once at startup) # Cached health check results (set once at startup)
_claude_available: bool | None = None _claude_available: bool | None = None
_ollama_available: bool | None = None
async def check_ollama_health() -> bool:
"""
Check if the Ollama server is reachable and has the configured model.
This should be called once at application startup.
The result is cached in `_ollama_available`.
Returns:
True if Ollama is reachable and OLLAMA_DEFAULT_MODEL is pulled.
"""
global _ollama_available
host = str(config.OLLAMA_HOST).rstrip("/")
model = config.OLLAMA_DEFAULT_MODEL
try:
async with httpx.AsyncClient(timeout=5.0) as client:
response = await client.get(f"{host}/api/tags")
response.raise_for_status()
names = [m.get("name", "") for m in response.json().get("models", [])]
if model in names or f"{model}:latest" in names:
_ollama_available = True
logger.info(
"ollama_health_check_passed",
host=host,
model=model,
)
return True
_ollama_available = False
logger.warning(
"ollama_health_check_failed",
reason="model_not_pulled",
host=host,
model=model,
hint=f"run `ollama pull {model}`",
)
return False
except Exception as e:
_ollama_available = False
logger.warning(
"ollama_health_check_failed",
reason="server_unreachable",
host=host,
error=str(e),
)
return False
async def check_claude_health() -> bool: async def check_claude_health() -> bool:
@@ -85,46 +146,93 @@ def is_claude_available() -> bool:
return _claude_available is True return _claude_available is True
def get_model(prefer_cloud: bool | None = None) -> Union[AnthropicModel, OpenAIChatModel]: def is_ollama_available() -> bool:
"""
Check if Ollama is available (from cached health check result).
Returns:
False only if the startup health check confirmed Ollama is down.
Unknown (check not run yet) counts as available so that contexts
without lifespan events keep the local-first behavior.
"""
return _ollama_available is not False
def resolve_backend(prefer_cloud: bool | None = None) -> str:
"""
Resolve which backend should serve requests.
Ollama is the primary backend. Claude is used when explicitly
preferred via PREFER_CLOUD_BACKEND, or as automatic fallback
when the startup health check found Ollama down.
Args:
prefer_cloud: Override config.PREFER_CLOUD_BACKEND for this call.
Returns:
"claude" or "ollama".
"""
use_cloud = prefer_cloud if prefer_cloud is not None else config.PREFER_CLOUD_BACKEND
if use_cloud and is_claude_available():
return "claude"
if not is_ollama_available() and is_claude_available():
logger.warning(
"backend_fallback_to_claude",
reason="ollama_unavailable",
)
return "claude"
return "ollama"
def get_model(prefer_cloud: bool | None = None) -> AnthropicModel | OpenAIChatModel:
""" """
Get the best available model. Get the best available model.
Returns Claude if available and preferred, otherwise Ollama. Returns Ollama unless Claude is preferred (or Ollama is down).
Args: Args:
prefer_cloud: Override config.PREFER_CLOUD_BACKEND for this call. prefer_cloud: Override config.PREFER_CLOUD_BACKEND for this call.
If None, uses the config value. If None, uses the config value.
Returns: Returns:
PydanticAI model instance (AnthropicModel or OpenAIChatModel). PydanticAI model instance (OpenAIChatModel or AnthropicModel).
Example: Example:
>>> model = get_model() >>> model = get_model()
>>> agent = Agent(model, system_prompt="...") >>> agent = Agent(model, system_prompt="...")
""" """
# Determine preference if resolve_backend(prefer_cloud) == "claude":
use_cloud = prefer_cloud if prefer_cloud is not None else config.PREFER_CLOUD_BACKEND try:
from pydantic_ai.models.anthropic import AnthropicModel
from pydantic_ai.providers.anthropic import AnthropicProvider
# Use Claude if available and preferred logger.debug(
if use_cloud and is_claude_available(): "model_selected",
logger.debug( backend="claude",
"model_selected", model=config.ANTHROPIC_MODEL,
backend="claude", )
model=config.ANTHROPIC_MODEL, return AnthropicModel(
) model_name=config.ANTHROPIC_MODEL,
return AnthropicModel( provider=AnthropicProvider(api_key=config.ANTHROPIC_API_KEY),
model_name=config.ANTHROPIC_MODEL, )
provider=AnthropicProvider(api_key=config.ANTHROPIC_API_KEY), except ImportError as e:
) logger.error(
"claude_backend_import_failed",
error=str(e),
hint="anthropic package missing or incompatible; using Ollama",
)
from pydantic_ai.models.openai import OpenAIChatModel
# Fall back to Ollama
from src.ollama.provider import get_ollama_provider from src.ollama.provider import get_ollama_provider
logger.debug( logger.debug(
"model_selected", "model_selected",
backend="ollama", backend="ollama",
model=config.OLLAMA_DEFAULT_MODEL, model=config.OLLAMA_DEFAULT_MODEL,
reason="fallback" if use_cloud else "preferred_local",
) )
return OpenAIChatModel( return OpenAIChatModel(
model_name=config.OLLAMA_DEFAULT_MODEL, model_name=config.OLLAMA_DEFAULT_MODEL,
@@ -132,7 +240,7 @@ def get_model(prefer_cloud: bool | None = None) -> Union[AnthropicModel, OpenAIC
) )
def get_tool_choice_settings() -> 'ModelSettings': def get_tool_choice_settings() -> ModelSettings:
""" """
Get model_settings for forcing tool calls on the first request. Get model_settings for forcing tool calls on the first request.
@@ -141,7 +249,7 @@ def get_tool_choice_settings() -> 'ModelSettings':
""" """
from pydantic_ai.settings import ModelSettings from pydantic_ai.settings import ModelSettings
if is_claude_available() and config.PREFER_CLOUD_BACKEND: if resolve_backend() == "claude":
# PydanticAI's Anthropic model handles tool_choice internally # PydanticAI's Anthropic model handles tool_choice internally
return ModelSettings() return ModelSettings()
else: else:
@@ -149,6 +257,20 @@ def get_tool_choice_settings() -> 'ModelSettings':
return ModelSettings(extra_body={"tool_choice": "required"}) return ModelSettings(extra_body={"tool_choice": "required"})
def get_sampling_settings(temperature: float) -> ModelSettings:
"""
Get model_settings with a sampling temperature where the backend allows it.
Ollama accepts a temperature; Claude Sonnet 5+ rejects sampling
parameters, so the Claude backend gets empty settings.
"""
from pydantic_ai.settings import ModelSettings
if resolve_backend() == "claude":
return ModelSettings()
return ModelSettings(temperature=temperature)
def get_model_info() -> dict: def get_model_info() -> dict:
""" """
Get information about the current model configuration. Get information about the current model configuration.
@@ -158,12 +280,14 @@ def get_model_info() -> dict:
Returns: Returns:
Dict with backend, model name, and availability info. Dict with backend, model name, and availability info.
""" """
use_cloud = config.PREFER_CLOUD_BACKEND and is_claude_available() backend = resolve_backend()
return { return {
"backend": "claude" if use_cloud else "ollama", "backend": backend,
"model": config.ANTHROPIC_MODEL if use_cloud else config.OLLAMA_DEFAULT_MODEL, "model": config.ANTHROPIC_MODEL if backend == "claude" else config.OLLAMA_DEFAULT_MODEL,
"claude_available": is_claude_available(), "claude_available": is_claude_available(),
"claude_configured": bool(config.ANTHROPIC_API_KEY), "claude_configured": bool(config.ANTHROPIC_API_KEY),
"ollama_available": is_ollama_available(),
"ollama_model": config.OLLAMA_DEFAULT_MODEL,
"prefer_cloud": config.PREFER_CLOUD_BACKEND, "prefer_cloud": config.PREFER_CLOUD_BACKEND,
} }
+91 -27
View File
@@ -6,9 +6,17 @@ from enum import Enum
from functools import lru_cache from functools import lru_cache
from pathlib import Path from pathlib import Path
from pydantic import Field, HttpUrl from pydantic import Field, HttpUrl, model_validator
from pydantic_settings import BaseSettings, SettingsConfigDict from pydantic_settings import BaseSettings, SettingsConfigDict
# Tenant isolation constants (see docs: tenant-based isolation, no separate
# test infrastructure). The production tenant owns real data in the shared
# services (Qdrant/Neo4j/Wiki.js/Redis); everything non-production must run
# under the reserved test tenant or an explicit test_-prefixed namespace.
PRODUCTION_TENANT = "jpmschweitzer"
TEST_TENANT = "llm_tester"
TEST_TENANT_PREFIX = "test_"
def _get_version_from_pyproject() -> str: def _get_version_from_pyproject() -> str:
""" """
@@ -42,7 +50,7 @@ class Environment(str, Enum):
class Config(BaseSettings): class Config(BaseSettings):
""" """
Global application configuration. Global application configuration.
Loads from environment variables and .env file. Loads from environment variables and .env file.
Domain-specific configs should be in their respective modules. Domain-specific configs should be in their respective modules.
""" """
@@ -52,45 +60,49 @@ class Config(BaseSettings):
case_sensitive=True, case_sensitive=True,
extra="ignore", extra="ignore",
) )
# Application # Application
APP_NAME: str = "OpenAI-Compatible API" APP_NAME: str = "OpenAI-Compatible API"
APP_VERSION: str = Field(default_factory=_get_version_from_pyproject) APP_VERSION: str = Field(default_factory=_get_version_from_pyproject)
ENVIRONMENT: Environment = Environment.DEVELOPMENT ENVIRONMENT: Environment = Environment.DEVELOPMENT
DEBUG: bool = Field(default=False, description="Debug mode") DEBUG: bool = Field(default=False, description="Debug mode")
# API Configuration # API Configuration
API_HOST: str = Field(default="0.0.0.0", description="API host") API_HOST: str = Field(default="0.0.0.0", description="API host")
API_PORT: int = Field(default=8000, description="API port") API_PORT: int = Field(default=8000, description="API port")
API_PREFIX: str = Field(default="/v1", description="API route prefix") API_PREFIX: str = Field(default="/v1", description="API route prefix")
# Anthropic Configuration (Claude - preferred backend) # Anthropic Configuration (Claude - cloud fallback)
ANTHROPIC_API_KEY: str | None = Field( ANTHROPIC_API_KEY: str | None = Field(
default=None, default=None,
description="Anthropic API key for Claude access" description="Anthropic API key for the Claude fallback backend"
) )
ANTHROPIC_MODEL: str = Field( ANTHROPIC_MODEL: str = Field(
default="claude-sonnet-4-20250514", default="claude-sonnet-5",
description="Claude model to use" description="Claude model for the fallback backend"
) )
PREFER_CLOUD_BACKEND: bool = Field( PREFER_CLOUD_BACKEND: bool = Field(
default=True, default=False,
description="Prefer Claude over Ollama when available" description="Prefer Claude over Ollama (default: local-first)"
) )
# Ollama Configuration (local fallback) # Ollama Configuration (local - primary backend)
OLLAMA_HOST: HttpUrl = Field( OLLAMA_HOST: HttpUrl = Field(
default="http://localhost:11434", default="http://localhost:11434",
description="Ollama server URL" description="Ollama server URL"
) )
OLLAMA_DEFAULT_MODEL: str = Field( OLLAMA_DEFAULT_MODEL: str = Field(
default="mistral-nemo:latest", default="gemma4:e2b",
description="Default Ollama model" description="Default Ollama model"
) )
OLLAMA_TIMEOUT: int = Field( OLLAMA_TIMEOUT: int = Field(
default=120, default=120,
description="Ollama request timeout in seconds" description="Ollama request timeout in seconds"
) )
STEWARD_TIMEOUT: int = Field(
default=60,
description="Steward analysis timeout in seconds (gemma4 needs ~35s warm)"
)
STREAM_TIMEOUT: int = Field( STREAM_TIMEOUT: int = Field(
default=20, default=20,
description="Timeout for each streaming turn in seconds" description="Timeout for each streaming turn in seconds"
@@ -98,8 +110,8 @@ class Config(BaseSettings):
# SearXNG Configuration # SearXNG Configuration
SEARXNG_HOST: HttpUrl = Field( SEARXNG_HOST: HttpUrl = Field(
default="http://localhost:8087", default="http://searxng:8080",
description="SearXNG server URL" description="SearXNG server URL (container name; internal port 8080)"
) )
SEARXNG_TIMEOUT: int = Field( SEARXNG_TIMEOUT: int = Field(
default=30, default=30,
@@ -121,9 +133,13 @@ class Config(BaseSettings):
) )
# Library-Desk Configuration (The Librarian backend) # Library-Desk Configuration (The Librarian backend)
LIBRARIAN_TIMEOUT: int = Field(
default=180,
description="Total time budget for a librarian delegation in seconds"
)
LIBRARY_DESK_HOST: HttpUrl = Field( LIBRARY_DESK_HOST: HttpUrl = Field(
default="http://localhost:8089", default="http://library-desk:8089",
description="Library-Desk API URL" description="Library-Desk API URL (container name; internal port 8089)"
) )
LIBRARY_DESK_API_KEY: str = Field( LIBRARY_DESK_API_KEY: str = Field(
default="", default="",
@@ -136,8 +152,8 @@ class Config(BaseSettings):
# Core-API Configuration (The Housekeeper backend) # Core-API Configuration (The Housekeeper backend)
CORE_API_HOST: HttpUrl = Field( CORE_API_HOST: HttpUrl = Field(
default="http://localhost:8090", default="http://core-api:8083",
description="Core-API URL for Home Assistant integration" description="Core-API URL for Home Assistant integration (container name; internal port 8083)"
) )
CORE_API_KEY: str = Field( CORE_API_KEY: str = Field(
default="", default="",
@@ -199,6 +215,38 @@ class Config(BaseSettings):
CORS_ALLOW_METHODS: list[str] = ["*"] CORS_ALLOW_METHODS: list[str] = ["*"]
CORS_ALLOW_HEADERS: list[str] = ["*"] CORS_ALLOW_HEADERS: list[str] = ["*"]
@model_validator(mode="after")
def _refuse_production_tenant_outside_production(self) -> "Config":
"""
Refuse startup when a non-production environment is explicitly
configured with the production tenant.
This is the hard stop of the tenant isolation guard: a dev/test
instance must never be able to read or write the production
tenant's data in the shared services.
The comparison is on the sanitized form: namespaces are derived
through sanitize_user_id(), so variants like "JPMSchweitzer" or
"jpmschweitzer." collide with the production namespaces and are
refused just as loudly.
"""
from src.core.multi_tenancy import sanitize_user_id
if (
self.ENVIRONMENT != Environment.PRODUCTION
and self.DEFAULT_USER is not None
and sanitize_user_id(self.DEFAULT_USER)
== sanitize_user_id(PRODUCTION_TENANT)
):
raise ValueError(
f"Refusing to start: ENVIRONMENT={self.ENVIRONMENT.value} is "
f"explicitly configured with the production tenant "
f"'{PRODUCTION_TENANT}'. Non-production environments must use "
f"'{TEST_TENANT}' or a '{TEST_TENANT_PREFIX}'-prefixed tenant. "
f"Unset DEFAULT_USER or set ENVIRONMENT=production."
)
return self
@property @property
def redis_memory_url(self) -> str: def redis_memory_url(self) -> str:
"""Construct Redis connection URL for memory cache.""" """Construct Redis connection URL for memory cache."""
@@ -239,23 +287,39 @@ class Config(BaseSettings):
@property @property
def effective_default_user(self) -> str: def effective_default_user(self) -> str:
""" """
Get effective default user, auto-determining from environment if not set. Get effective default user (tenant), enforcing tenant isolation.
- development/testing: llm_tester (isolated test scope) - production: DEFAULT_USER if set, else the production tenant
- production: jpmschweitzer (real user) - development/testing: FORCED to the reserved test tenant
("llm_tester") - the only accepted overrides are the test tenant
itself or a "test_"-prefixed namespace. Any other DEFAULT_USER
value is treated as misconfiguration and ignored.
""" """
if self.DEFAULT_USER is not None:
return self.DEFAULT_USER
if self.ENVIRONMENT == Environment.PRODUCTION: if self.ENVIRONMENT == Environment.PRODUCTION:
return "jpmschweitzer" return self.DEFAULT_USER or PRODUCTION_TENANT
return "llm_tester"
if self.DEFAULT_USER is not None and (
self.DEFAULT_USER == TEST_TENANT
or self.DEFAULT_USER.startswith(TEST_TENANT_PREFIX)
):
return self.DEFAULT_USER
return TEST_TENANT
@property
def tenant_forced(self) -> bool:
"""Whether the tenant guard overrode a misconfigured DEFAULT_USER."""
return (
self.ENVIRONMENT != Environment.PRODUCTION
and self.DEFAULT_USER is not None
and self.effective_default_user != self.DEFAULT_USER
)
@lru_cache @lru_cache
def get_config() -> Config: def get_config() -> Config:
""" """
Get cached configuration instance. Get cached configuration instance.
Uses lru_cache to ensure config is loaded once and reused. Uses lru_cache to ensure config is loaded once and reused.
""" """
return Config() return Config()
+38 -1
View File
@@ -41,6 +41,41 @@ current_conversation: ContextVar[str | None] = ContextVar(
) )
def apply_tenant_guard(user: str) -> str:
"""
Enforce tenant isolation at request-context resolution.
In non-production environments the production tenant must never be
the effective user - a request that explicitly asks for it is forced
to the reserved test tenant instead (with a loud log line).
Comparison happens on the *sanitized* form of the user: every local
namespace (Qdrant collection, Redis key) is derived through
sanitize_user_id(), so any raw variant that collides with the
production tenant after sanitization ("JPMSchweitzer",
"jpmschweitzer.", " jpmschweitzer", ...) would otherwise resolve to
the production namespaces. Those variants are forced too.
"""
# Import here to avoid circular dependency
from src.core.config import PRODUCTION_TENANT, TEST_TENANT, Environment, config
from src.core.multi_tenancy import sanitize_user_id
if (
config.ENVIRONMENT != Environment.PRODUCTION
and sanitize_user_id(user) == sanitize_user_id(PRODUCTION_TENANT)
):
from src.core.logging_config import get_logger
get_logger(__name__).warning(
"tenant_guard_forced",
environment=config.ENVIRONMENT.value,
requested_tenant=user,
forced_tenant=TEST_TENANT,
)
return TEST_TENANT
return user
def get_user() -> str: def get_user() -> str:
""" """
Get current user from request context. Get current user from request context.
@@ -48,6 +83,8 @@ def get_user() -> str:
Returns: Returns:
User identifier for the current request. User identifier for the current request.
Falls back to environment-aware default if not set. Falls back to environment-aware default if not set.
In non-production environments the production tenant is never
returned - the tenant guard forces the reserved test tenant.
Example: Example:
user = get_user() # "llm_tester" (dev) or "jpmschweitzer" (prod) user = get_user() # "llm_tester" (dev) or "jpmschweitzer" (prod)
@@ -55,7 +92,7 @@ def get_user() -> str:
user = current_user.get() user = current_user.get()
if user == _USER_NOT_SET: if user == _USER_NOT_SET:
return get_default_user() return get_default_user()
return user return apply_tenant_guard(user)
def get_conversation_id() -> str | None: def get_conversation_id() -> str | None:
-59
View File
@@ -273,65 +273,6 @@ class HouseholdRegistry:
return tools return tools
def get_streaming_delegation_tools(self, names: list[str]) -> list[Any]:
"""
Get streaming delegation wrapper tools for specified capabilities.
Similar to get_delegation_tools() but returns streaming wrappers
that yield butler-perspective think messages during execution.
These wrappers emit think slugs like:
- "Allow me to consult the archives, sir."
- "The Librarian has compiled the relevant findings."
Args:
names: List of member names to include
Returns:
List of streaming delegation wrappers and/or raw tools
Example:
>>> tools = registry.get_streaming_delegation_tools(["librarian"])
>>> async for chunk in tools[0](task="Search for Docker"):
... print(chunk) # Yields think messages then result
"""
from src.agents.delegation import STREAMING_DELEGATION_WRAPPERS
tools = []
for name in names:
member = self._members.get(name)
if not member:
logger.warning(
"household_member_not_found",
requested_name=name,
available_names=list(self._members.keys()),
)
continue
# Check if this member has a streaming delegation wrapper
if name in STREAMING_DELEGATION_WRAPPERS and member.agent is not None:
tools.append(STREAMING_DELEGATION_WRAPPERS[name])
logger.debug(
"streaming_delegation_wrapper_added",
member=name,
)
else:
# No agent = direct tools (e.g., tatlock_core)
tools.extend(member.tools)
logger.debug(
"raw_tools_added",
member=name,
tool_count=len(member.tools),
)
logger.info(
"streaming_delegation_tools_created",
requested_members=names,
total_tools=len(tools),
)
return tools
def list_members(self) -> list[str]: def list_members(self) -> list[str]:
""" """
List all registered member names. List all registered member names.
+38 -3
View File
@@ -9,13 +9,43 @@ from src.agents.biographer import register_biographer
from src.agents.housekeeper import register_housekeeper from src.agents.housekeeper import register_housekeeper
from src.agents.librarian import register_librarian from src.agents.librarian import register_librarian
from src.agents.tatlock_core import TATLOCK_CORE_CAPABILITY, tatlock_core_tools from src.agents.tatlock_core import TATLOCK_CORE_CAPABILITY, tatlock_core_tools
from src.anthropic.model_selector import check_claude_health, get_model_info from src.anthropic.model_selector import (
check_claude_health,
check_ollama_health,
get_model_info,
)
from src.core.config import Environment, config
from src.core.household_registry import get_household_registry from src.core.household_registry import get_household_registry
from src.core.logging_config import get_logger from src.core.logging_config import get_logger
logger = get_logger(__name__) logger = get_logger(__name__)
def log_tenant_guard() -> None:
"""
Emit one loud startup log line stating the effective tenant.
In non-production environments the tenant guard forces the reserved
test tenant regardless of DEFAULT_USER misconfiguration - this line
makes that override visible at startup.
"""
if config.ENVIRONMENT == Environment.PRODUCTION:
logger.info(
"tenant_guard_production",
environment=config.ENVIRONMENT.value,
tenant=config.effective_default_user,
)
return
logger.warning(
"tenant_guard_active",
environment=config.ENVIRONMENT.value,
forced_tenant=config.effective_default_user,
default_user_overridden=config.tenant_forced,
configured_default_user=config.DEFAULT_USER,
)
def register_household_members(): def register_household_members():
""" """
Register all household members with the registry. Register all household members with the registry.
@@ -87,7 +117,7 @@ async def initialize_application():
Initialize the application. Initialize the application.
Performs all startup tasks: Performs all startup tasks:
1. Check Claude API health (for backend selection) 1. Check Ollama (primary) and Claude (fallback) health for backend selection
2. Register household members 2. Register household members
3. (Future) Initialize connections 3. (Future) Initialize connections
@@ -95,13 +125,18 @@ async def initialize_application():
""" """
logger.info("application_initialization_starting") logger.info("application_initialization_starting")
# Check Claude API health for backend selection # Tenant isolation guard: state the effective tenant loudly
log_tenant_guard()
# Check backend health: Ollama is primary, Claude is the fallback
await check_ollama_health()
await check_claude_health() await check_claude_health()
model_info = get_model_info() model_info = get_model_info()
logger.info( logger.info(
"model_backend_configured", "model_backend_configured",
backend=model_info["backend"], backend=model_info["backend"],
model=model_info["model"], model=model_info["model"],
ollama_available=model_info["ollama_available"],
claude_available=model_info["claude_available"], claude_available=model_info["claude_available"],
) )
+21 -10
View File
@@ -40,14 +40,21 @@ class TatlockOllamaProvider(OllamaProvider):
# Override the client with our sanitized version # Override the client with our sanitized version
self._openai_client = _SanitizedAsyncOpenAI(base_url=base_url) self._openai_client = _SanitizedAsyncOpenAI(base_url=base_url)
logger.debug("tatlock_ollama_provider_created", base_url=base_url) logger.debug(
"tatlock_ollama_provider_created",
base_url=base_url,
timeout=config.OLLAMA_TIMEOUT,
)
class _SanitizedAsyncOpenAI(AsyncOpenAI): class _SanitizedAsyncOpenAI(AsyncOpenAI):
"""AsyncOpenAI client that sanitizes messages before sending.""" """AsyncOpenAI client that sanitizes messages before sending."""
def __init__(self, **kwargs: Any): def __init__(self, **kwargs: Any):
# Ollama doesn't need an API key # Ollama doesn't need an API key. Cap each LLM call at the
# configured Ollama timeout instead of the SDK default (~600s),
# so one stuck request cannot eat the whole delegation budget.
kwargs.setdefault("timeout", float(config.OLLAMA_TIMEOUT))
super().__init__(api_key="ollama", **kwargs) super().__init__(api_key="ollama", **kwargs)
@property @property
@@ -106,14 +113,18 @@ def _sanitize_messages(messages: list[dict[str, Any]]) -> list[dict[str, Any]]:
for msg in messages: for msg in messages:
msg_copy = dict(msg) msg_copy = dict(msg)
# Fix null content in assistant messages with tool calls # Fix null content in ANY message: Ollama rejects content: null with
if msg_copy.get("role") == "assistant": # "invalid message content type: <nil>". The tool-call-only assistant
if msg_copy.get("content") is None and msg_copy.get("tool_calls"): # case is the common one, but gemma thinking-only turns produce
msg_copy["content"] = "" # assistant messages with null content and NO tool_calls, which
logger.debug( # previously slipped through and 400'd the whole agent run.
"sanitized_null_content", if "content" in msg_copy and msg_copy.get("content") is None:
tool_call_count=len(msg_copy["tool_calls"]), msg_copy["content"] = ""
) logger.debug(
"sanitized_null_content",
role=msg_copy.get("role"),
tool_call_count=len(msg_copy.get("tool_calls") or []),
)
sanitized.append(msg_copy) sanitized.append(msg_copy)
+91 -50
View File
@@ -6,32 +6,32 @@ Tracks conversation history for analytics and future vector memory.
Integrates with Steward preprocessing for Phase 2 two-tier architecture. Integrates with Steward preprocessing for Phase 2 two-tier architecture.
""" """
import time import asyncio
import re
import secrets import secrets
from typing import AsyncGenerator import time
from collections.abc import AsyncGenerator
from src.agents.delegation import build_delegation_context, get_think_message
from src.agents.registry import ModelRegistry from src.agents.registry import ModelRegistry
from src.agents.steward.schemas import StewardRecommendation
from src.core.context import current_conversation, current_user, get_default_user
from src.core.logging_config import get_logger
from src.core.preprocessing import preprocess_request
from src.core.tool_tracking import ToolCallTracker
from src.core.tracing import SpanType, end_trace, start_span, start_trace
from src.responses.context import ContextWindow
from src.responses.history import ConversationHistory
from src.responses.schemas import ( from src.responses.schemas import (
FunctionCallOutputItem,
MessageOutputItem,
OutputTextContent,
ReasoningOutputItem,
Response, Response,
ResponseRequest, ResponseRequest,
ResponseUsage, ResponseUsage,
MessageOutputItem,
ReasoningOutputItem,
FunctionCallOutputItem,
OutputTextContent,
) )
from src.responses.streaming import StreamingCoordinator from src.responses.streaming import StreamingCoordinator
from src.responses.history import ConversationHistory
from src.responses.context import ContextWindow
from src.core.preprocessing import preprocess_request
from src.core.tool_tracking import ToolCallTracker
from src.core.logging_config import get_logger
from src.core.tracing import start_trace, end_trace, start_span, SpanType
from src.core.context import current_user, current_conversation, get_default_user
from src.agents.steward.schemas import StewardRecommendation
import re
import asyncio
logger = get_logger(__name__) logger = get_logger(__name__)
@@ -63,7 +63,8 @@ async def _execute_single_delegation(
agent_name: str, agent_name: str,
task: str, task: str,
tracker: "ToolCallTracker", tracker: "ToolCallTracker",
) -> tuple[str, str]: context: str = "",
) -> tuple[str, str, bool]:
""" """
Execute a single delegation to an agent. Execute a single delegation to an agent.
@@ -71,36 +72,38 @@ async def _execute_single_delegation(
agent_name: Name of agent (biographer, librarian, housekeeper) agent_name: Name of agent (biographer, librarian, housekeeper)
task: Task description task: Task description
tracker: Tool call tracker tracker: Tool call tracker
context: Trimmed conversation context for the expert
Returns: Returns:
tuple: (agent_name, result_summary) tuple: (agent_name, result_summary, success). On failure the
result summary is a curated user-safe sentence.
""" """
import time import time
start_time = time.time() start_time = time.time()
if agent_name == "biographer": if agent_name == "biographer":
from src.agents.delegation import delegate_to_biographer from src.agents.delegation import delegate_to_biographer
result = await delegate_to_biographer(task=task) result = await delegate_to_biographer(task=task, context=context)
duration = time.time() - start_time duration = time.time() - start_time
await tracker.track_call("delegate_to_biographer", duration) await tracker.track_call("delegate_to_biographer", duration)
return (agent_name, result.output) return (agent_name, result.output, result.success)
elif agent_name == "librarian": elif agent_name == "librarian":
from src.agents.delegation import delegate_to_librarian from src.agents.delegation import delegate_to_librarian
result = await delegate_to_librarian(task=task) result = await delegate_to_librarian(task=task, context=context)
duration = time.time() - start_time duration = time.time() - start_time
await tracker.track_call("delegate_to_librarian", duration) await tracker.track_call("delegate_to_librarian", duration)
return (agent_name, result.output) return (agent_name, result.output, result.success)
elif agent_name == "housekeeper": elif agent_name == "housekeeper":
from src.agents.delegation import delegate_to_housekeeper from src.agents.delegation import delegate_to_housekeeper
result = await delegate_to_housekeeper(task=task) result = await delegate_to_housekeeper(task=task, context=context)
duration = time.time() - start_time duration = time.time() - start_time
await tracker.track_call("delegate_to_housekeeper", duration) await tracker.track_call("delegate_to_housekeeper", duration)
return (agent_name, result.output) return (agent_name, result.output, result.success)
else: else:
return (agent_name, f"Unknown agent: {agent_name}") return (agent_name, f"Unknown agent: {agent_name}", False)
async def _handle_text_delegation( async def _handle_text_delegation(
@@ -175,13 +178,39 @@ async def _handle_text_delegation(
] ]
results = await asyncio.gather(*tasks, return_exceptions=True) results = await asyncio.gather(*tasks, return_exceptions=True)
# Combine results # asyncio.gather returns exactly one item per task; a length
# mismatch would mean results are attributed to the wrong
# agent, so fail loudly instead of mispairing silently.
if len(results) != len(matches):
logger.error(
"delegation_result_count_mismatch",
expected=len(matches),
got=len(results),
conversation_id=conversation_id,
)
return (
"I apologize, sir. I was unable to complete the "
"requested delegations."
)
# Combine results (failures carry curated user-safe sentences)
summaries = [] summaries = []
for agent_name, result in results: for (agent, task), item in zip(matches, results, strict=True):
if isinstance(result, Exception): agent_name = agent.lower()
summaries.append(f"**{agent_name}**: Error - {result}") if isinstance(item, BaseException):
logger.error(
"delegation_failed",
agent=agent_name,
error=str(item),
conversation_id=conversation_id,
)
summaries.append(
f"**{agent_name}**: "
f"{get_think_message(agent_name, task, 'error')}"
)
else: else:
summaries.append(f"**{agent_name}**: {result}") _, output, _ = item
summaries.append(f"**{agent_name}**: {output}")
return "\n\n".join(summaries) return "\n\n".join(summaries)
@@ -197,20 +226,19 @@ async def _handle_text_delegation(
conversation_id=conversation_id, conversation_id=conversation_id,
) )
try: try:
_, result = await _execute_single_delegation( _, output, _ = await _execute_single_delegation(
agent_name, task, tracker agent_name, task, tracker
) )
summaries.append(result) summaries.append(output)
except Exception as e: except Exception as e:
logger.error( logger.error(
"delegation_failed", "delegation_failed",
agent=agent_name, agent=agent_name,
error=str(e), error=str(e),
conversation_id=conversation_id, conversation_id=conversation_id,
exc_info=True,
) )
summaries.append( summaries.append(get_think_message(agent_name, task, "error"))
f"I apologize, sir. Delegation to {agent_name} failed: {e}"
)
return "\n\n".join(summaries) return "\n\n".join(summaries)
@@ -219,8 +247,9 @@ async def _handle_text_delegation(
"text_delegation_failed", "text_delegation_failed",
error=str(e), error=str(e),
conversation_id=conversation_id, conversation_id=conversation_id,
exc_info=True,
) )
return f"I apologize, sir. I encountered an error processing delegations: {e}" return "I apologize, sir. I was unable to complete the requested delegations."
async def _direct_delegation( async def _direct_delegation(
@@ -254,13 +283,14 @@ async def _direct_delegation(
results = [] results = []
for agent in recommendation.recommended_capabilities: for agent in recommendation.recommended_capabilities:
try: try:
agent_name, result = await _execute_single_delegation( agent_name, result, success = await _execute_single_delegation(
agent, user_message, tracker agent, user_message, tracker
) )
results.append(result) results.append(result)
logger.info( logger.info(
"direct_delegation_complete", "direct_delegation_complete",
agent=agent_name, agent=agent_name,
success=success,
result_preview=result[:100] if result else "empty", result_preview=result[:100] if result else "empty",
conversation_id=conversation_id, conversation_id=conversation_id,
) )
@@ -270,8 +300,9 @@ async def _direct_delegation(
agent=agent, agent=agent,
error=str(e), error=str(e),
conversation_id=conversation_id, conversation_id=conversation_id,
exc_info=True,
) )
results.append(f"I apologize, sir. Delegation to {agent} failed: {e}") results.append(get_think_message(agent, user_message, "error"))
return "\n\n".join(results) if results else "I apologize, sir. No delegation results available." return "\n\n".join(results) if results else "I apologize, sir. No delegation results available."
@@ -281,6 +312,7 @@ async def _direct_delegation_with_results(
recommendation: "StewardRecommendation", recommendation: "StewardRecommendation",
tracker: "ToolCallTracker", tracker: "ToolCallTracker",
conversation_id: str, conversation_id: str,
conversation_history: list | None = None,
) -> dict: ) -> dict:
""" """
Directly delegate to expert agents and return structured results. Directly delegate to expert agents and return structured results.
@@ -294,6 +326,7 @@ async def _direct_delegation_with_results(
recommendation: Steward's recommendation recommendation: Steward's recommendation
tracker: Tool call tracker tracker: Tool call tracker
conversation_id: Conversation ID conversation_id: Conversation ID
conversation_history: Prior turns, trimmed into expert context
Returns: Returns:
dict: Orchestration results with expert_results, tool_outputs, etc. dict: Orchestration results with expert_results, tool_outputs, etc.
@@ -306,18 +339,21 @@ async def _direct_delegation_with_results(
expert_results = {} expert_results = {}
tools_called = [] tools_called = []
context = build_delegation_context(conversation_history)
for agent in recommendation.recommended_capabilities: for agent in recommendation.recommended_capabilities:
try: try:
agent_name, result = await _execute_single_delegation( agent_name, result, success = await _execute_single_delegation(
agent, user_message, tracker agent, user_message, tracker, context=context
) )
expert_results[agent_name] = result expert_results[agent_name] = result
tools_called.append(f"delegate_to_{agent_name}") if success:
tools_called.append(f"delegate_to_{agent_name}")
logger.info( logger.info(
"direct_delegation_result", "direct_delegation_result",
agent=agent_name, agent=agent_name,
success=success,
result_preview=result[:100] if result else "empty", result_preview=result[:100] if result else "empty",
conversation_id=conversation_id, conversation_id=conversation_id,
) )
@@ -327,8 +363,9 @@ async def _direct_delegation_with_results(
agent=agent, agent=agent,
error=str(e), error=str(e),
conversation_id=conversation_id, conversation_id=conversation_id,
exc_info=True,
) )
expert_results[agent] = f"Error: {e}" expert_results[agent] = get_think_message(agent, user_message, "error")
return { return {
"tools_called": tools_called, "tools_called": tools_called,
@@ -439,7 +476,7 @@ async def create_response(request: ResponseRequest) -> Response:
user_input = _extract_user_input(request.input) user_input = _extract_user_input(request.input)
# Start trace # Start trace
trace = start_trace( start_trace(
conversation_id=conversation_id, conversation_id=conversation_id,
user=effective_user, user=effective_user,
request={ request={
@@ -451,7 +488,7 @@ async def create_response(request: ResponseRequest) -> Response:
) )
# Start service span # Start service span
service_span = start_span( start_span(
"create_response", "create_response",
SpanType.ROUTER, SpanType.ROUTER,
metadata={"model": request.model, "user": effective_user}, metadata={"model": request.model, "user": effective_user},
@@ -509,7 +546,7 @@ async def create_response(request: ResponseRequest) -> Response:
return response return response
except Exception as e: except Exception:
end_trace(status="error") end_trace(status="error")
raise raise
@@ -550,7 +587,7 @@ async def create_response_with_steward(request: ResponseRequest) -> Response:
user_input = _extract_user_input(request.input) user_input = _extract_user_input(request.input)
# Start trace # Start trace
trace = start_trace( start_trace(
conversation_id=conversation_id, conversation_id=conversation_id,
user=effective_user, user=effective_user,
request={ request={
@@ -562,7 +599,7 @@ async def create_response_with_steward(request: ResponseRequest) -> Response:
) )
# Start service span # Start service span
service_span = start_span( start_span(
"create_response_with_steward", "create_response_with_steward",
SpanType.ROUTER, SpanType.ROUTER,
metadata={"model": request.model, "user": effective_user}, metadata={"model": request.model, "user": effective_user},
@@ -617,7 +654,11 @@ async def create_response_with_steward(request: ResponseRequest) -> Response:
if delegation_only: if delegation_only:
# Direct delegation path - collect results then synthesize # Direct delegation path - collect results then synthesize
orchestration_results = await _direct_delegation_with_results( orchestration_results = await _direct_delegation_with_results(
effective_query, enriched.recommendation, tracker, conversation_id effective_query,
enriched.recommendation,
tracker,
conversation_id,
conversation_history=conversation_history,
) )
else: else:
# Phase 1: Orchestrate tool calls # Phase 1: Orchestrate tool calls
@@ -706,7 +747,7 @@ async def create_response_with_steward(request: ResponseRequest) -> Response:
return response return response
except Exception as e: except Exception:
end_trace(status="error") end_trace(status="error")
raise raise
+95 -58
View File
@@ -8,14 +8,22 @@ Handles Server-Sent Events (SSE) streaming with proper event types:
- response.done - response.done
""" """
from enum import Enum
from typing import Literal, AsyncGenerator
import json
import time import time
from collections.abc import AsyncGenerator
from enum import Enum
from typing import TYPE_CHECKING, Literal
from src.agents.registry import ModelRegistry
from src.core.logging_config import get_logger
from src.core.models import CustomBaseModel from src.core.models import CustomBaseModel
from src.responses.schemas import Response from src.responses.schemas import Response
from src.agents.registry import ModelRegistry
if TYPE_CHECKING:
from src.agents.steward.schemas import StewardRecommendation
from src.core.tool_tracking import ToolCallTracker
from src.responses.schemas import ResponseRequest
logger = get_logger(__name__)
# ============================================================================ # ============================================================================
@@ -131,18 +139,17 @@ class StreamingCoordinator:
Yields: Yields:
StreamEvent: Stream of SSE events StreamEvent: Stream of SSE events
""" """
from src.responses.service import ( import asyncio
_calculate_usage,
generate_id, from src.agents.tatlock import TatlockAgent
_conversation_history,
_direct_delegation_with_results,
)
from src.core.preprocessing import preprocess_request from src.core.preprocessing import preprocess_request
from src.core.tool_tracking import ToolCallTracker from src.core.tool_tracking import ToolCallTracker
from src.responses.schemas import MessageOutputItem, ReasoningOutputItem, OutputTextContent from src.responses.schemas import MessageOutputItem, OutputTextContent
from src.agents.tatlock import TatlockAgent from src.responses.service import (
from src.agents.delegation import get_think_message, STREAMING_DELEGATION_WRAPPERS _calculate_usage,
import asyncio _conversation_history,
generate_id,
)
output_items = [] output_items = []
@@ -182,20 +189,18 @@ class StreamingCoordinator:
tatlock = TatlockAgent() tatlock = TatlockAgent()
if delegation_only: if delegation_only:
# Direct delegation path with streaming think slugs # Direct delegation path - think slugs stream in real time,
orchestration_results = await self._stream_direct_delegation( # BEFORE and after each expert runs (not after the fact)
orchestration_results: dict = {}
async for event in self._stream_direct_delegation(
user_message=user_message, user_message=user_message,
recommendation=enriched.recommendation, recommendation=enriched.recommendation,
tracker=tracker, tracker=tracker,
conversation_id=conversation_id, conversation_id=conversation_id,
) conversation_history=conversation_history,
results=orchestration_results,
# Stream think slugs that were collected during delegation ):
# Each think message is complete, so we signal done after each yield event
for think_msg in orchestration_results.get("think_messages", []):
yield ReasoningSummaryDelta(delta=think_msg)
yield ReasoningSummaryDone()
await asyncio.sleep(0.05)
else: else:
# Phase 1: Orchestrate tool calls # Phase 1: Orchestrate tool calls
@@ -265,47 +270,66 @@ class StreamingCoordinator:
recommendation: "StewardRecommendation", # type: ignore recommendation: "StewardRecommendation", # type: ignore
tracker: "ToolCallTracker", # type: ignore tracker: "ToolCallTracker", # type: ignore
conversation_id: str, conversation_id: str,
) -> dict: conversation_history: list | None = None,
results: dict | None = None,
) -> AsyncGenerator[StreamEvent, None]:
""" """
Execute direct delegation with streaming think messages. Execute direct delegation, streaming think messages in real time.
Collects think messages as delegations execute for streaming to client. An async generator: the "start" think message for each expert is
yielded BEFORE its research runs (so the user sees 'Allow me to
consult the archives, sir.' while waiting), and the success/error
message right after it finishes.
Args: Args:
user_message: User's request user_message: User's request
recommendation: Steward's recommendation recommendation: Steward's recommendation
tracker: Tool call tracker tracker: Tool call tracker
conversation_id: Conversation ID conversation_id: Conversation ID
conversation_history: Prior turns, trimmed into expert context
results: Mutable dict populated with orchestration results
(expert_results, tools_called, think_messages, ...)
Returns: Yields:
dict: Orchestration results with think_messages list StreamEvent: Reasoning summary events as delegation progresses
""" """
import time as time_module
from src.agents.delegation import ( from src.agents.delegation import (
get_think_message, build_delegation_context,
delegate_to_librarian,
delegate_to_biographer, delegate_to_biographer,
delegate_to_housekeeper, delegate_to_housekeeper,
delegate_to_librarian,
get_think_message,
) )
import time as time_module
expert_results = {} expert_results = {}
tools_called = [] tools_called = []
think_messages = [] think_messages = []
context = build_delegation_context(conversation_history)
for agent in recommendation.recommended_capabilities: for agent in recommendation.recommended_capabilities:
# Emit start think message # Emit start think message BEFORE the expert runs
start_msg = get_think_message(agent, user_message, "start") start_msg = get_think_message(agent, user_message, "start")
think_messages.append(start_msg + "\n") think_messages.append(start_msg + "\n")
yield ReasoningSummaryDelta(delta=start_msg + "\n")
yield ReasoningSummaryDone()
start_time = time_module.time() start_time = time_module.time()
try: try:
# Execute delegation # Execute delegation
if agent == "librarian": if agent == "librarian":
result = await delegate_to_librarian(task=user_message) result = await delegate_to_librarian(
task=user_message, context=context
)
elif agent == "biographer": elif agent == "biographer":
result = await delegate_to_biographer(task=user_message) result = await delegate_to_biographer(
task=user_message, context=context
)
elif agent == "housekeeper": elif agent == "housekeeper":
result = await delegate_to_housekeeper(task=user_message) result = await delegate_to_housekeeper(
task=user_message, context=context
)
else: else:
result = None result = None
@@ -316,27 +340,39 @@ class StreamingCoordinator:
expert_results[agent] = result.output expert_results[agent] = result.output
tools_called.append(f"delegate_to_{agent}") tools_called.append(f"delegate_to_{agent}")
# Emit success think message # Emit success think message
success_msg = get_think_message(agent, user_message, "success") phase_msg = get_think_message(agent, user_message, "success")
think_messages.append(success_msg + "\n")
else: else:
error_msg = result.error if result else "Unknown error" # Failed delegations carry a curated user-safe sentence
expert_results[agent] = f"Error: {error_msg}" # in output; exception detail is already in the logs.
# Emit error think message phase_msg = get_think_message(agent, user_message, "error")
error_think = get_think_message(agent, user_message, "error") if result and result.output:
think_messages.append(error_think + "\n") expert_results[agent] = result.output
else:
expert_results[agent] = phase_msg
except Exception as e: except Exception as e:
expert_results[agent] = f"Error: {e}" logger.error(
error_think = get_think_message(agent, user_message, "error") "direct_delegation_stream_error",
think_messages.append(error_think + "\n") agent=agent,
error=str(e),
conversation_id=conversation_id,
exc_info=True,
)
phase_msg = get_think_message(agent, user_message, "error")
expert_results[agent] = phase_msg
return { think_messages.append(phase_msg + "\n")
"tools_called": tools_called, yield ReasoningSummaryDelta(delta=phase_msg + "\n")
"expert_results": expert_results, yield ReasoningSummaryDone()
"tool_outputs": {},
"raw_output": "", if results is not None:
"think_messages": think_messages, results.update({
} "tools_called": tools_called,
"expert_results": expert_results,
"tool_outputs": {},
"raw_output": "",
"think_messages": think_messages,
})
async def stream_response( async def stream_response(
self, self,
@@ -361,9 +397,10 @@ class StreamingCoordinator:
event: response.done event: response.done
data: {"response": {...}} data: {"response": {...}}
""" """
from src.responses.service import _calculate_usage, generate_id
import asyncio import asyncio
from src.responses.service import _calculate_usage, generate_id
output_items = [] output_items = []
last_message_text = "" # Track last streamed message text to compute deltas last_message_text = "" # Track last streamed message text to compute deltas
@@ -488,10 +525,10 @@ class StreamingCoordinator:
def _convert_output_items(self, items: list) -> list: def _convert_output_items(self, items: list) -> list:
"""Convert agent OutputItem objects to schema OutputItem objects.""" """Convert agent OutputItem objects to schema OutputItem objects."""
from src.responses.schemas import ( from src.responses.schemas import (
MessageOutputItem,
ReasoningOutputItem,
FunctionCallOutputItem, FunctionCallOutputItem,
MessageOutputItem,
OutputTextContent, OutputTextContent,
ReasoningOutputItem,
) )
converted = [] converted = []
@@ -521,9 +558,9 @@ class StreamingCoordinator:
def _create_error_event(self, error: Exception) -> ErrorEvent: def _create_error_event(self, error: Exception) -> ErrorEvent:
"""Create error event from exception.""" """Create error event from exception."""
from src.core.exceptions import ( from src.core.exceptions import (
RateLimitError,
ContextLengthError,
AppException, AppException,
ContextLengthError,
RateLimitError,
) )
if isinstance(error, RateLimitError): if isinstance(error, RateLimitError):
@@ -0,0 +1,268 @@
{
"query": "home server infrastructure",
"keywords": {
"core_keywords": [
"home",
"server",
"infrastructure"
],
"entities": [],
"synonyms": {},
"expansions": {}
},
"results": [
{
"source_type": "wiki",
"title": "Tower of Joy - AI Butler System",
"content": "",
"url": null,
"page_id": 146,
"page_path": "users/jpmschweitzer/projects/tower-of-joy",
"paperless_id": null,
"rrf_score": 0.01639344262295082,
"final_rank": 1,
"sources": [
"graph"
],
"related_dossiers": [
{
"page_id": 161,
"title": "Library Desk - Knowledge Management",
"path": "users/jpmschweitzer/projects/tower-of-joy/applications/library-desk",
"tag": "ai",
"shared_entities": 13
},
{
"page_id": 161,
"title": "Library Desk - Knowledge Management",
"path": "users/jpmschweitzer/projects/tower-of-joy/applications/library-desk",
"tag": "dossier:tatlock",
"shared_entities": 13
},
{
"page_id": 161,
"title": "Library Desk - Knowledge Management",
"path": "users/jpmschweitzer/projects/tower-of-joy/applications/library-desk",
"tag": "applications",
"shared_entities": 13
},
{
"page_id": 167,
"title": "Qdrant - Vector Database",
"path": "users/jpmschweitzer/projects/tower-of-joy/applications/qdrant",
"tag": "vector",
"shared_entities": 12
},
{
"page_id": 167,
"title": "Qdrant - Vector Database",
"path": "users/jpmschweitzer/projects/tower-of-joy/applications/qdrant",
"tag": "dossier:tatlock",
"shared_entities": 12
}
],
"metadata": {
"entity_matches": 3,
"matched_entities": [
"Infrastructure Services",
"Infrastructure Layer\nThe"
],
"engine": null
}
},
{
"source_type": "web",
"title": "What do I need to start a home server ? Can I go in almost blind",
"content": "You don't need industrial grade hardware to be a server. You may get better reliability and management options from that, but they can all run\u00a0...",
"url": "https://www.reddit.com/r/HomeServer/comments/1rx7udl/what_do_i_need_to_start_a_home_server_can_i_go_in/",
"page_id": null,
"page_path": null,
"paperless_id": null,
"rrf_score": 0.01639344262295082,
"final_rank": 2,
"sources": [
"web"
],
"related_dossiers": [],
"metadata": {
"entity_matches": null,
"matched_entities": null,
"engine": "startpage"
}
},
{
"source_type": "wiki",
"title": "PostgreSQL Shared - Database Server",
"content": "",
"url": null,
"page_id": 148,
"page_path": "users/jpmschweitzer/projects/tower-of-joy/infrastructure/postgres",
"paperless_id": null,
"rrf_score": 0.016129032258064516,
"final_rank": 3,
"sources": [
"graph"
],
"related_dossiers": [
{
"page_id": 149,
"title": "Redis Shared - Cache and Session Store",
"path": "users/jpmschweitzer/projects/tower-of-joy/infrastructure/redis",
"tag": "infrastructure",
"shared_entities": 5
},
{
"page_id": 149,
"title": "Redis Shared - Cache and Session Store",
"path": "users/jpmschweitzer/projects/tower-of-joy/infrastructure/redis",
"tag": "dossier:tatlock",
"shared_entities": 5
},
{
"page_id": 149,
"title": "Redis Shared - Cache and Session Store",
"path": "users/jpmschweitzer/projects/tower-of-joy/infrastructure/redis",
"tag": "cache",
"shared_entities": 5
},
{
"page_id": 105,
"title": "Google Cloud Platform",
"path": "users/jpmschweitzer/technology/cloud-platforms/gcp",
"tag": "technology",
"shared_entities": 4
},
{
"page_id": 105,
"title": "Google Cloud Platform",
"path": "users/jpmschweitzer/technology/cloud-platforms/gcp",
"tag": "cloud-platforms",
"shared_entities": 4
}
],
"metadata": {
"entity_matches": 1,
"matched_entities": [
"Database Server"
],
"engine": null
}
},
{
"source_type": "web",
"title": "25+ Must-Have Home Server Services for 2025 (Ultimate Guide)",
"content": "I\u2019ve been running a home server setup for years now, and it\u2019s been an incredible journey of discovery, learning, and practical benefits.\nIf you\u2019re considering setting up a home server or looking to expand your existing home lab, you\u2019re in the right place.\nIn this comprehensive guide, I\u2019ll walk you through the essential services that can transform your home server from a simple file storage system into a powerful, versatile hub that enhances your digital life.\nFrom media streaming to home automation, security, productivity, and more \u2013 we\u2019ll cover it all.\nWhether you\u2019re a seasoned self-hosting veteran or just taking your first steps into home server territory, this guide will help you discover new possibilities and build a setup that perfectly suits your needs.\nFoundation Services: The Building Blocks\nBefore diving into specific applications, let\u2019s cover the fundamental services that form the backbone of any robust home server setup.\nThese core components provide the infrastructure for everything else to run smoothly.\nHypervisors & Virtualization Platforms\nHypervisors allow you to run multiple virtual machines on a single physical server, making them crucial for efficient resource utilization.\nProxmox VE: The Homelab Virtualization King\nProxmox is my top recommendation for home server virtualization.\nThis open-source solution combines KVM virtualization with LXC containers, a powerful web interface, and built-in features like clustering, backups, and storage management.\nProxmox gives you the ability to run both full virtual machines and lightweight containers on the same hardware.\nIt\u2019s been extremely reliable in my setup, and its active community provides excellent support.\nI\u2019ve been using Proxmox for about three years, after switching from ESXi.\nThe transition was straightforward, and I\u2019ve found it much more suitable for home lab use.\nYou can create clusters with multiple nodes, run HA setups, and even use distributed storage with Ceph.\nTrueNAS SCALE: Storage with Ad...",
"url": "https://hostbor.com/25-must-have-home-server-services/",
"page_id": null,
"page_path": null,
"paperless_id": null,
"rrf_score": 0.016129032258064516,
"final_rank": 4,
"sources": [
"web"
],
"related_dossiers": [],
"metadata": {
"entity_matches": null,
"matched_entities": null,
"engine": "duckduckgo"
}
},
{
"source_type": "wiki",
"title": "Bazzite",
"content": "",
"url": null,
"page_id": 108,
"page_path": "users/jpmschweitzer/technology/linux_distributions/bazzite",
"paperless_id": null,
"rrf_score": 0.015873015873015872,
"final_rank": 5,
"sources": [
"graph"
],
"related_dossiers": [
{
"page_id": 110,
"title": "Zorin OS",
"path": "users/jpmschweitzer/technology/linux_distributions/zorin_os",
"tag": "technology",
"shared_entities": 22
},
{
"page_id": 110,
"title": "Zorin OS",
"path": "users/jpmschweitzer/technology/linux_distributions/zorin_os",
"tag": "linux_distributions",
"shared_entities": 22
},
{
"page_id": 110,
"title": "Zorin OS",
"path": "users/jpmschweitzer/technology/linux_distributions/zorin_os",
"tag": "zorin_os",
"shared_entities": 22
},
{
"page_id": 109,
"title": "Linux",
"path": "users/jpmschweitzer/technology/linux",
"tag": "technology",
"shared_entities": 21
},
{
"page_id": 109,
"title": "Linux",
"path": "users/jpmschweitzer/technology/linux",
"tag": "linux",
"shared_entities": 21
}
],
"metadata": {
"entity_matches": 1,
"matched_entities": [
"Display Server"
],
"engine": null
}
}
],
"context": "1. [WIKI] Tower of Joy - AI Butler System\n (no content)...\n Related research: ai, dossier:tatlock, applications\n\n2. [WEB] What do I need to start a home server ? Can I go in almost blind\n You don't need industrial grade hardware to be a server. You may get better reliability and management options from that, but they can all run\u00a0......\n\n3. [WIKI] PostgreSQL Shared - Database Server\n (no content)...\n Related research: infrastructure, dossier:tatlock, cache\n\n4. [WEB] 25+ Must-Have Home Server Services for 2025 (Ultimate Guide)\n I\u2019ve been running a home server setup for years now, and it\u2019s been an incredible journey of discovery, learning, and practical benefits.\nIf you\u2019re considering setting up a home server or looking to expand your existing home lab, you\u2019re in the right place.\nIn this comprehensive guide, I\u2019ll walk you t...\n\n5. [WIKI] Bazzite\n (no content)...\n Related research: technology, linux_distributions, zorin_os",
"source_counts": {
"graph": 3,
"web": 2
},
"total_results": 5,
"timing": {
"query_enhancement_ms": 30.002593994140625,
"vector_ms": 105.46708106994629,
"graph_ms": 21.07977867126465,
"web_ms": 1577.7764320373535,
"volatile_ms": 0.0,
"document_ms": 1.8155574798583984,
"fusion_ms": 0.1761913299560547,
"enrichment_ms": 14.33563232421875,
"reranking_ms": 0.0002384185791015625,
"persistence_ms": 33.80393981933594,
"total_ms": 1624.767780303955
},
"config_used": {
"vector_limit": 3,
"graph_limit": 3,
"web_limit": 2,
"volatile_limit": 1,
"document_limit": 2,
"enable_vector": true,
"enable_graph": true,
"enable_web": true,
"enable_volatile": false,
"enable_documents": true,
"enable_reranking": false,
"enable_enrichment": true,
"final_result_count": 6,
"rrf_k": 60,
"volatile_threshold": 0.8,
"document_threshold": 0.6
},
"search_id": "01062bc7-ca65-4d9a-a210-e1a8f44b93c1"
}
@@ -0,0 +1,39 @@
"""
Tests for structured failure behavior of The Librarian entry point.
run_librarian must raise AgentError on failure instead of returning
error text as if it were research output, and the raised error must
not leak exception detail (internal URLs etc.).
"""
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from src.agents.librarian.agent import run_librarian
from src.agents.protocol import AgentError
@pytest.mark.unit
class TestRunLibrarianFailures:
"""run_librarian raises structured errors instead of returning text."""
@pytest.mark.asyncio
async def test_run_librarian_raises_agent_error(self):
"""Failures raise AgentError rather than returning error prose."""
mock_agent = MagicMock()
mock_agent.run = AsyncMock(
side_effect=RuntimeError("Connection refused to http://internal:8089")
)
with patch(
"src.agents.librarian.agent.get_librarian_agent",
return_value=mock_agent,
):
with pytest.raises(AgentError) as exc_info:
await run_librarian(task="Find Docker docs")
assert exc_info.value.agent_name == "librarian"
# Exception detail stays in logs only
assert "internal" not in str(exc_info.value)
assert "Connection refused" not in str(exc_info.value)
+184 -8
View File
@@ -2,22 +2,23 @@
Tests for the Library-Desk HTTP client. Tests for the Library-Desk HTTP client.
""" """
import pytest
from unittest.mock import AsyncMock, MagicMock, patch from unittest.mock import AsyncMock, MagicMock, patch
import httpx import httpx
import pytest
from src.agents.librarian.client import ( from src.agents.librarian.client import (
LibraryDeskClient, Dossier,
EntityLinking,
GraphNode,
HybridRAGResponse, HybridRAGResponse,
HybridSearchResult, HybridSearchResult,
LibraryDeskClient,
ResearchSummary,
SmartCreateResponse,
VectorSearchResult,
WikiPage, WikiPage,
WikiSearchResult, WikiSearchResult,
VectorSearchResult,
GraphNode,
Dossier,
SmartCreateResponse,
ResearchSummary,
EntityLinking,
) )
@@ -549,6 +550,181 @@ class TestSmartCreateWikiPage:
assert result.research_summary.web_results == 0 assert result.research_summary.web_results == 0
# All tenant-scoped client methods with minimal call kwargs. Used to sweep
# the explicit-user contract: every library-desk request must carry a
# non-empty user (the service is removing its server-side default).
TENANT_SCOPED_METHODS = [
("hybrid_search", {"query": "q"}),
("search_wiki", {"query": "q"}),
("get_wiki_page", {"page_id": 1}),
("list_wiki_pages", {}),
("create_wiki_page", {"title": "t", "path": "/p", "content": "c"}),
("update_wiki_page", {"page_id": 1, "content": "c"}),
("smart_create_wiki_page", {"topic": "t", "tags": ["x"]}),
("list_dossiers", {}),
("semantic_search", {"query": "q"}),
("query_graph", {"cypher_query": "MATCH (n) RETURN n"}),
("list_graph_nodes", {}),
("get_graph_node", {"node_id": "n1"}),
("search_web", {"query": "q"}),
("extract_content", {"url": "http://example.com"}),
("extract_content_batch", {"urls": ["http://example.com"]}),
]
# One permissive response body that satisfies every method's parser
# (extra keys are ignored by the pydantic models).
UNIVERSAL_RESPONSE = {
"id": 1,
"path": "/p",
"title": "T",
"results": [],
"pages": [],
"dossiers": [],
"records": [],
"nodes": [],
"keywords": [],
"page": {"id": 1, "path": "/p", "title": "T"},
"result": {"url": "http://example.com", "success": True},
}
@pytest.mark.unit
class TestExplicitUserContract:
"""Every library-desk request sends a non-empty user explicitly."""
def _wire_client(self):
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = UNIVERSAL_RESPONSE
mock_response.raise_for_status = MagicMock()
mock_httpx = AsyncMock(spec=httpx.AsyncClient)
mock_httpx.get.return_value = mock_response
mock_httpx.post.return_value = mock_response
mock_httpx.put.return_value = mock_response
client = LibraryDeskClient(base_url="http://test:8089", api_key="k")
client._client = mock_httpx
return client, mock_httpx
def _sent_user(self, mock_httpx) -> str:
"""Extract the user sent on the single outgoing request."""
calls = (
mock_httpx.get.call_args_list
+ mock_httpx.post.call_args_list
+ mock_httpx.put.call_args_list
)
assert len(calls) == 1, "expected exactly one outgoing request"
kwargs = calls[0].kwargs
params = kwargs.get("params") or {}
payload = kwargs.get("json") or {}
return params.get("user") or payload.get("user") or ""
@pytest.mark.asyncio
@pytest.mark.parametrize("method_name,kwargs", TENANT_SCOPED_METHODS)
async def test_user_from_context_is_sent_on_the_wire(
self, method_name, kwargs
):
"""With no explicit user, the context user is resolved and sent."""
client, mock_httpx = self._wire_client()
with patch(
"src.agents.librarian.client.get_user", return_value="llm_tester"
):
await getattr(client, method_name)(**kwargs)
assert self._sent_user(mock_httpx) == "llm_tester"
@pytest.mark.asyncio
@pytest.mark.parametrize("method_name,kwargs", TENANT_SCOPED_METHODS)
async def test_explicit_user_is_sent_on_the_wire(self, method_name, kwargs):
"""An explicitly passed user is sent unchanged."""
client, mock_httpx = self._wire_client()
await getattr(client, method_name)(user="test_phase_b", **kwargs)
assert self._sent_user(mock_httpx) == "test_phase_b"
@pytest.mark.asyncio
@pytest.mark.parametrize("method_name,kwargs", TENANT_SCOPED_METHODS)
async def test_empty_context_user_fails_before_any_request(
self, method_name, kwargs
):
"""An empty resolved user raises before any bytes hit the wire."""
client, mock_httpx = self._wire_client()
with patch("src.agents.librarian.client.get_user", return_value=""):
with pytest.raises(ValueError, match="non-empty user"):
await getattr(client, method_name)(**kwargs)
mock_httpx.get.assert_not_called()
mock_httpx.post.assert_not_called()
mock_httpx.put.assert_not_called()
@pytest.mark.asyncio
async def test_explicit_whitespace_user_is_rejected(self):
"""A whitespace-only explicit user is rejected."""
client, mock_httpx = self._wire_client()
with pytest.raises(ValueError, match="non-empty user"):
await client.hybrid_search("q", user=" ")
mock_httpx.post.assert_not_called()
@pytest.mark.asyncio
async def test_explicit_padded_user_is_stripped_on_the_wire(self):
"""Padded explicit users are stripped, not sent verbatim."""
client, mock_httpx = self._wire_client()
await client.hybrid_search("q", user=" llm_tester ")
assert self._sent_user(mock_httpx) == "llm_tester"
@pytest.mark.asyncio
@pytest.mark.parametrize(
"explicit_user",
["jpmschweitzer", "JPMSchweitzer", "jpmschweitzer.", " jpmschweitzer"],
)
@pytest.mark.parametrize("method_name,kwargs", TENANT_SCOPED_METHODS)
async def test_explicit_production_tenant_is_guarded_in_dev(
self, monkeypatch, method_name, kwargs, explicit_user
):
"""
An explicit production-tenant argument (or a sanitization-collision
variant) never reaches library-desk from a non-production
environment - the client applies the same tenant guard as
context resolution.
"""
from src.core import config as config_module
from src.core.config import Environment
monkeypatch.setattr(
config_module.config, "ENVIRONMENT", Environment.DEVELOPMENT
)
client, mock_httpx = self._wire_client()
await getattr(client, method_name)(user=explicit_user, **kwargs)
assert self._sent_user(mock_httpx) == "llm_tester"
@pytest.mark.asyncio
async def test_explicit_production_tenant_passes_through_in_prod(
self, monkeypatch
):
"""In production the production tenant is sent unchanged."""
from src.core import config as config_module
from src.core.config import Environment
monkeypatch.setattr(
config_module.config, "ENVIRONMENT", Environment.PRODUCTION
)
client, mock_httpx = self._wire_client()
await client.hybrid_search("q", user="jpmschweitzer")
assert self._sent_user(mock_httpx) == "jpmschweitzer"
@pytest.mark.unit @pytest.mark.unit
class TestNewResponseModels: class TestNewResponseModels:
"""Tests for new response models.""" """Tests for new response models."""
+232
View File
@@ -0,0 +1,232 @@
"""
Tests for bounded retries, timeout wiring, and client reuse in
LibraryDeskClient, plus ModelRetry escalation from the read tools.
"""
from unittest.mock import AsyncMock, MagicMock, patch
import httpx
import pytest
from pydantic_ai import ModelRetry
from src.agents.librarian.client import (
LibraryDeskClient,
library_client_session,
)
from src.agents.librarian.tools import (
create_wiki_page,
hybrid_search,
search_wiki,
)
from src.core.config import config
@pytest.fixture(autouse=True)
def _no_backoff(monkeypatch):
"""Skip the retry backoff sleep in tests."""
monkeypatch.setattr("src.agents.librarian.client._RETRY_BACKOFF_SECONDS", 0)
def _ok_response(payload: dict) -> MagicMock:
response = MagicMock()
response.status_code = 200
response.json.return_value = payload
response.raise_for_status = MagicMock()
return response
@pytest.fixture
def client_with_mock():
client = LibraryDeskClient(base_url="http://test:8089", api_key="test-key")
client._client = AsyncMock(spec=httpx.AsyncClient)
return client
@pytest.mark.unit
class TestBoundedRetries:
"""2-attempt retry for GETs and read-only POST /query/*, /rag/search."""
@pytest.mark.asyncio
async def test_get_retries_once_on_transport_error(self, client_with_mock):
mock_httpx = client_with_mock._client
mock_httpx.get.side_effect = [
httpx.ConnectError("Connection refused"),
_ok_response({"results": []}),
]
results = await client_with_mock.search_wiki("docker", user="u")
assert results == []
assert mock_httpx.get.call_count == 2
@pytest.mark.asyncio
async def test_get_gives_up_after_two_attempts(self, client_with_mock):
mock_httpx = client_with_mock._client
mock_httpx.get.side_effect = httpx.ConnectError("Connection refused")
with pytest.raises(httpx.ConnectError):
await client_with_mock.search_wiki("docker", user="u")
assert mock_httpx.get.call_count == 2
@pytest.mark.asyncio
async def test_query_hybrid_retries_on_503(self, client_with_mock):
mock_httpx = client_with_mock._client
bad = MagicMock()
bad.status_code = 503
mock_httpx.post.side_effect = [
bad,
_ok_response({"results": [], "keywords": {}, "context": ""}),
]
response = await client_with_mock.hybrid_search("docker", user="u")
assert response.results == []
assert mock_httpx.post.call_count == 2
@pytest.mark.asyncio
async def test_wiki_write_is_never_retried(self, client_with_mock):
"""POST /wiki/pages must not retry - it could duplicate pages."""
mock_httpx = client_with_mock._client
mock_httpx.post.side_effect = httpx.ConnectError("Connection refused")
with pytest.raises(httpx.ConnectError):
await client_with_mock.create_wiki_page(
title="T", path="/t", content="c", user="u"
)
assert mock_httpx.post.call_count == 1
@pytest.mark.asyncio
async def test_smart_create_is_never_retried(self, client_with_mock):
mock_httpx = client_with_mock._client
mock_httpx.post.side_effect = httpx.ConnectError("Connection refused")
with pytest.raises(httpx.ConnectError):
await client_with_mock.smart_create_wiki_page(
topic="T", tags=["x"], user="u"
)
assert mock_httpx.post.call_count == 1
@pytest.mark.unit
class TestTimeoutWiring:
"""LIBRARY_DESK_TIMEOUT config replaces the hardcoded 60s/30s."""
def test_default_timeout_from_config(self):
client = LibraryDeskClient()
assert client.timeout == config.LIBRARY_DESK_TIMEOUT
def test_explicit_timeout_wins(self):
client = LibraryDeskClient(timeout=5)
assert client.timeout == 5
@pytest.mark.unit
class TestClientReuse:
"""One shared HTTP connection per librarian run."""
@pytest.mark.asyncio
async def test_clients_share_connection_inside_session(self):
async with library_client_session():
async with LibraryDeskClient() as c1:
http1 = c1._client
# shared connection survives client exit
assert http1 is not None
assert not http1.is_closed
async with LibraryDeskClient() as c2:
assert c2._client is http1
# session close tears the shared connection down
assert http1.is_closed
@pytest.mark.asyncio
async def test_nested_sessions_are_noops(self):
async with library_client_session():
async with LibraryDeskClient() as c1:
http1 = c1._client
async with library_client_session():
async with LibraryDeskClient() as c2:
assert c2._client is http1
# inner session exit must not close the shared connection
assert not http1.is_closed
@pytest.mark.asyncio
async def test_client_owns_connection_outside_session(self):
async with LibraryDeskClient() as client:
http_client = client._client
assert http_client.is_closed
@pytest.mark.asyncio
async def test_custom_target_does_not_reuse_shared(self):
async with library_client_session():
async with LibraryDeskClient() as shared_client:
shared_http = shared_client._client
async with LibraryDeskClient(base_url="http://other:9999") as custom:
assert custom._client is not shared_http
@pytest.mark.unit
class TestModelRetryEscalation:
"""Read tools raise ModelRetry on transient errors so Agent(retries=2) engages."""
def _patched_client(self, mock_client):
factory = MagicMock()
factory.return_value.__aenter__ = AsyncMock(return_value=mock_client)
factory.return_value.__aexit__ = AsyncMock(return_value=None)
return patch("src.agents.librarian.tools.LibraryDeskClient", factory)
@pytest.mark.asyncio
async def test_read_tool_raises_model_retry_on_transport_error(self):
mock_client = AsyncMock()
mock_client.hybrid_search.side_effect = httpx.ConnectError(
"Connection refused"
)
with self._patched_client(mock_client):
with pytest.raises(ModelRetry):
await hybrid_search("docker")
@pytest.mark.asyncio
async def test_read_tool_raises_model_retry_on_5xx(self):
request = httpx.Request("GET", "http://test:8089/wiki/search")
response = httpx.Response(502, request=request)
mock_client = AsyncMock()
mock_client.search_wiki.side_effect = httpx.HTTPStatusError(
"bad gateway", request=request, response=response
)
with self._patched_client(mock_client):
with pytest.raises(ModelRetry):
await search_wiki("docker")
@pytest.mark.asyncio
async def test_read_tool_returns_safe_message_on_non_transient(self):
mock_client = AsyncMock()
mock_client.hybrid_search.side_effect = ValueError("bad parse")
with self._patched_client(mock_client):
result = await hybrid_search("docker")
assert "unable" in result
assert "bad parse" not in result
@pytest.mark.asyncio
async def test_write_tool_never_raises_model_retry(self):
mock_client = AsyncMock()
mock_client.create_wiki_page.side_effect = httpx.ConnectError(
"Connection refused"
)
with self._patched_client(mock_client):
result = await create_wiki_page(
title="T", path="/t", content="c", tags=["x"]
)
assert "unable" in result
assert "Connection refused" not in result
@@ -0,0 +1,344 @@
"""
Contract tests for HybridRAG parsing against a recorded live response.
The fixture in fixtures/hybrid_query_recorded.json is a real (recorded)
response from library-desk's POST /query/hybrid. These tests pin the
field mapping (source_type/sources, rrf_score, context, per-item
related_dossiers, keywords dict with nested synonyms) so a drift in
either side shows up as a test failure instead of every result
rendering as "unknown (score: 0.00)".
"""
import json
from pathlib import Path
from unittest.mock import AsyncMock, MagicMock
import httpx
import pytest
from src.agents.librarian.client import HybridRAGResponse, LibraryDeskClient
from src.agents.librarian.tools import SOURCE_ICONS, _coverage_note, hybrid_search
FIXTURE_PATH = Path(__file__).parent / "fixtures" / "hybrid_query_recorded.json"
@pytest.fixture
def recorded_response() -> dict:
"""Load the recorded /query/hybrid response."""
return json.loads(FIXTURE_PATH.read_text())
@pytest.fixture
def client_with_recorded_response(recorded_response):
"""LibraryDeskClient whose httpx client replays the recorded response."""
mock_response = MagicMock()
mock_response.json.return_value = recorded_response
mock_response.raise_for_status = MagicMock()
mock_httpx = AsyncMock(spec=httpx.AsyncClient)
mock_httpx.post.return_value = mock_response
client = LibraryDeskClient(base_url="http://test:8089", api_key="test-key")
client._client = mock_httpx
return client
@pytest.mark.unit
class TestHybridRAGContract:
"""Contract tests for parsing the live /query/hybrid response shape."""
@pytest.mark.asyncio
async def test_sources_are_not_unknown(self, client_with_recorded_response):
"""Every result maps source_type - nothing falls back to 'unknown'."""
response = await client_with_recorded_response.hybrid_search(
"home server infrastructure", user="testuser"
)
assert isinstance(response, HybridRAGResponse)
assert response.results, "recorded fixture must contain results"
for result in response.results:
assert result.source != "unknown"
assert result.source in {"wiki", "web", "volatile", "document"}
@pytest.mark.asyncio
async def test_scores_are_non_zero(self, client_with_recorded_response):
"""rrf_score maps to score - no silent 0.00 fallback."""
response = await client_with_recorded_response.hybrid_search(
"home server infrastructure", user="testuser"
)
for result in response.results:
assert result.score > 0.0
@pytest.mark.asyncio
async def test_sources_list_and_icons(self, client_with_recorded_response):
"""Per-item sources list is parsed and every value has an icon."""
response = await client_with_recorded_response.hybrid_search(
"home server infrastructure", user="testuser"
)
for result in response.results:
assert result.sources, f"result '{result.title}' has empty sources"
for source in result.sources:
assert source in SOURCE_ICONS, f"no icon for source '{source}'"
@pytest.mark.asyncio
async def test_context_maps_to_formatted_context(
self, client_with_recorded_response
):
"""Top-level 'context' field maps to formatted_context."""
response = await client_with_recorded_response.hybrid_search(
"home server infrastructure", user="testuser"
)
assert response.formatted_context != ""
@pytest.mark.asyncio
async def test_keywords_and_synonyms_from_dict(
self, client_with_recorded_response
):
"""keywords is a dict: core_keywords + nested synonyms map."""
response = await client_with_recorded_response.hybrid_search(
"home server infrastructure", user="testuser"
)
assert response.keywords, "core_keywords should be extracted"
assert all(isinstance(k, str) for k in response.keywords)
# synonyms map in the fixture is empty, but must parse to a list
assert isinstance(response.synonyms, list)
@pytest.mark.asyncio
async def test_per_item_related_dossiers(self, client_with_recorded_response):
"""related_dossiers live per result and aggregate to unique titles."""
response = await client_with_recorded_response.hybrid_search(
"home server infrastructure", user="testuser"
)
per_item = [d for r in response.results for d in r.related_dossiers]
assert per_item, "recorded fixture contains per-item related_dossiers"
for dossier in per_item:
assert "title" in dossier
assert "tag" in dossier
assert response.related_dossiers, "top-level titles are aggregated"
assert len(response.related_dossiers) == len(set(response.related_dossiers))
@pytest.mark.asyncio
async def test_payload_never_sends_zero_limits(
self, client_with_recorded_response
):
"""The live service 422s on limits < 1; disabled legs use enable_* flags."""
await client_with_recorded_response.hybrid_search(
"home server infrastructure",
user="testuser",
web_limit=0,
document_limit=0,
volatile_limit=0,
)
payload = client_with_recorded_response._client.post.call_args.kwargs["json"]
config = payload["config"]
for key in (
"vector_limit",
"graph_limit",
"web_limit",
"document_limit",
"volatile_limit",
):
assert config[key] >= 1
assert config["enable_web"] is False
assert config["enable_documents"] is False
assert config["enable_volatile"] is False
@pytest.mark.asyncio
async def test_user_always_sent_as_query_param(
self, client_with_recorded_response
):
"""The tenant is always sent explicitly - library-desk is removing
its server-side default, so a missing user would 422."""
await client_with_recorded_response.hybrid_search(
"home server infrastructure", user="testuser"
)
params = client_with_recorded_response._client.post.call_args.kwargs[
"params"
]
assert params["user"] == "testuser"
@pytest.mark.asyncio
async def test_source_counts_and_timing_parsed(
self, client_with_recorded_response
):
"""source_counts and timing map into the response model."""
response = await client_with_recorded_response.hybrid_search(
"home server infrastructure", user="testuser"
)
assert response.source_counts == {"graph": 3, "web": 2}
assert response.timing.get("total_ms", 0) > 0
@pytest.mark.asyncio
async def test_source_status_absent_is_tolerated(
self, client_with_recorded_response
):
"""Recorded response predates source_status/degraded - defaults apply."""
response = await client_with_recorded_response.hybrid_search(
"home server infrastructure", user="testuser"
)
assert response.source_status == {}
assert response.degraded is False
@pytest.mark.asyncio
async def test_source_status_parsed_when_present(self, recorded_response):
"""Additive source_status/degraded fields parse when the service sends them."""
enriched = dict(recorded_response)
enriched["source_status"] = {
"vector": "ok",
"graph": "ok",
"web": "failed",
"volatile": "disabled",
"documents": "ok",
}
enriched["degraded"] = True
mock_response = MagicMock()
mock_response.json.return_value = enriched
mock_response.raise_for_status = MagicMock()
mock_httpx = AsyncMock(spec=httpx.AsyncClient)
mock_httpx.post.return_value = mock_response
client = LibraryDeskClient(base_url="http://test:8089", api_key="test-key")
client._client = mock_httpx
response = await client.hybrid_search("home server infrastructure", user="u")
assert response.degraded is True
assert response.source_status["web"] == "failed"
assert response.source_status["volatile"] == "disabled"
@pytest.mark.asyncio
async def test_tool_renders_no_unknown_results(self, client_with_recorded_response, monkeypatch):
"""The hybrid_search tool renders real sources and non-zero scores."""
class _Factory:
def __call__(self):
return self
async def __aenter__(self):
return client_with_recorded_response
async def __aexit__(self, *args):
return None
monkeypatch.setattr(
"src.agents.librarian.tools.LibraryDeskClient", _Factory()
)
output = await hybrid_search("home server infrastructure")
assert "unknown" not in output
assert "score: 0.00" not in output
assert "" not in output, "every source value should map to an icon"
@pytest.mark.unit
class TestCoverageNote:
"""Coverage note makes degraded searches visible to model and user."""
def _response(self, **kwargs) -> HybridRAGResponse:
return HybridRAGResponse(**kwargs)
def test_no_note_when_all_legs_report(self):
response = self._response(
source_counts={
"vector": 2,
"graph": 1,
"web": 2,
"documents": 1,
"volatile": 1,
},
)
note = _coverage_note(
response, include_web=True, include_documents=True, include_volatile=True
)
assert note == ""
def test_note_when_enabled_leg_missing_from_counts(self):
response = self._response(source_counts={"graph": 3, "web": 2})
note = _coverage_note(
response, include_web=True, include_documents=True, include_volatile=True
)
assert "Coverage note" in note
assert "documents" in note
assert "volatile" in note
# Always-on wiki legs are never inferred from count absence
assert "vector" not in note
assert "graph" not in note
def test_wiki_leg_absence_is_not_degradation(self):
"""vector/graph missing from top-N counts is healthy ranking, not outage."""
response = self._response(
source_counts={"web": 2, "documents": 1, "volatile": 1}
)
note = _coverage_note(
response, include_web=True, include_documents=True, include_volatile=True
)
assert note == ""
def test_disabled_legs_are_not_reported_missing(self):
response = self._response(source_counts={"vector": 2, "graph": 1})
note = _coverage_note(
response,
include_web=False,
include_documents=False,
include_volatile=False,
)
assert note == ""
def test_note_prefers_source_status_failures(self):
response = self._response(
source_counts={"vector": 2, "graph": 1},
source_status={
"vector": "ok",
"graph": "ok",
"web": "failed",
"volatile": "disabled",
"documents": "ok",
},
degraded=True,
)
note = _coverage_note(
response, include_web=True, include_documents=True, include_volatile=True
)
assert "failed" in note
assert "web" in note
# disabled legs are not reported as failures
assert "volatile" not in note
def test_no_note_when_status_all_ok(self):
response = self._response(
source_counts={"graph": 1},
source_status={
"vector": "ok",
"graph": "ok",
"web": "ok",
"volatile": "ok",
"documents": "ok",
},
degraded=False,
)
note = _coverage_note(
response, include_web=True, include_documents=True, include_volatile=True
)
assert note == ""
def test_degraded_without_named_failures(self):
response = self._response(
source_status={"vector": "ok", "graph": "ok"},
degraded=True,
)
note = _coverage_note(
response, include_web=True, include_documents=True, include_volatile=True
)
assert "partial" in note
@@ -0,0 +1,49 @@
"""
Snapshot tests for the JSON schemas emitted for librarian tools.
Ollama's OpenAI-compatible API mishandles anyOf[X, null] parameter
schemas, so tool parameters must use empty-string/empty-list sentinels
translated to None inside the tool (same pattern as the biographer
tools, commit 9d7ce39). This test fails if a X | None parameter ever
creeps back in.
"""
import pytest
from pydantic_ai.tools import Tool
from src.agents.librarian.tools import LIBRARIAN_TOOLS
def _nullable_anyof_paths(schema: object, path: str = "") -> list[str]:
"""Recursively collect JSON-schema paths that are anyOf[..., null]."""
offenders: list[str] = []
if isinstance(schema, dict):
any_of = schema.get("anyOf")
if isinstance(any_of, list) and any(
isinstance(sub, dict) and sub.get("type") == "null" for sub in any_of
):
offenders.append(path or "<root>")
for key, value in schema.items():
offenders.extend(_nullable_anyof_paths(value, f"{path}/{key}"))
elif isinstance(schema, list):
for i, item in enumerate(schema):
offenders.extend(_nullable_anyof_paths(item, f"{path}[{i}]"))
return offenders
@pytest.mark.unit
class TestLibrarianToolSchemas:
"""All registered librarian tools emit Ollama-safe parameter schemas."""
@pytest.mark.parametrize(
"tool_func", LIBRARIAN_TOOLS, ids=lambda f: f.__name__
)
def test_no_nullable_anyof_in_schema(self, tool_func):
schema = Tool(tool_func).function_schema.json_schema
offenders = _nullable_anyof_paths(schema)
assert offenders == [], (
f"{tool_func.__name__} emits anyOf[..., null] at {offenders}; "
"use empty-string/empty-list sentinels instead of X | None"
)
+78 -16
View File
@@ -5,21 +5,23 @@ Tests the tool functions that wrap the Library-Desk API,
including the new web search and content extraction tools. including the new web search and content extraction tools.
""" """
import pytest from unittest.mock import AsyncMock, patch
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from src.agents.librarian.client import (
BatchExtractionResponse,
ContentExtractionResult,
WebSearchResponse,
WebSearchResult,
WikiPage,
)
from src.agents.librarian.tools import ( from src.agents.librarian.tools import (
search_web, CLEAR_TAGS_SENTINEL,
read_url, read_url,
read_urls_batch, read_urls_batch,
hybrid_search, search_web,
search_wiki, update_wiki_page,
)
from src.agents.librarian.client import (
WebSearchResult,
WebSearchResponse,
ContentExtractionResult,
BatchExtractionResponse,
) )
@@ -104,8 +106,10 @@ class TestSearchWeb:
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_search_web_error_handling(self, mock_client): async def test_search_web_error_handling(self, mock_client):
"""Test web search error handling.""" """Test web search errors return a user-safe message without internals."""
mock_client.search_web.side_effect = Exception("Connection failed") mock_client.search_web.side_effect = Exception(
"Connection failed to http://internal-host:8089"
)
with patch( with patch(
"src.agents.librarian.tools.LibraryDeskClient" "src.agents.librarian.tools.LibraryDeskClient"
@@ -115,8 +119,10 @@ class TestSearchWeb:
result = await search_web("test query") result = await search_web("test query")
assert "Error" in result assert "unable to search the web" in result
assert "Connection failed" in result # Exception detail (internal URLs etc.) must not leak
assert "Connection failed" not in result
assert "internal-host" not in result
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_search_web_with_news_type(self, mock_client): async def test_search_web_with_news_type(self, mock_client):
@@ -227,7 +233,7 @@ class TestReadUrl:
mock_client_class.return_value.__aenter__.return_value = mock_client mock_client_class.return_value.__aenter__.return_value = mock_client
mock_client_class.return_value.__aexit__.return_value = None mock_client_class.return_value.__aexit__.return_value = None
result = await read_url("https://example.com/long", max_length=2000) await read_url("https://example.com/long", max_length=2000)
mock_client.extract_content.assert_called_with( mock_client.extract_content.assert_called_with(
url="https://example.com/long", url="https://example.com/long",
@@ -424,3 +430,59 @@ class TestWebSearchModels:
assert response.total_urls == 2 assert response.total_urls == 2
assert response.successful == 1 assert response.successful == 1
assert response.failed == 1 assert response.failed == 1
# ============================================================================
# Wiki Update Tests (tag sentinel behavior)
# ============================================================================
@pytest.mark.unit
class TestUpdateWikiPageTagSentinels:
"""Empty list leaves tags unchanged; the clear sentinel empties them."""
def _page(self, tags: list[str]) -> WikiPage:
return WikiPage(id=42, path="test/page", title="Test Page", tags=tags)
def _patched_client(self, mock_client):
patcher = patch("src.agents.librarian.tools.LibraryDeskClient")
mock_client_class = patcher.start()
mock_client_class.return_value.__aenter__.return_value = mock_client
mock_client_class.return_value.__aexit__.return_value = None
return patcher
@pytest.mark.asyncio
async def test_empty_tags_means_leave_unchanged(self, mock_client):
mock_client.update_wiki_page.return_value = self._page(["existing"])
patcher = self._patched_client(mock_client)
try:
await update_wiki_page(42, content="new content")
finally:
patcher.stop()
_, kwargs = mock_client.update_wiki_page.call_args
assert kwargs["tags"] is None
@pytest.mark.asyncio
async def test_clear_sentinel_sends_empty_tag_list(self, mock_client):
mock_client.update_wiki_page.return_value = self._page([])
patcher = self._patched_client(mock_client)
try:
result = await update_wiki_page(42, tags=[CLEAR_TAGS_SENTINEL])
finally:
patcher.stop()
_, kwargs = mock_client.update_wiki_page.call_args
assert kwargs["tags"] == []
assert "tags (cleared)" in result
@pytest.mark.asyncio
async def test_real_tags_are_passed_through(self, mock_client):
mock_client.update_wiki_page.return_value = self._page(["a", "b"])
patcher = self._patched_client(mock_client)
try:
await update_wiki_page(42, tags=["a", "b"])
finally:
patcher.stop()
_, kwargs = mock_client.update_wiki_page.call_args
assert kwargs["tags"] == ["a", "b"]
-339
View File
@@ -1,339 +0,0 @@
"""
Tests for multi-agent coordination engine.
"""
import pytest
from unittest.mock import AsyncMock, MagicMock, patch
from src.agents.coordination import (
CoordinationEngine,
get_coordination_engine,
delegate_to_librarian,
)
from src.agents.protocol import (
AgentResponse,
AgentUnavailableError,
DelegationIntent,
DelegationReason,
)
@pytest.fixture
def coordination_engine():
"""Create a fresh coordination engine for testing."""
return CoordinationEngine()
@pytest.fixture
def mock_registry():
"""Mock the household registry."""
with patch("src.agents.coordination.get_household_registry") as mock:
registry = MagicMock()
mock.return_value = registry
yield registry
@pytest.fixture
def librarian_intent():
"""Create a standard librarian delegation intent."""
return DelegationIntent(
target_agent="librarian",
task="Find information about Docker networking",
reason=DelegationReason.DOMAIN_EXPERTISE,
expected_outcome="Documentation and examples",
)
@pytest.mark.unit
class TestCoordinationEngine:
"""Tests for CoordinationEngine class."""
def test_initialization(self, coordination_engine):
"""Test engine initializes correctly."""
assert coordination_engine is not None
assert coordination_engine.registry is not None
def test_get_available_agents_empty(self, mock_registry):
"""Test getting available agents when none have agents."""
mock_registry.list_members.return_value = ["tatlock_core"]
mock_member = MagicMock()
mock_member.agent = None # No agent
mock_registry.get_member.return_value = mock_member
engine = CoordinationEngine()
available = engine.get_available_agents()
assert available == []
def test_get_available_agents_with_librarian(self, mock_registry):
"""Test getting available agents with librarian registered."""
mock_registry.list_members.return_value = ["tatlock_core", "librarian"]
# tatlock_core has no agent
core_member = MagicMock()
core_member.agent = None
# librarian has an agent
librarian_member = MagicMock()
librarian_member.agent = MagicMock()
def get_member_side_effect(name):
if name == "tatlock_core":
return core_member
elif name == "librarian":
return librarian_member
return None
mock_registry.get_member.side_effect = get_member_side_effect
engine = CoordinationEngine()
available = engine.get_available_agents()
assert "librarian" in available
assert "tatlock_core" not in available
def test_can_delegate_to_unknown_agent(self, mock_registry):
"""Test checking delegation to unknown agent."""
mock_registry.get_member.return_value = None
engine = CoordinationEngine()
assert engine.can_delegate_to("unknown_agent") is False
def test_can_delegate_to_librarian(self, mock_registry):
"""Test checking delegation to librarian."""
mock_member = MagicMock()
mock_member.agent = MagicMock() # Has an agent
mock_registry.get_member.return_value = mock_member
engine = CoordinationEngine()
assert engine.can_delegate_to("librarian") is True
@pytest.mark.unit
class TestDelegationExecution:
"""Tests for delegation execution."""
@pytest.mark.asyncio
async def test_execute_delegation_unavailable_agent(
self, mock_registry, librarian_intent
):
"""Test delegation fails for unavailable agent."""
mock_registry.get_member.return_value = None
engine = CoordinationEngine()
with pytest.raises(AgentUnavailableError) as exc_info:
await engine.execute_delegation(librarian_intent)
assert "librarian" in str(exc_info.value)
@pytest.mark.asyncio
async def test_execute_delegation_success(
self, mock_registry, librarian_intent
):
"""Test successful delegation execution."""
# Setup mock member with agent
mock_member = MagicMock()
mock_member.agent = MagicMock()
mock_registry.get_member.return_value = mock_member
# Mock the executor
with patch(
"src.agents.coordination.AGENT_EXECUTORS",
{"librarian": AsyncMock(return_value="Research results here")},
):
engine = CoordinationEngine()
response = await engine.execute_delegation(librarian_intent)
assert response.success is True
assert response.result == "Research results here"
# Duration might be 0 for very fast mock execution
assert response.duration_ms >= 0
@pytest.mark.asyncio
async def test_execute_delegation_error(
self, mock_registry, librarian_intent
):
"""Test delegation handles executor errors."""
mock_member = MagicMock()
mock_member.agent = MagicMock()
mock_registry.get_member.return_value = mock_member
# Mock executor that raises
async def failing_executor(**kwargs):
raise ValueError("API connection failed")
with patch(
"src.agents.coordination.AGENT_EXECUTORS",
{"librarian": failing_executor},
):
engine = CoordinationEngine()
response = await engine.execute_delegation(librarian_intent)
assert response.success is False
assert "API connection failed" in response.error_message
@pytest.mark.unit
class TestCoordinate:
"""Tests for multi-agent coordination."""
@pytest.mark.asyncio
async def test_coordinate_single_intent(self, mock_registry, librarian_intent):
"""Test coordinating a single delegation."""
mock_member = MagicMock()
mock_member.agent = MagicMock()
mock_registry.get_member.return_value = mock_member
with patch(
"src.agents.coordination.AGENT_EXECUTORS",
{"librarian": AsyncMock(return_value="Found docs")},
):
engine = CoordinationEngine()
result = await engine.coordinate([librarian_intent])
assert result.final_response == "Found docs"
assert "librarian" in result.agents_consulted
# Duration might be 0 for very fast mock execution
assert result.total_duration_ms >= 0
@pytest.mark.asyncio
async def test_coordinate_empty_intents(self, mock_registry):
"""Test coordinating with no intents."""
engine = CoordinationEngine()
result = await engine.coordinate([])
assert result.final_response == ""
assert result.agents_consulted == []
@pytest.mark.asyncio
async def test_coordinate_multiple_intents(self, mock_registry):
"""Test coordinating multiple delegations."""
mock_member = MagicMock()
mock_member.agent = MagicMock()
mock_registry.get_member.return_value = mock_member
intents = [
DelegationIntent(
target_agent="librarian",
task="Task 1",
reason=DelegationReason.DOMAIN_EXPERTISE,
expected_outcome="Result 1",
priority=1,
),
DelegationIntent(
target_agent="librarian",
task="Task 2",
reason=DelegationReason.DOMAIN_EXPERTISE,
expected_outcome="Result 2",
priority=2,
),
]
call_count = 0
async def mock_executor(**kwargs):
nonlocal call_count
call_count += 1
return f"Result {call_count}"
with patch(
"src.agents.coordination.AGENT_EXECUTORS",
{"librarian": mock_executor},
):
engine = CoordinationEngine()
result = await engine.coordinate(intents)
# Both intents were executed (check agents_consulted count)
assert len(result.agents_consulted) == 2
# Current implementation replaces same-agent responses in dict
# So final_response has the last result (or combined if different agents)
assert len(result.final_response) > 0
@pytest.mark.unit
class TestDelegateToLibrarian:
"""Tests for convenience delegation function."""
@pytest.mark.asyncio
async def test_delegate_to_librarian(self, mock_registry):
"""Test the delegate_to_librarian helper."""
mock_member = MagicMock()
mock_member.agent = MagicMock()
mock_registry.get_member.return_value = mock_member
with patch(
"src.agents.coordination.AGENT_EXECUTORS",
{"librarian": AsyncMock(return_value="Wiki search results")},
):
# Reset global engine
with patch(
"src.agents.coordination._coordination_engine",
None,
):
response = await delegate_to_librarian(
task="Search for Docker docs",
context="Setting up homelab",
)
assert response.success is True
assert response.result == "Wiki search results"
@pytest.mark.unit
class TestGetCoordinationEngine:
"""Tests for engine singleton."""
def test_get_coordination_engine_singleton(self):
"""Test engine is singleton."""
with patch("src.agents.coordination._coordination_engine", None):
engine1 = get_coordination_engine()
engine2 = get_coordination_engine()
# Should be same instance
assert engine1 is engine2
@pytest.mark.unit
class TestDelegationStreaming:
"""Tests for streaming delegation."""
@pytest.mark.asyncio
async def test_execute_delegation_stream_unavailable(
self, mock_registry, librarian_intent
):
"""Test streaming fails for unavailable agent."""
engine = CoordinationEngine()
# Change target to an agent that doesn't have a stream executor
librarian_intent.target_agent = "nonexistent_agent"
with pytest.raises(AgentUnavailableError):
async for _ in engine.execute_delegation_stream(librarian_intent):
pass
@pytest.mark.asyncio
async def test_execute_delegation_stream_success(
self, mock_registry, librarian_intent
):
"""Test successful streaming delegation."""
mock_member = MagicMock()
mock_member.agent = MagicMock()
mock_registry.get_member.return_value = mock_member
async def mock_stream(**kwargs):
yield "Hello "
yield "world"
with patch(
"src.agents.coordination.AGENT_STREAM_EXECUTORS",
{"librarian": mock_stream},
):
engine = CoordinationEngine()
chunks = []
async for chunk in engine.execute_delegation_stream(librarian_intent):
chunks.append(chunk)
assert chunks == ["Hello ", "world"]
+99 -27
View File
@@ -4,21 +4,79 @@ Tests for delegation infrastructure.
Tests the DelegationTask dataclass and delegation wrapper functions Tests the DelegationTask dataclass and delegation wrapper functions
that implement the agent-as-tool pattern. that implement the agent-as-tool pattern.
""" """
from unittest.mock import AsyncMock, patch
import pytest import pytest
from unittest.mock import AsyncMock, patch, MagicMock
from src.agents.delegation import ( from src.agents.delegation import (
ActionType,
DelegationTask,
DelegationResult,
HOUSEHOLD_THINK_MESSAGES, HOUSEHOLD_THINK_MESSAGES,
STREAMING_DELEGATION_WRAPPERS, ActionType,
DelegationResult,
DelegationTask,
_detect_action_type,
build_delegation_context,
delegate_to_librarian, delegate_to_librarian,
get_think_message, get_think_message,
_detect_action_type,
) )
@pytest.mark.unit
class TestBuildDelegationContext:
"""Tests for trimming conversation history into expert context."""
def test_empty_history_returns_empty(self):
assert build_delegation_context(None) == ""
assert build_delegation_context([]) == ""
def test_recent_turns_are_formatted(self):
history = [
{"role": "user", "content": "Tell me about Docker"},
{"role": "assistant", "content": "Docker is a container runtime."},
]
context = build_delegation_context(history)
assert "Recent conversation:" in context
assert "user: Tell me about Docker" in context
assert "assistant: Docker is a container runtime." in context
def test_only_last_max_turns_kept(self):
history = [
{"role": "user", "content": f"message {i}"} for i in range(10)
]
context = build_delegation_context(history, max_turns=6)
assert "message 3" not in context
assert "message 4" in context
assert "message 9" in context
def test_long_turns_are_truncated(self):
history = [{"role": "user", "content": "x" * 2000}]
context = build_delegation_context(history, max_chars_per_turn=500)
assert "x" * 500 in context
assert "x" * 501 not in context
def test_structured_content_parts_tolerated(self):
history = [
{"role": "user", "content": [{"type": "text", "text": "hello there"}]}
]
context = build_delegation_context(history)
assert "hello there" in context
def test_non_dict_entries_skipped(self):
history = ["garbage", {"role": "user", "content": "real message"}]
context = build_delegation_context(history)
assert "real message" in context
assert "garbage" not in context
@pytest.mark.unit @pytest.mark.unit
class TestDelegationTask: class TestDelegationTask:
"""Tests for the DelegationTask dataclass.""" """Tests for the DelegationTask dataclass."""
@@ -170,11 +228,11 @@ class TestDelegateToLibrarian:
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_delegate_to_librarian_handles_error(self): async def test_delegate_to_librarian_handles_error(self):
"""Test delegation handles Librarian errors gracefully.""" """Test delegation maps Librarian errors to a user-safe result."""
with patch( with patch(
"src.agents.librarian.agent.run_librarian", "src.agents.librarian.agent.run_librarian",
new_callable=AsyncMock, new_callable=AsyncMock,
side_effect=Exception("Connection refused"), side_effect=Exception("Connection refused to http://internal:8089"),
): ):
result = await delegate_to_librarian( result = await delegate_to_librarian(
task="Search for information", task="Search for information",
@@ -182,8 +240,39 @@ class TestDelegateToLibrarian:
assert isinstance(result, DelegationResult) assert isinstance(result, DelegationResult)
assert result.success is False assert result.success is False
assert result.output == "" # Output carries a curated butler-toned sentence
assert result.error == "Connection refused" assert result.output == get_think_message(
"librarian", "Search for information", "error"
)
# Exception detail stays in logs only - never in the result
assert "Connection refused" not in result.output
assert result.error is not None
assert "Connection refused" not in result.error
assert "internal" not in result.error
@pytest.mark.asyncio
async def test_delegate_to_librarian_timeout(self, monkeypatch):
"""Delegation is capped by LIBRARIAN_TIMEOUT and fails honestly."""
import asyncio
from src.core.config import config
async def slow_run(task, context=""):
await asyncio.sleep(5)
return "too late"
monkeypatch.setattr(config, "LIBRARIAN_TIMEOUT", 0.05)
with patch(
"src.agents.librarian.agent.run_librarian",
new=slow_run,
):
result = await delegate_to_librarian(task="Search for information")
assert result.success is False
assert "longer than expected" in result.output
assert result.error is not None
assert "time budget" in result.error
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_delegate_to_librarian_preserves_task(self): async def test_delegate_to_librarian_preserves_task(self):
@@ -340,20 +429,3 @@ class TestGetThinkMessage:
msg = get_think_message("unknown_expert", "some task", "start") msg = get_think_message("unknown_expert", "some task", "start")
assert "<think>" not in msg assert "<think>" not in msg
assert "unknown_expert" in msg.lower() assert "unknown_expert" in msg.lower()
@pytest.mark.unit
class TestStreamingDelegationWrappers:
"""Tests for streaming delegation wrapper mapping."""
def test_streaming_wrappers_exist(self):
"""Test streaming wrappers mapping has all experts."""
assert "librarian" in STREAMING_DELEGATION_WRAPPERS
assert "biographer" in STREAMING_DELEGATION_WRAPPERS
assert "housekeeper" in STREAMING_DELEGATION_WRAPPERS
def test_streaming_wrappers_are_async_generators(self):
"""Test streaming wrappers are async generator functions."""
import inspect
for name, wrapper in STREAMING_DELEGATION_WRAPPERS.items():
assert inspect.isasyncgenfunction(wrapper), f"{name} is not an async generator"
+15 -240
View File
@@ -1,256 +1,31 @@
""" """
Tests for agent communication protocol. Tests for the agent error protocol.
""" """
import pytest import pytest
from src.agents.protocol import ( from src.agents.protocol import AgentError
AgentError,
AgentRequest,
AgentResponse,
AgentTimeoutError,
AgentUnavailableError,
CoordinationResult,
DelegationIntent,
DelegationReason,
ToolCallRecord,
)
@pytest.mark.unit @pytest.mark.unit
class TestAgentRequest: class TestAgentError:
"""Tests for AgentRequest model.""" """Tests for the AgentError exception."""
def test_basic_request(self): def test_agent_error_defaults(self):
"""Test creating a basic agent request.""" """Test base AgentError with default agent name."""
request = AgentRequest(task="Find information about Docker")
assert request.task == "Find information about Docker"
assert request.context == ""
assert request.timeout_seconds == 60
def test_request_with_context(self):
"""Test request with additional context."""
request = AgentRequest(
task="Find Docker networking docs",
context="User is setting up a homelab",
delegation_reason=DelegationReason.DOMAIN_EXPERTISE,
)
assert request.task == "Find Docker networking docs"
assert request.context == "User is setting up a homelab"
assert request.delegation_reason == DelegationReason.DOMAIN_EXPERTISE
def test_request_serialization(self):
"""Test request can be serialized to dict."""
request = AgentRequest(
task="Research task",
context="Some context",
)
data = request.model_dump()
assert data["task"] == "Research task"
assert data["context"] == "Some context"
@pytest.mark.unit
class TestAgentResponse:
"""Tests for AgentResponse model."""
def test_successful_response(self):
"""Test creating a successful response."""
response = AgentResponse(
success=True,
result="Here are the findings...",
reasoning="Searched wiki and found relevant docs",
duration_ms=1500,
)
assert response.success is True
assert response.result == "Here are the findings..."
assert response.reasoning == "Searched wiki and found relevant docs"
assert response.duration_ms == 1500
assert response.error_message is None
def test_failed_response(self):
"""Test creating a failed response."""
response = AgentResponse(
success=False,
result="",
error_message="Connection timeout",
duration_ms=30000,
)
assert response.success is False
assert response.result == ""
assert response.error_message == "Connection timeout"
def test_response_with_tool_calls(self):
"""Test response tracking tool calls."""
tool_call = ToolCallRecord(
tool_name="hybrid_search",
arguments={"query": "Docker networking"},
result="Found 5 results",
duration_ms=500,
)
response = AgentResponse(
success=True,
result="Based on search...",
tool_calls=[tool_call],
)
assert len(response.tool_calls) == 1
assert response.tool_calls[0].tool_name == "hybrid_search"
@pytest.mark.unit
class TestDelegationIntent:
"""Tests for DelegationIntent model."""
def test_basic_intent(self):
"""Test creating a basic delegation intent."""
intent = DelegationIntent(
target_agent="librarian",
task="Research Docker networking",
reason=DelegationReason.DOMAIN_EXPERTISE,
expected_outcome="Documentation and examples",
)
assert intent.target_agent == "librarian"
assert intent.task == "Research Docker networking"
assert intent.reason == DelegationReason.DOMAIN_EXPERTISE
assert intent.priority == 1 # Default
def test_intent_with_priority(self):
"""Test intent with custom priority."""
intent = DelegationIntent(
target_agent="librarian",
task="Urgent research",
reason=DelegationReason.RESOURCE_EFFICIENCY,
expected_outcome="Quick answer",
priority=1,
)
assert intent.priority == 1
@pytest.mark.unit
class TestDelegationReason:
"""Tests for DelegationReason enum."""
def test_all_reasons_have_values(self):
"""Test all delegation reasons are defined."""
reasons = list(DelegationReason)
assert DelegationReason.DOMAIN_EXPERTISE in reasons
assert DelegationReason.TOOL_ACCESS in reasons
assert DelegationReason.RESOURCE_EFFICIENCY in reasons
assert DelegationReason.USER_PREFERENCE in reasons
@pytest.mark.unit
class TestCoordinationResult:
"""Tests for CoordinationResult model."""
def test_single_agent_result(self):
"""Test coordination with single agent."""
agent_response = AgentResponse(
success=True,
result="Research findings",
duration_ms=1000,
)
intent = DelegationIntent(
target_agent="librarian",
task="Research task",
reason=DelegationReason.DOMAIN_EXPERTISE,
expected_outcome="Findings",
)
result = CoordinationResult(
final_response="Research findings",
agent_responses={"librarian": agent_response},
delegation_intents=[intent],
total_duration_ms=1200,
agents_consulted=["librarian"],
)
assert result.final_response == "Research findings"
assert len(result.agent_responses) == 1
assert result.agents_consulted == ["librarian"]
def test_empty_result(self):
"""Test coordination with no delegations."""
result = CoordinationResult(
final_response="",
agent_responses={},
delegation_intents=[],
total_duration_ms=0,
agents_consulted=[],
)
assert result.final_response == ""
assert len(result.agents_consulted) == 0
@pytest.mark.unit
class TestAgentErrors:
"""Tests for agent error types."""
def test_agent_error(self):
"""Test base AgentError."""
error = AgentError("Something went wrong") error = AgentError("Something went wrong")
assert "Something went wrong" in str(error) assert "Something went wrong" in str(error)
assert error.agent_name == "unknown" assert error.agent_name == "unknown"
def test_agent_timeout_error(self): def test_agent_error_carries_agent_name(self):
"""Test AgentTimeoutError.""" """Agent name is stored and prefixed into the message."""
error = AgentTimeoutError( error = AgentError("Research task failed", agent_name="librarian")
"Timed out after 60s",
agent_name="librarian",
)
assert "Timed out" in str(error)
assert error.agent_name == "librarian" assert error.agent_name == "librarian"
assert str(error) == "[librarian] Research task failed"
assert error.message == "Research task failed"
def test_agent_unavailable_error(self): def test_agent_error_is_catchable_as_exception(self):
"""Test AgentUnavailableError.""" """AgentError participates in normal exception handling."""
error = AgentUnavailableError( with pytest.raises(AgentError):
"Agent not registered", raise AgentError("boom", agent_name="librarian")
agent_name="unknown_agent",
)
assert "not registered" in str(error)
assert error.agent_name == "unknown_agent"
@pytest.mark.unit
class TestToolCallRecord:
"""Tests for ToolCallRecord model."""
def test_tool_call_record(self):
"""Test creating a tool call record."""
record = ToolCallRecord(
tool_name="semantic_search",
arguments={"query": "networking concepts", "limit": 10},
result="Found 10 relevant documents",
duration_ms=250,
)
assert record.tool_name == "semantic_search"
assert record.arguments["query"] == "networking concepts"
assert record.duration_ms == 250
def test_tool_call_with_empty_result(self):
"""Test tool call with empty result."""
record = ToolCallRecord(
tool_name="query_graph",
arguments={"cypher": "MATCH (n) RETURN n"},
result="",
duration_ms=100,
)
assert record.result == ""
+3 -1
View File
@@ -410,10 +410,12 @@ async def test_tatlock_ollama_fallback(async_client: AsyncClient):
"stream": False "stream": False
} }
# 300s: this test forbids the Claude rescue, and the full local
# Steward -> orchestrate -> synthesize flow on gemma4 exceeds 120s
response = await async_client.post( response = await async_client.post(
"/v1/chat/completions", "/v1/chat/completions",
json=request_data, json=request_data,
timeout=120.0 timeout=300.0
) )
assert response.status_code == 200 assert response.status_code == 200
View File
+92
View File
@@ -0,0 +1,92 @@
"""
Unit tests for backend selection (Ollama primary, Claude fallback).
These tests set the cached health-check globals directly so they are
deterministic regardless of which services are reachable.
"""
import pytest
from src.anthropic import model_selector
from src.core.config import config
@pytest.fixture
def local_first(monkeypatch):
"""Baseline: local-first config, both backends healthy."""
monkeypatch.setattr(config, "PREFER_CLOUD_BACKEND", False)
monkeypatch.setattr(config, "ANTHROPIC_API_KEY", "sk-test-fake")
monkeypatch.setattr(model_selector, "_claude_available", True)
monkeypatch.setattr(model_selector, "_ollama_available", True)
class TestResolveBackend:
def test_default_is_ollama(self, local_first):
assert model_selector.resolve_backend() == "ollama"
def test_prefer_cloud_config_selects_claude(self, local_first, monkeypatch):
monkeypatch.setattr(config, "PREFER_CLOUD_BACKEND", True)
assert model_selector.resolve_backend() == "claude"
def test_prefer_cloud_override_selects_claude(self, local_first):
assert model_selector.resolve_backend(prefer_cloud=True) == "claude"
def test_prefer_cloud_without_claude_falls_back_to_ollama(self, local_first, monkeypatch):
monkeypatch.setattr(config, "PREFER_CLOUD_BACKEND", True)
monkeypatch.setattr(model_selector, "_claude_available", False)
assert model_selector.resolve_backend() == "ollama"
def test_ollama_down_falls_back_to_claude(self, local_first, monkeypatch):
monkeypatch.setattr(model_selector, "_ollama_available", False)
assert model_selector.resolve_backend() == "claude"
def test_ollama_down_without_claude_stays_ollama(self, local_first, monkeypatch):
monkeypatch.setattr(model_selector, "_ollama_available", False)
monkeypatch.setattr(model_selector, "_claude_available", False)
assert model_selector.resolve_backend() == "ollama"
def test_unknown_ollama_state_counts_as_available(self, local_first, monkeypatch):
monkeypatch.setattr(model_selector, "_ollama_available", None)
assert model_selector.resolve_backend() == "ollama"
class TestGetModel:
def test_ollama_backend_returns_openai_chat_model(self, local_first):
from pydantic_ai.models.openai import OpenAIChatModel
model = model_selector.get_model()
assert isinstance(model, OpenAIChatModel)
assert model.model_name == config.OLLAMA_DEFAULT_MODEL
def test_claude_backend_returns_anthropic_model(self, local_first):
from pydantic_ai.models.anthropic import AnthropicModel
model = model_selector.get_model(prefer_cloud=True)
assert isinstance(model, AnthropicModel)
assert model.model_name == config.ANTHROPIC_MODEL
class TestToolChoiceSettings:
def test_ollama_forces_tool_choice(self, local_first):
settings = model_selector.get_tool_choice_settings()
assert settings.get("extra_body") == {"tool_choice": "required"}
def test_claude_uses_native_tool_choice(self, local_first, monkeypatch):
monkeypatch.setattr(config, "PREFER_CLOUD_BACKEND", True)
settings = model_selector.get_tool_choice_settings()
assert not settings.get("extra_body")
class TestGetModelInfo:
def test_reports_ollama_primary(self, local_first):
info = model_selector.get_model_info()
assert info["backend"] == "ollama"
assert info["model"] == config.OLLAMA_DEFAULT_MODEL
assert info["ollama_available"] is True
assert info["claude_available"] is True
assert info["prefer_cloud"] is False
def test_reports_claude_when_ollama_down(self, local_first, monkeypatch):
monkeypatch.setattr(model_selector, "_ollama_available", False)
info = model_selector.get_model_info()
assert info["backend"] == "claude"
assert info["model"] == config.ANTHROPIC_MODEL
+41 -1
View File
@@ -12,11 +12,51 @@ from src.main import app
@pytest.fixture(scope="session", autouse=True) @pytest.fixture(scope="session", autouse=True)
def _initialize_app(): def _tenant_guard():
"""
Hard-fail the whole suite if the effective tenant resolves to the
production tenant ("jpmschweitzer").
Isolation is tenant-based: tests that touch shared services
(Qdrant memories collections, Wiki.js via library-desk, Neo4j,
Redis) must run under the reserved test tenant "llm_tester" (or a
test_-prefixed namespace). This mirrors the guard library-desk
applies on its side.
"""
from src.core.config import PRODUCTION_TENANT, config
from src.core.context import get_default_user
from src.core.multi_tenancy import get_memory_collection_name
effective = get_default_user()
if (
effective == PRODUCTION_TENANT
or config.effective_default_user == PRODUCTION_TENANT
):
pytest.exit(
f"TENANT GUARD: refusing to run the test suite - the effective "
f"tenant resolves to the production tenant '{PRODUCTION_TENANT}' "
f"(ENVIRONMENT={config.ENVIRONMENT.value}, "
f"DEFAULT_USER={config.DEFAULT_USER}). Tests must run under "
f"'llm_tester' or a test_-prefixed tenant.",
returncode=1,
)
# The Qdrant memories namespace derived from the effective tenant
# must never be the production collection.
assert get_memory_collection_name(effective) != get_memory_collection_name(
PRODUCTION_TENANT
), "test suite would target the production memories collection"
@pytest.fixture(scope="session", autouse=True)
def _initialize_app(_tenant_guard):
""" """
Run application lifespan (Claude health check, household registration, etc.) Run application lifespan (Claude health check, household registration, etc.)
once per test session. ASGITransport doesn't trigger lifespan events, once per test session. ASGITransport doesn't trigger lifespan events,
so we call it explicitly. so we call it explicitly.
Depends on _tenant_guard so the suite refuses to start under the
production tenant before any initialization happens.
""" """
from src.core.startup import initialize_application from src.core.startup import initialize_application
asyncio.run(initialize_application()) asyncio.run(initialize_application())
View File
+219
View File
@@ -0,0 +1,219 @@
"""
Wire-level contract tests for external service boundaries.
Each test sends the raw request the application code sends (no client
wrappers, no mocks) and asserts on the response shape, so boundary
breakage is caught directly instead of surfacing as agent misbehavior.
Semantics:
- Service unreachable -> skip (an outage is not a contract violation)
- Service reachable but wrong response shape -> fail
Run with: make test-contracts
"""
import json
import httpx
import pytest
from src.core.config import config
OLLAMA = str(config.OLLAMA_HOST).rstrip("/")
QDRANT = f"http://{config.QDRANT_HOST}:{config.QDRANT_PORT}"
SEARXNG = str(config.SEARXNG_HOST).rstrip("/")
CALCULATOR_TOOL = {
"type": "function",
"function": {
"name": "calculator",
"description": "Evaluate a math expression",
"parameters": {
"type": "object",
"properties": {"expression": {"type": "string"}},
"required": ["expression"],
},
},
}
async def _get_or_skip(url: str, service: str, timeout: float = 5.0) -> httpx.Response:
"""GET a URL, skipping the test if the service is unreachable."""
try:
async with httpx.AsyncClient(timeout=timeout) as client:
return await client.get(url)
except httpx.TransportError as e:
pytest.skip(f"{service} unreachable at {url}: {e}")
async def _post_or_skip(
url: str, service: str, payload: dict, timeout: float, headers: dict | None = None
) -> httpx.Response:
"""POST a payload, skipping the test if the service is unreachable."""
try:
async with httpx.AsyncClient(timeout=timeout) as client:
return await client.post(url, json=payload, headers=headers)
except httpx.TransportError as e:
pytest.skip(f"{service} unreachable at {url}: {e}")
@pytest.mark.contract
class TestOllamaContract:
"""Boundary: Ollama native API and its OpenAI-compat layer."""
async def test_tags_lists_configured_model(self):
# Mirrors check_ollama_health()
response = await _get_or_skip(f"{OLLAMA}/api/tags", "ollama")
assert response.status_code == 200
names = [m["name"] for m in response.json()["models"]]
model = config.OLLAMA_DEFAULT_MODEL
assert model in names or f"{model}:latest" in names, (
f"{model} not pulled; available: {names}"
)
async def test_generate_returns_plain_text(self):
# Mirrors StewardAgent._call_ollama()
response = await _post_or_skip(
f"{OLLAMA}/api/generate",
"ollama",
{
"model": config.OLLAMA_DEFAULT_MODEL,
"prompt": "Reply with the single word: pong",
"stream": False,
"options": {"temperature": 0.3, "top_p": 0.9},
},
timeout=config.OLLAMA_TIMEOUT,
)
assert response.status_code == 200
assert response.json()["response"].strip()
async def test_openai_compat_tool_calling(self):
# Mirrors the request PydanticAI's OpenAIChatModel sends for the
# orchestration phase, including the extra_body tool_choice.
response = await _post_or_skip(
f"{OLLAMA}/v1/chat/completions",
"ollama",
{
"model": config.OLLAMA_DEFAULT_MODEL,
"messages": [
{"role": "user", "content": "What is 6 * 7? Use the calculator."}
],
"tools": [CALCULATOR_TOOL],
"tool_choice": "required",
"stream": False,
},
timeout=config.OLLAMA_TIMEOUT,
)
assert response.status_code == 200
message = response.json()["choices"][0]["message"]
tool_calls = message.get("tool_calls")
assert tool_calls, f"model answered in text instead of calling the tool: {message}"
assert tool_calls[0]["function"]["name"] == "calculator"
arguments = json.loads(tool_calls[0]["function"]["arguments"])
assert "expression" in arguments
@pytest.mark.contract
class TestAnthropicContract:
"""Boundary: Anthropic Messages API (the Claude fallback backend)."""
HEADERS_KEY = "anthropic-version"
def _headers(self) -> dict:
if not config.ANTHROPIC_API_KEY:
pytest.skip("ANTHROPIC_API_KEY not configured")
return {
"x-api-key": config.ANTHROPIC_API_KEY,
"anthropic-version": "2023-06-01",
}
async def test_minimal_message_accepted(self):
# Mirrors check_claude_health(): tiny request, no sampling params
response = await _post_or_skip(
"https://api.anthropic.com/v1/messages",
"anthropic",
{
"model": config.ANTHROPIC_MODEL,
"max_tokens": 1,
"messages": [{"role": "user", "content": "hi"}],
},
timeout=30.0,
headers=self._headers(),
)
assert response.status_code == 200, response.text
async def test_temperature_rejected(self):
# Pins the Claude Sonnet 5+ contract that broke the Steward:
# sampling parameters are rejected with a 400 (and not billed).
response = await _post_or_skip(
"https://api.anthropic.com/v1/messages",
"anthropic",
{
"model": config.ANTHROPIC_MODEL,
"max_tokens": 1,
"messages": [{"role": "user", "content": "hi"}],
"temperature": 0.3,
},
timeout=30.0,
headers=self._headers(),
)
assert response.status_code == 400
assert "temperature" in response.text
@pytest.mark.contract
class TestQdrantContract:
"""Boundary: Qdrant REST API (Biographer's vector memory)."""
async def test_collections_endpoint(self):
response = await _get_or_skip(f"{QDRANT}/collections", "qdrant")
assert response.status_code == 200
assert "collections" in response.json()["result"]
@pytest.mark.contract
class TestSearxngContract:
"""Boundary: SearXNG JSON search API (web search tool)."""
async def test_json_search(self):
response = await _get_or_skip(
f"{SEARXNG}/search?q=test&format=json", "searxng", timeout=config.SEARXNG_TIMEOUT
)
assert response.status_code == 200
assert "results" in response.json()
@pytest.mark.contract
class TestLibraryDeskContract:
"""Boundary: library-desk research API (the Librarian's backend)."""
async def test_health(self):
host = getattr(config, "LIBRARY_DESK_HOST", None)
if not host:
pytest.skip("LIBRARY_DESK_HOST not configured")
response = await _get_or_skip(f"{str(host).rstrip('/')}/health", "library-desk")
assert response.status_code == 200
@pytest.mark.contract
class TestRedisContract:
"""Boundary: Redis on the configured memory DB."""
async def test_roundtrip(self):
import redis.asyncio as redis
client = redis.Redis(
host=config.REDIS_HOST,
port=config.REDIS_PORT,
db=config.REDIS_MEMORY_DB,
socket_connect_timeout=3,
)
try:
await client.ping()
except Exception as e:
pytest.skip(f"redis unreachable: {e}")
try:
await client.set("contract-test-key", "ok", ex=30)
assert await client.get("contract-test-key") == b"ok"
await client.delete("contract-test-key")
finally:
await client.aclose()
+300
View File
@@ -0,0 +1,300 @@
"""
Tests for the tenant isolation guard.
Isolation is tenant-based: the production tenant ("jpmschweitzer") owns
real data in the shared services, and every non-production environment
must run under the reserved test tenant ("llm_tester") or an explicit
"test_"-prefixed namespace.
Guard matrix covered here: dev/test/prod x default/explicit user, at
both config level (effective_default_user) and request-context
resolution (get_user).
"""
import pytest
from pydantic import ValidationError
from src.core.config import (
PRODUCTION_TENANT,
TEST_TENANT,
Config,
Environment,
)
from src.core.context import RequestContext, get_user
def make_config(**overrides) -> Config:
"""Build a Config isolated from the local .env file."""
return Config(_env_file=None, **overrides)
@pytest.mark.unit
class TestEffectiveDefaultUserMatrix:
"""Config-level guard: effective_default_user per environment."""
# --- development ---
def test_dev_without_default_user_forces_test_tenant(self):
config = make_config(ENVIRONMENT=Environment.DEVELOPMENT)
assert config.effective_default_user == TEST_TENANT
assert config.tenant_forced is False
def test_dev_with_test_tenant_is_kept(self):
config = make_config(ENVIRONMENT=Environment.DEVELOPMENT, DEFAULT_USER=TEST_TENANT)
assert config.effective_default_user == TEST_TENANT
assert config.tenant_forced is False
def test_dev_with_test_prefixed_override_is_kept(self):
config = make_config(ENVIRONMENT=Environment.DEVELOPMENT, DEFAULT_USER="test_phase_b")
assert config.effective_default_user == "test_phase_b"
assert config.tenant_forced is False
def test_dev_with_misconfigured_user_is_forced_to_test_tenant(self):
config = make_config(ENVIRONMENT=Environment.DEVELOPMENT, DEFAULT_USER="alice")
assert config.effective_default_user == TEST_TENANT
assert config.tenant_forced is True
def test_dev_with_production_tenant_refuses_startup(self):
with pytest.raises(ValidationError) as exc_info:
make_config(
ENVIRONMENT=Environment.DEVELOPMENT,
DEFAULT_USER=PRODUCTION_TENANT,
)
assert "Refusing to start" in str(exc_info.value)
assert PRODUCTION_TENANT in str(exc_info.value)
@pytest.mark.parametrize(
"variant",
[
"JPMSchweitzer",
"JPMSCHWEITZER",
"jpmschweitzer.",
" jpmschweitzer",
"jpmschweitzer ",
"_jpmschweitzer_",
"jpmschweitzer!",
],
)
def test_dev_with_production_tenant_variant_refuses_startup(self, variant):
"""Sanitization collisions with the production tenant are refused too."""
with pytest.raises(ValidationError, match="Refusing to start"):
make_config(
ENVIRONMENT=Environment.DEVELOPMENT,
DEFAULT_USER=variant,
)
# --- testing ---
def test_testing_without_default_user_forces_test_tenant(self):
config = make_config(ENVIRONMENT=Environment.TESTING)
assert config.effective_default_user == TEST_TENANT
def test_testing_with_misconfigured_user_is_forced_to_test_tenant(self):
config = make_config(ENVIRONMENT=Environment.TESTING, DEFAULT_USER="bob")
assert config.effective_default_user == TEST_TENANT
assert config.tenant_forced is True
def test_testing_with_production_tenant_refuses_startup(self):
with pytest.raises(ValidationError, match="Refusing to start"):
make_config(
ENVIRONMENT=Environment.TESTING,
DEFAULT_USER=PRODUCTION_TENANT,
)
def test_testing_with_test_prefixed_override_is_kept(self):
config = make_config(ENVIRONMENT=Environment.TESTING, DEFAULT_USER="test_ci_run")
assert config.effective_default_user == "test_ci_run"
# --- production ---
def test_prod_without_default_user_uses_production_tenant(self):
config = make_config(ENVIRONMENT=Environment.PRODUCTION)
assert config.effective_default_user == PRODUCTION_TENANT
assert config.tenant_forced is False
def test_prod_with_explicit_production_tenant_is_kept(self):
config = make_config(ENVIRONMENT=Environment.PRODUCTION, DEFAULT_USER=PRODUCTION_TENANT)
assert config.effective_default_user == PRODUCTION_TENANT
def test_prod_with_explicit_other_user_is_kept(self):
config = make_config(ENVIRONMENT=Environment.PRODUCTION, DEFAULT_USER="household_guest")
assert config.effective_default_user == "household_guest"
assert config.tenant_forced is False
@pytest.mark.unit
class TestRequestContextGuard:
"""Request-context resolution guard: get_user() per environment."""
def _patch_environment(self, monkeypatch, environment: Environment):
from src.core import config as config_module
monkeypatch.setattr(config_module.config, "ENVIRONMENT", environment)
def test_dev_default_resolution_is_test_tenant(self, monkeypatch):
self._patch_environment(monkeypatch, Environment.DEVELOPMENT)
assert get_user() == TEST_TENANT
def test_dev_explicit_production_tenant_is_forced(self, monkeypatch):
self._patch_environment(monkeypatch, Environment.DEVELOPMENT)
with RequestContext(user=PRODUCTION_TENANT):
assert get_user() == TEST_TENANT
def test_testing_explicit_production_tenant_is_forced(self, monkeypatch):
self._patch_environment(monkeypatch, Environment.TESTING)
with RequestContext(user=PRODUCTION_TENANT):
assert get_user() == TEST_TENANT
@pytest.mark.parametrize(
"variant",
[
"JPMSchweitzer",
"JPMSCHWEITZER",
"jpmschweitzer.",
" jpmschweitzer",
"jpmschweitzer ",
"_jpmschweitzer_",
"jpmschweitzer!",
],
)
@pytest.mark.parametrize(
"environment", [Environment.DEVELOPMENT, Environment.TESTING]
)
def test_production_tenant_sanitization_variants_are_forced(
self, monkeypatch, environment, variant
):
"""
Any raw user that sanitizes to the production tenant would resolve
to the production namespaces (memories_jpmschweitzer,
session:jpmschweitzer:*) - the guard must force it to the test
tenant in non-production environments.
"""
from src.core.multi_tenancy import get_memory_collection_name
self._patch_environment(monkeypatch, environment)
with RequestContext(user=variant):
effective = get_user()
assert effective == TEST_TENANT
assert (
get_memory_collection_name(effective)
!= get_memory_collection_name(PRODUCTION_TENANT)
)
def test_dev_non_colliding_user_is_not_forced(self, monkeypatch):
"""A user that sanitizes to a different namespace passes through."""
self._patch_environment(monkeypatch, Environment.DEVELOPMENT)
with RequestContext(user="jpm.schweitzer"):
# sanitizes to jpm_schweitzer != jpmschweitzer
assert get_user() == "jpm.schweitzer"
def test_dev_explicit_other_user_passes_through(self, monkeypatch):
self._patch_environment(monkeypatch, Environment.DEVELOPMENT)
with RequestContext(user="testuser"):
assert get_user() == "testuser"
def test_prod_explicit_production_tenant_passes_through(self, monkeypatch):
self._patch_environment(monkeypatch, Environment.PRODUCTION)
with RequestContext(user=PRODUCTION_TENANT):
assert get_user() == PRODUCTION_TENANT
def test_prod_explicit_other_user_passes_through(self, monkeypatch):
self._patch_environment(monkeypatch, Environment.PRODUCTION)
with RequestContext(user="alice"):
assert get_user() == "alice"
@pytest.mark.unit
class TestSuiteRunsUnderTestTenant:
"""
The live test session itself must resolve to the test tenant.
The session guard in tests/conftest.py hard-fails the suite when the
effective tenant is the production tenant; these tests assert the
namespaces every shared-service touch would use (Qdrant memories
collection, Redis session keys) are the llm_tester ones.
"""
def test_effective_tenant_is_not_production(self):
from src.core.context import get_default_user
assert get_default_user() != PRODUCTION_TENANT
def test_effective_tenant_is_the_reserved_test_tenant(self):
from src.core.context import get_default_user
assert get_default_user() == TEST_TENANT
def test_memories_collection_namespace_is_test_tenant(self):
from src.core.context import get_default_user
from src.core.multi_tenancy import get_memory_collection_name
assert (
get_memory_collection_name(get_default_user())
== f"memories_{TEST_TENANT}"
)
def test_redis_session_namespace_is_test_tenant(self):
from src.core.context import get_default_user
from src.core.multi_tenancy import get_session_key
key = get_session_key(get_default_user(), "conv_test")
assert key.startswith(f"session:{TEST_TENANT}:")
@pytest.mark.unit
class TestStartupTenantGuardLog:
"""One loud startup log line states the effective tenant."""
def test_non_production_logs_forced_tenant(self, monkeypatch):
from src.core import startup as startup_module
events = []
class _Recorder:
def warning(self, event, **kw):
events.append((event, kw))
def info(self, event, **kw):
events.append((event, kw))
monkeypatch.setattr(startup_module, "logger", _Recorder())
monkeypatch.setattr(startup_module.config, "ENVIRONMENT", Environment.DEVELOPMENT)
startup_module.log_tenant_guard()
assert events == [
(
"tenant_guard_active",
{
"environment": "development",
"forced_tenant": TEST_TENANT,
"default_user_overridden": startup_module.config.tenant_forced,
"configured_default_user": startup_module.config.DEFAULT_USER,
},
)
]
def test_production_logs_production_tenant(self, monkeypatch):
from src.core import startup as startup_module
events = []
class _Recorder:
def warning(self, event, **kw):
events.append(("warning", event, kw))
def info(self, event, **kw):
events.append(("info", event, kw))
monkeypatch.setattr(startup_module, "logger", _Recorder())
monkeypatch.setattr(startup_module.config, "ENVIRONMENT", Environment.PRODUCTION)
startup_module.log_tenant_guard()
assert events == [
(
"info",
"tenant_guard_production",
{"environment": "production", "tenant": PRODUCTION_TENANT},
)
]
+8 -2
View File
@@ -26,12 +26,18 @@ from typing import AsyncGenerator
from dataclasses import dataclass from dataclasses import dataclass
from src.core.config import PRODUCTION_TENANT, TEST_TENANT
# Test configuration # Test configuration
BASE_URL = "http://localhost:8777" BASE_URL = "http://localhost:8777"
QDRANT_URL = "http://localhost:6333" QDRANT_URL = "http://localhost:6333"
API_TIMEOUT = 120.0 # LLM calls can be slow API_TIMEOUT = 120.0 # LLM calls can be slow
TEST_USER = "llm_tester" # All e2e writes to shared services go to the reserved test tenant's
# namespaces (Qdrant memories_llm_tester, wiki llm_tester scope) - never
# the production tenant's.
TEST_USER = TEST_TENANT
TEST_COLLECTION = f"memories_{TEST_USER}" TEST_COLLECTION = f"memories_{TEST_USER}"
assert TEST_USER != PRODUCTION_TENANT
@dataclass @dataclass
@@ -986,7 +992,7 @@ class TestUserContextIsolation:
in_test_collection = any(unique_value in v for v in test_values) in_test_collection = any(unique_value in v for v in test_values)
# Check production collection (should NOT be there) # Check production collection (should NOT be there)
prod_collection = "memories_jpmschweitzer" prod_collection = f"memories_{PRODUCTION_TENANT}"
if await qdrant.collection_exists(prod_collection): if await qdrant.collection_exists(prod_collection):
prod_points = await qdrant.scroll_points(prod_collection) prod_points = await qdrant.scroll_points(prod_collection)
prod_values = [str(p.get("payload", {})) for p in prod_points] prod_values = [str(p.get("payload", {})) for p in prod_points]
+54
View File
@@ -0,0 +1,54 @@
"""
Tests for TatlockOllamaProvider configuration.
The AsyncOpenAI client must carry an explicit timeout from
config.OLLAMA_TIMEOUT instead of the SDK default (~600s), so a stuck
LLM call cannot consume the whole delegation budget.
"""
import pytest
from src.core.config import config
from src.ollama.provider import TatlockOllamaProvider, _sanitize_messages
@pytest.mark.unit
class TestProviderTimeout:
"""Timeout configuration on the underlying AsyncOpenAI client."""
def test_openai_client_timeout_from_config(self):
provider = TatlockOllamaProvider(base_url="http://localhost:11434/v1")
assert provider._openai_client.timeout == float(config.OLLAMA_TIMEOUT)
def test_timeout_is_not_sdk_default(self):
provider = TatlockOllamaProvider(base_url="http://localhost:11434/v1")
# The OpenAI SDK defaults to 600s; the configured cap must win
assert provider._openai_client.timeout < 600
@pytest.mark.unit
class TestMessageSanitization:
"""Null content sanitization for Ollama compatibility."""
def test_null_content_with_tool_calls_becomes_empty_string(self):
messages = [
{
"role": "assistant",
"content": None,
"tool_calls": [{"id": "call_1", "type": "function"}],
}
]
sanitized = _sanitize_messages(messages)
assert sanitized[0]["content"] == ""
def test_regular_messages_unchanged(self):
messages = [
{"role": "user", "content": "Hello"},
{"role": "assistant", "content": "Good day, sir."},
]
assert _sanitize_messages(messages) == messages
@@ -0,0 +1,185 @@
"""
Tests for real-time think message streaming and context plumbing in
the direct delegation paths.
_stream_direct_delegation must be an async generator that yields the
"start" think message BEFORE the expert runs (so 'Allow me to consult
the archives, sir.' streams while research is in flight), and both
direct delegation paths must pass trimmed conversation history as
expert context.
"""
from types import SimpleNamespace
from unittest.mock import AsyncMock, patch
import pytest
from src.agents.delegation import DelegationResult
from src.responses.streaming import (
ReasoningSummaryDelta,
ReasoningSummaryDone,
StreamingCoordinator,
)
HISTORY = [
{"role": "user", "content": "Tell me about my homelab wiki"},
{"role": "assistant", "content": "It documents your services, sir."},
]
def _librarian_result(output: str = "Findings.") -> DelegationResult:
return DelegationResult(
expert_name="librarian",
task="task",
success=True,
output=output,
)
@pytest.mark.unit
class TestStreamDirectDelegation:
"""Real-time streaming behavior of _stream_direct_delegation."""
@pytest.mark.asyncio
async def test_start_think_streams_before_research_runs(self):
coordinator = StreamingCoordinator()
tracker = AsyncMock()
results: dict = {}
with patch(
"src.agents.delegation.delegate_to_librarian",
new_callable=AsyncMock,
return_value=_librarian_result(),
) as mock_delegate:
gen = coordinator._stream_direct_delegation(
user_message="Search for Docker info",
recommendation=SimpleNamespace(
recommended_capabilities=["librarian"]
),
tracker=tracker,
conversation_id="conv_1",
conversation_history=HISTORY,
results=results,
)
# First event: the start think message, BEFORE any research
first = await gen.__anext__()
assert isinstance(first, ReasoningSummaryDelta)
assert first.delta.strip() != ""
assert mock_delegate.await_count == 0, (
"start think message must stream before the expert runs"
)
second = await gen.__anext__()
assert isinstance(second, ReasoningSummaryDone)
assert mock_delegate.await_count == 0
# Third event: completion message - research has now run
third = await gen.__anext__()
assert isinstance(third, ReasoningSummaryDelta)
assert mock_delegate.await_count == 1
remaining = [event async for event in gen]
assert any(isinstance(e, ReasoningSummaryDone) for e in remaining)
# Results dict is populated for Phase 2 synthesis
assert results["expert_results"] == {"librarian": "Findings."}
assert results["tools_called"] == ["delegate_to_librarian"]
assert len(results["think_messages"]) == 2
@pytest.mark.asyncio
async def test_conversation_history_passed_as_context(self):
coordinator = StreamingCoordinator()
tracker = AsyncMock()
with patch(
"src.agents.delegation.delegate_to_librarian",
new_callable=AsyncMock,
return_value=_librarian_result(),
) as mock_delegate:
events = [
event
async for event in coordinator._stream_direct_delegation(
user_message="And what services does it list?",
recommendation=SimpleNamespace(
recommended_capabilities=["librarian"]
),
tracker=tracker,
conversation_id="conv_1",
conversation_history=HISTORY,
results={},
)
]
assert events, "generator must yield think events"
context = mock_delegate.await_args.kwargs["context"]
assert "Tell me about my homelab wiki" in context
assert "It documents your services, sir." in context
@pytest.mark.asyncio
async def test_failed_delegation_streams_error_think(self):
coordinator = StreamingCoordinator()
tracker = AsyncMock()
results: dict = {}
failed = DelegationResult(
expert_name="librarian",
task="task",
success=False,
output="I'm afraid the archives proved difficult to access.",
error="The Librarian was unable to complete the task.",
)
with patch(
"src.agents.delegation.delegate_to_librarian",
new_callable=AsyncMock,
return_value=failed,
):
events = [
event
async for event in coordinator._stream_direct_delegation(
user_message="Search for Docker info",
recommendation=SimpleNamespace(
recommended_capabilities=["librarian"]
),
tracker=tracker,
conversation_id="conv_1",
results=results,
)
]
assert results["tools_called"] == []
# Expert result carries the curated user-safe sentence
assert "archives" in results["expert_results"]["librarian"]
deltas = [e.delta for e in events if isinstance(e, ReasoningSummaryDelta)]
assert len(deltas) == 2 # start + error think messages
@pytest.mark.unit
class TestServiceDelegationContext:
"""The non-streaming direct delegation path passes trimmed history."""
@pytest.mark.asyncio
async def test_direct_delegation_with_results_passes_context(self):
from src.responses.service import _direct_delegation_with_results
tracker = AsyncMock()
with patch(
"src.agents.delegation.delegate_to_librarian",
new_callable=AsyncMock,
return_value=_librarian_result(),
) as mock_delegate:
results = await _direct_delegation_with_results(
user_message="And what services does it list?",
recommendation=SimpleNamespace(
recommended_capabilities=["librarian"]
),
tracker=tracker,
conversation_id="conv_1",
conversation_history=HISTORY,
)
context = mock_delegate.await_args.kwargs["context"]
assert "Tell me about my homelab wiki" in context
assert results["expert_results"]["librarian"] == "Findings."