Compare commits

...
5 Commits
Author SHA1 Message Date
jpmschweitzerandClaude Fable 5 8808880145 fix(tests): T-7 — the unit suite is hermetic, the flake class is gone
Three flakes in one day, never a captured name — the name finally
came from a discriminating run instead of patience: pointing
OLLAMA_HOST at a dead port failed exactly eight tests, all in
tests/agents/test_tatlock_agent.py, all already marked integration,
all running in the unit gate anyway because make test excluded by
directory and they live outside the ignored directories. Eight tests
doing ~110 seconds of real LLM inference against the production
wrapper on every unit run, failing whenever the backend was
mid-deploy — which this week it constantly was.

The second coupling sat in the session init: conftest ran the real
health probes, an HTTP round trip to whatever answers behind
OLLAMA_HOST plus a real Anthropic API call whenever the dev .env
carries a key, so the cached backend globals followed the network of
the moment. The probes are stubbed to the deterministic local-first
state; tests needing other states patch the globals themselves, as
the selector tests always did.

make test now enforces -m "not integration" alongside the directory
ignores, and the acceptance is blunt: 679 passed in ~12 s, identical
against a dead backend and no API key — down from ~129 s of
infrastructure-coupled runtime. Both prongs mutation-checked: the
gate removed fails eight against a dead backend; the stub removed
fails the new session-globals test. T-8 files the orphaned
tests/integration directory that no make target runs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-13 11:29:10 +02:00
jpmschweitzerandClaude Fable 5 b62a6f4672 release v2.7.2
Build and Push / release (push) Successful in 4s
Build and Push / build (push) Successful in 1m51s
Production logs INFO to stdout: docker logs tatlock becomes a real
diagnostic surface for the first time.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-13 11:11:27 +02:00
jpmschweitzerandClaude Fable 5 aace128ef9 fix(logging): T-5 — production logs INFO to stdout
docker logs tatlock has been empty for months and the /app/logs
mount untouched since 2025-12. Not a broken handler: the stdout
StreamHandler was correct all along, but the production default
level was WARNING ("minimal noise") and a healthy service warns
roughly never — startup, backend flavor detection and the wrapper's
balancing/compaction signals were all suppressed, which is exactly
what made the serving cutover unverifiable from logs on 2026-09-12.

Production now defaults to INFO; LOG_LEVEL stays the override for
when noise is ever the real problem; development keeps DEBUG. The
mount was always decorative — no file handler exists, docker logs is
the log. Three tests pin the level table, the WARNING-revert
mutation shown to fail its test.

Also filed T-7: the full suite flaked once and passed on rerun for
the second time — next occurrence keeps the failing run's output.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-13 11:00:50 +02:00
jpmschweitzerandClaude Fable 5 cbec26835d release v2.7.1
Build and Push / release (push) Successful in 3s
Build and Push / build (push) Successful in 1m47s
The T-6 wiring fix reaches production: the sanitized client is in
the request path for the first time — null-content sanitizer,
OLLAMA_TIMEOUT cap and wrapper signal reading all become real.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-13 10:33:13 +02:00
jpmschweitzerandClaude Fable 5 1139d8ca38 fix(provider): T-6 — the sanitized client is wired for real
The provider assigned self._openai_client, an attribute nobody reads
— OllamaProvider.client serves self._client — so the null-content
sanitizer, the wrapper signal reading and the OLLAMA_TIMEOUT cap were
all silently dead in production (sessions unaffected; they ride
ModelSettings). Found via webber's identical bug on 2026-09-13.

Three fixes travel together, because wiring alone would have traded a
silent bypass for a loud crash: the client goes through the official
openai_client constructor parameter; the parent-chat lookup survives
openai's property/cached_property descriptor change (this venv's
2.11.0 is the latter — the old hardcoded .fget would have raised
AttributeError on the first wired completion; ported from webber's
2026-08-11 fix); and the balancing/compaction signals are read from
the response body's model_extra inside create() — the httpx
event-hook approach demonstrably never fires under the openai SDK.

The tests now assert through provider.client, the property
pydantic_ai actually reads. The old ones asserted on the dead
attribute and stayed green for the entire bypass — a check whose
subject is not the real thing cannot fail for the real reason
(workspace D-24's shape). Three mutations shown to fail their tests:
the dead-attribute wiring, the .fget-only lookup, and removing the
signal read from the request path. 683 tests green; a live dev turn
through the wired client answers normally.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-13 10:27:21 +02:00
12 changed files with 364 additions and 41 deletions
+12
View File
@@ -1,2 +1,14 @@
INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G99D07EM5DZWTS0WD9V8PSKM', 'status', 'backlog', 'in_progress', NULL, '2026-09-12 13:50:48', '2026-09-12 13:50:48.289', '2026-09-12 13:50:48.289', NULL, '2e88d155325b3e704787f48b2aa97591', 2) ON CONFLICT(hash) DO NOTHING;
INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G99D07EM5DZWTS0WD9V8PSKM', 'status', 'in_progress', 'done', NULL, '2026-09-12 14:13:29', '2026-09-12 14:13:29.027', '2026-09-12 14:13:29.027', NULL, '5b9dd9a62002db0fb4dc58f7f9ffcbb6', 2) ON CONFLICT(hash) DO NOTHING;
INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G9KH87TRDGW45GMDWVDRPABR', 'description', 'src/ollama/provider.py sets self._openai_client after super().__init__, but OllamaProvider.client serves self._client (pydantic_ai 1.27''s official openai_client= constructor param is the right wiring — webber hit the identical bug and fixed it 2026-09-13, exposed by the wrapper 503ing a session-less request). Silently dead in production as a result: the content-null sanitizer (moot on llama-server, live hazard if Ollama ever returns), the T-4 x-boilerroom-* signal hook (never logs), and the OLLAMA_TIMEOUT=120 cap (SDK default ~600s applies). Sessions were unaffected — they ride ModelSettings extra_body. Fix like webber: pass _SanitizedAsyncOpenAI through openai_client=, add a wiring test asserting provider.client is the sanitized type, then verify balancing/compaction logs actually appear.', 'src/ollama/provider.py sets self._openai_client after super().__init__, but OllamaProvider.client serves self._client (pydantic_ai 1.27''s official openai_client= constructor param is the right wiring — webber hit the identical bug and fixed it 2026-09-13, exposed by the wrapper 503ing a session-less request). Silently dead in production as a result: the content-null sanitizer (moot on llama-server, live hazard if Ollama ever returns), the T-4 x-boilerroom-* signal hook (never logs), and the OLLAMA_TIMEOUT=120 cap (SDK default ~600s applies). Sessions were unaffected — they ride ModelSettings extra_body. Fix like webber: pass _SanitizedAsyncOpenAI through openai_client=, add a wiring test asserting provider.client is the sanitized type, then verify balancing/compaction logs actually appear.
Second finding 2026-09-13, from webber''s twin fix: an httpx event-hook on the client (the T-4 signal-reading approach) never fires under the openai SDK even when correctly wired — webber verified headers arriving while the hook stayed silent. When rewiring the client, read the signals from the parsed body''s model_extra in the sanitized create() instead (the wrapper injects balancing/compaction_due into non-streamed JSON bodies; openai pydantic models retain unknown fields). The steward''s raw-call body read is unaffected and already correct.', NULL, '2026-09-13 08:00:17', '2026-09-13 08:00:17.002', '2026-09-13 08:00:17.002', NULL, '6fb4cc1d7d7222e3d570cd8c7fed4fd9', 2) ON CONFLICT(hash) DO NOTHING;
INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G9KH87TRDGW45GMDWVDRPABR', 'status', 'backlog', 'in_progress', NULL, '2026-09-13 08:19:35', '2026-09-13 08:19:35.028', '2026-09-13 08:19:35.028', NULL, '5bcf19379361a2f766cd4787ff734611', 2) ON CONFLICT(hash) DO NOTHING;
INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G9KH87TRDGW45GMDWVDRPABR', 'status', 'in_progress', 'done', NULL, '2026-09-13 08:27:21', '2026-09-13 08:27:21.779', '2026-09-13 08:27:21.779', NULL, '1c59422336b88a76c6f1c3720916d7d5', 2) ON CONFLICT(hash) DO NOTHING;
INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G99D07Z3FZY8EBT9PJ1RCGHW', 'status', 'backlog', 'in_progress', NULL, '2026-09-13 08:48:55', '2026-09-13 08:48:55.587', '2026-09-13 08:48:55.587', NULL, 'ca4f8fb42c961b0002aa87bb7b1d5af5', 2) ON CONFLICT(hash) DO NOTHING;
INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G99D07Z3FZY8EBT9PJ1RCGHW', 'status', 'in_progress', 'done', NULL, '2026-09-13 09:00:50', '2026-09-13 09:00:50.006', '2026-09-13 09:00:50.006', NULL, '5ea799ee4815e42ede80c3df4c81b28c', 2) ON CONFLICT(hash) DO NOTHING;
INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G9KZSCFR79G4DFCT4E5W5CAM', 'description', 'Observed 2026-09-13 twice: before the v2.7.0 release audit and again during T-5 — make test exits 1 with a failure that does not reproduce on the immediate rerun, and both times the failing test''s name went uncaptured because the diagnostic rerun passed. Next occurrence: keep the FULL output of the failing run (make test 2>&1 | tee /tmp/flake.log) before rerunning, then pin the named test. Suspects worth checking when a name exists: order-dependent state in the session-scoped app init (conftest asyncio.run + cached health globals), port/redis contention with the live containers, and the coverage plugin''s tmp handling.', 'Observed 2026-09-13 twice: before the v2.7.0 release audit and again during T-5 — make test exits 1 with a failure that does not reproduce on the immediate rerun, and both times the failing test''s name went uncaptured because the diagnostic rerun passed. Next occurrence: keep the FULL output of the failing run (make test 2>&1 | tee /tmp/flake.log) before rerunning, then pin the named test. Suspects worth checking when a name exists: order-dependent state in the session-scoped app init (conftest asyncio.run + cached health globals), port/redis contention with the live containers, and the coverage plugin''s tmp handling.
Third occurrence 2026-09-13, during the v2.7.2 release gate — and two process lessons captured the hard way: (1) the gate run was piped through grep, which swallowed make''s exit code, so the tag went out on a red run (release content was safe — version+changelog only on green-tested aace128 — but the gate did not gate); release runs must let make test''s exit code stand alone. (2) The failing name went uncaptured AGAIN because the tee capture only started on the rerun, which passed. The capture must wrap the FIRST gate run, every time, not begin after a failure.', NULL, '2026-09-13 09:14:10', '2026-09-13 09:14:10.345', '2026-09-13 09:14:10.345', NULL, '22909a81b656320b4a39405165f48612', 2) ON CONFLICT(hash) DO NOTHING;
INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G9KZSCFR79G4DFCT4E5W5CAM', 'status', 'backlog', 'in_progress', NULL, '2026-09-13 09:16:08', '2026-09-13 09:16:08.928', '2026-09-13 09:16:08.928', NULL, '34276614ec535d50d5fa6ddf9f5e0da6', 2) ON CONFLICT(hash) DO NOTHING;
INSERT INTO ticket_history (ticket_record_id, field, old_value, new_value, changed_by, changed_at, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G9KZSCFR79G4DFCT4E5W5CAM', 'status', 'in_progress', 'done', NULL, '2026-09-13 09:29:10', '2026-09-13 09:29:10.192', '2026-09-13 09:29:10.192', NULL, 'e040af9004e4510c4f8a9e9cacee9a8c', 2) ON CONFLICT(hash) DO NOTHING;
+3
View File
@@ -1,3 +1,6 @@
INSERT INTO ticket_idmap (record_id, ticket_id, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G94KC9XM3PW91C6B7VJA6VRM', 'T-3', '2026-09-11 21:09:10.267', '2026-09-11 21:09:10.267', NULL, 'a1ccf4d118877c230dffb961a7531cb4', 2) ON CONFLICT(record_id) DO UPDATE SET ticket_id=excluded.ticket_id, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= ticket_idmap.updated_at;
INSERT INTO ticket_idmap (record_id, ticket_id, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G99D07EM5DZWTS0WD9V8PSKM', 'T-4', '2026-09-12 08:20:21.494', '2026-09-12 08:20:21.494', NULL, '5b28e07bdc3be1f9ed0ed71d2da60c72', 2) ON CONFLICT(record_id) DO UPDATE SET ticket_id=excluded.ticket_id, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= ticket_idmap.updated_at;
INSERT INTO ticket_idmap (record_id, ticket_id, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G99D07Z3FZY8EBT9PJ1RCGHW', 'T-5', '2026-09-12 08:20:21.625', '2026-09-12 08:20:21.625', NULL, '0a417012779f612a4dad1a7dbd0787e3', 2) ON CONFLICT(record_id) DO UPDATE SET ticket_id=excluded.ticket_id, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= ticket_idmap.updated_at;
INSERT INTO ticket_idmap (record_id, ticket_id, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G9KH87TRDGW45GMDWVDRPABR', 'T-6', '2026-09-13 07:57:01.782', '2026-09-13 07:57:01.782', NULL, 'd6c184ba6d24e03c0dc77064bf71fead', 2) ON CONFLICT(record_id) DO UPDATE SET ticket_id=excluded.ticket_id, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= ticket_idmap.updated_at;
INSERT INTO ticket_idmap (record_id, ticket_id, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G9KZSCFR79G4DFCT4E5W5CAM', 'T-7', '2026-09-13 09:00:32.255', '2026-09-13 09:00:32.255', NULL, 'b0b0a4c3ccaabeb58ad1958e7c2c2602', 2) ON CONFLICT(record_id) DO UPDATE SET ticket_id=excluded.ticket_id, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= ticket_idmap.updated_at;
INSERT INTO ticket_idmap (record_id, ticket_id, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G9M68FC27YHQSP1R9SQQ87P0', 'T-8', '2026-09-13 09:28:48.737', '2026-09-13 09:28:48.737', NULL, 'd3a29143f5496ea4b4ee8a9a834c5290', 2) ON CONFLICT(record_id) DO UPDATE SET ticket_id=excluded.ticket_id, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= ticket_idmap.updated_at;
+23
View File
@@ -3,3 +3,26 @@ INSERT INTO tickets (record_id, type, parent_record_id, title, description, stat
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G99D07Z3FZY8EBT9PJ1RCGHW', 'bug', NULL, 'The container logs nowhere: stdout empty, /app/logs mount untouched', 'docker logs tatlock is 0 lines and the mounted ~/docker-data/tatlock/logs has been empty since 2025-12 — the running app writes logs to neither, which made the 2026-09-12 wrapper-cutover verification blind (flavor detection had to be proven behaviorally). Dev mode logs to build/logs/server.log via make run; production should log to stdout for docker logs at minimum.', 'backlog', 'medium', NULL, NULL, NULL, '2026-09-12 08:20:21.624', '2026-09-12 08:20:21.624', NULL, '3fd294efc9afa857aef15810d9003298', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at;
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G99D07EM5DZWTS0WD9V8PSKM', 'story', NULL, 'Adopt boilerroom named sessions with the usage ranking', 'Replace raw id_slot pinning (BACKEND_SLOT_PINNING) with the wrapper''s session extension fields, now that tatlock routes through boilerroom:8602 (since 2026-09-12). Ranking by likelihood of use, decided 2026-09-12 — lower eviction_order parks sooner: tatlock pipeline phases highest (eviction_order 40; e.g. sessions tatlock-steward/tatlock-orchestrate/tatlock-synthesize), librarian-routed calls as session librarian (30), webber will sit at 20 (workspace T-137), Open WebUI stays session-less by design (idle slots only, can never evict — and it is being deprecated for an own-webui build). Read the x-boilerroom-balancing and compaction_due signals; the compaction endpoint expects the client transcript (tatlock''s Redis/Qdrant context is the natural producer — boilerroom D-5 names tatlock the first producer).', 'in_progress', 'high', NULL, NULL, NULL, '2026-09-12 08:20:21.493', '2026-09-12 13:50:48.289', NULL, '62abeb4da2f1f7aedb8d92884e7e148c', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at;
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G99D07EM5DZWTS0WD9V8PSKM', 'story', NULL, 'Adopt boilerroom named sessions with the usage ranking', 'Replace raw id_slot pinning (BACKEND_SLOT_PINNING) with the wrapper''s session extension fields, now that tatlock routes through boilerroom:8602 (since 2026-09-12). Ranking by likelihood of use, decided 2026-09-12 — lower eviction_order parks sooner: tatlock pipeline phases highest (eviction_order 40; e.g. sessions tatlock-steward/tatlock-orchestrate/tatlock-synthesize), librarian-routed calls as session librarian (30), webber will sit at 20 (workspace T-137), Open WebUI stays session-less by design (idle slots only, can never evict — and it is being deprecated for an own-webui build). Read the x-boilerroom-balancing and compaction_due signals; the compaction endpoint expects the client transcript (tatlock''s Redis/Qdrant context is the natural producer — boilerroom D-5 names tatlock the first producer).', 'done', 'high', NULL, NULL, NULL, '2026-09-12 08:20:21.493', '2026-09-12 14:13:29.027', NULL, '89ce1be323ebb26f00c63569f197cab1', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at;
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G9KH87TRDGW45GMDWVDRPABR', 'bug', NULL, 'The sanitized OpenAI client is never used: the override pokes an attribute nobody reads', 'src/ollama/provider.py sets self._openai_client after super().__init__, but OllamaProvider.client serves self._client (pydantic_ai 1.27''s official openai_client= constructor param is the right wiring — webber hit the identical bug and fixed it 2026-09-13, exposed by the wrapper 503ing a session-less request). Silently dead in production as a result: the content-null sanitizer (moot on llama-server, live hazard if Ollama ever returns), the T-4 x-boilerroom-* signal hook (never logs), and the OLLAMA_TIMEOUT=120 cap (SDK default ~600s applies). Sessions were unaffected — they ride ModelSettings extra_body. Fix like webber: pass _SanitizedAsyncOpenAI through openai_client=, add a wiring test asserting provider.client is the sanitized type, then verify balancing/compaction logs actually appear.', 'backlog', 'high', NULL, NULL, NULL, '2026-09-13 07:57:01.782', '2026-09-13 07:57:01.782', NULL, '5e8ea3bf2d18ade9d17f0044ae1ff96b', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at;
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G9KH87TRDGW45GMDWVDRPABR', 'bug', NULL, 'The sanitized OpenAI client is never used: the override pokes an attribute nobody reads', 'src/ollama/provider.py sets self._openai_client after super().__init__, but OllamaProvider.client serves self._client (pydantic_ai 1.27''s official openai_client= constructor param is the right wiring — webber hit the identical bug and fixed it 2026-09-13, exposed by the wrapper 503ing a session-less request). Silently dead in production as a result: the content-null sanitizer (moot on llama-server, live hazard if Ollama ever returns), the T-4 x-boilerroom-* signal hook (never logs), and the OLLAMA_TIMEOUT=120 cap (SDK default ~600s applies). Sessions were unaffected — they ride ModelSettings extra_body. Fix like webber: pass _SanitizedAsyncOpenAI through openai_client=, add a wiring test asserting provider.client is the sanitized type, then verify balancing/compaction logs actually appear.
Second finding 2026-09-13, from webber''s twin fix: an httpx event-hook on the client (the T-4 signal-reading approach) never fires under the openai SDK even when correctly wired — webber verified headers arriving while the hook stayed silent. When rewiring the client, read the signals from the parsed body''s model_extra in the sanitized create() instead (the wrapper injects balancing/compaction_due into non-streamed JSON bodies; openai pydantic models retain unknown fields). The steward''s raw-call body read is unaffected and already correct.', 'backlog', 'high', NULL, NULL, NULL, '2026-09-13 07:57:01.782', '2026-09-13 08:00:17.002', NULL, '755abc3689e8ff4fc511deb164137fe5', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at;
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G9KH87TRDGW45GMDWVDRPABR', 'bug', NULL, 'The sanitized OpenAI client is never used: the override pokes an attribute nobody reads', 'src/ollama/provider.py sets self._openai_client after super().__init__, but OllamaProvider.client serves self._client (pydantic_ai 1.27''s official openai_client= constructor param is the right wiring — webber hit the identical bug and fixed it 2026-09-13, exposed by the wrapper 503ing a session-less request). Silently dead in production as a result: the content-null sanitizer (moot on llama-server, live hazard if Ollama ever returns), the T-4 x-boilerroom-* signal hook (never logs), and the OLLAMA_TIMEOUT=120 cap (SDK default ~600s applies). Sessions were unaffected — they ride ModelSettings extra_body. Fix like webber: pass _SanitizedAsyncOpenAI through openai_client=, add a wiring test asserting provider.client is the sanitized type, then verify balancing/compaction logs actually appear.
Second finding 2026-09-13, from webber''s twin fix: an httpx event-hook on the client (the T-4 signal-reading approach) never fires under the openai SDK even when correctly wired — webber verified headers arriving while the hook stayed silent. When rewiring the client, read the signals from the parsed body''s model_extra in the sanitized create() instead (the wrapper injects balancing/compaction_due into non-streamed JSON bodies; openai pydantic models retain unknown fields). The steward''s raw-call body read is unaffected and already correct.', 'in_progress', 'high', NULL, NULL, NULL, '2026-09-13 07:57:01.782', '2026-09-13 08:19:35.028', NULL, '5af3c96930a30bbe4bb64d13ddbc509e', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at;
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G9KH87TRDGW45GMDWVDRPABR', 'bug', NULL, 'The sanitized OpenAI client is never used: the override pokes an attribute nobody reads', 'src/ollama/provider.py sets self._openai_client after super().__init__, but OllamaProvider.client serves self._client (pydantic_ai 1.27''s official openai_client= constructor param is the right wiring — webber hit the identical bug and fixed it 2026-09-13, exposed by the wrapper 503ing a session-less request). Silently dead in production as a result: the content-null sanitizer (moot on llama-server, live hazard if Ollama ever returns), the T-4 x-boilerroom-* signal hook (never logs), and the OLLAMA_TIMEOUT=120 cap (SDK default ~600s applies). Sessions were unaffected — they ride ModelSettings extra_body. Fix like webber: pass _SanitizedAsyncOpenAI through openai_client=, add a wiring test asserting provider.client is the sanitized type, then verify balancing/compaction logs actually appear.
Second finding 2026-09-13, from webber''s twin fix: an httpx event-hook on the client (the T-4 signal-reading approach) never fires under the openai SDK even when correctly wired — webber verified headers arriving while the hook stayed silent. When rewiring the client, read the signals from the parsed body''s model_extra in the sanitized create() instead (the wrapper injects balancing/compaction_due into non-streamed JSON bodies; openai pydantic models retain unknown fields). The steward''s raw-call body read is unaffected and already correct.', 'done', 'high', NULL, NULL, NULL, '2026-09-13 07:57:01.782', '2026-09-13 08:27:21.778', NULL, 'ce3151ff27c70a1028ba7c03b07119a2', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at;
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G99D07Z3FZY8EBT9PJ1RCGHW', 'bug', NULL, 'The container logs nowhere: stdout empty, /app/logs mount untouched', 'docker logs tatlock is 0 lines and the mounted ~/docker-data/tatlock/logs has been empty since 2025-12 — the running app writes logs to neither, which made the 2026-09-12 wrapper-cutover verification blind (flavor detection had to be proven behaviorally). Dev mode logs to build/logs/server.log via make run; production should log to stdout for docker logs at minimum.', 'in_progress', 'medium', NULL, NULL, NULL, '2026-09-12 08:20:21.624', '2026-09-13 08:48:55.586', NULL, '4770f6c6759cb82d825eb550dd4e42e2', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at;
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G9KZSCFR79G4DFCT4E5W5CAM', 'bug', NULL, 'The full suite flakes: one run fails, the rerun passes, twice now', 'Observed 2026-09-13 twice: before the v2.7.0 release audit and again during T-5 — make test exits 1 with a failure that does not reproduce on the immediate rerun, and both times the failing test''s name went uncaptured because the diagnostic rerun passed. Next occurrence: keep the FULL output of the failing run (make test 2>&1 | tee /tmp/flake.log) before rerunning, then pin the named test. Suspects worth checking when a name exists: order-dependent state in the session-scoped app init (conftest asyncio.run + cached health globals), port/redis contention with the live containers, and the coverage plugin''s tmp handling.', 'backlog', 'medium', NULL, NULL, NULL, '2026-09-13 09:00:32.255', '2026-09-13 09:00:32.255', NULL, '36c6f97b5a309e169b7acb35f7ca46d4', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at;
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G99D07Z3FZY8EBT9PJ1RCGHW', 'bug', NULL, 'The container logs nowhere: stdout empty, /app/logs mount untouched', 'docker logs tatlock is 0 lines and the mounted ~/docker-data/tatlock/logs has been empty since 2025-12 — the running app writes logs to neither, which made the 2026-09-12 wrapper-cutover verification blind (flavor detection had to be proven behaviorally). Dev mode logs to build/logs/server.log via make run; production should log to stdout for docker logs at minimum.', 'done', 'medium', NULL, NULL, NULL, '2026-09-12 08:20:21.624', '2026-09-13 09:00:50.006', NULL, 'd04562943a55527d6036ec38c2657308', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at;
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G9KZSCFR79G4DFCT4E5W5CAM', 'bug', NULL, 'The full suite flakes: one run fails, the rerun passes, twice now', 'Observed 2026-09-13 twice: before the v2.7.0 release audit and again during T-5 — make test exits 1 with a failure that does not reproduce on the immediate rerun, and both times the failing test''s name went uncaptured because the diagnostic rerun passed. Next occurrence: keep the FULL output of the failing run (make test 2>&1 | tee /tmp/flake.log) before rerunning, then pin the named test. Suspects worth checking when a name exists: order-dependent state in the session-scoped app init (conftest asyncio.run + cached health globals), port/redis contention with the live containers, and the coverage plugin''s tmp handling.
Third occurrence 2026-09-13, during the v2.7.2 release gate — and two process lessons captured the hard way: (1) the gate run was piped through grep, which swallowed make''s exit code, so the tag went out on a red run (release content was safe — version+changelog only on green-tested aace128 — but the gate did not gate); release runs must let make test''s exit code stand alone. (2) The failing name went uncaptured AGAIN because the tee capture only started on the rerun, which passed. The capture must wrap the FIRST gate run, every time, not begin after a failure.', 'backlog', 'medium', NULL, NULL, NULL, '2026-09-13 09:00:32.255', '2026-09-13 09:14:10.345', NULL, 'ba4c6ab575af14f50acd3a29ff5e7ef8', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at;
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G9KZSCFR79G4DFCT4E5W5CAM', 'bug', NULL, 'The full suite flakes: one run fails, the rerun passes, twice now', 'Observed 2026-09-13 twice: before the v2.7.0 release audit and again during T-5 — make test exits 1 with a failure that does not reproduce on the immediate rerun, and both times the failing test''s name went uncaptured because the diagnostic rerun passed. Next occurrence: keep the FULL output of the failing run (make test 2>&1 | tee /tmp/flake.log) before rerunning, then pin the named test. Suspects worth checking when a name exists: order-dependent state in the session-scoped app init (conftest asyncio.run + cached health globals), port/redis contention with the live containers, and the coverage plugin''s tmp handling.
Third occurrence 2026-09-13, during the v2.7.2 release gate — and two process lessons captured the hard way: (1) the gate run was piped through grep, which swallowed make''s exit code, so the tag went out on a red run (release content was safe — version+changelog only on green-tested aace128 — but the gate did not gate); release runs must let make test''s exit code stand alone. (2) The failing name went uncaptured AGAIN because the tee capture only started on the rerun, which passed. The capture must wrap the FIRST gate run, every time, not begin after a failure.', 'in_progress', 'medium', NULL, NULL, NULL, '2026-09-13 09:00:32.255', '2026-09-13 09:16:08.928', NULL, '660f20aa80dccf32427c3c3e26c46c8c', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at;
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G9M68FC27YHQSP1R9SQQ87P0', 'task', NULL, 'tests/integration/ runs under no make target', 'Surfaced during T-7: make test ignores tests/integration by directory, and make test-integration runs only tests/agents/test_tatlock_agent.py — so the marked streaming tests in tests/integration/test_tatlock_streaming.py execute under no target at all. Fold them into test-integration (and check they still pass against the current wrapper topology) or retire them deliberately.', 'backlog', 'medium', NULL, NULL, NULL, '2026-09-13 09:28:48.737', '2026-09-13 09:28:48.737', NULL, '50b6e6d25ab009bd17a23544b08e722a', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at;
INSERT INTO tickets (record_id, type, parent_record_id, title, description, status, priority, assigned_to, team, decision_ref, created_at, updated_at, deleted_at, hash, canonical_version) VALUES ('06G9KZSCFR79G4DFCT4E5W5CAM', 'bug', NULL, 'The full suite flakes: one run fails, the rerun passes, twice now', 'Observed 2026-09-13 twice: before the v2.7.0 release audit and again during T-5 — make test exits 1 with a failure that does not reproduce on the immediate rerun, and both times the failing test''s name went uncaptured because the diagnostic rerun passed. Next occurrence: keep the FULL output of the failing run (make test 2>&1 | tee /tmp/flake.log) before rerunning, then pin the named test. Suspects worth checking when a name exists: order-dependent state in the session-scoped app init (conftest asyncio.run + cached health globals), port/redis contention with the live containers, and the coverage plugin''s tmp handling.
Third occurrence 2026-09-13, during the v2.7.2 release gate — and two process lessons captured the hard way: (1) the gate run was piped through grep, which swallowed make''s exit code, so the tag went out on a red run (release content was safe — version+changelog only on green-tested aace128 — but the gate did not gate); release runs must let make test''s exit code stand alone. (2) The failing name went uncaptured AGAIN because the tee capture only started on the rerun, which passed. The capture must wrap the FIRST gate run, every time, not begin after a failure.', 'done', 'medium', NULL, NULL, NULL, '2026-09-13 09:00:32.255', '2026-09-13 09:29:10.191', NULL, 'b15a45c0eaa13b417afe5257236ed0ec', 2) ON CONFLICT(record_id) DO UPDATE SET type=excluded.type, parent_record_id=excluded.parent_record_id, title=excluded.title, description=excluded.description, status=excluded.status, priority=excluded.priority, assigned_to=excluded.assigned_to, team=excluded.team, decision_ref=excluded.decision_ref, updated_at=excluded.updated_at, deleted_at=excluded.deleted_at, hash=excluded.hash, canonical_version=excluded.canonical_version WHERE excluded.updated_at >= tickets.updated_at;
+52
View File
@@ -7,6 +7,58 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Fixed
- The unit suite is hermetic and the flake class is gone (T-7). Two
live couplings hid inside `make test`: the session init ran the real
backend health probes (an HTTP round trip to OLLAMA_HOST and a real
Anthropic API call whenever the dev .env carries a key), and the
eight integration-marked tests in `tests/agents/test_tatlock_agent.py`
ran in the unit gate because it excluded by directory while they
live outside the ignored directories — ~110 seconds of real LLM
inference per "unit" run, failing whenever the backend was
mid-deploy. The probes are now stubbed to the deterministic
local-first state (a session test pins it), the gate enforces
`-m "not integration"`, and the suite runs in ~12 s with identical
results against a dead backend. T-8 files the orphaned
`tests/integration/` directory that no make target runs.
## [2.7.2] - 2026-09-13
### Fixed
- Production logs exist now (T-5): the production default log level
was WARNING ("minimal noise"), and a healthy service warns roughly
never — so `docker logs tatlock` was empty for months, startup,
backend flavor detection and the wrapper's balancing/compaction
signals all suppressed, and the serving cutover could not be
verified from logs. The production default is INFO on stdout;
`LOG_LEVEL` remains the override for when noise is ever the real
problem. (The `/app/logs` mount was always decorative — no file
handler exists; `docker logs` is the log.)
## [2.7.1] - 2026-09-13
### Fixed
- The sanitized OpenAI client was never used (T-6): the provider
assigned `self._openai_client`, an attribute nobody reads —
`OllamaProvider.client` serves `self._client` — so the null-content
sanitizer, the wrapper signal reading and the `OLLAMA_TIMEOUT` cap
were all silently dead (sessions were unaffected; they ride
ModelSettings). Found via webber's identical bug. The client now
goes through the constructor's official `openai_client` parameter,
the parent-chat lookup survives openai's property/cached_property
descriptor change (ported from webber's 2026-08-11 fix — the old
`.fget` access would have crashed the first wired completion on
openai 2.11), and the wrapper's `balancing`/`compaction_due`
signals are read from the response body's `model_extra` in the
request path — the httpx event-hook approach demonstrably never
fires under the openai SDK. The provider tests now assert through
`provider.client`, the property pydantic_ai actually reads; the
old ones asserted on the dead attribute and stayed green
throughout.
## [2.7.0] - 2026-09-13
### Added
+1 -1
View File
@@ -40,7 +40,7 @@ run: ## Start the development server on port 8777
$(UVICORN) src.main:app --reload --host $(HOST) --port $(PORT) 2>&1 | tee build/logs/server.log
test: ## Run unit tests (no external services needed)
$(PYTEST) --ignore=tests/e2e --ignore=tests/integration --ignore=tests/contracts
$(PYTEST) --ignore=tests/e2e --ignore=tests/integration --ignore=tests/contracts -m "not integration"
test-unit: test ## Alias for test
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "tatlock"
version = "2.7.0"
version = "2.7.2"
description = "OpenAI-compatible API with Ollama backend"
requires-python = ">=3.12"
dependencies = [
+9 -3
View File
@@ -230,15 +230,21 @@ class Config(BaseSettings):
Get effective log level, auto-determining from environment if not set.
- development: DEBUG (maximum verbosity)
- production: WARNING (minimal noise)
- production: INFO
- testing: INFO
Production ran at WARNING ("minimal noise") until T-5: a healthy
service warns roughly never, so `docker logs` was empty for
months — startup, backend flavor detection and the wrapper's
balancing/compaction signals all suppressed, which made the
2026-09-12 serving cutover unverifiable from logs. INFO on
stdout is the operational floor; LOG_LEVEL quiets it when noise
ever becomes the real problem.
"""
if self.LOG_LEVEL is not None:
return self.LOG_LEVEL
if self.ENVIRONMENT == Environment.DEVELOPMENT:
return "DEBUG"
if self.ENVIRONMENT == Environment.PRODUCTION:
return "WARNING"
return "INFO"
@property
+63 -25
View File
@@ -4,11 +4,21 @@ PydanticAI provider for Ollama with message sanitization.
Ollama's OpenAI-compatible API rejects messages with `content: null`,
which PydanticAI sends for assistant messages that only contain tool calls.
This provider sanitizes messages to use empty strings instead of null.
History worth keeping (repo T-6): until 2026-09-13 the sanitized client
was never in the request path — the provider assigned
`self._openai_client`, an attribute nobody reads, while
`OllamaProvider.client` serves `self._client`. The sanitizer, the
wrapper-signal reading and the OLLAMA_TIMEOUT cap were all silently
dead (sessions were unaffected — they ride ModelSettings). webber hit
the identical bug the same day; both now wire the client through the
constructor's official `openai_client` parameter, and the signals are
read from the response body — an httpx event-hook variant demonstrably
never fires under the openai SDK.
"""
from typing import Any
import httpx
from openai import AsyncOpenAI
from pydantic_ai.providers.ollama import OllamaProvider
@@ -18,21 +28,25 @@ from src.core.logging_config import get_logger
logger = get_logger(__name__)
async def _log_wrapper_signals(response: httpx.Response) -> None:
def _log_wrapper_signals(response: Any) -> None:
"""
Surface the boilerroom wrapper's per-response signals (repo T-4).
balancing names the sessions parked or restored to serve this
request (wrapper D-4 — never silent); compaction-due says a session
crossed its authored context budget (wrapper D-5 — a signal, not an
interrupt). Reading them is this ticket; producing the compacted
transcript is a later one. The headers are absent on any other
backend, so the hook costs nothing there.
interrupt). Read from the parsed body's extra fields: the wrapper
injects them into non-streamed JSON answers and openai's pydantic
models retain unknown fields in `model_extra`. Absent on any other
backend, so this costs nothing there. Streams carry the signals in
headers only and go unlogged here; the steward's raw call reads its
own body.
"""
balancing = response.headers.get("x-boilerroom-balancing")
if balancing and balancing != "[]":
extra = getattr(response, "model_extra", None) or {}
balancing = extra.get("balancing")
if balancing:
logger.info("backend_balancing", moves=balancing)
if response.headers.get("x-boilerroom-compaction-due") == "true":
if extra.get("compaction_due"):
logger.warning(
"backend_compaction_due",
hint="GET /sessions on the wrapper names the session over budget",
@@ -58,10 +72,12 @@ class TatlockOllamaProvider(OllamaProvider):
clean_host = str(config.OLLAMA_HOST).rstrip("/")
base_url = f"{clean_host}/v1"
super().__init__(base_url=base_url)
# Override the client with our sanitized version
self._openai_client = _SanitizedAsyncOpenAI(base_url=base_url)
# The sanitized client goes through the official constructor
# parameter: the provider's `.client` property serves `_client`,
# and poking a lookalike attribute after super().__init__ had
# built its own client is how the sanitizer sat bypassed in
# production (T-6).
super().__init__(openai_client=_SanitizedAsyncOpenAI(base_url=base_url))
logger.debug(
"tatlock_ollama_provider_created",
@@ -78,29 +94,49 @@ class _SanitizedAsyncOpenAI(AsyncOpenAI):
# configured Ollama timeout instead of the SDK default (~600s),
# so one stuck request cannot eat the whole delegation budget.
kwargs.setdefault("timeout", float(config.OLLAMA_TIMEOUT))
# The response hook reads the wrapper's balancing/compaction
# headers on every call this client makes, streams included.
kwargs.setdefault(
"http_client",
httpx.AsyncClient(
event_hooks={"response": [_log_wrapper_signals]},
timeout=float(config.OLLAMA_TIMEOUT),
),
)
super().__init__(api_key="ollama", **kwargs)
@property
def chat(self) -> "_SanitizedChat":
"""Return sanitized chat interface."""
def chat(self) -> "_SanitizedChat": # type: ignore[override]
"""Return sanitized chat interface.
Deliberately incompatible with AsyncOpenAI.chat, which is a Chat
resource. Replacing it is the entire mechanism of this class;
typing it as the parent's Chat would describe an object this
class does not return. The suppression is on this member alone.
"""
return _SanitizedChat(self)
def _parent_chat(client: AsyncOpenAI) -> Any:
"""Get AsyncOpenAI's own `chat`, bypassing the subclass override.
openai has shipped `chat` as both a property (getter `.fget`) and a
functools.cached_property (getter `.func`) — this venv's 2.11.0 is
the latter, so the old hardcoded `.fget` access raised
AttributeError the moment anything touched `.chat`. It went
unnoticed here for the same reason the whole class did: the client
was never wired (T-6). Reading whichever getter the descriptor
exposes keeps this working across both shapes, and raises something
legible if openai adopts a third. Ported from webber's 2026-08-11
fix.
"""
descriptor = AsyncOpenAI.__dict__["chat"]
getter = getattr(descriptor, "func", None) or getattr(descriptor, "fget", None)
if getter is None: # pragma: no cover - defensive
raise TypeError(
f"AsyncOpenAI.chat is a {type(descriptor).__name__} with neither "
"'func' nor 'fget'; the sanitising wrapper needs updating"
)
return getter(client)
class _SanitizedChat:
"""Chat interface wrapper with sanitized completions."""
def __init__(self, client: _SanitizedAsyncOpenAI):
self._client = client
self._original_chat = AsyncOpenAI.chat.fget(client) # type: ignore
self._original_chat = _parent_chat(client)
@property
def completions(self) -> "_SanitizedCompletions":
@@ -124,7 +160,9 @@ class _SanitizedCompletions:
if "messages" in kwargs:
kwargs["messages"] = _sanitize_messages(kwargs["messages"])
return await self._original.create(**kwargs)
response = await self._original.create(**kwargs)
_log_wrapper_signals(response)
return response
def _sanitize_messages(messages: list[dict[str, Any]]) -> list[dict[str, Any]]:
+31 -3
View File
@@ -49,16 +49,44 @@ def _tenant_guard():
@pytest.fixture(scope="session", autouse=True)
def _initialize_app(_tenant_guard):
"""
Run application lifespan (Claude health check, household registration, etc.)
Run application lifespan (household registration, backend globals)
once per test session. ASGITransport doesn't trigger lifespan events,
so we call it explicitly.
The backend health probes are stubbed (T-7): `make test` promises
"no external services", but the real probes coupled every unit run
to live infrastructure — an HTTP round trip to whatever answers
behind OLLAMA_HOST, and a real Anthropic API call whenever the dev
.env carries a key. The suite flaked whenever the local backend was
mid-deploy: the ollama probe failed, the cached globals flipped,
and any test that consults them unpatched changed behavior. The
stubs land the globals in the deterministic local-first state a
healthy startup produces; tests needing other states patch the
globals themselves, and `make test-integration` runs real probes.
Depends on _tenant_guard so the suite refuses to start under the
production tenant before any initialization happens.
"""
from src.core.startup import initialize_application
from unittest.mock import patch
asyncio.run(initialize_application())
from src.anthropic import model_selector
from src.core import startup
async def _healthy_local_backend() -> bool:
model_selector._ollama_available = True
model_selector._local_flavor = "boilerroom"
return True
async def _no_cloud_fallback() -> bool:
model_selector._claude_available = False
return False
# Patch the names startup actually calls (imported into its module).
with (
patch.object(startup, "check_ollama_health", _healthy_local_backend),
patch.object(startup, "check_claude_health", _no_cloud_fallback),
):
asyncio.run(startup.initialize_application())
@pytest.fixture
+35
View File
@@ -241,6 +241,41 @@ class TestLoggingConfiguration:
with patch.object(config, "ENVIRONMENT", Environment.PRODUCTION):
assert config.log_format == "json"
def test_production_default_level_is_info(self):
"""Production logs INFO to stdout (T-5).
The old WARNING default made `docker logs` empty for months — a
healthy service warns roughly never, so startup, flavor
detection and the wrapper signals were all suppressed and the
serving cutover could not be verified from logs.
"""
from src.core.config import Environment, config
with (
patch.object(config, "ENVIRONMENT", Environment.PRODUCTION),
patch.object(config, "LOG_LEVEL", None),
):
assert config.effective_log_level == "INFO"
def test_log_level_env_still_quiets_production(self):
"""LOG_LEVEL stays the override for when noise is the problem."""
from src.core.config import Environment, config
with (
patch.object(config, "ENVIRONMENT", Environment.PRODUCTION),
patch.object(config, "LOG_LEVEL", "WARNING"),
):
assert config.effective_log_level == "WARNING"
def test_development_default_stays_debug(self):
from src.core.config import Environment, config
with (
patch.object(config, "ENVIRONMENT", Environment.DEVELOPMENT),
patch.object(config, "LOG_LEVEL", None),
):
assert config.effective_log_level == "DEBUG"
def test_multiple_loggers_independent(self):
"""Test multiple loggers are independent."""
logger1 = get_logger("test.logger1")
+19
View File
@@ -0,0 +1,19 @@
"""
The suite's backend globals are stubbed, not probed (T-7).
`make test` promises "no external services". The session init used to
run the real health probes — an HTTP round trip to whatever answers
behind OLLAMA_HOST and a real Anthropic API call whenever the dev .env
carries a key — so the cached globals depended on live infrastructure,
and the suite flaked whenever the backend was mid-deploy. This test
discriminates: under the stub the globals are always the local-first
healthy state; under real probes they follow the network of the moment.
"""
from src.anthropic import model_selector
def test_session_globals_are_deterministic():
assert model_selector._ollama_available is True
assert model_selector._local_flavor == "boilerroom"
assert model_selector._claude_available is False
+115 -8
View File
@@ -1,31 +1,73 @@
"""
Tests for TatlockOllamaProvider configuration.
Tests for TatlockOllamaProvider configuration and wiring.
The AsyncOpenAI client must carry an explicit timeout from
config.OLLAMA_TIMEOUT instead of the SDK default (~600s), so a stuck
LLM call cannot consume the whole delegation budget.
Every assertion here goes through `provider.client` — the property
pydantic_ai actually reads. The previous version asserted on
`_openai_client`, a lookalike attribute nobody read, and stayed green
for the whole period the sanitizer was bypassed in production (T-6):
a test whose subject is not the real thing cannot fail for the real
reason.
"""
import pytest
from src.core.config import config
from src.ollama.provider import TatlockOllamaProvider, _sanitize_messages
from src.ollama import provider as provider_module
from src.ollama.provider import (
TatlockOllamaProvider,
_parent_chat,
_sanitize_messages,
_SanitizedAsyncOpenAI,
_SanitizedCompletions,
)
@pytest.mark.unit
class TestClientWiring:
"""The sanitized client must be the one the provider serves (T-6)."""
def test_provider_serves_the_sanitized_client(self):
provider = TatlockOllamaProvider(base_url="http://localhost:11434/v1")
assert isinstance(provider.client, _SanitizedAsyncOpenAI)
def test_the_full_chain_reaches_completions(self):
# Walks provider.client -> sanitized chat -> parent lookup ->
# completions. Under openai's cached_property `chat`, the old
# hardcoded `.fget` access raised AttributeError right here.
provider = TatlockOllamaProvider(base_url="http://localhost:11434/v1")
assert provider.client.chat.completions is not None
def test_parent_lookup_survives_either_descriptor_shape(self):
from openai import AsyncOpenAI
descriptor = AsyncOpenAI.__dict__["chat"]
assert hasattr(descriptor, "func") or hasattr(
descriptor, "fget"
), "AsyncOpenAI.chat exposes neither getter; _parent_chat needs updating"
client = _SanitizedAsyncOpenAI(base_url="http://localhost:11434/v1")
parent = _parent_chat(client)
assert parent is not None
assert (
type(parent).__name__ != "_SanitizedChat"
), "parent lookup must not recurse into the override"
@pytest.mark.unit
class TestProviderTimeout:
"""Timeout configuration on the underlying AsyncOpenAI client."""
"""Timeout configuration on the client the provider actually serves."""
def test_openai_client_timeout_from_config(self):
provider = TatlockOllamaProvider(base_url="http://localhost:11434/v1")
assert provider._openai_client.timeout == float(config.OLLAMA_TIMEOUT)
assert provider.client.timeout == float(config.OLLAMA_TIMEOUT)
def test_timeout_is_not_sdk_default(self):
provider = TatlockOllamaProvider(base_url="http://localhost:11434/v1")
# The OpenAI SDK defaults to 600s; the configured cap must win
assert provider._openai_client.timeout < 600
assert provider.client.timeout < 600
@pytest.mark.unit
@@ -52,3 +94,68 @@ class TestMessageSanitization:
]
assert _sanitize_messages(messages) == messages
class _StubLogger:
def __init__(self):
self.calls = []
def info(self, event, **kwargs):
self.calls.append(("info", event, kwargs))
def warning(self, event, **kwargs):
self.calls.append(("warning", event, kwargs))
class _BusyResponse:
def __init__(self):
self.model_extra = {
"balancing": [{"parked": "librarian"}],
"compaction_due": True,
}
class _QuietResponse:
def __init__(self):
self.model_extra = {"balancing": [], "compaction_due": False}
@pytest.mark.unit
class TestWrapperSignals:
"""The wrapper's body signals must not pass silently (T-4/T-6).
Read from the parsed body's model_extra: an httpx event-hook
variant demonstrably never fires under the openai SDK.
"""
def test_busy_response_logs_balancing_and_compaction(self, monkeypatch):
stub = _StubLogger()
monkeypatch.setattr(provider_module, "logger", stub)
provider_module._log_wrapper_signals(_BusyResponse())
assert any(kind == "info" and event == "backend_balancing" for kind, event, _ in stub.calls)
assert any(kind == "warning" for kind, _, _ in stub.calls)
def test_quiet_response_logs_nothing(self, monkeypatch):
stub = _StubLogger()
monkeypatch.setattr(provider_module, "logger", stub)
provider_module._log_wrapper_signals(_QuietResponse())
assert stub.calls == []
async def test_create_surfaces_the_signals(self, monkeypatch):
# The read must sit in the request path, not merely exist:
# removing the call from create() has to fail this test.
stub = _StubLogger()
monkeypatch.setattr(provider_module, "logger", stub)
class FakeOriginal:
async def create(self, **kwargs):
return _BusyResponse()
completions = _SanitizedCompletions(FakeOriginal())
await completions.create(messages=[{"role": "user", "content": "hi"}])
assert any(event == "backend_balancing" for _, event, _ in stub.calls)