The trailing wildcard on the three rm -rf deny entries spanned path
separators, so Bash(rm -rf /*) matched every absolute path on the
machine rather than the filesystem root, and the ~ and $HOME entries
had the same shape. Narrowed to the exact literal forms.
These rules match literal command text, so they still stop a typo on
rm -rf /, rm -rf ~ or rm -rf $HOME exactly, but they no longer stop a
recursive delete aimed at any other path. That reduced cover is
deliberate, not an oversight.
D-26 requires `make test` to pass with no network; T-55's audit measured 426
passed/29 skipped with network vs. 412 passed/23 skipped/20 ERRORS inside an
unprivileged network namespace. All 20 errors trace to a real Bolt connection
opened at fixture setup (neo4j_client -> client.connect()), not to test logic.
The ticket's own summary said all 20 were in test_entity_linking.py; tracing
the actual error list showed only 5 were (TestEntityLinkingIntegration,
TestEntityLinkingMultiTenancy, plus the trailing module-level cleanup test).
The other 15 are every test in test_hybrid_rag.py, whose hybrid_rag_service
fixture resolves graph_service -> neo4j_client regardless of what the test
body itself exercises -- including the RRF-fusion and context-formatting
classes that read as pure logic. There is no unit/integration split inside
that file without restructuring its fixture graph, which is out of scope
here; the whole module is marked instead of picking classes apart from
underneath a shared fixture chain.
The fix is the mechanism this repo already had and had never wired to a
target: tests/conftest.py's `integration` pytest marker plus its
RUN_INTEGRATION_TESTS/TEST_TENANT gate (test_integration.py,
test_tenant_isolation_live.py, test_quality_report_live.py and
TestWikiChangeListenerIntegration already used it). Applying the same marker
here means `make test` skips these 20 the same way it already skipped the
other 23 -- no file move, no new fixture layer, matching repo precedent
exactly rather than inventing a second convention beside it.
`make test-integration` is the D-26 home: sets RUN_INTEGRATION_TESTS=1,
selects `-m integration`, and treats pytest's own "no tests collected" exit
code (5) as a hard failure rather than a pass, so a marker that gets renamed
or lost fails loudly instead of the target quietly collecting zero and going
green.
Verified (unshare -rn sh -c 'ip link set lo up; ...' after confirming the
positive control -- a live :8089 returning HTTP 200 outside returns curl exit
7 inside):
make test, no network: 412 passed, 43 skipped, exit 0 (was 20 ERRORS)
make test, with network: 412 passed, 43 skipped, exit 0 (unchanged; the 14
of these 20 that were previously counted in the 426 passed now skip by
default -- reclassified, not lost; the other 6 already skipped for an
unrelated reason before this change)
make test-integration, these 20, with network: 14 passed, 6 skipped
(test_wiki_page's own pytest.skip when it can't create a wiki page -- a
pre-existing soft-skip, unrelated to this change), 0 failed, exit 0
make test-integration mutated to select a nonexistent marker: FAIL,
"selected 0 tests", exit 2 -- confirmed loud, then reverted
Not fixed here: the other 23 tests already carrying `integration` include
three files (test_integration.py, test_tenant_isolation_live.py,
test_quality_report_live.py) that fail under `make test-integration` today
because they call the local dev server on :8778, which was not running in
this session -- a pre-existing "never proven runnable" gap this same ticket
family exists to find, but a different set of tests than the one measured
here.
Co-Authored-By: Claude <noreply@anthropic.com>
`make setup` exited 0 whether or not the environment it produced was
usable — per D-24, a step whose job is to not fail has a passing state
indistinguishable from its broken state. Adds two cheap checks at the
end: `pip check` for version drift between installed packages, and
`pytest --collect-only` to walk the full src/ import graph and catch a
missing declared dependency, which is what core-api's undetected
missing sqlalchemy looked like (T-47). Neither needs any of the five
backing services running — dependencies.py only constructs clients
inside lru_cache getters, never at import/collection time.
Also fixes `setup` to install requirements-dev.txt rather than
requirements.txt. It only ever installed the latter since the
Makefile's introduction, so `make test` and `make lint` — both of
which need pytest and ruff — were never actually reachable from a
clean `make setup`. requirements-dev.txt pulls in requirements.txt via
-r, so the runtime set installed is unchanged; only the tooling to
prove it is added. Found because the new check failed honestly on its
first clean-tree run, before this fix.
Verified: clean-tree run installs everything and passes (455 tests
collected); a second run is a fast no-op; uninstalling a declared
runtime dependency (asyncpg) makes the check fail with
ModuleNotFoundError, and rerunning setup restores and re-passes it.
Co-Authored-By: Claude <noreply@anthropic.com>
The comment still said five, and named only neo4j and qdrant as
unbounded. Both were true one commit ago. Also states the property that
makes adding probes safe: concurrent bounds do not sum, so wall time is
one bound regardless of count.
Co-Authored-By: Claude <noreply@anthropic.com>
paperless, system_settings and scheduler ran serially after the five
probes bounded in 80229f2, with no timeout of their own (scheduler
defaults to 30.0s, system_settings' connect() is unbounded) — the same
defect class, just further down the same function.
All eight probes now join one asyncio.gather(return_exceptions=True),
each bounded at 2s via the existing _bounded_probe. paperless and
system_settings keep their three-state result: an unconfigured service
is excluded from the probe list entirely rather than run through the
bound, so "not configured" (None) cannot collapse into "unhealthy"
(False) the way it would if the bound's bool-only wrapper were applied
uniformly. scheduler has no config gate and joins the gather plainly.
Neither is read by /health today — only src/main.py and
tests/test_integration.py call check_service_health(), and both read
only the original five — so this is unreachable from the live
endpoint. Bounded rather than deleted: bounding cannot break a
consumer that reads these keys later, deleting could.
check_service_health() ran neo4j, qdrant, wikijs, searxng and ollama
serially with await, and only ollama's client carried its own timeout.
Neo4j (connection_timeout=30.0) and Qdrant (timeout=30.0) fell back to
driver defaults far past the container healthcheck's 10s timeout, so a
hung (not failing) dependency blocked the whole chain and flipped the
container unhealthy for a reason unrelated to its own liveness.
Each probe now runs under asyncio.wait_for bounded at 2s — chosen
against the 10s healthcheck timeout so five concurrent bounded probes
cannot approach it even if all five hang — and all five run
concurrently under asyncio.gather(return_exceptions=True), so one
probe timing out or raising cannot block or cancel the others.
Gating (neo4j+qdrant only), the unconditional 200 response, ollama's
existing 5.0s client-level timeout, and the paperless/system_settings/
scheduler probes are unchanged.
105 findings to zero. Most were mechanical — 67 unused imports, and assorted
f-strings without placeholders. Three groups needed a decision.
The 15 F821 "undefined name" were forward references, not runtime errors. Each
annotation is quoted — `-> "WikiService"`, `Optional["IngestionService"]` — with
the real import inside the function body to break an import cycle. A quoted
annotation is never evaluated, so the code ran; the names were simply
unresolvable to any checker. They now have a TYPE_CHECKING block, which costs
nothing at import time and keeps the cycle broken.
The 6 E402 split two ways. `import secrets`, `Security`, `Request` and
`HTTPBearer` in dependencies.py had drifted below several hundred lines of
factory functions for no reason — stdlib and fastapi, no cycle to avoid — and
moved up. The other three are deliberate and now say so: the VectorService and
GraphService aliases import back into dependencies.py, and main.py's routers
expect a configured app, so both must stay put.
Bare `except:` narrowed to `except Exception:` in three places, which stops them
swallowing KeyboardInterrupt and SystemExit.
The 5 unused locals were all genuinely dead. One is worth naming rather than
fixing: qdrant_client.delete()'s return value was bound and never read, so a
failed delete is indistinguishable from a successful one — the assignment is
gone, but nothing checks the status either way and that has not changed here.
`timing = {}` in _retrieve_parallel looked like it might mean the reported
per-leg timings were always zero; traced, and they come from output["timing"],
so the local was only vestigial.
426 passed, 29 skipped, unchanged. The app imports and the service aliases still
resolve, which is the check that mattered after moving imports in
dependencies.py.
The gate still prints "not gated here yet: test (T-56)" — lint is green, tests
remain unwired, and that is left visible rather than silently absent.
Co-Authored-By: Claude <noreply@anthropic.com>
The hook carried ~50 lines of gitleaks logic and a comment explaining it was
self-contained because "this repo has no Makefile". It has one now, so the
reason is gone and the arrangement is backwards: a hook is a trigger, and
logic belongs where it can be read, run by hand, and changed under review.
.githooks/pre-push is now a byte-identical shim onto `make pre-push` in every
repo in the workspace. The scan itself moves to ci/secrets.sh unchanged, and
`make secrets` runs it on its own.
The call surface is identical everywhere; what it runs is not, and should not
be — each repo gates what it actually has. That is the point of standardising
the name rather than the contents: nobody has to read a repo to find out how
to check it.
secrets runs first, deliberately. It is the only failure here that cannot be
undone by fixing it afterwards — a failed lint costs another commit, a pushed
credential is cached and indexed whether or not it is later deleted.
Some of these gates fail today, on lint debt that predates them, and they are
left wired anyway. The board was measured once and written down in T-56
instead of being worked around here. Narrowing each gate to whatever already
passes would produce a gate that reports success for doing nothing, which is
the failure this workspace keeps rediscovering.
Co-Authored-By: Claude <noreply@anthropic.com>
Environment guards now exit 69 rather than 1, so a caller can tell a suite
that could not start from one that ran and failed. The first toj test sweep
reported "3 repositories failed" and none of the three had executed a test —
two could not find go, one had no venv. That points the reader at the tests
when the fault is in the environment.
Only the environment guards change. A gitleaks finding, a failed test run and
a vulncheck hit still exit 1, because those did run and did fail.
Co-Authored-By: Claude <noreply@anthropic.com>
Every repo gets one at the root: help, plus test and lint where those exist.
The point is that a target name means the same thing in every repo, so an
agent or a person can act without reading the repo first.
Paths resolve here rather than in callers (D-10). python3 on this host is 3.8
and cannot parse these sources, and a bare pytest or ruff resolves only in a
login shell — so both are named explicitly through the venv, and a missing
venv fails with the command to fix it rather than a bare no-such-file.
Co-Authored-By: Claude <noreply@anthropic.com>
pql is now a bare word on PATH, which removed the long incantation that had
been forcing --vault into every call by habit. Convenience lowered the cost
of the wrong thing without lowering the cost of the right one: a three-word
pql ticket new targets whichever vault the cwd happens to sit in, and there
are nine of them with colliding id sequences.
PQL_VAULT in each project settings file makes the vault a property of the
session rather than of the working directory — the same lesson Rule 3 records
for git -C, applied to pql. Verified the env var overrides cwd discovery,
that an explicit --vault still beats the env var, and that the harness
hot-reloads it without a restart.
This does not make provenance visible: no output says which vault answered,
so a forgotten --vault still returns a well-formed answer about the wrong
dataset. That remains T-37.
Co-Authored-By: Claude <noreply@anthropic.com>
toj is now on the global PATH as /usr/local/bin/toj, so its scope boundary
had to stop being "the absolute path is inconvenient to type" and start
being a rule. Its repo and settings verbs operate on the workspace root; run
from inside this repo they answer about the wrong tree.
Both spellings are denied, bare and absolute, because a deny with one
spelling left open is decorative.
Co-Authored-By: Claude <noreply@anthropic.com>
No repo here scanned for committed credentials. The hook is self-contained
rather than delegating to a Makefile, because this repo has none and a hook
reaching into a sibling repo breaks the moment this one is cloned elsewhere.
Scans the outgoing range rather than full history: history carries settled
findings — test fixtures, vendored third-party code — and a gate that fails
on something unfixable gets bypassed within a week.
Setting core.hooksPath means pql init must replant its replication shims into
.githooks, which is why they are gitignored here alongside the tracked
pre-push. Same layout pql itself uses.
Co-Authored-By: Claude <noreply@anthropic.com>
One agent doc per repo, and it is CLAUDE.md. Two agent docs drift, and
the one nobody read is always the one holding the rule that mattered —
this repo had a CLAUDE.md whose entire content was an instruction to go
read the other file.
Composed fresh rather than reformatted. Everything factual carries over;
the structure follows what someone working here actually needs first.
Three corrections made while carrying content across:
- The release flow instructed `git add -A`. That is denied by policy
and sweeps in whatever else is dirty, including secrets. Now: stage
by name.
- The feature-branch mandate is gone. Linear history everywhere, no
per-repo exceptions as of 2026-08-08.
- The 8778/8089 split is stated explicitly rather than left implicit
in two separate sections. 8778 is wakeup.sh's reload server, 8089 is
the container — testing the wrong one silently exercises the wrong
build. Both ports verified against wakeup.sh and the deployed stack
before writing them down.
Adds the live double-ingestion defect (T-1) where someone touching
ingestion will see it, including the part that is not yet established:
whether it merely wastes GPU or actually corrupts Qdrant and Neo4j.
Co-Authored-By: Claude <noreply@anthropic.com>
Migrates TODO.md into pql and removes it. Six tickets: the double-
ingestion bug with its full investigation preserved, and an epic
covering the four stub endpoints in src/main.py.
Markdown TODO lists cannot express blocking, parentage or status, and
nothing notices when they go stale. Tickets travel with the repo —
.pql/changelog/ is committed and replayed by the git hooks, while the
databases are ignored and rebuildable with `pql plan rebuild`.
Replaces the feature-branch mandate with the workspace convention:
linear history, no merge commits, work on main or a short-lived branch
that is fast-forwarded away. tatlock remains the one repo that requires
branches.
Adds a committed .claude/settings.json. `pql init` writes one containing
only allow rules, which is the wrong shape — an allowlist with no floor
under it. Every git deny appears in both `git <verb>` and `git * <verb>`
form; the second catches `git -C <path>`, and without it the git denies
would be decorative.
.gitignore gains two entries. `.claude/settings.local.json` was only
protected by a global gitignore on this machine, so the protection did
not travel with the repo. The .pql rules ignore everything except the
changelog, deliberately, since that file is what makes tickets portable.
Co-Authored-By: Claude <noreply@anthropic.com>
Two uvicorn workers each hold a LISTEN connection on wiki_page_changes, so
Postgres delivers every notification to both and each page edit is ingested
twice. Found while verifying the v1.9.1 reconnect fix — the reconnect logged
twice, which is what gave it away.
Proven with a colon-free pg_notify payload that is rejected before ingestion
runs: one NOTIFY, two "Invalid notification payload" lines. Recorded with the
reproduction rather than the conclusion alone, since whether this corrupts data
or is only wasteful has not been established.
Co-Authored-By: Claude <noreply@anthropic.com>
Ships the Wiki.js change listener supervision fix. Patch release: no API
change, no migration — the listener now reconnects after a database restart
instead of going silently deaf, and /health reports its subscription state.
Co-Authored-By: Claude <noreply@anthropic.com>
The listener opened one asyncpg connection, called add_listener, and set
running = True. Nothing watched that connection afterwards. When it dropped, the
subscription was gone for good while running still reported True, so the service
stayed healthy in every way anything could observe and silently stopped indexing
page edits. Recovery needed a manual container restart.
That happened on 2026-08-08 when postgres-shared was redeployed. The sibling
settings_client survived the same event because it uses asyncpg.create_pool,
which replaces dead connections; a bare LISTEN connection has no such recovery.
A supervisor task now waits on asyncpg's termination callback and reconnects
with bounded exponential backoff, 1s doubling to a 60s cap. It retries forever
rather than giving up after N attempts: a database under maintenance does come
back, and a listener that stopped trying would reproduce exactly the silent
deafness this exists to prevent. The termination listener is re-registered on
every new connection because asyncpg clears its listener list as soon as it
fires them, so a one-time registration survives exactly one drop.
running is now derived from the connection rather than assigned, and stop() sets
a flag the termination callback and supervisor both check so a deliberate
shutdown cannot race into a reconnect.
NOTIFY is fire-and-forget, so events emitted during an outage are lost and
cannot be replayed. The reconnect logs the gap and names
POST /maintenance/integrity-check rather than reporting a clean recovery.
Reconciling automatically is left out on purpose: deriving the tenant for a
changed page is subtle here, and getting it wrong writes into the wrong user's
namespace.
Verified against the real database by terminating the listener's backend with
pg_terminate_backend. Old code: running=True with is_closed()=True, dead
forever. New code: reconnects on its own onto a new server pid. The same probe
was run against both implementations so the check is known to discriminate.
One existing test mocked the connection with a bare AsyncMock, which models
asyncpg's synchronous is_closed() as a coroutine — always truthy, so the
connection read as closed once running started deriving from it. Corrected.
Co-Authored-By: Claude <noreply@anthropic.com>
The wikijs-integration.js embedded a full-privilege API key that was
served to every wiki visitor — it unlocked all 66 authenticated
endpoints, including page/vector deletes and index purges. That key
has been rotated out of service.
The two browser endpoints (/ingest/page, /entity-linking/link-page)
now authenticate via the NPM /library-desk/ proxy location instead of a
key: Authentik forward-auth for external users, LAN bypass for internal,
verified by a trusted proxy marker header. This is safe because
library-desk binds loopback-only, so NPM is the sole path that can set
that header. The browser holds no secret; the script calls same-origin
with credentials. Machine callers (the Scheduler) keep the Bearer key
on the container-network endpoints. verify_api_key now compares in
constant time.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Network-migration release: .env.example service URL defaults moved to
docker-dataplane container names ahead of the Phase 4 port lockdown, CI
image pushes routed via git.schweitz.net, AGENTS.md health-check URL
corrected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The homelab is retiring *.schweitz.internal and will rebind most
published container ports from 0.0.0.0 to 127.0.0.1 (Phase 4), so
host-IP:published-port URLs will stop working for container-to-container
traffic. Point the .env.example defaults at docker-dataplane container
names and INTERNAL ports instead: wiki:3000, neo4j:7687, searxng:8080
(internal port, not the 8087 host publish), paperless:8000, ollama:11434.
All names and ports verified against the running containers.
Also correct the AGENTS.md deploy health-check URL, which claimed the
service runs on port 8000; it runs on 8089.
static/wikijs-integration.js is left unchanged: it already derives the
API base from its own script URL (document.currentScript.src, split on
/static/) and only uses the hardcoded IP:8089 as a last-resort fallback.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The .internal registry domain is being retired; git.schweitz.net now
serves the registry without SSO on /v2/.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
get_wikijs_namespace() returns '/users/{user}' with a leading slash while
Wiki.js search results carry paths without one, so the prefix filter in
search_pages rejected every result - wiki search returned empty for every
tenant. Found by cross-repo integration verification; the librarian now
gets real search results. Compare slash-normalized on both sides.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tenant-scoping fixture still stubbed embed_text, which does not
exist on OllamaClient - the exact mock-a-nonexistent-method pattern
that hid the original HybridRAG document-leg bug (406143e). A
regression reintroducing embed_text would have passed this suite.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
ContentExtractor._fetch used client.get(), buffering the whole body in
memory before the MAX_RESPONSE_BYTES check truncated it - the cap
protected Trafilatura but not memory/bandwidth (a multi-hundred-MB URL
was still fully downloaded, on up to max_urls_per_batch concurrent
fetches, bounded only by the read timeout).
Fetches now stream via client.stream + aiter_bytes and close the
connection as soon as the cap is reached; charset still comes from the
Content-Type header, available before the body is read.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
DocumentSyncService._index_vectors ran delete_by_filter on the
document's existing chunks FIRST and only then embedded; if the
embedding pass failed (Ollama down) the Paperless document was left
with zero vectors until the next successful sync - the same
zero-vector hazard already fixed for wiki pages in
VectorService.update_from_page.
Chunk ids are now deterministic uuid5 (document_{id}_chunk_{i}) so
re-upserting overwrites in place; new points are upserted first, then
stale points (including legacy random-uuid4 ones) are pruned via
scroll + delete_by_ids, and only after a successful upsert.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
SchedulerClient.register_volatile_fetch (consolidation's prefetch
routing) registered tasks that were dead on arrival:
- JSON body stored under 'body', which rest_api_executor ignores
(it only reads config['payload'])
- no auth block, so the scheduled POST to /volatile/fetch would 401
against library-desk's verify_api_key
- user placed in the body while /volatile/fetch endpoints require it
as a query parameter (RequiredUserQuery) - would 422 regardless
The task config now carries user in the URL query string (encoded),
an empty payload, and auth {type: bearer, token: ${LIBRARY_API_KEY}}
substituted Scheduler-side (never stored raw).
SchedulerClient also sent no Authorization to the Scheduler API itself,
so registration 401'd silently at consolidation time; it now sends
Bearer auth from the new scheduler_api_key setting.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
The Scheduler's task-management endpoints (GET/POST /tasks, PUT
/tasks/{name}) are guarded by verify_api_key, but execute() built a bare
httpx.Client with no Authorization header: the existence probe 401'd
(misread as 'task absent') and every POST/PUT registration failed, so
--execute was never runnable end-to-end against the real Scheduler.
--execute now requires SCHEDULER_API_KEY from the environment (never
stored) and sends Authorization: Bearer on all registrar HTTP calls.
Deploy notes updated alongside the LIBRARY_API_KEY requirement.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
- CORS: drop allow_credentials (wildcard origin + credentials told
browsers to attach credentials for any site); origins configurable via
CORS_ALLOW_ORIGINS (default * is safe without credentials). Verified
live: preflight no longer advertises access-control-allow-credentials.
- Scheduler tasks: auth moved from a plain Authorization header (which
the Scheduler's rest_api_executor does NOT env-substitute) to its
auth {type: bearer, token: ${LIBRARY_API_KEY}} block, substituted from
the Scheduler's own environment at execution time. The registrar no
longer resolves the real key client-side, so it can never be persisted
into the scheduled_tasks.config JSONB column. Also fixed: JSON bodies
moved from the ignored "body" key to "payload" (the executor only
reads config["payload"], so the tasks would have POSTed empty bodies
and failed required-user validation).
- Reranker: parsed ranking indices are deduplicated preserving first
occurrence (an LLM answer like "3,3,1" duplicated a result).
- HybridRAG wiring consolidated into dependencies.get_hybrid_rag_service
(now including volatile_service); the inline copies in /query/hybrid
and /wiki/pages/smart-create are gone - smart-create previously ran
without the volatile leg, and the singleton was unused.
- Remaining Neo4j writes (GraphService ingestion/deletes/purges/entity
mentions, webhook rename+delete cleanup, document-sync _index_graph,
consolidation mark-processed/add-entity) moved from auto-commit
execute_query to execute_write managed transactions with retry.
Verified end-to-end on the local dev server as llm_tester: /query/hybrid
200 with all five legs ok (volatile now active), background persistence
landed as one transaction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
- Pages are fetched with httpx.AsyncClient under real connect (3s) and
read timeouts on the event loop; only the CPU-bound Trafilatura parse
runs in the thread pool. trafilatura.fetch_url previously ran inside
the worker thread with no caller-side timeout control, so an
asyncio.wait_for timeout abandoned the thread while it kept
downloading for up to ~30s.
- Trafilatura now runs ONCE per document via bare_extraction (text and
metadata together). The old path parsed three times: extract() for
text, extract(output_format='xml') whose result was discarded, and
bare_extraction for metadata.
- extract_batch caps full-page extractions per call (default 8,
configurable); overflow URLs return unsuccessful results so the web
leg falls back to the search snippet instead of fanning out unbounded
downloads per search.
- Responses over 5MB are truncated before parsing; thread-pool queue
depth is logged for backpressure visibility.
Verified live against a real URL (fetch + single-parse extraction OK).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
Phase 3 enrichment ran a sequential Neo4j query per fused result and the
final trim then discarded most of the output. Enrichment now covers only
results that can still reach the response - the Phase 4 rerank slice
(RERANK_SLICE_SIZE = 20, results beyond it are dropped when reranking)
or final_result_count, whichever applies - and resolves every page in a
single UNWIND $page_ids Cypher query via the new tenant-scoped
GraphService.get_related_documents_batch (per-page ordering by
shared_entities and per-page limit preserved via ORDER BY + collect()).
Unenriched tail results still carry related_dossiers: [] so the response
shape is unchanged. Query validated with EXPLAIN against the live Neo4j.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
Phase 6 persistence gated every /query/hybrid response with ~21+
sequential auto-commit Neo4j queries (SearchQuery node, then one query
per FOUND document link, then one per WebResult). The search_id is now
generated up front and returned immediately; the persistence runs as a
background asyncio task (strong references held against mid-flight GC).
The write itself is collapsed into ONE UNWIND-based execute_write
transaction with aggregating CALL subqueries (so an empty doc-link list
cannot swallow the web-result branch), meaning a mid-way failure can no
longer leave a partial SearchQuery graph behind.
The persisted shape consumed by the consolidation repair loop is
unchanged - SearchQuery {id, query, user, timestamp, processed:false,
total_results, web_count, keywords}, tenant labels, FOUND {rank,
rrf_score} -> WebResult {url, title, content} - and is now pinned by
tests/test_search_persistence.py against exactly what
consolidation_service queries. Tenant scoping of the document MATCH is
preserved and asserted.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
- embed_batch now issues one batched /api/embed request (the old loop
made one /api/embeddings round-trip per chunk) with a per-text
fallback that preserves None-for-failed semantics
- update_from_page embeds all chunks in that single call and stores
them in one Qdrant batch upsert (upsert_points)
- reindex order reversed: upsert new points first, then prune stale ids
(deterministic uuid5 ids make overwrite safe) so a mid-way failure no
longer leaves the page with zero vectors
- VectorUpdateSummary gains status (success/partial/failed) and
chunks_skipped; all-embeddings-failed keeps old vectors and reports
failure instead of success=True
Measured on a 7-chunk page ingest (local server, llm_tester):
~375ms -> ~181ms median over 3 runs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
- ensure_collection was called without await, so the coroutine never ran
and fresh tenants had no collection when the upsert hit Qdrant
- a single None entry from embed_batch poisoned the point batch and
aborted the whole document upsert; failed chunks are now skipped with
a warning (all-failed raises and the IndexResult reports failure)
- raw client.delete/client.upsert calls now go through the async wrapper
(delete_by_filter and the new batch upsert_points method)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
Every vector call ran on the sync QdrantClient inside async wrapper
methods, blocking the FastAPI event loop per Qdrant round-trip. The
wrapper now holds an AsyncQdrantClient (timeout via QDRANT_TIMEOUT,
default 30s) and awaits all client calls; the wrapper API is unchanged.
Call sites off the wrapper were fixed too: the HybridRAG document leg
now uses the async search_vectors wrapper instead of the deprecated raw
client.search (also fixing its call to the nonexistent ollama.embed_text
which made the leg permanently report 'failed'), the health check awaits
get_collections, and document_sync's raw delete/upsert calls are awaited
(routed through wrappers in the next commit).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
- job_cleanup_loop (src/jobs/job_manager.py): hourly in-process pass over
JobManager.cleanup_expired_jobs, started at app startup and cancelled
at shutdown; Redis job payloads auto-expire but set memberships do not.
- docs/scheduler-tasks.md: the four production Scheduler task payloads
for the deploy checklist - nightly integrity check 04:30, weekly
quality report Sunday 03:00 (day_of_week=6, 0=Monday), daily Paperless
orphan-cleanup 05:00 hitting the existing
/maintenance/cleanup/paperless?user=jpmschweitzer&dry_run=false
endpoint, and disabling test_example_task - with exact HTTP bodies
(explicit user=jpmschweitzer, Authorization: Bearer ${LIBRARY_API_KEY}
placeholder).
- scripts/register_scheduler_tasks.py: reads SCHEDULER_URL from env,
DRY-RUN BY DEFAULT (prints the exact payloads, provably contacts
nothing), --execute gated and requiring LIBRARY_API_KEY to fill the
placeholder. NOT executed - definitions delivered for the deploy
checklist only.
9 new offline tests (loop passes/error-resilience/cancellation, payload
schedules, explicit production user, placeholder, dry-run default).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
ROOT CAUSE (investigated read-only against prod): the production
container sets OLLAMA_MODEL=nomic-embed-text (the embedding model), which
the pre-rename generation setting also read, so every consolidation
/api/generate call failed with HTTP 400 ('"nomic-embed-text" does not
support generate' - confirmed in prod logs and by a direct Ollama probe).
_classify_web_results_unified swallowed that as an empty classification,
and consolidate_knowledge marked EVERY SearchQuery processed anyway -
permanently draining the queue with zero pages ever created. Live Neo4j
shows 197/200 SearchQuery nodes processed=true with no output; every
subsequent 30-minute run then logged 'No unprocessed searches found'.
The label/tenant scoping was NOT at fault: persistence writes both the
tenant label and the plain :SearchQuery label the loop matches on.
The model resolution itself was already fixed in Phase A (94482bc,
ollama_llm_model / OLLAMA_LLM_MODEL). This commit repairs the pipeline
defect that masked it:
- LLM infrastructure failure (no output from generate) now raises
ConsolidationLLMUnavailableError instead of returning an empty routing
- consolidate_knowledge leaves those searches UNPROCESSED for the next
run, aborts the rest of the batch (the LLM is down for all of them),
and reports searches_deferred
- unparseable-but-present model output is still consumed (avoids
retrying a bad prompt forever); low-web skips unchanged
- every run logs 'Consolidation run complete: searches_processed=N
searches_deferred=M duration_ms=X'; both fields added to the response
- lookback boundary is now timezone-aware UTC (Neo4j datetime() reads
naive strings as UTC, shifting the window on CET hosts)
10 new offline regression tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
POST /maintenance/quality-report {user}:
- runs the tenant-scoped duplicate scan (cosine >= threshold, default 0.9)
- flags stale pages: not updated in stale_days AND <= max_search_hits
SearchQuery FOUND hits from the tenant's graph data
- lists pages missing tags/description (report subtree exempt)
- folds in the latest integrity-check results (Redis cache from
/maintenance/integrity-check, or run inline when absent)
- writes the dated report to users/{user}/system/quality-reports/YYYY-MM-DD
via the existing wiki write path; same-day reruns update the same page
(page id remembered in Redis because the Wiki.js listing lags creation)
- response returns the full markdown report + page path + counts +
duration_ms
Also fixes WikiJSClient.update_page: Wiki.js 2.x requires tags on the
update mutation (server maps over it unconditionally); calls without tags
failed with "Cannot read properties of undefined (reading 'map')" -
which also silently broke the consolidation page-update path. Current
tags are now preserved when the caller supplies none.
Verified live end-to-end on the local dev server as llm_tester
(tests/test_quality_report_live.py, integration-marked): probe page
flagged for missing metadata, report page written and fetched back,
same-day rerun updates in place, teardown leaves zero llm_tester pages.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
Live evidence during quality-report verification: pages.list(limit=100)
returned 43 pages while 140 existed; limit=500 returned all 140. Wiki.js
applies the limit BEFORE its own visibility filtering, so a response with
fewer pages than requested does NOT prove the listing is complete. The
Phase A limit-growth loop stopped on len < limit and silently truncated
listings (page counts, orphan cleanups, integrity scans, and the quality
report all consume this listing).
The loop now doubles the limit until the returned count stops increasing
(fixed point), at the cost of one confirming fetch. Offline pagination
tests updated, including a regression test simulating the pre-filter
limit behavior.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
POST /maintenance/integrity-check {user} reports per tenant, without
ever fixing anything:
- wiki pages with ZERO vectors in Qdrant (silent-skip reindex victims)
- orphaned vectors whose wiki page no longer exists
- unexpected Qdrant collections vs known tenant patterns (test-tenant
residue and unknown namespaces flagged; foreign services counted)
- Neo4j Document nodes without wiki counterparts
- counts + duration_ms
The latest report is cached in Redis (library:integrity:latest:{user},
30-day TTL) so the weekly quality report can fold it in. Explicit user
required per Phase B. Offline tests assert the report contents, the
collection classification rules, and that no destructive client method
is ever invoked.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
Replace the four stub endpoints with real implementations, all requiring
an explicit tenant user (Phase B rule):
- /ingest/check-updates: GraphService now records a SHA-256 content_hash
on every Document node at ingestion time; the endpoint compares those
stored hashes against current Wiki.js page content in one UNWIND Cypher
query per tenant and returns changed/new/deleted page lists (entity-stub
pages excluded, pre-hash-tracking documents flagged stored_hash_missing).
- /ingest/status/{job_id}: backed by the Redis JobManager; jobs are
tenant-scoped (foreign jobs 404). /ingest/page, /ingest/batch and
/ingest/all now create job records and return job_id.
- /ingest/repo-status/{repository}: wiki page count vs indexed Document
nodes under users/{tenant}/{repository} plus tenant job stats.
- /deduplicate/check: tenant-scoped Qdrant similarity scan; chunk pairs
above ~0.9 cosine from different pages grouped per page pair with best
score and page references (read-only).
Supporting changes: get_job_manager dependency (+ shutdown close),
scroll_all_points can return vectors, VectorService.find_duplicate_pairs,
src/core/hashing.compute_content_hash. 13 new offline unit tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
The pre-existing untracked handover note was swept into 84e9185 by a
broad git add; restore it to its untracked working-tree state.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
scripts/purge_test_artifacts.py removes confirmed test residue from the
shared stores:
- Qdrant: library_desk_llm_tester, test_user, library_desk_test_user,
memories_llm_tester, volatile_llm_tester, core_ai_user_test_* and any
collection containing llm_tester / llm-tester
- Neo4j: nodes labelled User_Llm_Tester* (SearchQuery/Document/WebResult
and sub-tenants) plus legacy llm-tester Document nodes matched by
users/llm* path
- Redis: *llm_tester* / *llm-tester* keys on the service DB
Safety: --dry-run is the DEFAULT (prints identifiers and counts only);
--execute is required for real deletion; the script exits fatally if a
target rule ever matches a jpmschweitzer-namespaced identifier; the
snapshot prerequisite (Qdrant snapshot API, neo4j-admin database dump)
is documented in the module docstring. Connection settings come from
the repo .env; secrets are never printed.
Verified with a read-only --dry-run against the live stores.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Guard-gated (RUN_INTEGRATION_TESTS=1 + reserved-tenant guard) test that
runs against the local wakeup server (8778, never the production
container on 8089) with the shared backing services:
- creates a wiki page with a unique marker and ingests it (vectors +
graph) as llm_tester,
- /query/hybrid as llm_tester must return the tenant's own page and
ZERO results from the jpmschweitzer tenant (paths, sources, and the
formatted LLM context are all checked),
- /query/hybrid as a third nonexistent tenant (llm_tester_void, inside
the reserved namespace so even its persisted SearchQuery stays in
test space - nothing is ever written as jpmschweitzer) must return
zero results entirely, on both the marker query and a broad query,
- module teardown deletes the created page; the conftest session
teardown purges all remaining llm_tester artifacts.
Verified live: 3 passed in 23.55s; post-run checks show 0 *_llm_tester
Qdrant collections, 0 User_Llm_Tester* Neo4j nodes, and 0 wiki pages
under users/llm_tester.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rewrite tests/conftest.py for the shared-services testing model where
tenancy is the only isolation wall:
- Remove the hardcoded production host default (192.168.86.149):
TEST_HOST env with a safe localhost default; LIBRARY_DESK_URL selects
the local wakeup server (8778), never the production container (8089).
- Pin the suite to the reserved test tenant llm_tester (TEST_TENANT may
only choose a tenant inside the reserved llm_tester* namespace).
- Session guard (autouse) hard-aborts the whole run if the effective
tenant is jpmschweitzer or outside the reserved namespace.
- Integration-marked tests only run with RUN_INTEGRATION_TESTS=1 and a
passing guard; they are skipped otherwise.
- Session-scoped teardown deletes ALL llm_tester artifacts created
during the run: Qdrant *_llm_tester collections, Neo4j
User_Llm_Tester* nodes, wiki subtree users/llm_tester (and hyphen
variant), llm_tester Redis keys on the service DB - with hard
assert_safe_test_tenant() checks before every delete. Uses a sync
fixture + asyncio.run to avoid the session loop-scope mismatch.
- Legacy tests/test_integration.py marked integration and pinned to the
test tenant (taxonomy/list reads no longer touch the production
namespace; Qdrant tests use the tenant-scoped collection name).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ensure_collection() was called with the raw user string while the
upsert targeted get_collection_name(user), creating stray bare-name
collections (e.g. 'test_user') and failing the actual upsert when the
scoped collection did not exist yet.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The TTLs were doubled in 4cfad2e (v1.7.2) to survive missed scheduler
runs, but the unit tests kept the old expectations and have been failing
since. Update weather (7200), financial (600), and sports (120)
assertions to the current defaults.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A live /query/hybrid probe as user=llm_tester returned jpmschweitzer
pages. Audit of all legs (vector, graph, web-persistence, volatile,
documents) plus enrichment/persistence found and fixed these unscoped
paths:
- vector_service.update_from_page and graph_service.update_from_page now
refuse pages outside users/{user}/ - previously any tenant could
ingest any wiki page (incl. another tenant's) into its own collection
and graph labels, which is how foreign content entered the vector leg.
- ingestion_service.ingest_all_pages clamps path_prefix to the caller's
namespace (segment-exact, sanitized comparison) and defaults to
users/{user}; /ingest/all returns 400 on cross-tenant prefixes.
- hybrid_rag_service._persist_search_for_librarian linked SearchQuery
nodes to unscoped (d:Document {page_id}); now matches only
User_{Tenant}_Document nodes.
- graph_service: _get_entity_mention_count, entity-stub mention/related
queries, generate_entity_stubs, find/purge_orphan_entities matched
unscoped Document nodes; cleanup_broken_relationships matched all
tenants' SearchQuery nodes; _entity_has_wiki_page listed all wiki
pages. All are now tenant-label / namespace scoped.
- volatile_service collection names now use the sanitized user id.
- is_path_in_user_namespace enforces a path-segment boundary
(users/llm_tester2 is not llm_tester's namespace) and treats
hyphen/underscore tenant spellings as the same sanitized tenant.
- New offline unit tests per leg (mocked clients) assert the
tenant-scoped collection/label/path is used and cross-tenant access
is refused.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
