fix: stop consolidation from consuming searches when the LLM is down
ROOT CAUSE (investigated read-only against prod): the production
container sets OLLAMA_MODEL=nomic-embed-text (the embedding model), which
the pre-rename generation setting also read, so every consolidation
/api/generate call failed with HTTP 400 ('"nomic-embed-text" does not
support generate' - confirmed in prod logs and by a direct Ollama probe).
_classify_web_results_unified swallowed that as an empty classification,
and consolidate_knowledge marked EVERY SearchQuery processed anyway -
permanently draining the queue with zero pages ever created. Live Neo4j
shows 197/200 SearchQuery nodes processed=true with no output; every
subsequent 30-minute run then logged 'No unprocessed searches found'.
The label/tenant scoping was NOT at fault: persistence writes both the
tenant label and the plain :SearchQuery label the loop matches on.
The model resolution itself was already fixed in Phase A (94482bc,
ollama_llm_model / OLLAMA_LLM_MODEL). This commit repairs the pipeline
defect that masked it:
- LLM infrastructure failure (no output from generate) now raises
ConsolidationLLMUnavailableError instead of returning an empty routing
- consolidate_knowledge leaves those searches UNPROCESSED for the next
run, aborts the rest of the batch (the LLM is down for all of them),
and reports searches_deferred
- unparseable-but-present model output is still consumed (avoids
retrying a bad prompt forever); low-web skips unchanged
- every run logs 'Consolidation run complete: searches_processed=N
searches_deferred=M duration_ms=X'; both fields added to the response
- lookback boundary is now timezone-aware UTC (Neo4j datetime() reads
naive strings as UTC, shifting the window on CET hosts)
10 new offline regression tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbFZyDvYksazX6nYQYZ67L
This commit is contained in:
@@ -50,9 +50,14 @@ class ConsolidationResponse(BaseModel):
|
||||
volatile_cached: int = Field(default=0, description="Items cached to volatile storage")
|
||||
files_queued: int = Field(default=0, description="Files queued for Paperless")
|
||||
prefetch_registered: int = Field(default=0, description="Prefetch patterns registered")
|
||||
searches_deferred: int = Field(
|
||||
default=0,
|
||||
description="Searches left unprocessed for the next run because the generation LLM was unavailable"
|
||||
)
|
||||
errors: List[str] = Field(default=[], description="Error messages")
|
||||
results: List[ConsolidationResult] = Field(description="Per-search results")
|
||||
dry_run: bool = Field(description="Whether this was a dry run")
|
||||
duration_ms: float = Field(default=0.0, description="Run duration in milliseconds")
|
||||
|
||||
|
||||
class MemoryRouteClassification(BaseModel):
|
||||
|
||||
Reference in New Issue
Block a user