feat: integrate Paperless documents and volatile cache into Librarian

- Add Paperless document search to HybridRAG pipeline
- Add volatile cache (weather, forecast, news, stocks) to HybridRAG
- Add include_documents and include_volatile params to hybrid_search
- Add 📑 and ⚡ icons for document/volatile sources
- Update Librarian prompt with new data source awareness
- Fix Biographer routing: personal memory queries now route correctly
- Add location keywords to Steward pre-fetch logic

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

CHANGELOG.md

@@ -7,6 +7,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [1.11.0] - 2025-12-30
+
+### Added
+
+- **Paperless document integration** - HybridRAG now includes indexed PDFs and scanned documents from Paperless-ngx
+  - New `include_documents` parameter in `hybrid_search` tool
+  - 📑 icon for document sources in search results
+  - Librarian prompt updated with document awareness
+
+- **Volatile cache integration** - HybridRAG now includes pre-fetched real-time data
+  - New `include_volatile` parameter in `hybrid_search` tool
+  - ⚡ icon for volatile sources in search results
+  - Supports weather, forecast, news, stock, crypto, sun, air_quality namespaces
+  - Librarian prompt updated with volatile cache awareness (user-configured items only)
+
+- **Biographer routing in Steward** - Personal memory queries now correctly route to The Biographer
+  - Added explicit routing rules for "where do I live", "what car do I drive", etc.
+  - Added biographer delegation examples to Steward prompt
+  - Location keywords ("live", "where", "home") now trigger profile pre-fetch
+
+### Changed
+
+- **LibraryDeskClient.hybrid_search** - Now passes full config including `document_limit`, `volatile_limit`, and enable flags
+- **Steward guidelines** - Clarified that research queries about TOPICS go to Librarian, queries about USER go to Biographer
+
 ## [1.10.1] - 2025-12-23
 
 ### Fixed

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "tatlock"
-version = "1.10.1"
+version = "1.11.0"
 description = "OpenAI-compatible API with Ollama backend"
 requires-python = ">=3.12"
 dependencies = []

src/agents/librarian/agent.py

@@ -39,7 +39,14 @@ Your role is to help users find, understand, synthesize, and manage information
 - The personal wiki (Wiki.js) containing documentation and notes
 - The knowledge graph (Neo4j) with entities and relationships
 - Vector embeddings (Qdrant) for semantic search
-- Web search (SearXNG) for current information
+- Paperless documents (📑) - indexed PDFs, scanned documents, invoices, receipts from the user's document archive
+- Volatile cache (⚡) - pre-fetched real-time data for user-relevant locations and items:
+  - weather/forecast: conditions and forecasts for user's configured cities
+  - news: headlines from user's preferred sources
+  - stock/crypto: quotes for user's watched symbols
+  - sun/air_quality: data for user's locations
+  - Note: volatile data may not exist for arbitrary queries - falls back to web search
+- Web search (SearXNG) for current information not available in cache
 
 ## Your Personality
 - Scholarly and thorough in your research
@@ -60,7 +67,13 @@ Your role is to help users find, understand, synthesize, and manage information
   - Use for: comparing multiple sources, gathering info from several pages
 
 ### Internal Research Tools
-- **hybrid_search**: Your primary research tool - searches wiki, graph, and web at once
+- **hybrid_search**: Your primary research tool - searches ALL sources at once:
+  - Wiki pages (vector similarity)
+  - Knowledge graph (entity relationships)
+  - Paperless documents (📑 indexed PDFs, scans)
+  - Volatile cache (⚡ weather, news, stocks - when available)
+  - Web search (current information)
+  Results are fused and re-ranked by relevance. Volatile data gets priority when fresh.
 - **search_wiki**: Find specific wiki pages by keyword
 - **semantic_search**: Find conceptually similar content
 - **explore_knowledge_graph** / **find_related_entities**: Discover connections

src/agents/librarian/client.py

@@ -225,18 +225,22 @@ class LibraryDeskClient:
         vector_limit: int = 10,
         graph_limit: int = 10,
         web_limit: int = 5,
+        document_limit: int = 5,
+        volatile_limit: int = 3,
         enable_reranking: bool = True,
         final_result_count: int = 10,
     ) -> HybridRAGResponse:
         """
-        Execute HybridRAG search combining vector, graph, and web results.
+        Execute HybridRAG search combining vector, graph, documents, volatile, and web.
 
         Args:
             query: Search query
             user: User identifier for multi-tenancy (defaults to request context)
-            vector_limit: Max results from vector search
+            vector_limit: Max results from vector search (wiki pages)
             graph_limit: Max results from graph search
-            web_limit: Max results from web search
+            web_limit: Max results from web search (0 to disable)
+            document_limit: Max results from Paperless documents (0 to disable)
+            volatile_limit: Max results from volatile cache (0 to disable)
             enable_reranking: Whether to rerank with LLM
             final_result_count: Number of final results after fusion
 
@@ -252,6 +256,11 @@ class LibraryDeskClient:
                 "vector_limit": vector_limit,
                 "graph_limit": graph_limit,
                 "web_limit": web_limit,
+                "document_limit": document_limit,
+                "volatile_limit": volatile_limit,
+                "enable_documents": document_limit > 0,
+                "enable_volatile": volatile_limit > 0,
+                "enable_web": web_limit > 0,
                 "enable_reranking": enable_reranking,
                 "final_result_count": final_result_count,
             },

src/agents/librarian/tools.py

@@ -17,20 +17,26 @@ logger = get_logger(__name__)
 async def hybrid_search(
     query: str,
     include_web: bool = True,
+    include_documents: bool = True,
+    include_volatile: bool = True,
 ) -> str:
     """
     Search across all knowledge sources using HybridRAG.
 
     This is the primary research tool, combining:
-    - Vector search (semantic similarity over documents)
+    - Vector search (semantic similarity over wiki pages)
     - Knowledge graph (entities and relationships)
+    - Paperless documents (📑 indexed PDFs, scans, invoices)
+    - Volatile cache (⚡ weather, news, stocks - for user's configured items)
     - Web search (current information from SearXNG)
 
-    Results are fused and re-ranked by relevance.
+    Results are fused and re-ranked by relevance. Volatile data gets priority when fresh.
 
     Args:
         query: Natural language research query
         include_web: Whether to include web results (default: True)
+        include_documents: Whether to include Paperless documents (default: True)
+        include_volatile: Whether to include volatile cache data (default: True)
 
     Returns:
         Formatted search results with sources and context
@@ -38,12 +44,16 @@ async def hybrid_search(
     Examples:
         hybrid_search("How does Docker orchestration work with Kubernetes?")
         hybrid_search("What projects use Neo4j?", include_web=False)
+        hybrid_search("Find my electricity invoices", include_web=False, include_volatile=False)
+        hybrid_search("What's the weather in Rotterdam?")  # May hit volatile cache
     """
     try:
         async with LibraryDeskClient() as client:
             response = await client.hybrid_search(
                 query=query,
                 web_limit=5 if include_web else 0,
+                document_limit=5 if include_documents else 0,
+                volatile_limit=3 if include_volatile else 0,
             )
 
             if not response.results:
@@ -70,6 +80,8 @@ async def hybrid_search(
                     "vector": "📄",
                     "graph": "🔗",
                     "web": "🌐",
+                    "document": "📑",
+                    "volatile": "⚡",
                 }.get(result.source, "•")
 
                 output_parts.append(

src/agents/steward/agent.py

@@ -56,14 +56,22 @@ USER QUERY: {query}
 GUIDELINES:
 - Be conservative - only recommend truly necessary capabilities
 - Simple greetings/chat → no capabilities needed (conversational response only)
-- Questions about prior conversation ("what did I say", "my name", "what we discussed") → no capabilities (Tatlock has full history)
+- Questions about prior conversation ("what did I say", "what we discussed") → no capabilities (Tatlock has full history)
 - Math/calculations → tatlock_core
 - Time/date queries → tatlock_core
+- PERSONAL MEMORY queries → biographer to recall (ALWAYS use for questions about the user themselves):
+  - "where do I live", "what's my location", "my address" → biographer to recall location
+  - "what's my name", "who am I" → biographer to recall name
+  - "what car do I drive", "my vehicle" → biographer to recall car
+  - "what do you know about me", "what have I told you" → biographer to recall or list_memories
+  - "remember that I...", "store that..." → biographer to store_insight
+  - "forget my...", "delete..." → biographer to forget_memory
+  - "my timezone", "my preferences" → biographer to recall preferences
 - Web searches, weather, news, current information → librarian with search_web
 - Read a URL or article → librarian with read_url
 - Wiki creation ("create a page about X", "add X to wiki") → librarian with smart_create
 - Wiki updates ("update the page", "add to dossier") → librarian with update
-- Research queries ("find info", "what do we know about", "search for") → librarian with hybrid_search
+- Research queries about TOPICS (not about the user) → librarian with hybrid_search
 - In-depth research, knowledge synthesis, document lookup → librarian with hybrid_search
 - If conversation history is relevant, note which previous turns matter
 - Assess complexity: simple (1 tool), moderate (2-3 tools), complex (multiple steps)
@@ -75,6 +83,10 @@ COMPLEXITY: [simple/moderate/complex]
 CONTEXT: [any relevant conversation context, or "none"]
 
 EXAMPLES:
+- "DELEGATE: biographer to recall the user's location" (for "where do I live?")
+- "DELEGATE: biographer to recall the user's car" (for "what car do I drive?")
+- "DELEGATE: biographer to list_memories about the user" (for "what do you know about me?")
+- "DELEGATE: biographer to store_insight about user's pet" (for "remember that I have a dog named Max")
 - "DELEGATE: librarian to search_web for tomorrow's weather forecast"
 - "DELEGATE: librarian to create a wiki page about CI/CD pipelines"
 - "DELEGATE: librarian to hybrid_search for information about Docker networking"

src/agents/steward/service.py

@@ -236,7 +236,9 @@ async def _prefetch_memory_context(user_request: str) -> dict[str, Any]:
     # Location-related queries
     if any(word in request_lower for word in [
         "weather", "temperature", "forecast", "nearby", "local",
-        "directions", "distance", "map", "here"
+        "directions", "distance", "map", "here",
+        # Direct location questions
+        "live", "where", "home", "reside", "location", "address",
     ]):
         profile_keys.append("location")