Commit Graph
12 Commits
Author SHA1 Message Date
jpmschweitzerandClaude Opus 4.5 4ff3fc4c7a feat: add Paperless-ngx document storage integration
- Add /documents router with webhook, upload, search, health endpoints
- Create DocumentSyncService for indexing documents to vectors/graph
- Add PaperlessClient for REST API integration
- Configure dependency injection for Paperless client
- Add document models for webhook payloads and responses
- Event-driven architecture via Paperless workflow webhooks

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-25 14:16:40 +01:00
jpmschweitzerandClaude Opus 4.5 37f8e1819e feat: refactor volatile cache to vector storage with HybridRAG integration
Build and Push / build (release) Successful in 28s
- Migrate volatile backend from Redis to Qdrant for semantic search
- Add natural language conversion for structured data embedding
- Simplify API: /volatile/search, /volatile/store, /{namespace}/{key}
- Integrate volatile into HybridRAG with priority boost in RRF fusion
- Add POST /maintenance/cleanup/volatile for expiry purging
- Update tests for new Qdrant-based architecture (37/37 pass)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-24 20:03:37 +01:00
jpmschweitzerandClaude Opus 4.5 7297e6b9f1 feat: add volatile cache system for ephemeral data
Phase 2 of Memory Management System - volatile memory tier:

- VolatileCacheService: Redis-backed TTL storage
- Volatile router with full CRUD operations
- Predefined namespaces: weather, news, financial, transit, traffic,
  air_quality, sports, social, system, context, custom
- Each namespace has appropriate default TTL (1min to 1hr)
- Refresh schedule support via cron expressions
- Scheduler integration endpoint: GET /volatile/scheduled

Endpoints:
- GET/POST/DELETE /volatile/{namespace}/{key}
- GET/DELETE /volatile/{namespace}
- GET /volatile/stats
- GET /volatile/scheduled
- GET /volatile/namespaces

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-24 17:31:45 +01:00
jpmschweitzerandClaude Opus 4.5 f756ca9490 feat: add maintenance system with index reconciliation
Complete maintenance subsystem for index health and cleanup:

Endpoints:
- GET /maintenance/health - lightweight (or detailed) health check
- POST /maintenance/cleanup/all - full orphan cleanup
- POST /maintenance/cleanup/vectors - purge orphan vector chunks
- POST /maintenance/cleanup/graph - purge orphan graph nodes
- POST /maintenance/reconcile-index - cleanup + reindex missing pages

Bidirectional orphan detection:
- find_documents_without_vectors() in GraphService
- find_chunks_without_graph_nodes() in VectorService

Redis integration:
- Tracks last_cleanup timestamp for scheduler visibility

Config additions:
- Document store, volatile cache, and maintenance settings
- VectorServiceDep and GraphServiceDep type aliases

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-24 16:22:39 +01:00
jpmschweitzerandClaude Opus 4.5 b976da0092 refactor: improve web results analysis prompt
Apply llm-findings.md recommendations:

Web results analysis (temp 0.0):
- Add ANALYSIS STEPS for chain-of-thought reasoning
- Strict RULES section with negative constraints:
  - "Do NOT suggest pages with insufficient info"
  - "Do NOT invent entities not mentioned"
  - "Do NOT suggest paths outside taxonomy"
- Conservative approach: quality over quantity
- Removed "be INCLUSIVE" guidance (caused over-suggestion)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-23 17:22:12 +01:00
jpmschweitzerandClaude Opus 4.5 edfe11f0fb refactor: improve wiki page writer prompts
Apply llm-findings.md recommendations:

Conflict detection (temp 0.0):
- Add explicit analysis steps (CoT)
- Strict rules: only flag direct contradictions
- Negative constraints for false positives

Page creation (temp 0.3):
- Add CRITICAL CONSTRAINTS section
- "Do NOT invent facts not in source"
- "Do NOT fill sections with placeholders"
- Omit sections if information unavailable

Page reconstruction (temp 0.2):
- Add preservation constraints
- "Do NOT rephrase facts changing meaning"
- "Preserve exact quotes, dates, numbers verbatim"

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-23 17:21:58 +01:00
jpmschweitzerandClaude Opus 4.5 8e003bb9e8 refactor: improve keyword extraction and re-ranking prompts
Apply llm-findings.md recommendations:

Keyword extraction (temp 0.0):
- Add negative constraints: "Do NOT invent terms"
- Simplify output format
- Remove verbose example

LLM re-ranking (temp 0.0):
- Add explicit rules section
- Negative constraints: "Do NOT consider document length"
- Clearer output format specification

Both prompts now use temperature=0.0 for deterministic,
consistent outputs.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-23 17:21:41 +01:00
jpmschweitzerandClaude Opus 4.5 a1832e3245 refactor: consolidate Ollama model configuration
Build and Push / build (release) Successful in 27s
- Add OLLAMA_EMBEDDING_MODEL for embeddings (nomic-embed-text)
- OLLAMA_MODEL now used for all LLM operations (mistral-nemo-large:latest)
- Remove separate reranker_model setting
- Update WikiPageWriter to use settings instead of hardcoded model
- Improves VRAM efficiency by keeping one model hot

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-22 11:15:27 +01:00
jpmschweitzerandClaude Opus 4.5 c359fcbcd8 feat: two-stage RRF for fair wiki vs web ranking
Build and Push / build (release) Successful in 28s
- Merge vector+graph into single wiki source before RRF with web
- Wiki pages no longer get 2x advantage from dual retrieval
- Add vector similarity threshold (0.7 default)
- Skip synonyms in graph search to reduce noise
- Fix duplicate entity links bug in graph search

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-15 17:49:31 +01:00
jpmschweitzerandClaude Opus 4.5 61863ff597 feat: add RAG search endpoint with content extraction
Build and Push / build (release) Successful in 1m2s
- Add /rag/search endpoint for web, news, and image search via SearXNG
- Add /content/extract and /content/extract/batch endpoints
- Add ContentExtractor client using Trafilatura for content extraction
- Enhance HybridRAG web search with full content extraction
- Add Redis caching for search results
- Add new configuration options for search and extraction timeouts

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-15 15:50:48 +01:00
jpmschweitzerandClaude Opus 4.5 e05e7aeae3 feat: add smart page creation endpoint with HybridRAG research
Add POST /wiki/pages/smart-create endpoint that combines research with
content generation for the librarian agent:

- Run HybridRAG search on topic (wiki + graph + web)
- Use LLM (WikiPageWriter) to synthesize findings into wiki content
- Create page with proper attribution and sources
- Schedule background tasks for vector/graph indexing
- Apply bidirectional entity linking (forward + backward links)

New files:
- src/services/entity_linking_utils.py - shared entity linking helper

Modified:
- src/models/wiki.py - WikiSmartCreateRequest/Response models
- src/services/wiki_service.py - smart_create_page() method
- src/routers/wiki.py - /pages/smart-create endpoint

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-11 20:23:46 +01:00
jpmschweitzer 95852190ba Initial commit: library-desk service extraction from portainer-core
Build and Push / build (release) Successful in 36s
2025-12-11 17:28:23 +01:00