Commit Graph
28 Commits
Author SHA1 Message Date
jpmschweitzerandClaude Opus 4.5 f139f518ff docs: add scheduler integration and memory system plan
Documentation updates:
- AGENTS.md: Added wakeup.sh usage and local testing instructions
- LIBRARIAN_INTEGRATION.md: Complete maintenance scheduler docs
  - Scheduled task configuration for reconcile-index
  - Endpoint specifications and response formats
- docs/MEMORY_SYSTEM_PLAN.md: Three-tier memory architecture
  - Volatile (Redis TTL) for ephemeral context
  - Documents (TBD) for git mirrors, PDFs, images
  - Knowledge (Wiki + Neo4j) for permanent research

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-24 16:33:06 +01:00
jpmschweitzerandClaude Opus 4.5 f756ca9490 feat: add maintenance system with index reconciliation
Complete maintenance subsystem for index health and cleanup:

Endpoints:
- GET /maintenance/health - lightweight (or detailed) health check
- POST /maintenance/cleanup/all - full orphan cleanup
- POST /maintenance/cleanup/vectors - purge orphan vector chunks
- POST /maintenance/cleanup/graph - purge orphan graph nodes
- POST /maintenance/reconcile-index - cleanup + reindex missing pages

Bidirectional orphan detection:
- find_documents_without_vectors() in GraphService
- find_chunks_without_graph_nodes() in VectorService

Redis integration:
- Tracks last_cleanup timestamp for scheduler visibility

Config additions:
- Document store, volatile cache, and maintenance settings
- VectorServiceDep and GraphServiceDep type aliases

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-24 16:22:39 +01:00
jpmschweitzerandClaude Opus 4.5 0e6c3619eb feat: add scroll and batch delete methods to Qdrant client
Add bulk operations needed for maintenance:
- scroll_all_points(): paginated iteration over all points
- delete_by_ids(): batch delete points by ID list

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-24 16:21:57 +01:00
jpmschweitzerandClaude Opus 4.5 9446d6bf9a refactor: switch Wiki.js client to API token authentication
Replace username/password login flow with simpler API token auth:
- Use WIKI_GRAPHQL_API environment variable for JWT token
- Remove login() method and session management
- Add list_pages() method for fetching all pages
- Keep legacy auth fields in config for backwards compatibility

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-24 16:21:39 +01:00
jpmschweitzerandClaude Opus 4.5 318636d33d release: v1.3.3 - LLM prompt improvements and dead code cleanup
Build and Push / build (release) Successful in 28s
- Improved LLM prompts with temperature control and negative constraints
- Removed dead code and unused imports
- Wired /query/semantic and /query/graph to real implementations
- Updated test fixtures for external service access

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-23 17:23:44 +01:00
jpmschweitzerandClaude Opus 4.5 b976da0092 refactor: improve web results analysis prompt
Apply llm-findings.md recommendations:

Web results analysis (temp 0.0):
- Add ANALYSIS STEPS for chain-of-thought reasoning
- Strict RULES section with negative constraints:
  - "Do NOT suggest pages with insufficient info"
  - "Do NOT invent entities not mentioned"
  - "Do NOT suggest paths outside taxonomy"
- Conservative approach: quality over quantity
- Removed "be INCLUSIVE" guidance (caused over-suggestion)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-23 17:22:12 +01:00
jpmschweitzerandClaude Opus 4.5 edfe11f0fb refactor: improve wiki page writer prompts
Apply llm-findings.md recommendations:

Conflict detection (temp 0.0):
- Add explicit analysis steps (CoT)
- Strict rules: only flag direct contradictions
- Negative constraints for false positives

Page creation (temp 0.3):
- Add CRITICAL CONSTRAINTS section
- "Do NOT invent facts not in source"
- "Do NOT fill sections with placeholders"
- Omit sections if information unavailable

Page reconstruction (temp 0.2):
- Add preservation constraints
- "Do NOT rephrase facts changing meaning"
- "Preserve exact quotes, dates, numbers verbatim"

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-23 17:21:58 +01:00
jpmschweitzerandClaude Opus 4.5 8e003bb9e8 refactor: improve keyword extraction and re-ranking prompts
Apply llm-findings.md recommendations:

Keyword extraction (temp 0.0):
- Add negative constraints: "Do NOT invent terms"
- Simplify output format
- Remove verbose example

LLM re-ranking (temp 0.0):
- Add explicit rules section
- Negative constraints: "Do NOT consider document length"
- Clearer output format specification

Both prompts now use temperature=0.0 for deterministic,
consistent outputs.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-23 17:21:41 +01:00
jpmschweitzerandClaude Opus 4.5 dfd1f19bf9 feat: add temperature parameter to Ollama generate_text
Add temperature control for LLM text generation:
- temperature=0.0 for deterministic outputs (JSON, rankings)
- temperature=0.3-0.5 for controlled creative content
- None uses model default (~0.7 for mistral-nemo)

Based on llm-findings.md recommendations for improving
mistral-nemo output consistency.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-23 17:21:18 +01:00
jpmschweitzerandClaude Opus 4.5 1aca286703 test: update fixtures to use real service host
- Add TEST_HOST config (default: 192.168.86.149) in conftest.py
- Update all test fixtures to use configurable host instead of
  docker hostnames (neo4j, qdrant, wiki, etc.)
- Fix test_integration.py WikiJS client to use username/password auth
- Fix ollama_client fixture to use ollama_embedding_model setting

This allows tests to run against real services from outside Docker.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-23 17:21:02 +01:00
jpmschweitzerandClaude Opus 4.5 262a58b0d2 refactor: wire query endpoints and remove stub endpoints
- Wire /query/semantic to VectorService.search()
- Wire /query/graph to GraphService.execute_query()
- Remove stub endpoints:
  - /stats (returns zeros)
  - /ingest/document (shadowed by router)
  - /ingest/batch (shadowed by router)
- Remove unused StatsResponse model
- Add TODO.md tracking remaining stubs to implement:
  - /ingest/check-updates
  - /ingest/status/{document_id}
  - /ingest/repo-status/{repository}
  - /deduplicate/check

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-23 17:20:45 +01:00
jpmschweitzerandClaude Opus 4.5 02d728ac5b refactor: remove dead code and unused imports
- Remove unused get_default_user() from dependencies.py
- Remove unused imports from routers:
  - wiki.py: HTTPAuthorizationCredentials, Security
  - graph.py: Neo4jClient, WikiJSClient
  - hybrid_rag.py: VectorService, GraphService (duplicates)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-23 17:20:27 +01:00
jpmschweitzerandClaude Opus 4.5 a1832e3245 refactor: consolidate Ollama model configuration
Build and Push / build (release) Successful in 27s
- Add OLLAMA_EMBEDDING_MODEL for embeddings (nomic-embed-text)
- OLLAMA_MODEL now used for all LLM operations (mistral-nemo-large:latest)
- Remove separate reranker_model setting
- Update WikiPageWriter to use settings instead of hardcoded model
- Improves VRAM efficiency by keeping one model hot

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-22 11:15:27 +01:00
jpmschweitzerandClaude Opus 4.5 5be31a5a00 docs: add release flow section to AGENTS.md
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-18 18:51:49 +01:00
jpmschweitzerandClaude Opus 4.5 376284f90e fix: add content_extractor to smart-create endpoint
Build and Push / build (release) Successful in 30s
The POST /wiki/pages/smart-create endpoint was failing with 500
Internal Server Error because HybridRAGService.__init__() was
missing the required content_extractor parameter.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-16 09:31:23 +01:00
jpmschweitzerandClaude Opus 4.5 f095de1162 docs: add HybridRAG architecture documentation
Documents two-stage RRF, configuration options, and notes
potential vector search noise improvements for future reference.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-15 17:54:38 +01:00
jpmschweitzerandClaude Opus 4.5 c359fcbcd8 feat: two-stage RRF for fair wiki vs web ranking
Build and Push / build (release) Successful in 28s
- Merge vector+graph into single wiki source before RRF with web
- Wiki pages no longer get 2x advantage from dual retrieval
- Add vector similarity threshold (0.7 default)
- Skip synonyms in graph search to reduce noise
- Fix duplicate entity links bug in graph search

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-15 17:49:31 +01:00
jpmschweitzerandClaude Opus 4.5 464ec5380c fix: add content_extractor to hybrid_rag router dependency
Build and Push / build (release) Successful in 29s
The router had its own local get_hybrid_rag_service factory that was
missing the new content_extractor parameter, causing 500 errors.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-15 17:00:38 +01:00
jpmschweitzerandClaude Opus 4.5 61863ff597 feat: add RAG search endpoint with content extraction
Build and Push / build (release) Successful in 1m2s
- Add /rag/search endpoint for web, news, and image search via SearXNG
- Add /content/extract and /content/extract/batch endpoints
- Add ContentExtractor client using Trafilatura for content extraction
- Enhance HybridRAG web search with full content extraction
- Add Redis caching for search results
- Add new configuration options for search and extraction timeouts

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-15 15:50:48 +01:00
jpmschweitzerandClaude Opus 4.5 16b86a7c1b feat: add Watchtower update trigger to build workflow
Build and Push / build (release) Successful in 30s
Automatically triggers Watchtower to update containers after successful build and push.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-14 12:59:49 +01:00
jpmschweitzerandClaude Opus 4.5 2ac4595b37 fix: update registry login URL in Gitea workflow
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-14 11:21:29 +01:00
jpmschweitzerandClaude Opus 4.5 37552b926f fix: update registry URL and add smart create tests
Build and Push / build (release) Failing after 12s
- Updated container registry URL in Gitea workflow (git.schweitz.net → git.schweitz.internal)
- Added comprehensive tests for smart page creation feature

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-14 11:16:44 +01:00
jpmschweitzer 23ccd5ca5b added AGENTS.md llm instruction file 2025-12-11 21:16:49 +01:00
jpmschweitzerandClaude Opus 4.5 2bfb1e29d7 fix: include pyproject.toml in Docker image for version info
The version was showing as 0.0.0 because pyproject.toml wasn't
being copied into the Docker image.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-11 21:08:53 +01:00
jpmschweitzerandClaude Opus 4.5 0f224b460e docs: add AGENTS.md for Claude Code context
Build and Push / build (release) Failing after 16s
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-11 20:29:04 +01:00
jpmschweitzerandClaude Opus 4.5 3deed7cbcb chore: add version management and changelog
- Add pyproject.toml with project metadata and version (1.1.0)
- Update config.py to read version from pyproject.toml
- Update main.py to use centralized version in FastAPI app
- Add CHANGELOG.md documenting v1.0.0 and v1.1.0 changes

Version is now the single source of truth in pyproject.toml and is
displayed in the health check endpoint and API docs.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-11 20:23:56 +01:00
jpmschweitzerandClaude Opus 4.5 e05e7aeae3 feat: add smart page creation endpoint with HybridRAG research
Add POST /wiki/pages/smart-create endpoint that combines research with
content generation for the librarian agent:

- Run HybridRAG search on topic (wiki + graph + web)
- Use LLM (WikiPageWriter) to synthesize findings into wiki content
- Create page with proper attribution and sources
- Schedule background tasks for vector/graph indexing
- Apply bidirectional entity linking (forward + backward links)

New files:
- src/services/entity_linking_utils.py - shared entity linking helper

Modified:
- src/models/wiki.py - WikiSmartCreateRequest/Response models
- src/services/wiki_service.py - smart_create_page() method
- src/routers/wiki.py - /pages/smart-create endpoint

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-11 20:23:46 +01:00
jpmschweitzer 95852190ba Initial commit: library-desk service extraction from portainer-core
Build and Push / build (release) Successful in 36s
2025-12-11 17:28:23 +01:00