Commit Graph
16 Commits
Author SHA1 Message Date
jpmschweitzerandClaude Fable 5 a4c299bf9b fix(security)!: make raw Cypher endpoints read-only with write-clause denylist
/query/graph scoping was a documented no-op (graph_service returned the
query unscoped) and neo4j_client permitted writes; a live probe showed a
nonexistent user could read the whole graph.

- Add Neo4jClient.execute_read() that opens the session with
  default_access_mode=READ_ACCESS so the database refuses writes even if
  validation is bypassed.
- GraphService.execute_query() now rejects queries containing
  CREATE/MERGE/DELETE/DETACH/SET/REMOVE/DROP/FOREACH/LOAD or any CALL
  (conservative word-boundary denylist on the uppercased query) and
  executes through the read-only session; the no-op _scope_query_to_user
  is removed.
- Remove the false user-scoping claims from /query/graph (main.py) and
  /graph/query docs and the CypherQueryRequest model: the endpoints are
  documented as admin/debug, unscoped read-only (per-tenant label
  injection for arbitrary Cypher would need a real parser; /graph/nodes
  remains the tenant-scoped path).
- Offline unit tests: denylist coverage (incl. lowercase/multiline/CALL),
  word-boundary false-positive check, and READ_ACCESS session assertion.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 11:08:40 +02:00
jpmschweitzerandClaude Fable 5 84e9185371 feat!: require explicit user on every tenant-data endpoint
Remove the implicit jpmschweitzer default tenant (DEFAULT_USER) from
src/core/multi_tenancy.py and every endpoint and request model that
inherited it (~40 endpoints across /query, /wiki, /vector, /graph,
/ingest, /volatile, /documents, /stats, /rag).

- Add validate_required_user() + RequiredUser pydantic type in
  multi_tenancy and a shared require_user FastAPI dependency
  (RequiredUserQuery) that rejects missing, empty, and whitespace-only
  users with 422, following the /maintenance/* pattern.
- Wiki page create / smart-create / dossier request models now require
  user (no fallback in wiki_service).
- /maintenance/cleanup/test-data derives the tenant from the page path
  instead of using the production tenant collection.
- Wiki.js change listener skips changes when no tenant user can be
  derived from the notification email instead of defaulting to the
  production tenant.
- Consolidation service internal helpers no longer default to the
  production tenant.
- Tool catalog marks user as required with honest descriptions.
- OpenAPI descriptions updated honestly; CHANGELOG notes that callers
  (tatlock, Scheduler ingest tasks) must now send explicit user.
- Offline tests: 422 coverage for query/body endpoints, required-user
  validator tests; updated legacy tests that assumed a default tenant.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 11:06:11 +02:00
jpmschweitzerandClaude Fable 5 a66d801abd test: add offline unit tests for Phase A reliability fixes
Mock-based tests (no live services) covering:
- ollama_llm_model resolution under the OLLAMA_MODEL env collision
- per-leg retrieval failure -> source_status/degraded signaling
- Phase 0/Phase 4 LLM timeout fallbacks
- /stats wiki page count using the users/{user} path prefix
- Wiki.js list_all_pages limit-growth pagination and
  list_pages limit-after-filter behavior

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 10:05:51 +02:00
jpmschweitzerandClaude Fable 5 94482bcb59 fix: rename generation model setting to ollama_llm_model to avoid OLLAMA_MODEL env collision
The deployed container sets OLLAMA_MODEL=nomic-embed-text for embeddings,
which shadowed the generation-model setting and broke Phase 0 keyword
extraction and Phase 4 LLM re-ranking on every request. The setting is now
ollama_llm_model (env: OLLAMA_LLM_MODEL, default gemma4:e2b), startup logs
the resolved generation model, and Phase 0/Phase 4 LLM calls are wrapped in
a 12s asyncio.wait_for with graceful fallback so a hung call cannot gate
retrieval for the full 120s client timeout.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 09:57:57 +02:00
jpmschweitzerandClaude Opus 4.5 46b9bcd7a0 feat: separate current weather from forecast into distinct namespaces
- Add FORECAST namespace for multi-day outlook (12hr TTL)
- WEATHER namespace now stores only current conditions (1hr TTL)
- Split fetch_weather into fetch_current_weather + fetch_forecast
- Add POST /volatile/fetch/forecast/{city} endpoint
- Different update frequencies for efficient caching

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-30 12:44:16 +01:00
jpmschweitzerandClaude Opus 4.5 d6b30570a0 release: v1.6.1 - Weather forecasts, sun times, air quality
Build and Push / build (release) Successful in 52s
- Weather fetch now returns 7-day forecasts with UV index
- New /volatile/fetch/sun/{city} endpoint for sunrise/sunset
- New /volatile/fetch/air_quality/{city} endpoint for AQI and pollutants
- OpenMeteoProvider now implements AirQualityProvider interface

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-30 10:13:06 +01:00
jpmschweitzerandClaude Opus 4.5 6b0530ed79 fix: remove dead automated user filtering code
The _is_automated_user method was never called - loop prevention is
handled by debouncing instead. User email filtering was intentionally
removed because the notification email is the page CREATOR, not editor.

- Remove unused _is_automated_user method
- Update test to verify notifications are processed regardless of user
- Remove obsolete test_automated_user_filtering test

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-29 21:47:55 +01:00
jpmschweitzerandClaude Opus 4.5 1f47b052d8 feat: add unified memory routing to consolidation service
- Add MemoryRouteClassification and MemoryRoutingResult models
- Implement unified classifier (_classify_web_results_unified) that routes
  web results to: wiki, volatile, file (Paperless), prefetch, or skip
- Add routing methods: _route_to_volatile, _route_to_files, _register_prefetch
- Update _process_search to use unified classifier instead of separate analysis
- Add get_volatile_cache_service factory to dependencies
- Wire volatile_service and settings_client into ConsolidationService
- Update ConsolidationResult/Response with new routing counters

Test fixes:
- Fix WikiJSClient fixtures to use api_token instead of username/password
- Fix entity linking test assertions to expect full user-namespaced paths
- Add sample_unified_classification fixture for new classifier format

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-28 17:19:20 +01:00
jpmschweitzerandClaude Opus 4.5 37f8e1819e feat: refactor volatile cache to vector storage with HybridRAG integration
Build and Push / build (release) Successful in 28s
- Migrate volatile backend from Redis to Qdrant for semantic search
- Add natural language conversion for structured data embedding
- Simplify API: /volatile/search, /volatile/store, /{namespace}/{key}
- Integrate volatile into HybridRAG with priority boost in RRF fusion
- Add POST /maintenance/cleanup/volatile for expiry purging
- Update tests for new Qdrant-based architecture (37/37 pass)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-24 20:03:37 +01:00
jpmschweitzerandClaude Opus 4.5 7297e6b9f1 feat: add volatile cache system for ephemeral data
Phase 2 of Memory Management System - volatile memory tier:

- VolatileCacheService: Redis-backed TTL storage
- Volatile router with full CRUD operations
- Predefined namespaces: weather, news, financial, transit, traffic,
  air_quality, sports, social, system, context, custom
- Each namespace has appropriate default TTL (1min to 1hr)
- Refresh schedule support via cron expressions
- Scheduler integration endpoint: GET /volatile/scheduled

Endpoints:
- GET/POST/DELETE /volatile/{namespace}/{key}
- GET/DELETE /volatile/{namespace}
- GET /volatile/stats
- GET /volatile/scheduled
- GET /volatile/namespaces

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-24 17:31:45 +01:00
jpmschweitzerandClaude Opus 4.5 f756ca9490 feat: add maintenance system with index reconciliation
Complete maintenance subsystem for index health and cleanup:

Endpoints:
- GET /maintenance/health - lightweight (or detailed) health check
- POST /maintenance/cleanup/all - full orphan cleanup
- POST /maintenance/cleanup/vectors - purge orphan vector chunks
- POST /maintenance/cleanup/graph - purge orphan graph nodes
- POST /maintenance/reconcile-index - cleanup + reindex missing pages

Bidirectional orphan detection:
- find_documents_without_vectors() in GraphService
- find_chunks_without_graph_nodes() in VectorService

Redis integration:
- Tracks last_cleanup timestamp for scheduler visibility

Config additions:
- Document store, volatile cache, and maintenance settings
- VectorServiceDep and GraphServiceDep type aliases

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-24 16:22:39 +01:00
jpmschweitzerandClaude Opus 4.5 1aca286703 test: update fixtures to use real service host
- Add TEST_HOST config (default: 192.168.86.149) in conftest.py
- Update all test fixtures to use configurable host instead of
  docker hostnames (neo4j, qdrant, wiki, etc.)
- Fix test_integration.py WikiJS client to use username/password auth
- Fix ollama_client fixture to use ollama_embedding_model setting

This allows tests to run against real services from outside Docker.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-23 17:21:02 +01:00
jpmschweitzerandClaude Opus 4.5 c359fcbcd8 feat: two-stage RRF for fair wiki vs web ranking
Build and Push / build (release) Successful in 28s
- Merge vector+graph into single wiki source before RRF with web
- Wiki pages no longer get 2x advantage from dual retrieval
- Add vector similarity threshold (0.7 default)
- Skip synonyms in graph search to reduce noise
- Fix duplicate entity links bug in graph search

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-15 17:49:31 +01:00
jpmschweitzerandClaude Opus 4.5 61863ff597 feat: add RAG search endpoint with content extraction
Build and Push / build (release) Successful in 1m2s
- Add /rag/search endpoint for web, news, and image search via SearXNG
- Add /content/extract and /content/extract/batch endpoints
- Add ContentExtractor client using Trafilatura for content extraction
- Enhance HybridRAG web search with full content extraction
- Add Redis caching for search results
- Add new configuration options for search and extraction timeouts

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-15 15:50:48 +01:00
jpmschweitzerandClaude Opus 4.5 37552b926f fix: update registry URL and add smart create tests
Build and Push / build (release) Failing after 12s
- Updated container registry URL in Gitea workflow (git.schweitz.net → git.schweitz.internal)
- Added comprehensive tests for smart page creation feature

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-14 11:16:44 +01:00
jpmschweitzer 95852190ba Initial commit: library-desk service extraction from portainer-core
Build and Push / build (release) Successful in 36s
2025-12-11 17:28:23 +01:00