Commit Graph
105 Commits
Author SHA1 Message Date
jpmschweitzerandClaude Opus 4.5 75c3950dd8 feat: add Docker healthchecks, remove Uptime Kuma
- Add healthcheck configurations to 13 stacks for Portainer status monitoring
- Remove Uptime Kuma service (replaced by Docker healthchecks)
- Clean up stale Heimdall references
- Update documentation and service counts

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-14 12:05:05 +01:00
jpmschweitzer 8793ff1779 update with library desk api ENV vars 2025-12-11 19:20:40 +01:00
jpmschweitzerandClaude Opus 4.5 7c62577377 fix: use JSON array format for CORS_ORIGINS env var
Pydantic settings expects list[str] fields as JSON arrays, not plain strings.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-11 18:46:42 +01:00
jpmschweitzerandClaude Opus 4.5 235b694356 feat: add tatlock stack for homelab butler API
- OpenAI-compatible API server with LLM agent orchestration
- Port 8000, connects to docker-dataplane network
- Integrates with ollama, searxng, redis-shared services
- Watchtower auto-update enabled

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-11 18:28:04 +01:00
jpmschweitzerandClaude Opus 4.5 6581a743ac refactor: extract library-desk to standalone repository
- Move library-desk service to git.schweitz.net/jpmschweitzer/library-desk
- Update stacks/library-desk.yml to use container image
- Add library-desk to External Repositories in CONTAINERS.md
- Remove source code (now in external repo)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-11 17:46:48 +01:00
jpmschweitzer 98c92a1211 cleanup of docs 2025-12-11 16:32:37 +01:00
jpmschweitzerandClaude Opus 4.5 35077ed5ca chore: prune obsolete documentation and consolidate
- Remove obsolete plans/ directory (AI orchestrator moved to separate services)
- Remove docs/sessions/ (historical session notes)
- Remove docs/reference/ (duplicated in CONTAINERS.md)
- Remove docs/architecture/ (duplicated in CONTAINERS.md)
- Remove maintenance stack and backup-procedures guide (decommissioned)
- Remove scripts/ directory (unused)
- Fold EXTERNAL_SERVICES.md into CONTAINERS.md
- Update Nextcloud to reflect PostgreSQL shared (was MariaDB)
- Clean up README.md links

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-11 16:29:06 +01:00
jpmschweitzerandClaude Opus 4.5 0815f02529 refactor: extract core-api to standalone repository
Core-API service moved to git.schweitz.net/jpmschweitzer/core-api
- Source code removed from this repository
- Stack file updated to use container image from Gitea registry
- Documentation updated with external service reference
- Removed obsolete docs/services/core-api.md (docs now in core-api repo)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-11 16:10:17 +01:00
jpmschweitzerandClaude Opus 4.5 ca83bbffb5 refactor: extract scheduler to standalone repository
Scheduler service moved to git.schweitz.net/jpmschweitzer/scheduler
- Source code removed from this repository
- Stack file updated to use container image from Gitea registry
- Added Gitea Actions runner to gitea stack for CI/CD
- Documentation updated with external service reference

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-11 15:18:42 +01:00
jpmschweitzerandClaude Opus 4.5 9af231d72d fix(stacks): configure ollama for stability and persistent model loading
- OLLAMA_NUM_PARALLEL=1: Process requests sequentially to avoid batch
  overflow panics when multiple services call embeddings concurrently
- OLLAMA_KEEP_ALIVE=-1: Keep models loaded in VRAM indefinitely
- OLLAMA_MAX_LOADED_MODELS=2: Allow both mistral-nemo and nomic-embed-text
  to stay loaded simultaneously

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-11 11:01:03 +01:00
jpmschweitzerandClaude Opus 4.5 375913ec9d fix(library-desk): ensure page_id is int in get_page GraphQL call
GraphQL requires Int type, but page_id might be passed as string.
Explicit int() conversion prevents 400 Bad Request errors.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-10 22:31:55 +01:00
jpmschweitzerandClaude Opus 4.5 4cc2002fa6 fix(library-desk): reduce ollama health check to single API call
Previously called /api/tags twice (once for server check, once for
model check). Now reuses response from single call.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-10 22:31:07 +01:00
jpmschweitzerandClaude Opus 4.5 c91f77bf28 chore(stacks): change wiki.js port to 3000 (internal = external)
Simplifies access by using the same port internally and externally.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-10 22:13:28 +01:00
jpmschweitzerandClaude Opus 4.5 a607d7a660 fix(library-desk): resolve warnings and errors in logs
- Fix WikiChangeListener: use get_page() instead of non-existent get_page_by_id()
- Fix webhooks: use delete_page_chunks() instead of non-existent delete_page_vectors()
- Fix Neo4j: replace deprecated id() with elementId() in entity query
- Change some warnings to debug level to reduce log noise

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-10 22:03:54 +01:00
jpmschweitzerandClaude Opus 4.5 40aafb0331 docs: update changelog for v0.12.0-library-desk-enhancements
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-10 21:44:20 +01:00
jpmschweitzerandClaude Opus 4.5 4b9092d631 feat(library-desk): improve entity linking with fuzzy matching
- Add fuzzy_match_entity_to_document with confidence scoring
- Filter self-referential links (entity linking to current page)
- Fix path cleaning to preserve full wiki paths
- Add longest-first matching to prevent partial matches

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-10 21:36:24 +01:00
jpmschweitzerandClaude Opus 4.5 651de9ac92 chore(stacks): update library-desk env vars for wiki change detection
Replace WIKIJS_API_KEY with WIKIJS_USERNAME/PASSWORD and add
WIKIJS_DB_PASSWORD for PostgreSQL LISTEN/NOTIFY connection.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-10 21:36:02 +01:00
jpmschweitzerandClaude Opus 4.5 14d152bd64 test(library-desk): add test coverage for new features
- test_consolidation: taxonomy fetching, ingestion_service injection
- test_graph_service: Document node tags, entity-stub skipping
- test_ingestion: list_all_pages usage, batch ingestion
- test_integration: list_all_pages, get_taxonomy_structure
- test_wiki_change_listener: PostgreSQL notification handling

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-10 21:35:39 +01:00
jpmschweitzerandClaude Opus 4.5 b74b5bc2c8 feat(library-desk): add wiki change detection via PostgreSQL LISTEN/NOTIFY
- Add WikiChangeListener service for PostgreSQL notifications
- Add webhooks router for HTTP webhook fallback
- Add asyncpg dependency for PostgreSQL async support
- Include setup scripts and documentation for triggers

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-10 21:35:18 +01:00
jpmschweitzerandClaude Opus 4.5 bce6b71f9b fix(library-desk): use list_all_pages instead of search_pages for bulk re-index
search_pages with empty query uses stale search index. list_all_pages
fetches actual page list from GraphQL for reliable bulk re-indexing.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-10 21:34:40 +01:00
jpmschweitzerandClaude Opus 4.5 d100212da5 fix(library-desk): add tags property to Document nodes in graph_service
Set d.tags = \$tags in MERGE query to prevent Neo4j warnings about
missing tags property in related documents query.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-10 21:32:37 +01:00
jpmschweitzerandClaude Opus 4.5 aba981ff21 fix(library-desk): inject ingestion_service into ConsolidationService
Pass ingestion_service to ConsolidationService constructor so pages
created/updated during consolidation are properly indexed in Qdrant
and Neo4j.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-10 21:31:15 +01:00
jpmschweitzerandClaude Opus 4.5 6ddf2a1bd9 feat(library-desk): add taxonomy-aware classification to consolidation
- Fetch existing wiki taxonomy structure before LLM analysis
- Include existing paths in prompt to prefer existing categories
- Add _format_taxonomy_for_prompt helper
- Mark searches as processed even when skipped/errored (prevents buildup)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-10 21:30:58 +01:00
jpmschweitzerandClaude Opus 4.5 9baea7ee6c feat(library-desk): add list_all_pages and get_taxonomy_structure to wikijs client
- list_all_pages: fetch all pages with path prefix filter
- get_taxonomy_structure: extract category/subcategory structure for
  taxonomy-aware classification

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-10 21:30:47 +01:00
jpmschweitzerandClaude Opus 4.5 ed7792ef98 feat(library-desk): fetch page ID via GraphQL in wikijs-integration.js
Use pages.singleByPath(path, locale) GraphQL query to get page ID
during initialization instead of searching through page list.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-10 21:30:18 +01:00
jpmschweitzer d582f27098 more cleanup 2025-12-10 20:26:44 +01:00
jpmschweitzer 953bcc5388 cleanup 2025-12-10 20:26:06 +01:00
jpmschweitzer b80f43745a cleanup 2025-12-10 20:24:07 +01:00
jpmschweitzer c670b10630 obsolete 2025-12-10 20:23:57 +01:00
jpmschweitzer 32c4805a07 remove obsolete core-ai 2025-12-10 20:21:26 +01:00
jpmschweitzerandClaude Opus 4.5 65114cb477 fix(library-desk): fix idempotency in create_entity_mentions
Fix create_entity_mentions to correctly count only newly created relationships,
not all relationships processed by MERGE.

**Problem**: count(r) was returning ALL relationships touched by MERGE (both
created and matched), breaking idempotency tests.

**Solution**: Use temporary flag 'just_created' set only ON CREATE, filter to
those relationships, count them, then remove the flag. This ensures the count
only includes new relationships.

Now properly returns:
- N on first run (N new relationships created)
- 0 on subsequent runs (no new relationships, all already exist)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-10 02:10:31 +01:00
jpmschweitzerandClaude Opus 4.5 6ea6e4d0ad fix(library-desk): fix entity linking path cleaning and longest-first matching
Two critical bug fixes for entity linking:

1. **Path cleaning**: Replace hardcoded "users/jpmschweitzer/" with regex pattern
   to handle any user namespace. Now properly cleans paths for all users.

2. **Longest-first matching**: Move protected_ranges computation inside entity loop
   to recompute after each entity is processed. Prevents nested links like
   [[Machine](/machine) Learning](/ml) when processing multi-word entities.

These fixes ensure entity linking works correctly across all users and prevents
nested markdown links when entity names overlap.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-10 02:09:59 +01:00
jpmschweitzerandClaude Opus 4.5 60b0100861 test(library-desk): add comprehensive entity linking tests
Add test suite covering:
- Entity mention detection (case-insensitive, whole-word matching, sorting)
- Content link addition (protection of existing links, longest-first matching)
- Integration tests (get entities with paths, create relationships, idempotency)
- Multi-tenancy isolation tests

All 17 tests passing. Validates entity linking functionality end-to-end.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-10 02:09:38 +01:00
jpmschweitzer 1f0a894291 docs(library-desk): add architecture and API documentation
Documentation:
- Architecture overview and design decisions
- API endpoint reference
- Service interaction diagrams
- Configuration guide
- Multi-tenancy patterns
- Knowledge graph schema
2025-12-10 01:40:23 +01:00
jpmschweitzer ae8edf6111 chore(library-desk): add maintenance and cleanup scripts
Cleanup Scripts:
- cleanup_graph.py - Clean up duplicate entities and orphaned nodes
- cleanup_wiki.py - Remove orphaned pages and fix broken links
- Utility scripts for database maintenance
- Not part of main application, run manually
2025-12-10 01:40:10 +01:00
jpmschweitzer 9fe5faa3e7 feat(library-desk): add core API routers for wiki, vector, and ingestion
Ingestion Router:
- POST /ingest/page - Index single page
- POST /ingest/batch - Batch indexing
- POST /ingest/all - Full knowledge base refresh
- Support vector and graph ingestion

Vector Router:
- POST /vector/search - Semantic search via Qdrant
- GET /vector/stats - Collection statistics
- DELETE /vector/page - Remove page embeddings

Wiki Router:
- GET /wiki/pages - List wiki pages
- GET /wiki/pages/{id} - Get page details
- PUT /wiki/pages/{id} - Update page
- POST /wiki/search - Search wiki content
- Full Wiki.js GraphQL integration
2025-12-10 01:36:07 +01:00
jpmschweitzer 3ab44c2ab0 feat(library-desk): add AI agent tools router
Tools Router:
- Expose library-desk capabilities as AI tool endpoints
- Support function calling for LLM agents
- Query wiki pages, search knowledge base
- Access graph entities and relationships

Tools Models:
- ToolDefinition for function schemas
- ToolParameter specifications
- ToolResponse format
- OpenAI function calling compatible
2025-12-10 01:30:23 +01:00
jpmschweitzer dca8b63a50 feat(library-desk): add graph query router and models
Graph Router:
- GET /graph/entities - List all entities for user
- GET /graph/relationships - Query entity relationships
- GET /graph/search - Search entities by name/type
- GET /graph/stats - Knowledge graph statistics

Graph Models:
- Entity, Relationship models
- GraphStats for analytics
- SearchFilters for queries
- Support multi-tenancy with user isolation
2025-12-10 01:28:42 +01:00
jpmschweitzer 8f29bfe064 feat(library-desk): add knowledge consolidation system
Consolidation Service:
- Automated wiki page creation from research results
- Entity extraction and disambiguation
- Multi-source fact integration
- Confidence scoring and source citation
- Template-based page generation
- Schema.org taxonomy integration

Wiki Page Writer:
- Structured markdown generation
- Standard templates (person, organization, technology)
- Metadata formatting (tags, categories, timestamps)
- Citation and source linking

Router:
- POST /consolidate/research endpoint
- Batch consolidation support
- Manual and automated triggers

Models:
- ConsolidationRequest with source data
- ConsolidationResult with page details
- Entity resolution metadata

Tests:
- Page generation validation
- Entity extraction accuracy
- Multi-source merging logic
2025-12-10 01:28:29 +01:00
jpmschweitzer 15930a9600 feat(library-desk): implement HybridRAG query system
HybridRAG Service:
- Combine vector (Qdrant), graph (Neo4j), and web (SearXNG) search
- Reciprocal Rank Fusion (RRF) for result merging
- LLM re-ranking with mistral-nemo
- Graph enrichment with related dossiers
- Query enhancement with keyword/synonym extraction
- Search result persistence for offline processing

Router:
- POST /query/hybrid endpoint
- Configurable search limits per source
- Enable/disable individual sources
- Timing breakdown for performance monitoring

Models:
- HybridRAGRequest, HybridRAGResponse
- HybridRAGResult with source tracking
- KeywordExtraction for query analysis
- TimingBreakdown for performance metrics

Tests:
- End-to-end HybridRAG query tests
- RRF fusion algorithm validation
- Multi-source result merging
2025-12-10 01:28:13 +01:00
jpmschweitzer 8c0ced68eb feat(scheduler): add Pydantic models and improve task API
Scheduler API:
- Add Pydantic models for request/response validation
- Improve API documentation with examples
- Add detailed schedule pattern documentation
- Document priority levels and executor types

Models:
- TaskCreate, TaskUpdate, TaskResponse models
- Field validation and constraints
- Type safety for task operations

Documentation:
- Add TASK_REGISTRATION.md guide
- Document schedule patterns and executor configs
2025-12-10 01:26:21 +01:00
jpmschweitzer 0bd4f8056a feat(library-desk): add configuration for HybridRAG and enhance multi-tenancy
Configuration:
- Add HybridRAG settings (reranker model, search limits)
- Change Wiki.js auth from API key to username/password
- Configure vector, graph, and web search limits

Multi-tenancy:
- Add get_neo4j_user_base_label() for entity node labeling
- Support title-cased labels following Neo4j conventions
- Maintain namespace isolation for entities vs documents
2025-12-10 01:25:40 +01:00
jpmschweitzer 12fe7e55c0 refactor(library-desk): improve client implementations
Ollama Client:
- Improve model checking to handle :latest tag variants
- Match models with or without explicit tag

Qdrant Client:
- Add collection_exists() method for checking collection presence
- Refactor ensure_collection() to accept collection name directly
- Better separation of concerns

SearXNG Client:
- Add health_check() method for service monitoring
- Simple endpoint check without full search
- Used by health check endpoint
2025-12-10 01:25:07 +01:00
jpmschweitzer 67a124af91 feat(library-desk): add vector service for semantic search
Vector Service:
- Manage document embeddings in Qdrant
- Update vectors from wiki pages
- Handle chunking and embedding generation
- Support force refresh and incremental updates

Vector Models:
- VectorSearchResult for search responses
- VectorUpdateSummary for indexing metrics
- Track chunks created/deleted

Used by ingestion_service for page embedding
2025-12-10 01:21:56 +01:00
jpmschweitzer c2faddf54a feat(library-desk): add ingestion models for page processing
- Add IngestionResult model for single page ingestion
- Add BatchIngestionResult for batch operations
- Track vector chunks, graph entities, and relationships
- Include processing time metrics

Used by ingestion_service for page indexing
2025-12-10 01:21:21 +01:00
jpmschweitzer b777098957 feat(library-desk): add wiki models for page operations
- Add WikiPage model for page data
- Add WikiPageUpdate model for partial updates
- Add field validators for tags and descriptions
- Support optional fields for flexible updates

Used by wiki_service and entity_linking router
2025-12-10 01:20:15 +01:00
jpmschweitzer 7354a66a3d feat(library-desk): integrate entity linking into main app
Main App:
- Mount /static directory for serving Wiki.js integration scripts
- Register entity_linking router
- Refactor API key verification to dependencies module

Dependencies:
- Add service factory functions for all services
- Add get_wiki_service() for wiki operations
- Add get_graph_service() for entity operations
- Add get_ingestion_service() for auto entity linking
- Improve health check for SearXNG
2025-12-10 01:18:40 +01:00
jpmschweitzer d80f063186 feat(library-desk): add Wiki.js integration buttons for entity linking
- Add combined integration script with both re-index and entity linking
- Add standalone entity linking button
- Add standalone re-index button
- Auto-detect Library Desk URL from script tag
- Support both toolbar and floating button positions
- Show real-time status updates and notifications
- Auto-reload page after successful entity linking

Usage: Inject via Wiki.js Code Injection settings
<script src="http://IP:8089/static/wikijs-integration.js"></script>
2025-12-10 01:16:46 +01:00
jpmschweitzer 30b9825365 feat(library-desk): add entity management to graph and ingestion services
Graph Service:
- Add get_all_entities() to retrieve entities with wiki page paths
- Add create_entity_mentions() for MENTIONS relationship creation
- Support entity-to-document linking via title matching

Ingestion Service:
- Add _link_existing_entities() for automatic entity linking
- Auto-link entities during page ingestion
- Support skip_entity_linking parameter for granular control
2025-12-10 01:16:24 +01:00
jpmschweitzer 5a4a203beb fix(library-desk): preserve published status when updating wiki pages
- Add is_published parameter to WikiJS client update_page() method
- Update wiki_service to always pass is_published=True
- Prevents pages from being unpublished during entity linking updates
- Important for internal wikis where all pages should remain published
2025-12-10 01:16:11 +01:00