jpmschweitzerandClaude Opus 4.5
a1832e3245
Build and Push / build (release) Successful in 27s
- Add OLLAMA_EMBEDDING_MODEL for embeddings (nomic-embed-text) - OLLAMA_MODEL now used for all LLM operations (mistral-nemo-large:latest) - Remove separate reranker_model setting - Update WikiPageWriter to use settings instead of hardcoded model - Improves VRAM efficiency by keeping one model hot 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
238 lines
6.4 KiB
Markdown
238 lines
6.4 KiB
Markdown
# Library Desk - API Coordination Service
|
|
|
|
**FastAPI service that coordinates all Library operations**
|
|
|
|
## Overview
|
|
|
|
Library Desk is the central coordination layer for The Library system, providing a unified API for:
|
|
|
|
- **HybridRAG Queries** - Combines Neo4j (structure) + Qdrant (semantics) + SearXNG (web)
|
|
- **Document Ingestion** - Parse, chunk, embed, and index documents
|
|
- **Entity Extraction** - NLP to identify classes, functions, concepts
|
|
- **Relationship Mapping** - Link entities in Neo4j knowledge graph
|
|
- **Mind Map Generation** - Query Neo4j graph → Render D3.js visualizations
|
|
- **Wiki.js Proxy** - CRUD operations for dossiers
|
|
- **Deduplication** - Vector similarity + graph analysis
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Library Desk API (FastAPI)
|
|
├── Neo4j (knowledge graph)
|
|
├── Qdrant (vector search)
|
|
├── Wiki.js (wiki operations)
|
|
├── SearXNG (web search)
|
|
├── Ollama (embeddings)
|
|
└── Redis (caching)
|
|
```
|
|
|
|
## Requirements
|
|
|
|
- **Python**: 3.12+
|
|
- **Dependencies**: See `requirements.txt`
|
|
|
|
## Configuration
|
|
|
|
Environment variables (set in Portainer stack):
|
|
|
|
```bash
|
|
# Required
|
|
LIBRARY_API_KEY=<generate-with-openssl-rand-hex-32>
|
|
NEO4J_PASSWORD=<neo4j-password>
|
|
WIKIJS_API_KEY=<from-wiki-admin-panel>
|
|
|
|
# Optional (defaults provided)
|
|
NEO4J_URI=bolt://neo4j:7687
|
|
NEO4J_USER=neo4j
|
|
QDRANT_HOST=qdrant
|
|
QDRANT_PORT=6333
|
|
WIKIJS_URL=http://wiki:3000
|
|
SEARXNG_URL=http://searxng:8080
|
|
OLLAMA_URL=http://ollama:11434
|
|
OLLAMA_MODEL=mistral-nemo-large:latest
|
|
OLLAMA_EMBEDDING_MODEL=nomic-embed-text
|
|
REDIS_HOST=redis-shared
|
|
REDIS_PORT=6379
|
|
REDIS_DB=2
|
|
```
|
|
|
|
## API Endpoints
|
|
|
|
### System
|
|
|
|
- `GET /` - Root endpoint
|
|
- `GET /health` - Health check (public)
|
|
- `GET /stats` - System statistics (authenticated)
|
|
|
|
### Query (All require API key)
|
|
|
|
- `POST /query/hybrid` - HybridRAG (graph + vector + web)
|
|
- `POST /query/semantic` - Vector search only
|
|
- `POST /query/graph` - Graph traversal only
|
|
- `GET /query/related/{id}` - Find related content
|
|
|
|
### Content Management (Future)
|
|
|
|
- `POST /ingest/document` - Index new document
|
|
- `POST /ingest/wiki-page` - Sync Wiki.js page
|
|
- `POST /wiki/dossier` - Create dossier (proxies to Wiki.js)
|
|
- `PUT /wiki/dossier/{id}` - Update dossier
|
|
- `DELETE /wiki/dossier/{id}` - Delete dossier
|
|
|
|
### Graph Operations (Future)
|
|
|
|
- `GET /graph/entities` - List entities
|
|
- `GET /graph/mindmap/{id}` - Generate mind map
|
|
- `POST /graph/query` - Execute Cypher query
|
|
|
|
### Deduplication (Future)
|
|
|
|
- `POST /deduplicate/find` - Find duplicates
|
|
- `POST /deduplicate/merge` - Merge duplicates
|
|
|
|
## Development
|
|
|
|
### Local Setup
|
|
|
|
```bash
|
|
# Create virtual environment
|
|
python3 -m venv venv
|
|
source venv/bin/activate
|
|
|
|
# Install dependencies
|
|
pip install -r requirements.txt
|
|
|
|
# Run development server
|
|
uvicorn src.main:app --reload --host 0.0.0.0 --port 8089
|
|
```
|
|
|
|
### Project Structure
|
|
|
|
Following [FastAPI Best Practices](https://github.com/zhanymkanov/fastapi-best-practices):
|
|
|
|
```
|
|
library-desk/
|
|
├── src/
|
|
│ ├── __init__.py # Package initialization
|
|
│ ├── main.py # FastAPI application
|
|
│ ├── config.py # Pydantic settings
|
|
│ └── [future modules] # Domain-specific modules
|
|
├── requirements.txt # Python dependencies
|
|
└── README.md # This file
|
|
```
|
|
|
|
Future structure (as features are added):
|
|
|
|
```
|
|
library-desk/
|
|
├── src/
|
|
│ ├── query/ # Query domain
|
|
│ │ ├── router.py
|
|
│ │ ├── schemas.py
|
|
│ │ ├── service.py
|
|
│ │ └── dependencies.py
|
|
│ ├── graph/ # Graph domain
|
|
│ ├── wiki/ # Wiki domain
|
|
│ └── ingest/ # Ingestion domain
|
|
```
|
|
|
|
## Best Practices Implemented
|
|
|
|
✅ **Async routes** for I/O operations
|
|
✅ **Dependency injection** for configuration and auth
|
|
✅ **Pydantic models** for request/response validation
|
|
✅ **Modular settings** using Pydantic Settings
|
|
✅ **API key authentication** with Bearer tokens
|
|
✅ **OpenAPI documentation** auto-generated
|
|
✅ **Proper logging** with structured format
|
|
✅ **CORS middleware** configured
|
|
✅ **Health checks** for monitoring
|
|
✅ **Minor version locking** (`~=`) in requirements
|
|
✅ **CVE-checked dependencies** (Dec 2025)
|
|
|
|
## API Documentation
|
|
|
|
Once running, access:
|
|
|
|
- **Interactive docs**: http://localhost:8089/docs
|
|
- **ReDoc**: http://localhost:8089/redoc
|
|
- **OpenAPI spec**: http://localhost:8089/openapi.json
|
|
|
|
## Authentication
|
|
|
|
All protected endpoints require a Bearer token:
|
|
|
|
```bash
|
|
curl -H "Authorization: Bearer ${LIBRARY_API_KEY}" \
|
|
http://localhost:8089/stats
|
|
```
|
|
|
|
## Testing
|
|
|
|
```bash
|
|
# Run tests (when implemented)
|
|
pytest
|
|
|
|
# With coverage
|
|
pytest --cov=src --cov-report=term
|
|
```
|
|
|
|
## Deployment
|
|
|
|
Deployed via Portainer stack: `/stacks/library-desk.yml`
|
|
|
|
The container:
|
|
- Runs on port 8089
|
|
- Auto-creates venv on startup
|
|
- Installs dependencies from requirements.txt
|
|
- Starts uvicorn with 2 workers
|
|
- Mounts source code for live editing
|
|
|
|
## Monitoring
|
|
|
|
- **Uptime Kuma**: Monitor `/health` endpoint
|
|
- **Logs**: `docker logs library-desk`
|
|
- **Stats**: `GET /stats` (requires API key)
|
|
|
|
## Scheduler Integration
|
|
|
|
See [LIBRARIAN_INTEGRATION.md](./LIBRARIAN_INTEGRATION.md) for details on how The Scheduler (Librarian) integrates with Library Desk for automated documentation indexing.
|
|
|
|
**Key Workflow:**
|
|
1. Scheduler mirrors docs to Gitea (daily 03:00)
|
|
2. Scheduler syncs to Library Desk (daily 03:30)
|
|
3. Library Desk ingests, chunks, embeds, and indexes
|
|
4. Content becomes searchable via HybridRAG
|
|
|
|
## Future Enhancements
|
|
|
|
- [ ] Implement HybridRAG query logic
|
|
- [ ] Add Neo4j connection pooling
|
|
- [ ] Add Qdrant client initialization
|
|
- [ ] Implement Wiki.js API proxy
|
|
- [ ] Add entity extraction (spaCy/NLP)
|
|
- [ ] Implement mind map generation
|
|
- [ ] Add deduplication logic
|
|
- [ ] **Implement Scheduler integration endpoints** (see LIBRARIAN_INTEGRATION.md)
|
|
- [ ] Add comprehensive tests
|
|
- [ ] Add rate limiting
|
|
- [ ] Add request tracing
|
|
|
|
## References
|
|
|
|
- [FastAPI Best Practices](https://github.com/zhanymkanov/fastapi-best-practices)
|
|
- [FastAPI Documentation](https://fastapi.tiangolo.com/)
|
|
- [Pydantic Documentation](https://docs.pydantic.dev/)
|
|
- [Neo4j Python Driver](https://neo4j.com/docs/python-manual/current/)
|
|
- [Qdrant Python Client](https://python-client.qdrant.tech/)
|
|
|
|
## License
|
|
|
|
Part of Portainer Core infrastructure.
|
|
|
|
## Support
|
|
|
|
- Check logs: `docker logs library-desk`
|
|
- Health check: `curl http://localhost:8089/health`
|
|
- API docs: http://localhost:8089/docs
|