Initial commit: library-desk service extraction from portainer-core
Build and Push / build (release) Successful in 36s
Build and Push / build (release) Successful in 36s
This commit is contained in:
@@ -0,0 +1,236 @@
|
||||
# Library Desk - API Coordination Service
|
||||
|
||||
**FastAPI service that coordinates all Library operations**
|
||||
|
||||
## Overview
|
||||
|
||||
Library Desk is the central coordination layer for The Library system, providing a unified API for:
|
||||
|
||||
- **HybridRAG Queries** - Combines Neo4j (structure) + Qdrant (semantics) + SearXNG (web)
|
||||
- **Document Ingestion** - Parse, chunk, embed, and index documents
|
||||
- **Entity Extraction** - NLP to identify classes, functions, concepts
|
||||
- **Relationship Mapping** - Link entities in Neo4j knowledge graph
|
||||
- **Mind Map Generation** - Query Neo4j graph → Render D3.js visualizations
|
||||
- **Wiki.js Proxy** - CRUD operations for dossiers
|
||||
- **Deduplication** - Vector similarity + graph analysis
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Library Desk API (FastAPI)
|
||||
├── Neo4j (knowledge graph)
|
||||
├── Qdrant (vector search)
|
||||
├── Wiki.js (wiki operations)
|
||||
├── SearXNG (web search)
|
||||
├── Ollama (embeddings)
|
||||
└── Redis (caching)
|
||||
```
|
||||
|
||||
## Requirements
|
||||
|
||||
- **Python**: 3.12+
|
||||
- **Dependencies**: See `requirements.txt`
|
||||
|
||||
## Configuration
|
||||
|
||||
Environment variables (set in Portainer stack):
|
||||
|
||||
```bash
|
||||
# Required
|
||||
LIBRARY_API_KEY=<generate-with-openssl-rand-hex-32>
|
||||
NEO4J_PASSWORD=<neo4j-password>
|
||||
WIKIJS_API_KEY=<from-wiki-admin-panel>
|
||||
|
||||
# Optional (defaults provided)
|
||||
NEO4J_URI=bolt://neo4j:7687
|
||||
NEO4J_USER=neo4j
|
||||
QDRANT_HOST=qdrant
|
||||
QDRANT_PORT=6333
|
||||
WIKIJS_URL=http://wiki:3000
|
||||
SEARXNG_URL=http://searxng:8080
|
||||
OLLAMA_URL=http://ollama:11434
|
||||
OLLAMA_MODEL=nomic-embed-text
|
||||
REDIS_HOST=redis-shared
|
||||
REDIS_PORT=6379
|
||||
REDIS_DB=2
|
||||
```
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### System
|
||||
|
||||
- `GET /` - Root endpoint
|
||||
- `GET /health` - Health check (public)
|
||||
- `GET /stats` - System statistics (authenticated)
|
||||
|
||||
### Query (All require API key)
|
||||
|
||||
- `POST /query/hybrid` - HybridRAG (graph + vector + web)
|
||||
- `POST /query/semantic` - Vector search only
|
||||
- `POST /query/graph` - Graph traversal only
|
||||
- `GET /query/related/{id}` - Find related content
|
||||
|
||||
### Content Management (Future)
|
||||
|
||||
- `POST /ingest/document` - Index new document
|
||||
- `POST /ingest/wiki-page` - Sync Wiki.js page
|
||||
- `POST /wiki/dossier` - Create dossier (proxies to Wiki.js)
|
||||
- `PUT /wiki/dossier/{id}` - Update dossier
|
||||
- `DELETE /wiki/dossier/{id}` - Delete dossier
|
||||
|
||||
### Graph Operations (Future)
|
||||
|
||||
- `GET /graph/entities` - List entities
|
||||
- `GET /graph/mindmap/{id}` - Generate mind map
|
||||
- `POST /graph/query` - Execute Cypher query
|
||||
|
||||
### Deduplication (Future)
|
||||
|
||||
- `POST /deduplicate/find` - Find duplicates
|
||||
- `POST /deduplicate/merge` - Merge duplicates
|
||||
|
||||
## Development
|
||||
|
||||
### Local Setup
|
||||
|
||||
```bash
|
||||
# Create virtual environment
|
||||
python3 -m venv venv
|
||||
source venv/bin/activate
|
||||
|
||||
# Install dependencies
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Run development server
|
||||
uvicorn src.main:app --reload --host 0.0.0.0 --port 8089
|
||||
```
|
||||
|
||||
### Project Structure
|
||||
|
||||
Following [FastAPI Best Practices](https://github.com/zhanymkanov/fastapi-best-practices):
|
||||
|
||||
```
|
||||
library-desk/
|
||||
├── src/
|
||||
│ ├── __init__.py # Package initialization
|
||||
│ ├── main.py # FastAPI application
|
||||
│ ├── config.py # Pydantic settings
|
||||
│ └── [future modules] # Domain-specific modules
|
||||
├── requirements.txt # Python dependencies
|
||||
└── README.md # This file
|
||||
```
|
||||
|
||||
Future structure (as features are added):
|
||||
|
||||
```
|
||||
library-desk/
|
||||
├── src/
|
||||
│ ├── query/ # Query domain
|
||||
│ │ ├── router.py
|
||||
│ │ ├── schemas.py
|
||||
│ │ ├── service.py
|
||||
│ │ └── dependencies.py
|
||||
│ ├── graph/ # Graph domain
|
||||
│ ├── wiki/ # Wiki domain
|
||||
│ └── ingest/ # Ingestion domain
|
||||
```
|
||||
|
||||
## Best Practices Implemented
|
||||
|
||||
✅ **Async routes** for I/O operations
|
||||
✅ **Dependency injection** for configuration and auth
|
||||
✅ **Pydantic models** for request/response validation
|
||||
✅ **Modular settings** using Pydantic Settings
|
||||
✅ **API key authentication** with Bearer tokens
|
||||
✅ **OpenAPI documentation** auto-generated
|
||||
✅ **Proper logging** with structured format
|
||||
✅ **CORS middleware** configured
|
||||
✅ **Health checks** for monitoring
|
||||
✅ **Minor version locking** (`~=`) in requirements
|
||||
✅ **CVE-checked dependencies** (Dec 2025)
|
||||
|
||||
## API Documentation
|
||||
|
||||
Once running, access:
|
||||
|
||||
- **Interactive docs**: http://localhost:8089/docs
|
||||
- **ReDoc**: http://localhost:8089/redoc
|
||||
- **OpenAPI spec**: http://localhost:8089/openapi.json
|
||||
|
||||
## Authentication
|
||||
|
||||
All protected endpoints require a Bearer token:
|
||||
|
||||
```bash
|
||||
curl -H "Authorization: Bearer ${LIBRARY_API_KEY}" \
|
||||
http://localhost:8089/stats
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
# Run tests (when implemented)
|
||||
pytest
|
||||
|
||||
# With coverage
|
||||
pytest --cov=src --cov-report=term
|
||||
```
|
||||
|
||||
## Deployment
|
||||
|
||||
Deployed via Portainer stack: `/stacks/library-desk.yml`
|
||||
|
||||
The container:
|
||||
- Runs on port 8089
|
||||
- Auto-creates venv on startup
|
||||
- Installs dependencies from requirements.txt
|
||||
- Starts uvicorn with 2 workers
|
||||
- Mounts source code for live editing
|
||||
|
||||
## Monitoring
|
||||
|
||||
- **Uptime Kuma**: Monitor `/health` endpoint
|
||||
- **Logs**: `docker logs library-desk`
|
||||
- **Stats**: `GET /stats` (requires API key)
|
||||
|
||||
## Scheduler Integration
|
||||
|
||||
See [LIBRARIAN_INTEGRATION.md](./LIBRARIAN_INTEGRATION.md) for details on how The Scheduler (Librarian) integrates with Library Desk for automated documentation indexing.
|
||||
|
||||
**Key Workflow:**
|
||||
1. Scheduler mirrors docs to Gitea (daily 03:00)
|
||||
2. Scheduler syncs to Library Desk (daily 03:30)
|
||||
3. Library Desk ingests, chunks, embeds, and indexes
|
||||
4. Content becomes searchable via HybridRAG
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
- [ ] Implement HybridRAG query logic
|
||||
- [ ] Add Neo4j connection pooling
|
||||
- [ ] Add Qdrant client initialization
|
||||
- [ ] Implement Wiki.js API proxy
|
||||
- [ ] Add entity extraction (spaCy/NLP)
|
||||
- [ ] Implement mind map generation
|
||||
- [ ] Add deduplication logic
|
||||
- [ ] **Implement Scheduler integration endpoints** (see LIBRARIAN_INTEGRATION.md)
|
||||
- [ ] Add comprehensive tests
|
||||
- [ ] Add rate limiting
|
||||
- [ ] Add request tracing
|
||||
|
||||
## References
|
||||
|
||||
- [FastAPI Best Practices](https://github.com/zhanymkanov/fastapi-best-practices)
|
||||
- [FastAPI Documentation](https://fastapi.tiangolo.com/)
|
||||
- [Pydantic Documentation](https://docs.pydantic.dev/)
|
||||
- [Neo4j Python Driver](https://neo4j.com/docs/python-manual/current/)
|
||||
- [Qdrant Python Client](https://python-client.qdrant.tech/)
|
||||
|
||||
## License
|
||||
|
||||
Part of Portainer Core infrastructure.
|
||||
|
||||
## Support
|
||||
|
||||
- Check logs: `docker logs library-desk`
|
||||
- Health check: `curl http://localhost:8089/health`
|
||||
- API docs: http://localhost:8089/docs
|
||||
Reference in New Issue
Block a user