jpmschweitzer/library-desk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 5 Wiki Activity

feat: add Paperless-ngx document storage integration

Browse Source 
- Add /documents router with webhook, upload, search, health endpoints
- Create DocumentSyncService for indexing documents to vectors/graph
- Add PaperlessClient for REST API integration
- Configure dependency injection for Paperless client
- Add document models for webhook payloads and responses
- Event-driven architecture via Paperless workflow webhooks

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
jpmschweitzer
2025-12-25 14:16:40 +01:00
co-authored by Claude Opus 4.5
parent e6e65d6d78
commit 4ff3fc4c7a
12 changed files with 1856 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/DOCUMENT_STORAGE_PLAN.md
+509
View File
@@ -0,0 +1,509 @@
# Phase 3: Document Storage System - Implementation Plan
## Overview
Document storage tier for Library Desk - storing and indexing PDFs, images, videos, and git documentation mirrors.
**User Decisions:**
- Paperless-ngx container for OCR
- Ebooks deferred to future phase
- Video.js player deferred to after core implementation
| Phase | Status | Version |
|-------|--------|---------|
| Phase 1: Cleanup System | Complete | v1.4.0 |
| Phase 2: Volatile Memory | Complete | v1.4.3 |
| Phase 3: Document Storage | Planning | - |
| Phase 4: Test Data Cleanup | Complete | v1.4.4 |
---
## Architecture
**Paperless-ngx as primary document store** (no SeaweedFS needed):
```
┌─────────────────────────────────────────────────────────────────┐
│ External Sources │
│ ┌─────────┐ ┌────────────┐ ┌──────────────┐ │
│ │ GitHub │ │ Direct │ │ Email/Folder │ │
│ │ Docs │ │ Upload │ │ Ingestion │ │
│ └────┬────┘ └─────┬──────┘ └──────┬───────┘ │
└───────┼─────────────┼────────────────┼──────────────────────────┘
│ │ │
▼ ▼ ▼
┌─────────────────────────────────────────────────────────────────┐
│ Paperless-ngx │
│ ┌───────────────────────────────────────────────────────────┐ │
│ │ - Document storage (PDFs, images, videos) │ │
│ │ - OCR via Tesseract (PDFs, images) │ │
│ │ - Web UI for browsing/tagging │ │
│ │ - REST API for integration │ │
│ └─────────────────────────┬─────────────────────────────────┘ │
└────────────────────────────┼────────────────────────────────────┘
│ REST API (sync)
▼
┌─────────────────────────────────────────────────────────────────┐
│ Library Desk │
│ ┌───────────────────────────────────────────────────────────┐ │
│ │ DocumentSyncService │ │
│ │ - Polls Paperless for new/updated docs │ │
│ │ - Extracts text + metadata via API │ │
│ │ - Sends to vector/graph pipelines │ │
│ └─────────────────────────┬─────────────────────────────────┘ │
│ │ │
│ ┌────────────────┼────────────────┐ │
│ ▼ ▼ ▼ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Qdrant │ │ Neo4j │ │ Wiki.js │ │
│ │ (vectors)│ │ (graph) │ │ (catalog)│ │
│ └──────────┘ └──────────┘ └──────────┘ │
└─────────────────────────────────────────────────────────────────┘
```
**File handling by type:**
| File Type | Paperless | Library Desk |
|-----------|-----------|--------------|
| PDFs | OCR → text | Index text → vectors/graph |
| Images | OCR → text | Index text → vectors/graph |
| Videos | Storage only | Index metadata → vectors/graph |
---
## Technology Stack
| Component | Purpose | Rationale |
|-----------|---------|-----------|
| **Paperless-ngx** | Document storage + OCR | All-in-one: storage, OCR, web UI, REST API |
| **ClamAV** | Virus scanning | Host OS install, pyclamd integration, better isolation |
| **PDF.js** | PDF viewer | Embeddable in Wiki.js (deferred) |
**Why Paperless-ngx as primary store:**
- Eliminates need for separate blob storage (SeaweedFS/MinIO)
- Built-in web UI for browsing and tagging
- Tesseract OCR with 100+ language support
- REST API for Library Desk integration
- Handles videos as raw files (no OCR, but stored)
- Email and folder watching for automatic ingestion
- Active community, well-maintained
---
## Paperless-ngx API Deep Dive
### Authentication
```
POST /api/token/
Body: {"username": "...", "password": "..."}
Response: {"token": "..."}
Header: Authorization: Token <token>
```
### Document Upload (for HybridRAG → Paperless)
```
POST /api/documents/post_document/
Content-Type: multipart/form-data
Fields:
- document (file, required)
- title (string)
- created (datetime)
- correspondent (ID)
- document_type (ID)
- storage_path (ID)
- tags (repeatable IDs)
- custom_fields (JSON array)
Response: {"task_id": "uuid"}
```
Track consumption: `GET /api/tasks/?task_id={uuid}` → returns document ID when complete
### Document Search
```
GET /api/documents/?query=search+terms # Full-text search
GET /api/documents/?more_like_id=123 # Similarity search
Response includes __search_hit__:
{
"score": 0.95,
"highlights": "<span>matched</span> text",
"rank": 0
}
```
### Custom Field Filtering
```
GET /api/documents/?custom_field_query=field_name__operation=value
Operations:
- exact, in, isnull, exists (all types)
- icontains, istartswith, iendswith (text)
- gt, gte, lt, lte, range (numeric/date)
- contains (document links)
```
### Bulk Operations
```
POST /api/documents/bulk_edit/
{
"documents": [1, 2, 3],
"method": "add_tag|remove_tag|set_correspondent|set_document_type|merge|split|...",
"parameters": {...}
}
```
### Webhooks (Push to Library Desk!)
Paperless workflows can trigger webhooks on document events:
| Trigger | When | Available Data |
|---------|------|----------------|
| Consumption Started | Before OCR | file_path, source, filename |
| Document Added | After OCR | content, tags, doc_type, correspondent, `{doc_url}` |
| Document Updated | On change | Same as Added |
| Scheduled | Time-based | Date offsets from document dates |
**Webhook Action**: POST to Library Desk endpoint with document data
### Organization Features
| Feature | Purpose | API Endpoint |
|---------|---------|--------------|
| Tags | Nested labels (5 levels deep) | `/api/tags/` |
| Correspondents | Source/destination | `/api/correspondents/` |
| Document Types | Classification | `/api/document_types/` |
| Storage Paths | File organization | `/api/storage_paths/` |
| Custom Fields | Extensible metadata | `/api/custom_fields/` |
### Custom Fields We Should Create
| Field Name | Type | Purpose |
|------------|------|---------|
| `source_url` | URL | Original download URL (for HybridRAG uploads) |
| `library_indexed` | Boolean | Sync status with Library Desk |
| `library_doc_id` | Text | Library Desk document reference |
| `collection` | Text | Logical grouping (e.g., "fastapi-docs") |
### External LLM Add-ons (Optional)
Community tools exist for Ollama integration:
- **[paperless-ai](https://github.com/clusterzx/paperless-ai)** - Auto-tagging, RAG chat
- **[paperless-gpt](https://github.com/icereed/paperless-gpt)** - LLM-enhanced OCR, auto-titling
**Recommendation:** Skip these - Library Desk already has Ollama integration for:
- Embedding (nomic-embed-text)
- LLM analysis (mistral-nemo)
- Entity extraction
- HybridRAG
We'll do our own classification/tagging via Library Desk after sync.
---
## Virus Scanning Integration
**ClamAV daemon + pyclamd** (no third-party REST wrappers):
```
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ File Upload │────►│ Library Desk │────►│ ClamAV Daemon │
│ (URL or file) │ │ (pyclamd) │ │ (clamd:3310) │
└─────────────────┘ └────────┬────────┘ └─────────────────┘
│
┌────────────┴────────────┐
▼ ▼
┌──────────┐ ┌──────────┐
│ Clean │ │ Infected │
│ ✓ │ │ ✗ │
└────┬─────┘ └────┬─────┘
│ │
▼ ▼
Upload to Paperless Reject + Log
```
### ClamAV Deployment (Host OS)
ClamAV runs on the host OS (not containerized) for better security isolation:
```bash
# Installed via apt on Ubuntu/Debian
# Config: /etc/clamav/clamd.conf
# TCPSocket 3310
# TCPAddr 0.0.0.0
```
Benefits: scans outside container isolation, single virus DB, survives container restarts.
### Library Desk Integration
```python
# src/clients/clamav_client.py
import pyclamd
class ClamAVClient:
def __init__(self, host: str, port: int = 3310):
self.cd = pyclamd.ClamdNetworkSocket(host, port)
async def scan_bytes(self, data: bytes) -> ScanResult:
"""Scan file bytes, return clean/infected status."""
result = self.cd.scan_stream(data)
if result is None:
return ScanResult(clean=True)
return ScanResult(clean=False, virus_name=result['stream'][1])
def ping(self) -> bool:
"""Health check."""
return self.cd.ping()
```
### Scan Points
| Location | When | Action on Infected |
|----------|------|-------------------|
| `/documents/upload` | Before Paperless upload | Reject with 400, log threat |
| HybridRAG web fetch | Before saving PDF | Skip file, log threat |
| `/documents/webhook` | Optional re-scan | Quarantine in Paperless |
### Config Settings
```python
# src/config.py
CLAMAV_HOST: str = "192.168.86.149" # Host OS IP (not container)
CLAMAV_PORT: int = 3310
CLAMAV_ENABLED: bool = True # Bypass for testing
CLAMAV_TIMEOUT: int = 30 # seconds
```
---
## Integration Strategy
### Option A: Webhook Push (Preferred)
```
Paperless Workflow → POST webhook → Library Desk /documents/webhook
```
- Real-time indexing when documents added/updated
- Configure in Paperless: Workflow → Document Added → Webhook Action
- Library Desk receives document ID, fetches content via API
### Option B: Polling Pull (Fallback)
```
Scheduler → POST /documents/sync → Library Desk polls Paperless
```
- Periodic sync for missed webhooks or initial bulk import
- Track `library_indexed` custom field to skip already-processed docs
### Option C: HybridRAG Upload (New!)
```
HybridRAG web search → finds PDF → POST to Paperless → webhook → indexed
```
- When HybridRAG finds a relevant PDF/document in web results
- Download and upload to Paperless with `source_url` custom field
- Paperless OCRs it, triggers webhook, Library Desk indexes
---
## Library Desk API Design
### Documents Router (`/documents`)
| Endpoint | Method | Purpose |
|----------|--------|---------|
| `/documents/webhook` | POST | Receive Paperless webhook (Document Added/Updated) |
| `/documents/sync` | POST | Pull new/updated docs from Paperless → index |
| `/documents/upload` | POST | Upload file to Paperless (for HybridRAG) |
| `/documents/sync-from-git` | POST | Pull docs from Gitea → upload to Paperless → index |
| `/documents/{document_id}` | GET | Get document metadata |
| `/documents/{document_id}/text` | GET | Get extracted text |
| `/documents/search` | POST | Semantic search across documents |
| `/documents/collection/{name}` | GET | List documents in collection |
| `/documents/collection/{name}/catalog` | POST | Generate wiki catalog page |
**Upload flow (HybridRAG → Paperless):**
1. HybridRAG finds PDF in web results
2. POST `/documents/upload` with URL or file
3. Library Desk downloads, uploads to Paperless with metadata
4. Returns task_id for async tracking
5. Paperless webhook triggers indexing when OCR complete
### Viewers Router (`/viewers`) - Deferred
| Endpoint | Method | Purpose |
|----------|--------|---------|
| `/viewers/pdf/{document_id}` | GET | Serve PDF.js viewer |
| `/viewers/image/{document_id}` | GET | Serve image lightbox |
| `/viewers/video/{document_id}` | GET | Serve Video.js player |
---
## Data Flow: Document Processing Pipeline
```
1. INTAKE (Paperless-ngx handles this)
└─ Upload via Paperless UI, email, or folder watch
└─ Paperless assigns document ID and stores file
2. OCR EXTRACTION (Paperless-ngx handles this)
├─ PDFs → Tesseract → Plain text
├─ Images → Tesseract → Plain text
└─ Videos → Metadata only (no OCR)
3. SYNC TO LIBRARY DESK (scheduled or manual)
└─ Poll Paperless API for new/updated documents
└─ Fetch text content + metadata
4. TEXT CHUNKING
└─ VectorService._chunk_text() (existing)
5. EMBEDDING
└─ OllamaClient.embed() (existing)
6. VECTOR STORAGE (Qdrant)
└─ Payload: {doc_type: "document", paperless_id, ...}
7. GRAPH STORAGE (Neo4j)
└─ Document node + MENTIONS relationships
8. WIKI CATALOG (optional)
└─ Auto-generate catalog page via ConsolidationService
```
---
## Git Docs Integration
Extends existing `scheduler/src/executors/doc_sync_executor.py`:
1. **Scheduler** syncs docs from GitHub → Gitea (existing)
2. **Post-sync hook** calls `POST /documents/sync-from-git`
3. **Library Desk** indexes docs into vectors/graph
4. **Auto-generate** wiki catalog page for collection
---
## Wiki.js Viewer Integration
Since Wiki.js v2 requires disabled HTML sanitization for iframes:
```markdown
<!-- In wiki catalog page -->
## Document Preview
<iframe
src="http://library-desk:8089/viewers/pdf/abc123"
width="100%" height="600px">
</iframe>
```
**Wiki.js Settings Required:**
- `Administration > Security > Allowed HTML Elements: iframe`
- `Content Security Policy: frame-src http://library-desk:8089`
---
## Implementation Phases
### Phase 3.1: Infrastructure Setup
- [ ] Deploy Paperless-ngx container (Docker Compose)
- [x] ClamAV installed on host OS (port 3310)
- [ ] Configure Paperless: storage path, OCR settings, API token
- [ ] Create custom fields in Paperless: `source_url`, `library_indexed`, `library_doc_id`, `collection`
- [ ] Create `src/clients/paperless_client.py`
- [ ] Create `src/clients/clamav_client.py` (pyclamd wrapper)
- [ ] Create `src/models/document.py`
- [ ] Add config settings to `src/config.py` (PAPERLESS_*, CLAMAV_*)
### Phase 3.2: Webhook Integration (Push)
- [ ] Create `src/routers/documents.py`
- [ ] Implement `/documents/webhook` endpoint (receives Paperless events)
- [ ] Configure Paperless Workflow: Document Added → Webhook → Library Desk
- [ ] Create `src/services/document_sync_service.py`
- [ ] Implement document indexing pipeline (fetch text → chunk → embed → graph)
### Phase 3.3: Polling Sync (Pull Fallback)
- [ ] Implement `/documents/sync` endpoint
- [ ] Poll Paperless for docs where `library_indexed=false`
- [ ] Track sync state (last_sync timestamp in Redis)
- [ ] Update `library_indexed` after successful indexing
### Phase 3.4: HybridRAG Upload Integration
- [ ] Implement `/documents/upload` endpoint
- [ ] Download file from URL
- [ ] **Virus scan before upload** (reject if infected, log threat)
- [ ] Upload clean files to Paperless with metadata
- [ ] Set `source_url` custom field
- [ ] Extend HybridRAG service to detect and upload relevant PDFs
- [ ] Add `save_to_documents` option to HybridRAG config
### Phase 3.5: Indexing Pipeline
- [ ] Extend VectorService for `doc_type: "document"`
- [ ] Extend GraphService for Document nodes (link to Paperless ID)
- [ ] Implement `/documents/search` endpoint
- [ ] Add dependency injection
### Phase 3.6: Git Docs Integration
- [ ] Create `src/clients/gitea_client.py`
- [ ] Implement `/documents/sync-from-git` → bulk upload to Paperless
- [ ] Create collection auto-cataloging (wiki pages)
- [ ] Add scheduler task for periodic git sync
### Phase 3.7: Viewers (Deferred)
*After core implementation is working*
- [ ] Create `static/pdf-viewer.html` (PDF.js)
- [ ] Create `static/image-viewer.html`
- [ ] Create `static/video-player.html` (Video.js)
- [ ] Create `src/routers/viewers.py`
### Phase 3.8: Maintenance & Testing
- [ ] Extend cleanup for document orphans
- [ ] Add document orphan detection (Paperless deleted but still in Qdrant/Neo4j)
- [ ] Create `tests/test_document_sync.py`
- [ ] Create `tests/test_paperless_client.py`
---
## Files to Create
| Path | Purpose |
|------|---------|
| `src/clients/paperless_client.py` | Paperless-ngx REST API client |
| `src/clients/clamav_client.py` | ClamAV scanner (pyclamd wrapper) |
| `src/clients/gitea_client.py` | Gitea repo access |
| `src/models/document.py` | Document/Collection/ScanResult models |
| `src/services/document_sync_service.py` | Sync orchestrator |
| `src/routers/documents.py` | Document endpoints (webhook, sync, upload, search) |
| `tests/test_document_sync.py` | Sync service tests |
| `tests/test_paperless_client.py` | API client tests |
| `tests/test_clamav_client.py` | Virus scanner tests |
| `docker/docker-compose.documents.yml` | Paperless + ClamAV deployment |
**Deferred files (Phase 3.7):**
| Path | Purpose |
|------|---------|
| `src/routers/viewers.py` | Viewer endpoints |
| `static/pdf-viewer.html` | PDF.js viewer |
| `static/image-viewer.html` | Image lightbox |
| `static/video-player.html` | Video.js player |
## Files to Modify
| Path | Changes |
|------|---------|
| `src/config.py` | `PAPERLESS_*`, `CLAMAV_*` settings |
| `src/core/dependencies.py` | DocumentSyncService, PaperlessClient, ClamAVClient DI |
| `src/main.py` | Register documents router |
| `src/services/vector_service.py` | `doc_type: "document"` handling |
| `src/services/graph_service.py` | Document node with Paperless ID |
| `src/services/hybrid_rag_service.py` | Add `save_to_documents` option + virus scan |
| `src/models/hybrid_rag.py` | Add `save_to_documents` config |
| `src/routers/maintenance.py` | Document orphan cleanup, ClamAV health check |
| `requirements.txt` | Add `pyclamd` |
## Paperless Custom Fields Setup
Create these in Paperless UI (Administration → Custom Fields):
| Field | Type | Purpose |
|-------|------|---------|
| `source_url` | URL | Original download URL |
| `library_indexed` | Boolean | Sync status |
| `library_doc_id` | Text | Library Desk reference |
| `collection` | Text | Logical grouping |
docs/MEMORY_SYSTEM_PLAN.md
+2 -2
View File
@@ -7,7 +7,7 @@ A three-tier memory architecture for Library Desk with intelligent orchestration
| Tier | Storage | Purpose | TTL |
|------|---------|---------|-----|
| **Volatile** | Qdrant (vectors) | Weather, news, financial, ephemeral context | 5min - 2hr |
| **Documents** | TBD (research) | Git mirrors, PDFs, video, images | Permanent |
| **Documents** | Paperless-ngx + ClamAV (host) | Git mirrors, PDFs, video, images | Permanent |
| **Knowledge** | Wiki + Neo4j | Personal dossiers, research, summaries | Permanent |
**Implementation Priority**: Cleanup → Volatile → Documents → Test Data Cleanup
@@ -18,7 +18,7 @@ A three-tier memory architecture for Library Desk with intelligent orchestration
|-------|--------|---------|
| Phase 1: Cleanup System | ✅ Complete | v1.4.0 |
| Phase 2: Volatile Memory | ✅ Complete | v1.4.3 |
| Phase 3: Document Storage | ⏳ Pending | - |
| Phase 3: Document Storage | ✅ Planned | See [DOCUMENT_STORAGE_PLAN.md](DOCUMENT_STORAGE_PLAN.md) |
| Phase 4: Test Data Cleanup | ✅ Complete | v1.4.4 |
---