library-desk/docs/DOCUMENT_STORAGE_PLAN.md

# Phase 3: Document Storage System - Implementation Plan

## Overview

Document storage tier for Library Desk - storing and indexing PDFs, images, videos, and git documentation mirrors.

**User Decisions:**
- Paperless-ngx container for OCR
- Ebooks deferred to future phase
- Video.js player deferred to after core implementation

| Phase | Status | Version |
|-------|--------|---------|
| Phase 1: Cleanup System | Complete | v1.4.0 |
| Phase 2: Volatile Memory | Complete | v1.4.3 |
| Phase 3: Document Storage | Planning | - |
| Phase 4: Test Data Cleanup | Complete | v1.4.4 |

---

## Architecture

**Paperless-ngx as primary document store** (no SeaweedFS needed):

```
┌─────────────────────────────────────────────────────────────────┐
│                     External Sources                             │
│  ┌─────────┐  ┌────────────┐  ┌──────────────┐                  │
│  │ GitHub  │  │ Direct     │  │ Email/Folder │                  │
│  │ Docs    │  │ Upload     │  │ Ingestion    │                  │
│  └────┬────┘  └─────┬──────┘  └──────┬───────┘                  │
└───────┼─────────────┼────────────────┼──────────────────────────┘
        │             │                │
        ▼             ▼                ▼
┌─────────────────────────────────────────────────────────────────┐
│                      Paperless-ngx                               │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │  - Document storage (PDFs, images, videos)                │  │
│  │  - OCR via Tesseract (PDFs, images)                       │  │
│  │  - Web UI for browsing/tagging                            │  │
│  │  - REST API for integration                               │  │
│  └─────────────────────────┬─────────────────────────────────┘  │
└────────────────────────────┼────────────────────────────────────┘
                             │ REST API (sync)
                             ▼
┌─────────────────────────────────────────────────────────────────┐
│                        Library Desk                              │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │              DocumentSyncService                          │  │
│  │  - Polls Paperless for new/updated docs                   │  │
│  │  - Extracts text + metadata via API                       │  │
│  │  - Sends to vector/graph pipelines                        │  │
│  └─────────────────────────┬─────────────────────────────────┘  │
│                            │                                     │
│           ┌────────────────┼────────────────┐                   │
│           ▼                ▼                ▼                   │
│      ┌──────────┐    ┌──────────┐    ┌──────────┐              │
│      │  Qdrant  │    │  Neo4j   │    │ Wiki.js  │              │
│      │ (vectors)│    │ (graph)  │    │ (catalog)│              │
│      └──────────┘    └──────────┘    └──────────┘              │
└─────────────────────────────────────────────────────────────────┘
```

**File handling by type:**

| File Type | Paperless | Library Desk |
|-----------|-----------|--------------|
| PDFs | OCR → text | Index text → vectors/graph |
| Images | OCR → text | Index text → vectors/graph |
| Videos | Storage only | Index metadata → vectors/graph |

---

## Technology Stack

| Component | Purpose | Rationale |
|-----------|---------|-----------|
| **Paperless-ngx** | Document storage + OCR | All-in-one: storage, OCR, web UI, REST API |
| **ClamAV** | Virus scanning | Host OS install, pyclamd integration, better isolation |
| **PDF.js** | PDF viewer | Embeddable in Wiki.js (deferred) |

**Why Paperless-ngx as primary store:**
- Eliminates need for separate blob storage (SeaweedFS/MinIO)
- Built-in web UI for browsing and tagging
- Tesseract OCR with 100+ language support
- REST API for Library Desk integration
- Handles videos as raw files (no OCR, but stored)
- Email and folder watching for automatic ingestion
- Active community, well-maintained

---

## Paperless-ngx API Deep Dive

### Authentication
```
POST /api/token/
Body: {"username": "...", "password": "..."}
Response: {"token": "..."}

Header: Authorization: Token <token>
```

### Document Upload (for HybridRAG → Paperless)
```
POST /api/documents/post_document/
Content-Type: multipart/form-data

Fields:
- document (file, required)
- title (string)
- created (datetime)
- correspondent (ID)
- document_type (ID)
- storage_path (ID)
- tags (repeatable IDs)
- custom_fields (JSON array)

Response: {"task_id": "uuid"}
```

Track consumption: `GET /api/tasks/?task_id={uuid}` → returns document ID when complete

### Document Search
```
GET /api/documents/?query=search+terms     # Full-text search
GET /api/documents/?more_like_id=123       # Similarity search

Response includes __search_hit__:
{
  "score": 0.95,
  "highlights": "<span>matched</span> text",
  "rank": 0
}
```

### Custom Field Filtering
```
GET /api/documents/?custom_field_query=field_name__operation=value

Operations:
- exact, in, isnull, exists (all types)
- icontains, istartswith, iendswith (text)
- gt, gte, lt, lte, range (numeric/date)
- contains (document links)
```

### Bulk Operations
```
POST /api/documents/bulk_edit/
{
  "documents": [1, 2, 3],
  "method": "add_tag|remove_tag|set_correspondent|set_document_type|merge|split|...",
  "parameters": {...}
}
```

### Webhooks (Push to Library Desk!)
Paperless workflows can trigger webhooks on document events:

| Trigger | When | Available Data |
|---------|------|----------------|
| Consumption Started | Before OCR | file_path, source, filename |
| Document Added | After OCR | content, tags, doc_type, correspondent, `{doc_url}` |
| Document Updated | On change | Same as Added |
| Scheduled | Time-based | Date offsets from document dates |

**Webhook Action**: POST to Library Desk endpoint with document data

### Organization Features

| Feature | Purpose | API Endpoint |
|---------|---------|--------------|
| Tags | Nested labels (5 levels deep) | `/api/tags/` |
| Correspondents | Source/destination | `/api/correspondents/` |
| Document Types | Classification | `/api/document_types/` |
| Storage Paths | File organization | `/api/storage_paths/` |
| Custom Fields | Extensible metadata | `/api/custom_fields/` |

### Custom Fields We Should Create
| Field Name | Type | Purpose |
|------------|------|---------|
| `source_url` | URL | Original download URL (for HybridRAG uploads) |
| `library_indexed` | Boolean | Sync status with Library Desk |
| `library_doc_id` | Text | Library Desk document reference |
| `collection` | Text | Logical grouping (e.g., "fastapi-docs") |

### External LLM Add-ons (Optional)
Community tools exist for Ollama integration:
- **[paperless-ai](https://github.com/clusterzx/paperless-ai)** - Auto-tagging, RAG chat
- **[paperless-gpt](https://github.com/icereed/paperless-gpt)** - LLM-enhanced OCR, auto-titling

**Recommendation:** Skip these - Library Desk already has Ollama integration for:
- Embedding (nomic-embed-text)
- LLM analysis (mistral-nemo)
- Entity extraction
- HybridRAG

We'll do our own classification/tagging via Library Desk after sync.

---

## Virus Scanning Integration

**ClamAV daemon + pyclamd** (no third-party REST wrappers):

```
┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│  File Upload    │────►│  Library Desk   │────►│  ClamAV Daemon  │
│  (URL or file)  │     │  (pyclamd)      │     │  (clamd:3310)   │
└─────────────────┘     └────────┬────────┘     └─────────────────┘
                                 │
                    ┌────────────┴────────────┐
                    ▼                         ▼
              ┌──────────┐              ┌──────────┐
              │  Clean   │              │ Infected │
              │  ✓       │              │  ✗       │
              └────┬─────┘              └────┬─────┘
                   │                         │
                   ▼                         ▼
          Upload to Paperless         Reject + Log
```

### ClamAV Deployment (Host OS)

ClamAV runs on the host OS (not containerized) for better security isolation:

```bash
# Installed via apt on Ubuntu/Debian
# Config: /etc/clamav/clamd.conf
# TCPSocket 3310
# TCPAddr 0.0.0.0
```

Benefits: scans outside container isolation, single virus DB, survives container restarts.

### Library Desk Integration
```python
# src/clients/clamav_client.py
import pyclamd

class ClamAVClient:
    def __init__(self, host: str, port: int = 3310):
        self.cd = pyclamd.ClamdNetworkSocket(host, port)

    async def scan_bytes(self, data: bytes) -> ScanResult:
        """Scan file bytes, return clean/infected status."""
        result = self.cd.scan_stream(data)
        if result is None:
            return ScanResult(clean=True)
        return ScanResult(clean=False, virus_name=result['stream'][1])

    def ping(self) -> bool:
        """Health check."""
        return self.cd.ping()
```

### Scan Points
| Location | When | Action on Infected |
|----------|------|-------------------|
| `/documents/upload` | Before Paperless upload | Reject with 400, log threat |
| HybridRAG web fetch | Before saving PDF | Skip file, log threat |
| `/documents/webhook` | Optional re-scan | Quarantine in Paperless |

### Config Settings
```python
# src/config.py
CLAMAV_HOST: str = "192.168.86.149"  # Host OS IP (not container)
CLAMAV_PORT: int = 3310
CLAMAV_ENABLED: bool = True  # Bypass for testing
CLAMAV_TIMEOUT: int = 30  # seconds
```

---

## Integration Strategy

### Option A: Webhook Push (Preferred)
```
Paperless Workflow → POST webhook → Library Desk /documents/webhook
```
- Real-time indexing when documents added/updated
- Configure in Paperless: Workflow → Document Added → Webhook Action
- Library Desk receives document ID, fetches content via API

### Option B: Polling Pull (Fallback)
```
Scheduler → POST /documents/sync → Library Desk polls Paperless
```
- Periodic sync for missed webhooks or initial bulk import
- Track `library_indexed` custom field to skip already-processed docs

### Option C: HybridRAG Upload (New!)
```
HybridRAG web search → finds PDF → POST to Paperless → webhook → indexed
```
- When HybridRAG finds a relevant PDF/document in web results
- Download and upload to Paperless with `source_url` custom field
- Paperless OCRs it, triggers webhook, Library Desk indexes

---

## Library Desk API Design

### Documents Router (`/documents`)

| Endpoint | Method | Purpose |
|----------|--------|---------|
| `/documents/webhook` | POST | Receive Paperless webhook (Document Added/Updated) |
| `/documents/sync` | POST | Pull new/updated docs from Paperless → index |
| `/documents/upload` | POST | Upload file to Paperless (for HybridRAG) |
| `/documents/sync-from-git` | POST | Pull docs from Gitea → upload to Paperless → index |
| `/documents/{document_id}` | GET | Get document metadata |
| `/documents/{document_id}/text` | GET | Get extracted text |
| `/documents/search` | POST | Semantic search across documents |
| `/documents/collection/{name}` | GET | List documents in collection |
| `/documents/collection/{name}/catalog` | POST | Generate wiki catalog page |

**Upload flow (HybridRAG → Paperless):**
1. HybridRAG finds PDF in web results
2. POST `/documents/upload` with URL or file
3. Library Desk downloads, uploads to Paperless with metadata
4. Returns task_id for async tracking
5. Paperless webhook triggers indexing when OCR complete

### Viewers Router (`/viewers`) - Deferred

| Endpoint | Method | Purpose |
|----------|--------|---------|
| `/viewers/pdf/{document_id}` | GET | Serve PDF.js viewer |
| `/viewers/image/{document_id}` | GET | Serve image lightbox |
| `/viewers/video/{document_id}` | GET | Serve Video.js player |

---

## Data Flow: Document Processing Pipeline

```
1. INTAKE (Paperless-ngx handles this)
   └─ Upload via Paperless UI, email, or folder watch
   └─ Paperless assigns document ID and stores file

2. OCR EXTRACTION (Paperless-ngx handles this)
   ├─ PDFs → Tesseract → Plain text
   ├─ Images → Tesseract → Plain text
   └─ Videos → Metadata only (no OCR)

3. SYNC TO LIBRARY DESK (scheduled or manual)
   └─ Poll Paperless API for new/updated documents
   └─ Fetch text content + metadata

4. TEXT CHUNKING
   └─ VectorService._chunk_text() (existing)

5. EMBEDDING
   └─ OllamaClient.embed() (existing)

6. VECTOR STORAGE (Qdrant)
   └─ Payload: {doc_type: "document", paperless_id, ...}

7. GRAPH STORAGE (Neo4j)
   └─ Document node + MENTIONS relationships

8. WIKI CATALOG (optional)
   └─ Auto-generate catalog page via ConsolidationService
```

---

## Git Docs Integration

Extends existing `scheduler/src/executors/doc_sync_executor.py`:

1. **Scheduler** syncs docs from GitHub → Gitea (existing)
2. **Post-sync hook** calls `POST /documents/sync-from-git`
3. **Library Desk** indexes docs into vectors/graph
4. **Auto-generate** wiki catalog page for collection

---

## Wiki.js Viewer Integration

Since Wiki.js v2 requires disabled HTML sanitization for iframes:

```markdown
<!-- In wiki catalog page -->
## Document Preview

<iframe
  src="http://library-desk:8089/viewers/pdf/abc123"
  width="100%" height="600px">
</iframe>
```

**Wiki.js Settings Required:**
- `Administration > Security > Allowed HTML Elements: iframe`
- `Content Security Policy: frame-src http://library-desk:8089`

---

## Implementation Phases

### Phase 3.1: Infrastructure Setup
- [ ] Deploy Paperless-ngx container (Docker Compose)
- [x] ClamAV installed on host OS (port 3310)
- [ ] Configure Paperless: storage path, OCR settings, API token
- [ ] Create custom fields in Paperless: `source_url`, `library_indexed`, `library_doc_id`, `collection`
- [ ] Create `src/clients/paperless_client.py`
- [ ] Create `src/clients/clamav_client.py` (pyclamd wrapper)
- [ ] Create `src/models/document.py`
- [ ] Add config settings to `src/config.py` (PAPERLESS_*, CLAMAV_*)

### Phase 3.2: Webhook Integration (Push)
- [ ] Create `src/routers/documents.py`
- [ ] Implement `/documents/webhook` endpoint (receives Paperless events)
- [ ] Configure Paperless Workflow: Document Added → Webhook → Library Desk
- [ ] Create `src/services/document_sync_service.py`
- [ ] Implement document indexing pipeline (fetch text → chunk → embed → graph)

### Phase 3.3: Polling Sync (Pull Fallback)
- [ ] Implement `/documents/sync` endpoint
- [ ] Poll Paperless for docs where `library_indexed=false`
- [ ] Track sync state (last_sync timestamp in Redis)
- [ ] Update `library_indexed` after successful indexing

### Phase 3.4: HybridRAG Upload Integration
- [ ] Implement `/documents/upload` endpoint
- [ ] Download file from URL
- [ ] **Virus scan before upload** (reject if infected, log threat)
- [ ] Upload clean files to Paperless with metadata
- [ ] Set `source_url` custom field
- [ ] Extend HybridRAG service to detect and upload relevant PDFs
- [ ] Add `save_to_documents` option to HybridRAG config

### Phase 3.5: Indexing Pipeline
- [ ] Extend VectorService for `doc_type: "document"`
- [ ] Extend GraphService for Document nodes (link to Paperless ID)
- [ ] Implement `/documents/search` endpoint
- [ ] Add dependency injection

### Phase 3.6: Git Docs Integration
- [ ] Create `src/clients/gitea_client.py`
- [ ] Implement `/documents/sync-from-git` → bulk upload to Paperless
- [ ] Create collection auto-cataloging (wiki pages)
- [ ] Add scheduler task for periodic git sync

### Phase 3.7: Viewers (Deferred)
*After core implementation is working*
- [ ] Create `static/pdf-viewer.html` (PDF.js)
- [ ] Create `static/image-viewer.html`
- [ ] Create `static/video-player.html` (Video.js)
- [ ] Create `src/routers/viewers.py`

### Phase 3.8: Maintenance & Testing
- [ ] Extend cleanup for document orphans
- [ ] Add document orphan detection (Paperless deleted but still in Qdrant/Neo4j)
- [ ] Create `tests/test_document_sync.py`
- [ ] Create `tests/test_paperless_client.py`

---

## Files to Create

| Path | Purpose |
|------|---------|
| `src/clients/paperless_client.py` | Paperless-ngx REST API client |
| `src/clients/clamav_client.py` | ClamAV scanner (pyclamd wrapper) |
| `src/clients/gitea_client.py` | Gitea repo access |
| `src/models/document.py` | Document/Collection/ScanResult models |
| `src/services/document_sync_service.py` | Sync orchestrator |
| `src/routers/documents.py` | Document endpoints (webhook, sync, upload, search) |
| `tests/test_document_sync.py` | Sync service tests |
| `tests/test_paperless_client.py` | API client tests |
| `tests/test_clamav_client.py` | Virus scanner tests |
| `docker/docker-compose.documents.yml` | Paperless + ClamAV deployment |

**Deferred files (Phase 3.7):**

| Path | Purpose |
|------|---------|
| `src/routers/viewers.py` | Viewer endpoints |
| `static/pdf-viewer.html` | PDF.js viewer |
| `static/image-viewer.html` | Image lightbox |
| `static/video-player.html` | Video.js player |

## Files to Modify

| Path | Changes |
|------|---------|
| `src/config.py` | `PAPERLESS_*`, `CLAMAV_*` settings |
| `src/core/dependencies.py` | DocumentSyncService, PaperlessClient, ClamAVClient DI |
| `src/main.py` | Register documents router |
| `src/services/vector_service.py` | `doc_type: "document"` handling |
| `src/services/graph_service.py` | Document node with Paperless ID |
| `src/services/hybrid_rag_service.py` | Add `save_to_documents` option + virus scan |
| `src/models/hybrid_rag.py` | Add `save_to_documents` config |
| `src/routers/maintenance.py` | Document orphan cleanup, ClamAV health check |
| `requirements.txt` | Add `pyclamd` |

## Paperless Custom Fields Setup

Create these in Paperless UI (Administration → Custom Fields):

| Field | Type | Purpose |
|-------|------|---------|
| `source_url` | URL | Original download URL |
| `library_indexed` | Boolean | Sync status |
| `library_doc_id` | Text | Library Desk reference |
| `collection` | Text | Logical grouping |