library-desk/docs/DOCUMENT_STORAGE_PLAN.md

Phase 3: Document Storage System - Implementation Plan

Overview

Document storage tier for Library Desk - storing and indexing PDFs, images, videos, and git documentation mirrors.

User Decisions:

• Paperless-ngx container for OCR
• Ebooks deferred to future phase
• Video.js player deferred to after core implementation

Phase	Status	Version
Phase 1: Cleanup System	Complete	v1.4.0
Phase 2: Volatile Memory	Complete	v1.4.3
Phase 3: Document Storage	Planning	-
Phase 4: Test Data Cleanup	Complete	v1.4.4

---

Architecture

Paperless-ngx as primary document store (no SeaweedFS needed):

┌─────────────────────────────────────────────────────────────────┐
│                     External Sources                             │
│  ┌─────────┐  ┌────────────┐  ┌──────────────┐                  │
│  │ GitHub  │  │ Direct     │  │ Email/Folder │                  │
│  │ Docs    │  │ Upload     │  │ Ingestion    │                  │
│  └────┬────┘  └─────┬──────┘  └──────┬───────┘                  │
└───────┼─────────────┼────────────────┼──────────────────────────┘
        │             │                │
        ▼             ▼                ▼
┌─────────────────────────────────────────────────────────────────┐
│                      Paperless-ngx                               │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │  - Document storage (PDFs, images, videos)                │  │
│  │  - OCR via Tesseract (PDFs, images)                       │  │
│  │  - Web UI for browsing/tagging                            │  │
│  │  - REST API for integration                               │  │
│  └─────────────────────────┬─────────────────────────────────┘  │
└────────────────────────────┼────────────────────────────────────┘
                             │ REST API (sync)
                             ▼
┌─────────────────────────────────────────────────────────────────┐
│                        Library Desk                              │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │              DocumentSyncService                          │  │
│  │  - Polls Paperless for new/updated docs                   │  │
│  │  - Extracts text + metadata via API                       │  │
│  │  - Sends to vector/graph pipelines                        │  │
│  └─────────────────────────┬─────────────────────────────────┘  │
│                            │                                     │
│           ┌────────────────┼────────────────┐                   │
│           ▼                ▼                ▼                   │
│      ┌──────────┐    ┌──────────┐    ┌──────────┐              │
│      │  Qdrant  │    │  Neo4j   │    │ Wiki.js  │              │
│      │ (vectors)│    │ (graph)  │    │ (catalog)│              │
│      └──────────┘    └──────────┘    └──────────┘              │
└─────────────────────────────────────────────────────────────────┘

File handling by type:

File Type	Paperless	Library Desk
PDFs	OCR → text	Index text → vectors/graph
Images	OCR → text	Index text → vectors/graph
Videos	Storage only	Index metadata → vectors/graph

---

Technology Stack

Component	Purpose	Rationale
Paperless-ngx	Document storage + OCR	All-in-one: storage, OCR, web UI, REST API
ClamAV	Virus scanning	Host OS install, pyclamd integration, better isolation
PDF.js	PDF viewer	Embeddable in Wiki.js (deferred)

Why Paperless-ngx as primary store:

• Eliminates need for separate blob storage (SeaweedFS/MinIO)
• Built-in web UI for browsing and tagging
• Tesseract OCR with 100+ language support
• REST API for Library Desk integration
• Handles videos as raw files (no OCR, but stored)
• Email and folder watching for automatic ingestion
• Active community, well-maintained

---

Paperless-ngx API Deep Dive

Authentication

POST /api/token/
Body: {"username": "...", "password": "..."}
Response: {"token": "..."}

Header: Authorization: Token <token>

Document Upload (for HybridRAG → Paperless)

POST /api/documents/post_document/
Content-Type: multipart/form-data

Fields:
- document (file, required)
- title (string)
- created (datetime)
- correspondent (ID)
- document_type (ID)
- storage_path (ID)
- tags (repeatable IDs)
- custom_fields (JSON array)

Response: {"task_id": "uuid"}

Track consumption: GET /api/tasks/?task_id={uuid} → returns document ID when complete

Document Search

GET /api/documents/?query=search+terms     # Full-text search
GET /api/documents/?more_like_id=123       # Similarity search

Response includes __search_hit__:
{
  "score": 0.95,
  "highlights": "<span>matched</span> text",
  "rank": 0
}

Custom Field Filtering

GET /api/documents/?custom_field_query=field_name__operation=value

Operations:
- exact, in, isnull, exists (all types)
- icontains, istartswith, iendswith (text)
- gt, gte, lt, lte, range (numeric/date)
- contains (document links)

Bulk Operations

POST /api/documents/bulk_edit/
{
  "documents": [1, 2, 3],
  "method": "add_tag|remove_tag|set_correspondent|set_document_type|merge|split|...",
  "parameters": {...}
}

Webhooks (Push to Library Desk!)

Paperless workflows can trigger webhooks on document events:

Trigger	When	Available Data
Consumption Started	Before OCR	file_path, source, filename
Document Added	After OCR	content, tags, doc_type, correspondent, {doc_url}
Document Updated	On change	Same as Added
Scheduled	Time-based	Date offsets from document dates

Webhook Action: POST to Library Desk endpoint with document data

Organization Features

Feature	Purpose	API Endpoint
Tags	Nested labels (5 levels deep)	/api/tags/
Correspondents	Source/destination	/api/correspondents/
Document Types	Classification	/api/document_types/
Storage Paths	File organization	/api/storage_paths/
Custom Fields	Extensible metadata	/api/custom_fields/

Custom Fields We Should Create

Field Name	Type	Purpose
source_url	URL	Original download URL (for HybridRAG uploads)
library_indexed	Boolean	Sync status with Library Desk
library_doc_id	Text	Library Desk document reference
collection	Text	Logical grouping (e.g., "fastapi-docs")

External LLM Add-ons (Optional)

Community tools exist for Ollama integration:

• paperless-ai - Auto-tagging, RAG chat
• paperless-gpt - LLM-enhanced OCR, auto-titling

Recommendation: Skip these - Library Desk already has Ollama integration for:

• Embedding (nomic-embed-text)
• LLM analysis (mistral-nemo)
• Entity extraction
• HybridRAG

We'll do our own classification/tagging via Library Desk after sync.

---

Virus Scanning Integration

ClamAV daemon + pyclamd (no third-party REST wrappers):

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│  File Upload    │────►│  Library Desk   │────►│  ClamAV Daemon  │
│  (URL or file)  │     │  (pyclamd)      │     │  (clamd:3310)   │
└─────────────────┘     └────────┬────────┘     └─────────────────┘
                                 │
                    ┌────────────┴────────────┐
                    ▼                         ▼
              ┌──────────┐              ┌──────────┐
              │  Clean   │              │ Infected │
              │  ✓       │              │  ✗       │
              └────┬─────┘              └────┬─────┘
                   │                         │
                   ▼                         ▼
          Upload to Paperless         Reject + Log

ClamAV Deployment (Host OS)

ClamAV runs on the host OS (not containerized) for better security isolation:

# Installed via apt on Ubuntu/Debian
# Config: /etc/clamav/clamd.conf
# TCPSocket 3310
# TCPAddr 0.0.0.0

Benefits: scans outside container isolation, single virus DB, survives container restarts.

Library Desk Integration

# src/clients/clamav_client.py
import pyclamd

class ClamAVClient:
    def __init__(self, host: str, port: int = 3310):
        self.cd = pyclamd.ClamdNetworkSocket(host, port)

    async def scan_bytes(self, data: bytes) -> ScanResult:
        """Scan file bytes, return clean/infected status."""
        result = self.cd.scan_stream(data)
        if result is None:
            return ScanResult(clean=True)
        return ScanResult(clean=False, virus_name=result['stream'][1])

    def ping(self) -> bool:
        """Health check."""
        return self.cd.ping()

Scan Points

Location	When	Action on Infected
/documents/upload	Before Paperless upload	Reject with 400, log threat
HybridRAG web fetch	Before saving PDF	Skip file, log threat
/documents/webhook	Optional re-scan	Quarantine in Paperless

Config Settings

# src/config.py
CLAMAV_HOST: str = "192.168.86.149"  # Host OS IP (not container)
CLAMAV_PORT: int = 3310
CLAMAV_ENABLED: bool = True  # Bypass for testing
CLAMAV_TIMEOUT: int = 30  # seconds

---

Integration Strategy

Option A: Webhook Push (Preferred)

Paperless Workflow → POST webhook → Library Desk /documents/webhook

• Real-time indexing when documents added/updated
• Configure in Paperless: Workflow → Document Added → Webhook Action
• Library Desk receives document ID, fetches content via API

Option B: Polling Pull (Fallback)

Scheduler → POST /documents/sync → Library Desk polls Paperless

• Periodic sync for missed webhooks or initial bulk import
• Track library_indexed custom field to skip already-processed docs

Option C: HybridRAG Upload (New!)

HybridRAG web search → finds PDF → POST to Paperless → webhook → indexed

• When HybridRAG finds a relevant PDF/document in web results
• Download and upload to Paperless with source_url custom field
• Paperless OCRs it, triggers webhook, Library Desk indexes

---

Library Desk API Design

Documents Router (/documents)

Endpoint	Method	Purpose
/documents/webhook	POST	Receive Paperless webhook (Document Added/Updated)
/documents/sync	POST	Pull new/updated docs from Paperless → index
/documents/upload	POST	Upload file to Paperless (for HybridRAG)
/documents/sync-from-git	POST	Pull docs from Gitea → upload to Paperless → index
/documents/{document_id}	GET	Get document metadata
/documents/{document_id}/text	GET	Get extracted text
/documents/search	POST	Semantic search across documents
/documents/collection/{name}	GET	List documents in collection
/documents/collection/{name}/catalog	POST	Generate wiki catalog page

Upload flow (HybridRAG → Paperless):

1. HybridRAG finds PDF in web results
2. POST /documents/upload with URL or file
3. Library Desk downloads, uploads to Paperless with metadata
4. Returns task_id for async tracking
5. Paperless webhook triggers indexing when OCR complete

Viewers Router (/viewers) - Deferred

Endpoint	Method	Purpose
/viewers/pdf/{document_id}	GET	Serve PDF.js viewer
/viewers/image/{document_id}	GET	Serve image lightbox
/viewers/video/{document_id}	GET	Serve Video.js player

---

Data Flow: Document Processing Pipeline

1. INTAKE (Paperless-ngx handles this)
   └─ Upload via Paperless UI, email, or folder watch
   └─ Paperless assigns document ID and stores file

2. OCR EXTRACTION (Paperless-ngx handles this)
   ├─ PDFs → Tesseract → Plain text
   ├─ Images → Tesseract → Plain text
   └─ Videos → Metadata only (no OCR)

3. SYNC TO LIBRARY DESK (scheduled or manual)
   └─ Poll Paperless API for new/updated documents
   └─ Fetch text content + metadata

4. TEXT CHUNKING
   └─ VectorService._chunk_text() (existing)

5. EMBEDDING
   └─ OllamaClient.embed() (existing)

6. VECTOR STORAGE (Qdrant)
   └─ Payload: {doc_type: "document", paperless_id, ...}

7. GRAPH STORAGE (Neo4j)
   └─ Document node + MENTIONS relationships

8. WIKI CATALOG (optional)
   └─ Auto-generate catalog page via ConsolidationService

---

Git Docs Integration

Extends existing scheduler/src/executors/doc_sync_executor.py:

1. Scheduler syncs docs from GitHub → Gitea (existing)
2. Post-sync hook calls POST /documents/sync-from-git
3. Library Desk indexes docs into vectors/graph
4. Auto-generate wiki catalog page for collection

---

Wiki.js Viewer Integration

Since Wiki.js v2 requires disabled HTML sanitization for iframes:

<!-- In wiki catalog page -->
## Document Preview

<iframe
  src="http://library-desk:8089/viewers/pdf/abc123"
  width="100%" height="600px">
</iframe>

Wiki.js Settings Required:

• Administration > Security > Allowed HTML Elements: iframe
• Content Security Policy: frame-src http://library-desk:8089

---

Implementation Phases

Phase 3.1: Infrastructure Setup

• Deploy Paperless-ngx container (Docker Compose)
• ClamAV installed on host OS (port 3310)
• Configure Paperless: storage path, OCR settings, API token
• Create custom fields in Paperless: source_url, library_indexed, library_doc_id, collection
• Create src/clients/paperless_client.py
• Create src/clients/clamav_client.py (pyclamd wrapper)
• Create src/models/document.py
• Add config settings to src/config.py (PAPERLESS_, CLAMAV_)

Phase 3.2: Webhook Integration (Push)

• Create src/routers/documents.py
• Implement /documents/webhook endpoint (receives Paperless events)
• Configure Paperless Workflow: Document Added → Webhook → Library Desk
• Create src/services/document_sync_service.py
• Implement document indexing pipeline (fetch text → chunk → embed → graph)

Phase 3.3: Polling Sync (Pull Fallback)

• Implement /documents/sync endpoint
• Poll Paperless for docs where library_indexed=false
• Track sync state (last_sync timestamp in Redis)
• Update library_indexed after successful indexing

Phase 3.4: HybridRAG Upload Integration

• Implement /documents/upload endpoint
• Download file from URL
• Virus scan before upload (reject if infected, log threat)
• Upload clean files to Paperless with metadata
• Set source_url custom field
• Extend HybridRAG service to detect and upload relevant PDFs
• Add save_to_documents option to HybridRAG config

Phase 3.5: Indexing Pipeline

• Extend VectorService for doc_type: "document"
• Extend GraphService for Document nodes (link to Paperless ID)
• Implement /documents/search endpoint
• Add dependency injection

Phase 3.6: Git Docs Integration

• Create src/clients/gitea_client.py
• Implement /documents/sync-from-git → bulk upload to Paperless
• Create collection auto-cataloging (wiki pages)
• Add scheduler task for periodic git sync

Phase 3.7: Viewers (Deferred)

After core implementation is working

• Create static/pdf-viewer.html (PDF.js)
• Create static/image-viewer.html
• Create static/video-player.html (Video.js)
• Create src/routers/viewers.py

Phase 3.8: Maintenance & Testing

• Extend cleanup for document orphans
• Add document orphan detection (Paperless deleted but still in Qdrant/Neo4j)
• Create tests/test_document_sync.py
• Create tests/test_paperless_client.py

---

Files to Create

Path	Purpose
src/clients/paperless_client.py	Paperless-ngx REST API client
src/clients/clamav_client.py	ClamAV scanner (pyclamd wrapper)
src/clients/gitea_client.py	Gitea repo access
src/models/document.py	Document/Collection/ScanResult models
src/services/document_sync_service.py	Sync orchestrator
src/routers/documents.py	Document endpoints (webhook, sync, upload, search)
tests/test_document_sync.py	Sync service tests
tests/test_paperless_client.py	API client tests
tests/test_clamav_client.py	Virus scanner tests
docker/docker-compose.documents.yml	Paperless + ClamAV deployment

Deferred files (Phase 3.7):

Path	Purpose
src/routers/viewers.py	Viewer endpoints
static/pdf-viewer.html	PDF.js viewer
static/image-viewer.html	Image lightbox
static/video-player.html	Video.js player

Files to Modify

Path	Changes
src/config.py	PAPERLESS_*, CLAMAV_* settings
src/core/dependencies.py	DocumentSyncService, PaperlessClient, ClamAVClient DI
src/main.py	Register documents router
src/services/vector_service.py	doc_type: "document" handling
src/services/graph_service.py	Document node with Paperless ID
src/services/hybrid_rag_service.py	Add save_to_documents option + virus scan
src/models/hybrid_rag.py	Add save_to_documents config
src/routers/maintenance.py	Document orphan cleanup, ClamAV health check
requirements.txt	Add pyclamd

Paperless Custom Fields Setup

Create these in Paperless UI (Administration → Custom Fields):

Field	Type	Purpose
source_url	URL	Original download URL
library_indexed	Boolean	Sync status
library_doc_id	Text	Library Desk reference
collection	Text	Logical grouping