portainer-core/services/library-desk/WIKI_CHANGE_DETECTION.md

Wiki.js Page Change Detection

Automatic processing system for user-edited Wiki.js pages using PostgreSQL database triggers.

Overview

Problem: Wiki.js doesn't have built-in webhooks (feature is planned but not released)

Solution: PostgreSQL NOTIFY/LISTEN triggers to detect page changes in real-time

When you manually edit pages in Wiki.js, the change detection system ensures they're processed identically to AI-generated content:

• ✅ Vectors updated - Qdrant embeddings refreshed
• ✅ Graph updated - Entities extracted, relationships created
• ✅ Bidirectional links - Automatic entity linking both ways
• ✅ Deletions handled - Clean removal from vectors + graph

---

How It Works

┌─────────────────────────────────────────────┐
│  Wiki.js (User edits page)                 │
└──────────────────┬──────────────────────────┘
                   │
                   ↓
┌─────────────────────────────────────────────┐
│  PostgreSQL "pages" table                   │
│  INSERT/UPDATE/DELETE triggers fire         │
└──────────────────┬──────────────────────────┘
                   │ pg_notify('wiki_page_changes', ...)
                   ↓
┌─────────────────────────────────────────────┐
│  library-desk WikiChangeListener            │
│  Receives notification via LISTEN           │
└──────────────────┬──────────────────────────┘
                   │
                   ↓
┌─────────────────────────────────────────────┐
│  Processing Pipeline (same as webhooks)     │
│  ├─ Ingest page (vectors + graph)          │
│  ├─ Extract entities                        │
│  ├─ Forward linking (page → entities)       │
│  └─ Backward linking (entities → page)      │
└─────────────────────────────────────────────┘

Key Components:

1. PostgreSQL Triggers - Installed in Wiki.js database, emit NOTIFY events on page changes
2. WikiChangeListener - Python service that LISTENs for notifications and processes changes
3. Processing Pipeline - Same logic as webhook handler (uses process_wiki_page_change())

---

Setup Instructions

Prerequisites

• Library-desk has network access to Wiki.js PostgreSQL database
• Database user has CREATE FUNCTION and CREATE TRIGGER permissions

Step 1: Configure Environment Variables

Add these to your .env file:

# Wiki.js Database Configuration (for change detection)
WIKIJS_DB_HOST=postgres          # PostgreSQL host (usually same as Wiki.js)
WIKIJS_DB_PORT=5432              # PostgreSQL port
WIKIJS_DB_NAME=wiki              # Wiki.js database name
WIKIJS_DB_USER=wikijs            # Database user (needs trigger permissions)
WIKIJS_DB_PASSWORD=your_password # Database password

Step 2: Install Database Triggers

Run the setup script from within the library-desk container:

# Enter the container
docker exec -it library-desk bash

# Run trigger setup script
cd /app
python3 setup_wiki_triggers.py

Expected output:

======================================================================
Wiki.js Database Trigger Setup
======================================================================

Connecting to Wiki.js database at postgres:5432
Database: wiki
User: wikijs

✓ Connected to Wiki.js database

Installing triggers...

✓ Triggers installed successfully:

  - wiki_page_delete_trigger (DELETE)
  - wiki_page_insert_trigger (INSERT)
  - wiki_page_update_trigger (UPDATE)

======================================================================
Setup complete!
======================================================================

Next steps:
  1. Restart library-desk service to activate the listener
  2. Edit a page in Wiki.js to test
  3. Check library-desk logs for processing messages

Step 3: Restart Library-Desk

docker restart library-desk

Check logs for successful startup:

docker logs library-desk --tail 50 | grep -E "(Wiki.js change listener|wiki_page_changes)"

You should see:

INFO - Starting Wiki.js database change listener
INFO - Listening for Wiki.js page changes via PostgreSQL NOTIFY
INFO - Wiki.js change listener started successfully

Step 4: Test the System

1. Create a new page in Wiki.js

   • Add content mentioning existing entities (e.g., "Docker", "Kubernetes")
   • Save the page

2. Check library-desk logs:

   docker logs library-desk --follow | grep -E "(Received|Processing|Entity linking)"
   

   Expected log output:

   INFO - Received INSERT notification for page 123 by user@example.com
   INFO - Processing page.create for page 123 ('Test Page')
   INFO - Ingesting page 123 into knowledge base
   INFO - Ingestion complete for page 123
   INFO - Applying bidirectional entity linking for page 123
   INFO - Entity linking complete for page 123: 5 forward links, 3 backward links (2 pages updated)
   INFO - Successfully processed page.create for page 123
   

3. Verify results:

   • Check vectors in Qdrant
   • Check Neo4j for Document node and entities
   • Visit the page in Wiki.js - entities should be hyperlinked

---

Event Processing Details

page.create (INSERT trigger)

User creates page → PostgreSQL INSERT trigger fires
  ├─ pg_notify('wiki_page_changes', 'INSERT:123:user@example.com')
  ├─ WikiChangeListener receives notification
  ├─ Ingest page (vector + graph)
  ├─ Extract entities
  ├─ Forward linking (page → entities)
  └─ Backward linking (entities → page)

page.update (UPDATE trigger)

User edits page → PostgreSQL UPDATE trigger fires
  ├─ pg_notify('wiki_page_changes', 'UPDATE:123:user@example.com')
  ├─ WikiChangeListener receives notification
  ├─ Re-ingest page (force refresh)
  ├─ Update entities
  ├─ Refresh forward links
  └─ Refresh backward links

page.delete (DELETE trigger)

User deletes page → PostgreSQL DELETE trigger fires
  ├─ pg_notify('wiki_page_changes', 'DELETE:123:user@example.com')
  ├─ WikiChangeListener receives notification
  ├─ Remove vectors from Qdrant
  ├─ Find orphaned entities
  ├─ Delete Document node
  ├─ Delete orphaned entities
  └─ Clean broken relationships

---

Monitoring

Check Listener Status

# Check if listener is running
docker logs library-desk --tail 50 | grep "Wiki.js change listener"

# Should show:
# INFO - Wiki.js change listener started successfully
# INFO - Listening for Wiki.js page changes via PostgreSQL NOTIFY

Monitor Page Changes

# Watch for page change notifications
docker logs library-desk --follow | grep -E "(Received.*notification|Processing page)"

Verify Trigger Installation

Connect to Wiki.js database and check:

-- List installed triggers
SELECT trigger_name, event_manipulation, event_object_table
FROM information_schema.triggers
WHERE event_object_table = 'pages'
ORDER BY trigger_name;

-- Expected result:
--   wiki_page_delete_trigger | DELETE | pages
--   wiki_page_insert_trigger | INSERT | pages
--   wiki_page_update_trigger | UPDATE | pages

Test Notification Manually

You can manually test the NOTIFY/LISTEN system:

-- In one session, listen for notifications:
LISTEN wiki_page_changes;

-- In another session, manually trigger:
NOTIFY wiki_page_changes, 'TEST:999:test@example.com';

-- The first session should receive the notification

---

Troubleshooting

Change Listener Not Starting

Problem: Logs show "Failed to start Wiki.js change listener"

Solutions:

1. Check database connection settings in .env
2. Verify library-desk can reach Wiki.js PostgreSQL:

   docker exec library-desk ping postgres
   

3. Test database credentials:

   docker exec library-desk psql -h postgres -U wikijs -d wiki -c "\dt"
   

Changes Not Being Processed

Problem: Pages edited but no processing logs appear

Solutions:

1. Check triggers are installed:

   docker exec -it postgres psql -U wikijs -d wiki -c "
   SELECT trigger_name FROM information_schema.triggers
   WHERE event_object_table = 'pages';
   "
   

2. Verify listener is running:

   docker logs library-desk | grep "Listening for Wiki.js page changes"
   

3. Test notification manually:

   docker exec -it postgres psql -U wikijs -d wiki -c "
   NOTIFY wiki_page_changes, 'INSERT:123:test@example.com';
   "
   

   Check library-desk logs for "Received INSERT notification"

4. Check authorEmail field exists: The triggers use authorEmail field from Wiki.js pages table. Verify:

   SELECT column_name
   FROM information_schema.columns
   WHERE table_name = 'pages' AND column_name = 'authorEmail';
   

Trigger Permission Denied

Problem: Setup script fails with "permission denied to create trigger"

Solutions:

1. Grant necessary permissions to database user:

   GRANT CREATE ON DATABASE wiki TO wikijs;
   ALTER TABLE pages OWNER TO wikijs;
   

2. Or run setup script with database admin user:

   # Modify WIKIJS_DB_USER temporarily to postgres
   WIKIJS_DB_USER=postgres python3 setup_wiki_triggers.py
   

Processing Errors

Problem: Notification received but processing fails

Solutions:

1. Check library-desk logs for specific error:

   docker logs library-desk --tail 100 | grep -A 10 "Failed to handle notification"
   

2. Verify user mapping works:

   • Triggers extract user from authorEmail field
   • Check extract_user_from_email() in wiki_change_listener.py
   • Default: user@domain.com → user

3. Ensure page_id exists and is accessible

---

User Mapping

By default, the user is extracted from the Wiki.js authorEmail field:

# user@example.com → user
user = user_email.split('@')[0]

Custom mapping: Edit src/services/wiki_change_listener.py:

async def _handle_notification(self, connection, pid, channel, payload):
    # ... existing code ...

    # Custom user mapping
    user_map = {
        "admin@example.com": "admin",
        "john.doe@example.com": "jpmschweitzer"
    }
    user = user_map.get(user_email, user_email.split('@')[0])

---

Comparison: Webhooks vs Database Triggers

Feature	Webhooks (Not Available)	PostgreSQL Triggers
Availability	Planned, not released	Available now
Setup	Simple (UI configuration)	Requires database access
Reliability	HTTP-based, can fail	Database-native, highly reliable
Performance	Network overhead	Minimal overhead
Maintenance	None	Must survive database migrations
Isolation	Loose coupling	Tight coupling to database

Recommendation: Use database triggers until Wiki.js webhooks are officially released. When webhooks become available, migrate to that approach for better isolation.

---

Uninstalling

To remove the triggers (if switching to webhooks later):

docker exec -it postgres psql -U wikijs -d wiki -c "
DROP TRIGGER IF EXISTS wiki_page_insert_trigger ON pages;
DROP TRIGGER IF EXISTS wiki_page_update_trigger ON pages;
DROP TRIGGER IF EXISTS wiki_page_delete_trigger ON pages;
DROP FUNCTION IF EXISTS notify_page_change();
"

Then remove the listener startup from src/main.py (lines 354-364).

---

Summary

With database triggers configured:

User Action	Automatic Processing
Create page	Ingest + entity link (bidirectional)
Edit page	Re-ingest + refresh links
Delete page	Clean vectors + graph + orphaned entities

Result: User-edited and AI-generated pages are indistinguishable in the knowledge base!

---

Related Files

• src/services/wiki_change_listener.py - LISTEN service implementation
• src/routers/webhooks.py - Processing logic (shared with future webhook approach)
• src/services/consolidation_service.py - Bidirectional entity linking
• setup_wiki_triggers.py - Trigger installation script
• src/main.py - Application startup/shutdown hooks

---

Future: Migration to Webhooks

When Wiki.js webhooks are released:

1. Configure webhook in Wiki.js admin panel
2. Point to http://library-desk:8089/webhooks/wikijs
3. Remove database triggers (see Uninstalling section)
4. Disable listener startup in src/main.py

The webhook handler already exists and uses the same processing logic!