# Wiki.js Page Change Detection

Automatic processing system for user-edited Wiki.js pages using PostgreSQL database triggers.

## Overview

**Problem:** Wiki.js doesn't have built-in webhooks (feature is [planned but not released](https://requarks.canny.io/wiki/p/webhooks))

**Solution:** PostgreSQL NOTIFY/LISTEN triggers to detect page changes in real-time

When you manually edit pages in Wiki.js, the change detection system ensures they're processed identically to AI-generated content:

- ✅ **Vectors updated** - Qdrant embeddings refreshed
- ✅ **Graph updated** - Entities extracted, relationships created
- ✅ **Bidirectional links** - Automatic entity linking both ways
- ✅ **Deletions handled** - Clean removal from vectors + graph

---

## How It Works

```
┌─────────────────────────────────────────────┐
│  Wiki.js (User edits page)                 │
└──────────────────┬──────────────────────────┘
                   │
                   ↓
┌─────────────────────────────────────────────┐
│  PostgreSQL "pages" table                   │
│  INSERT/UPDATE/DELETE triggers fire         │
└──────────────────┬──────────────────────────┘
                   │ pg_notify('wiki_page_changes', ...)
                   ↓
┌─────────────────────────────────────────────┐
│  library-desk WikiChangeListener            │
│  Receives notification via LISTEN           │
└──────────────────┬──────────────────────────┘
                   │
                   ↓
┌─────────────────────────────────────────────┐
│  Processing Pipeline (same as webhooks)     │
│  ├─ Ingest page (vectors + graph)          │
│  ├─ Extract entities                        │
│  ├─ Forward linking (page → entities)       │
│  └─ Backward linking (entities → page)      │
└─────────────────────────────────────────────┘
```

**Key Components:**

1. **PostgreSQL Triggers** - Installed in Wiki.js database, emit NOTIFY events on page changes
2. **WikiChangeListener** - Python service that LISTENs for notifications and processes changes
3. **Processing Pipeline** - Same logic as webhook handler (uses `process_wiki_page_change()`)

---

## Setup Instructions

### Prerequisites

- Library-desk has network access to Wiki.js PostgreSQL database
- Database user has `CREATE FUNCTION` and `CREATE TRIGGER` permissions

### Step 1: Configure Environment Variables

Add these to your `.env` file:

```bash
# Wiki.js Database Configuration (for change detection)
WIKIJS_DB_HOST=postgres          # PostgreSQL host (usually same as Wiki.js)
WIKIJS_DB_PORT=5432              # PostgreSQL port
WIKIJS_DB_NAME=wiki              # Wiki.js database name
WIKIJS_DB_USER=wikijs            # Database user (needs trigger permissions)
WIKIJS_DB_PASSWORD=your_password # Database password
```

### Step 2: Install Database Triggers

Run the setup script from within the library-desk container:

```bash
# Enter the container
docker exec -it library-desk bash

# Run trigger setup script
cd /app
python3 setup_wiki_triggers.py
```

**Expected output:**
```
======================================================================
Wiki.js Database Trigger Setup
======================================================================

Connecting to Wiki.js database at postgres:5432
Database: wiki
User: wikijs

✓ Connected to Wiki.js database

Installing triggers...

✓ Triggers installed successfully:

  - wiki_page_delete_trigger (DELETE)
  - wiki_page_insert_trigger (INSERT)
  - wiki_page_update_trigger (UPDATE)

======================================================================
Setup complete!
======================================================================

Next steps:
  1. Restart library-desk service to activate the listener
  2. Edit a page in Wiki.js to test
  3. Check library-desk logs for processing messages
```

### Step 3: Restart Library-Desk

```bash
docker restart library-desk
```

**Check logs for successful startup:**
```bash
docker logs library-desk --tail 50 | grep -E "(Wiki.js change listener|wiki_page_changes)"
```

You should see:
```
INFO - Starting Wiki.js database change listener
INFO - Listening for Wiki.js page changes via PostgreSQL NOTIFY
INFO - Wiki.js change listener started successfully
```

### Step 4: Test the System

1. **Create a new page in Wiki.js**
   - Add content mentioning existing entities (e.g., "Docker", "Kubernetes")
   - Save the page

2. **Check library-desk logs:**
   ```bash
   docker logs library-desk --follow | grep -E "(Received|Processing|Entity linking)"
   ```

   **Expected log output:**
   ```
   INFO - Received INSERT notification for page 123 by user@example.com
   INFO - Processing page.create for page 123 ('Test Page')
   INFO - Ingesting page 123 into knowledge base
   INFO - Ingestion complete for page 123
   INFO - Applying bidirectional entity linking for page 123
   INFO - Entity linking complete for page 123: 5 forward links, 3 backward links (2 pages updated)
   INFO - Successfully processed page.create for page 123
   ```

3. **Verify results:**
   - Check vectors in Qdrant
   - Check Neo4j for Document node and entities
   - Visit the page in Wiki.js - entities should be hyperlinked

---

## Event Processing Details

### page.create (INSERT trigger)

```
User creates page → PostgreSQL INSERT trigger fires
  ├─ pg_notify('wiki_page_changes', 'INSERT:123:user@example.com')
  ├─ WikiChangeListener receives notification
  ├─ Ingest page (vector + graph)
  ├─ Extract entities
  ├─ Forward linking (page → entities)
  └─ Backward linking (entities → page)
```

### page.update (UPDATE trigger)

```
User edits page → PostgreSQL UPDATE trigger fires
  ├─ pg_notify('wiki_page_changes', 'UPDATE:123:user@example.com')
  ├─ WikiChangeListener receives notification
  ├─ Re-ingest page (force refresh)
  ├─ Update entities
  ├─ Refresh forward links
  └─ Refresh backward links
```

### page.delete (DELETE trigger)

```
User deletes page → PostgreSQL DELETE trigger fires
  ├─ pg_notify('wiki_page_changes', 'DELETE:123:user@example.com')
  ├─ WikiChangeListener receives notification
  ├─ Remove vectors from Qdrant
  ├─ Find orphaned entities
  ├─ Delete Document node
  ├─ Delete orphaned entities
  └─ Clean broken relationships
```

---

## Monitoring

### Check Listener Status

```bash
# Check if listener is running
docker logs library-desk --tail 50 | grep "Wiki.js change listener"

# Should show:
# INFO - Wiki.js change listener started successfully
# INFO - Listening for Wiki.js page changes via PostgreSQL NOTIFY
```

### Monitor Page Changes

```bash
# Watch for page change notifications
docker logs library-desk --follow | grep -E "(Received.*notification|Processing page)"
```

### Verify Trigger Installation

Connect to Wiki.js database and check:

```sql
-- List installed triggers
SELECT trigger_name, event_manipulation, event_object_table
FROM information_schema.triggers
WHERE event_object_table = 'pages'
ORDER BY trigger_name;

-- Expected result:
--   wiki_page_delete_trigger | DELETE | pages
--   wiki_page_insert_trigger | INSERT | pages
--   wiki_page_update_trigger | UPDATE | pages
```

### Test Notification Manually

You can manually test the NOTIFY/LISTEN system:

```sql
-- In one session, listen for notifications:
LISTEN wiki_page_changes;

-- In another session, manually trigger:
NOTIFY wiki_page_changes, 'TEST:999:test@example.com';

-- The first session should receive the notification
```

---

## Troubleshooting

### Change Listener Not Starting

**Problem:** Logs show "Failed to start Wiki.js change listener"

**Solutions:**
1. Check database connection settings in `.env`
2. Verify library-desk can reach Wiki.js PostgreSQL:
   ```bash
   docker exec library-desk ping postgres
   ```
3. Test database credentials:
   ```bash
   docker exec library-desk psql -h postgres -U wikijs -d wiki -c "\dt"
   ```

### Changes Not Being Processed

**Problem:** Pages edited but no processing logs appear

**Solutions:**

1. **Check triggers are installed:**
   ```bash
   docker exec -it postgres psql -U wikijs -d wiki -c "
   SELECT trigger_name FROM information_schema.triggers
   WHERE event_object_table = 'pages';
   "
   ```

2. **Verify listener is running:**
   ```bash
   docker logs library-desk | grep "Listening for Wiki.js page changes"
   ```

3. **Test notification manually:**
   ```bash
   docker exec -it postgres psql -U wikijs -d wiki -c "
   NOTIFY wiki_page_changes, 'INSERT:123:test@example.com';
   "
   ```
   Check library-desk logs for "Received INSERT notification"

4. **Check authorEmail field exists:**
   The triggers use `authorEmail` field from Wiki.js pages table. Verify:
   ```sql
   SELECT column_name
   FROM information_schema.columns
   WHERE table_name = 'pages' AND column_name = 'authorEmail';
   ```

### Trigger Permission Denied

**Problem:** Setup script fails with "permission denied to create trigger"

**Solutions:**

1. Grant necessary permissions to database user:
   ```sql
   GRANT CREATE ON DATABASE wiki TO wikijs;
   ALTER TABLE pages OWNER TO wikijs;
   ```

2. Or run setup script with database admin user:
   ```bash
   # Modify WIKIJS_DB_USER temporarily to postgres
   WIKIJS_DB_USER=postgres python3 setup_wiki_triggers.py
   ```

### Processing Errors

**Problem:** Notification received but processing fails

**Solutions:**

1. Check library-desk logs for specific error:
   ```bash
   docker logs library-desk --tail 100 | grep -A 10 "Failed to handle notification"
   ```

2. Verify user mapping works:
   - Triggers extract user from `authorEmail` field
   - Check `extract_user_from_email()` in `wiki_change_listener.py`
   - Default: `user@domain.com` → `user`

3. Ensure page_id exists and is accessible

---

## User Mapping

By default, the user is extracted from the Wiki.js `authorEmail` field:

```python
# user@example.com → user
user = user_email.split('@')[0]
```

**Custom mapping:** Edit `src/services/wiki_change_listener.py`:

```python
async def _handle_notification(self, connection, pid, channel, payload):
    # ... existing code ...

    # Custom user mapping
    user_map = {
        "admin@example.com": "admin",
        "john.doe@example.com": "jpmschweitzer"
    }
    user = user_map.get(user_email, user_email.split('@')[0])
```

---

## Comparison: Webhooks vs Database Triggers

| Feature | Webhooks (Not Available) | PostgreSQL Triggers |
|---------|-------------------------|---------------------|
| **Availability** | Planned, not released | Available now |
| **Setup** | Simple (UI configuration) | Requires database access |
| **Reliability** | HTTP-based, can fail | Database-native, highly reliable |
| **Performance** | Network overhead | Minimal overhead |
| **Maintenance** | None | Must survive database migrations |
| **Isolation** | Loose coupling | Tight coupling to database |

**Recommendation:** Use database triggers until Wiki.js webhooks are officially released. When webhooks become available, migrate to that approach for better isolation.

---

## Uninstalling

To remove the triggers (if switching to webhooks later):

```bash
docker exec -it postgres psql -U wikijs -d wiki -c "
DROP TRIGGER IF EXISTS wiki_page_insert_trigger ON pages;
DROP TRIGGER IF EXISTS wiki_page_update_trigger ON pages;
DROP TRIGGER IF EXISTS wiki_page_delete_trigger ON pages;
DROP FUNCTION IF EXISTS notify_page_change();
"
```

Then remove the listener startup from `src/main.py` (lines 354-364).

---

## Summary

With database triggers configured:

| User Action | Automatic Processing |
|-------------|---------------------:|
| Create page | Ingest + entity link (bidirectional) |
| Edit page | Re-ingest + refresh links |
| Delete page | Clean vectors + graph + orphaned entities |

**Result:** User-edited and AI-generated pages are **indistinguishable** in the knowledge base!

---

## Related Files

- `src/services/wiki_change_listener.py` - LISTEN service implementation
- `src/routers/webhooks.py` - Processing logic (shared with future webhook approach)
- `src/services/consolidation_service.py` - Bidirectional entity linking
- `setup_wiki_triggers.py` - Trigger installation script
- `src/main.py` - Application startup/shutdown hooks

---

## Future: Migration to Webhooks

When Wiki.js webhooks are released:

1. Configure webhook in Wiki.js admin panel
2. Point to `http://library-desk:8089/webhooks/wikijs`
3. Remove database triggers (see Uninstalling section)
4. Disable listener startup in `src/main.py`

The webhook handler already exists and uses the same processing logic!