# Memory "Remember" Triggers - Implementation Plan

## Overview

This document outlines the implementation of "remember" triggers for the memory system. Currently, we have recall (search) working for volatile and documents, but no automated triggers to populate these memory tiers.

**Key architectural principle:**
- **Scheduler-driven**: Prefetch data that's useful on a repeating schedule (weather, news)
- **HybridRAG-driven**: Cache ad-hoc ephemeral data discovered during searches
- **Learning loop**: HybridRAG can register scheduler tasks when it discovers prefetch-worthy patterns

---

## Current State

| Memory Tier | Remember Trigger | Recall | Status |
|-------------|------------------|--------|--------|
| Wiki | Wiki.js webhook, Consolidation | HybridRAG vector+graph | ✅ Complete |
| Documents | Paperless webhook | Manual `/documents/search` | ⚠️ Partial (no HybridRAG recall) |
| Volatile | Manual `/volatile/store` only | HybridRAG volatile search | ⚠️ Partial (no auto-triggers) |

---

## Architecture

```
┌─────────────────────────────────────────────────────────────────────────┐
│                         REMEMBER TRIGGERS                                │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│                        ┌──────────────────────┐                         │
│                        │  HybridRAG Search    │                         │
│                        │  Post-processor      │                         │
│                        └──────────┬───────────┘                         │
│                                   │                                      │
│                    ┌──────────────┼──────────────┐                      │
│                    ▼              ▼              ▼                      │
│            ┌─────────────┐ ┌───────────┐ ┌─────────────────┐           │
│            │ Classify    │ │ Store     │ │ Register        │           │
│            │ web results │ │ immediate │ │ scheduler task  │           │
│            └──────┬──────┘ │ (volatile)│ │ (if prefetch    │           │
│                   │        │ short TTL │ │  worthy)        │           │
│                   │        └───────────┘ └────────┬────────┘           │
│                   │                               │                     │
│     ┌─────────────┼─────────────┐                 │                     │
│     ▼             ▼             ▼                 ▼                     │
│ ┌───────┐   ┌──────────┐  ┌──────────┐    ┌─────────────┐              │
│ │ PDF   │   │ Ephemeral│  │ Prefetch │    │  Scheduler  │              │
│ │       │   │ (1x use) │  │ worthy   │    │  (external) │              │
│ └───┬───┘   └────┬─────┘  └────┬─────┘    └──────┬──────┘              │
│     │            │             │                  │                     │
│     ▼            ▼             │                  │                     │
│ ┌────────┐  ┌─────────┐        │                  │                     │
│ │Paperless│ │Volatile │        │    ┌─────────────┘                     │
│ │Documents│ │short TTL│        │    │                                   │
│ └────────┘  └─────────┘        │    ▼                                   │
│                                │  ┌─────────────────┐                   │
│                                └─►│ POST /volatile/ │                   │
│                                   │ fetch (cron)    │                   │
│                                   └────────┬────────┘                   │
│                                            │                            │
│                                            ▼                            │
│                                   ┌─────────────────┐                   │
│                                   │ Volatile        │                   │
│                                   │ long TTL        │                   │
│                                   └─────────────────┘                   │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘
```

---

## Central Settings Database

### Rationale

External API credentials (NewsAPI, etc.) and configs (Open-Meteo) should NOT be in environment variables because:
- They're not deployment-specific (same across all environments)
- They change independently of deployments
- Multiple services across Tatlock need access to shared credentials
- Environment variables require container restarts to update

### Database Choice: PostgreSQL

**Decision:** Use `postgres-shared` container (existing Tatlock infrastructure).

Create a new database `system_settings` on the shared PostgreSQL instance. This container exists specifically for cross-service databases.

### Schema Design

```sql
-- Run on postgres-shared as admin user

-- Create database
CREATE DATABASE system_settings;

-- Create settings user (shared across all Tatlock services)
CREATE USER settings WITH PASSWORD 'changeme';
GRANT ALL PRIVILEGES ON DATABASE system_settings TO settings;

-- Connect to system_settings database
\c system_settings

-- Create table
CREATE TABLE settings (
    key VARCHAR(255) NOT NULL,
    user_scope VARCHAR(100) NOT NULL DEFAULT 'global',  -- 'global' or specific username
    value JSONB NOT NULL,
    schema JSONB,              -- JSON Schema for UI rendering (nullable)
    description TEXT,
    updated_at TIMESTAMP DEFAULT NOW(),
    updated_by VARCHAR(100),
    PRIMARY KEY (key, user_scope)
);

-- Index for user-scoped lookups
CREATE INDEX idx_settings_user_scope ON settings(user_scope);

-- Grant full access
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO settings;
```

### Query Pattern

```sql
-- Get setting with user override, fallback to global
SELECT value, schema FROM settings
WHERE key = $1 AND user_scope IN ($2, 'global')
ORDER BY CASE WHEN user_scope = $2 THEN 0 ELSE 1 END
LIMIT 1;
```

### Data Types with JSON Schema

The `schema` column contains JSON Schema for UI widget rendering:

| JSON Schema | UI Widget |
|-------------|-----------|
| `{"type": "string", "format": "password"}` | Masked input |
| `{"type": "string", "enum": [...]}` | Dropdown/select |
| `{"type": "boolean"}` | Toggle switch |
| `{"type": "array", "items": {"type": "string"}}` | Multi-select or list |
| `{"type": "number", "minimum": 0, "maximum": 100}` | Slider or number input |
| No schema | Raw JSON editor |

### Example Data

```sql
-- Global API keys (with schemas for CRUD UI)
INSERT INTO settings (key, user_scope, value, schema, description) VALUES
('api.openmeteo', 'global',
 '{"base_url": "https://api.open-meteo.com/v1/forecast", "timezone": "Europe/Amsterdam"}',
 '{
   "type": "object",
   "properties": {
     "base_url": {"type": "string", "format": "uri", "title": "Base URL"},
     "timezone": {"type": "string", "title": "Default Timezone"}
   }
 }',
 'Open-Meteo weather API (no API key required)'),

('api.newsapi', 'global',
 '{"api_key": "xxx"}',
 '{
   "type": "object",
   "properties": {
     "api_key": {"type": "string", "format": "password", "title": "API Key"}
   },
   "required": ["api_key"]
 }',
 'NewsAPI.org credentials'),

('api.nos_rss', 'global',
 '{"feed_url": "https://feeds.nos.nl/nosnieuwsalgemeen"}',
 '{
   "type": "object",
   "properties": {
     "feed_url": {"type": "string", "format": "uri", "title": "Feed URL"}
   }
 }',
 'NOS.nl RSS feed');

-- User-specific preferences (explicit choices)
INSERT INTO settings (key, user_scope, value, schema, description) VALUES
('weather.units', 'jpmschweitzer',
 '"metric"',
 '{"type": "string", "enum": ["metric", "imperial"], "title": "Temperature Units"}',
 'Preferred temperature units'),

('news.sources', 'jpmschweitzer',
 '["nos", "reuters"]',
 '{
   "type": "array",
   "items": {"type": "string"},
   "uniqueItems": true,
   "title": "News Sources"
 }',
 'Preferred news sources');
```

### What Goes Where

| Data Type | Storage | Examples |
|-----------|---------|----------|
| **API credentials/config** | Settings DB (global) | `api.openmeteo`, `api.nos`, `api.alphavantage` |
| **Explicit user preferences** | Settings DB (user-scoped) | `weather.units`, `news.sources` |
| **Learned user facts** | Biographer knowledge graph | Location, interests, schedule |
| **Internal service URLs** | ENV vars | `SCHEDULER_URL`, `REDIS_HOST` |

**Key principle:** Settings DB stores explicit choices. Biographer stores learned context.

**Example flow for weather fetch:**
1. Scheduler triggers `/volatile/fetch/weather`
2. Fetch service queries biographer: "Where does this user live?"
3. Biographer returns "Rotterdam" from knowledge graph
4. Fetch service reads `weather.units` preference from settings
5. Calls Open-Meteo API (geocode city → lat/long → forecast) with units from settings
6. Stores result in volatile cache

### Library-Desk Integration

**ENV vars (deployment-specific only):**
```bash
# Central settings database
SYSTEM_SETTINGS_HOST=postgres-shared
SYSTEM_SETTINGS_PORT=5432
SYSTEM_SETTINGS_DB=system_settings
SYSTEM_SETTINGS_USER=settings
SYSTEM_SETTINGS_PASSWORD=xxx

# Internal service URLs (plumbing, not in settings DB)
SCHEDULER_URL=http://scheduler:8080
BIOGRAPHER_URL=http://biographer:8080
```

**New file: `src/clients/settings_client.py`**

```python
"""
Client for central Tatlock settings database.

Library-desk reads settings. Writes are done via psql CLI or future CRUD manager.
"""

import asyncpg
import logging
from typing import Optional, Any

logger = logging.getLogger(__name__)


class SettingsClient:
    """Client for system_settings database."""

    def __init__(self, dsn: str):
        self.dsn = dsn
        self._pool: Optional[asyncpg.Pool] = None

    async def connect(self):
        """Initialize connection pool."""
        if not self._pool:
            self._pool = await asyncpg.create_pool(self.dsn, min_size=1, max_size=5)

    async def close(self):
        """Close connection pool."""
        if self._pool:
            await self._pool.close()

    async def get(self, key: str, user_scope: str = "global") -> Optional[Any]:
        """
        Get a setting by key with user fallback to global.

        Returns user-specific value if exists, otherwise global.
        """
        await self.connect()
        async with self._pool.acquire() as conn:
            row = await conn.fetchrow(
                """
                SELECT value FROM settings
                WHERE key = $1 AND user_scope IN ($2, 'global')
                ORDER BY CASE WHEN user_scope = $2 THEN 0 ELSE 1 END
                LIMIT 1
                """,
                key, user_scope
            )
            return row["value"] if row else None

    async def get_by_prefix(self, prefix: str, user_scope: str = "global") -> dict[str, Any]:
        """Get all settings matching a key prefix (e.g., 'api.')."""
        await self.connect()
        async with self._pool.acquire() as conn:
            rows = await conn.fetch(
                """
                SELECT DISTINCT ON (key) key, value FROM settings
                WHERE key LIKE $1 AND user_scope IN ($2, 'global')
                ORDER BY key, CASE WHEN user_scope = $2 THEN 0 ELSE 1 END
                """,
                f"{prefix}%", user_scope
            )
            return {row["key"]: row["value"] for row in rows}

    async def get_api_key(self, service: str) -> Optional[str]:
        """Convenience method to get API key for a service."""
        value = await self.get(f"api.{service}")
        if isinstance(value, dict):
            return value.get("api_key")
        return value
```

### CLI Management

Settings are managed via direct psql commands (future CRUD manager for UI):

```bash
# Connect to settings database
psql -h postgres-shared -U settings -d system_settings

# Add global API key
INSERT INTO settings (key, value, description)
VALUES ('api.alpha_vantage', '{"api_key": "YOUR_KEY"}', 'Alpha Vantage financial API');

# Add global API key with schema for UI
INSERT INTO settings (key, value, schema, description)
VALUES ('api.alpha_vantage', '{"api_key": "YOUR_KEY"}',
        '{"type": "object", "properties": {"api_key": {"type": "string", "format": "password"}}}',
        'Alpha Vantage financial API');

# Add user-specific preference
INSERT INTO settings (key, user_scope, value, description)
VALUES ('weather.units', 'jpmschweitzer', '"metric"', 'Preferred temperature units');

# Update NewsAPI key
UPDATE settings
SET value = '{"api_key": "NEW_KEY"}', updated_at = NOW()
WHERE key = 'api.newsapi' AND user_scope = 'global';

# List all API keys
SELECT key, description FROM settings WHERE key LIKE 'api.%';

# List user settings with fallback
SELECT DISTINCT ON (key) key, user_scope, value FROM settings
WHERE user_scope IN ('jpmschweitzer', 'global')
ORDER BY key, CASE WHEN user_scope = 'jpmschweitzer' THEN 0 ELSE 1 END;

# View specific setting
SELECT * FROM settings WHERE key = 'api.openmeteo';
```

---

## Phase A: Scheduler-Driven Volatile (Prefetch)

### A.1 New Endpoint: `/volatile/fetch`

**File:** `src/routers/volatile.py`

```python
@router.post("/fetch/{namespace}/{key}")
async def fetch_and_store(
    namespace: str,  # "weather", "news"
    key: str,        # "rotterdam", "nos-headlines"
    user: str = Query(default=DEFAULT_USER),
):
    """
    Fetch fresh data from external API and store in volatile cache.

    Called by scheduler on cron schedule. Combines:
    1. Call appropriate API client based on namespace
    2. Store result in volatile cache with appropriate TTL

    API credentials are read from system_settings database.
    """
```

### A.2 API Clients

**New files in `src/clients/`:**

| File | API | Data Type | Refresh |
|------|-----|-----------|---------|
| `weather_client.py` | Open-Meteo (free, no key) | Current + forecast | Daily |
| `news_client.py` | NOS.nl RSS (free, no key) | Headlines | Every 6h |
| `financial_client.py` | Alpha Vantage / Yahoo | Stocks, crypto | On-demand |

**Example: `src/clients/weather_client.py`**

```python
class WeatherClient:
    """Open-Meteo API client with geocoding support."""

    def __init__(self, settings_client: SettingsClient):
        self.settings = settings_client
        self._geo_cache: dict[str, tuple[float, float]] = {}

    async def _get_config(self) -> dict:
        """Get Open-Meteo config from central settings."""
        return await self.settings.get("api.openmeteo")

    async def _geocode(self, city: str) -> tuple[float, float]:
        """Convert city name to lat/long coordinates."""
        if city.lower() in self._geo_cache:
            return self._geo_cache[city.lower()]

        config = await self._get_config()
        url = f"{config['geocoding_url']}?name={city}&count=1"
        async with httpx.AsyncClient() as client:
            resp = await client.get(url)
            data = resp.json()
            if data.get("results"):
                lat = data["results"][0]["latitude"]
                lon = data["results"][0]["longitude"]
                self._geo_cache[city.lower()] = (lat, lon)
                return (lat, lon)
        raise ValueError(f"Could not geocode city: {city}")

    async def get_current(self, city: str) -> dict:
        """Get current weather for city."""
        config = await self._get_config()
        lat, lon = await self._geocode(city)

        url = (f"{config['forecast_url']}?"
               f"latitude={lat}&longitude={lon}"
               f"&current=temperature_2m,weather_code,relative_humidity_2m,wind_speed_10m"
               f"&timezone={config['timezone']}")

        async with httpx.AsyncClient() as client:
            resp = await client.get(url)
            data = resp.json()

        current = data["current"]
        return {
            "temperature": current["temperature_2m"],
            "weather_code": current["weather_code"],
            "humidity": current["relative_humidity_2m"],
            "wind_speed": current["wind_speed_10m"],
            "text": f"Currently {current['temperature_2m']}°C in {city}."
        }
```

### A.3 Fetch Service

**New file:** `src/services/volatile_fetch_service.py`

```python
class VolatileFetchService:
    """Service to fetch external data and store in volatile cache."""

    def __init__(
        self,
        weather_client: WeatherClient,
        news_client: NewsClient,
        volatile_service: VolatileCacheService,
    ):
        self.weather = weather_client
        self.news = news_client
        self.volatile = volatile_service

    async def fetch_weather(self, user: str, city: str) -> VolatileRecordResponse:
        """Fetch weather and store in volatile cache."""
        data = await self.weather.get_current(city)
        return await self.volatile.store(
            user=user,
            namespace="weather",
            key=city.lower(),
            data=data,
            source="openmeteo",
            ttl=86400,  # 24h
        )
```

### A.4 Scheduler Configuration

| Task | Schedule | Endpoint |
|------|----------|----------|
| `volatile_weather` | `0 6 * * *` | `POST /volatile/fetch/weather/rotterdam?user=jpmschweitzer` |
| `volatile_news_nos` | `0 */6 * * *` | `POST /volatile/fetch/news/nos?user=jpmschweitzer` |

---

## Phase B: HybridRAG-Driven Memory (Reactive)

### B.1 Post-Processor Classification

**Modify:** `src/services/hybrid_rag_service.py`

Add Phase 6.5 after persistence:

```python
async def _postprocess_for_memory(
    self,
    web_results: List[Dict],
    query: str,
    user: str,
    config: HybridRAGConfig,
) -> Dict[str, Any]:
    """
    Phase 6.5: Classify web results and store/register appropriately.
    """
    stats = {"volatile": 0, "documents": 0, "prefetch_registered": 0}

    for result in web_results:
        url = result.get("url", "")
        content = result.get("content", "")
        content_type = self._classify_content(url, content)

        if content_type == "pdf" and config.save_documents:
            await self._save_to_documents(url, result.get("title"))
            stats["documents"] += 1

        elif content_type == "ephemeral":
            if config.save_volatile:
                await self._save_to_volatile(user, query, result, ttl=3600)
                stats["volatile"] += 1

            if config.register_prefetch:
                prefetch_spec = self._should_register_prefetch(url, content, query)
                if prefetch_spec:
                    if await self._register_prefetch_task(user, prefetch_spec):
                        stats["prefetch_registered"] += 1

    return stats
```

### B.2 Content Classification

```python
def _classify_content(self, url: str, content: str) -> str:
    """
    Classify web result for memory routing.

    Returns: "pdf", "ephemeral", "skip"
    """
    if url.endswith(".pdf"):
        return "pdf"

    ephemeral_domains = [
        "weather.com", "open-meteo.com", "buienradar",
        "nos.nl", "nu.nl", "reuters.com",
        "yahoo.com/finance", "marketwatch.com",
    ]
    if any(domain in url for domain in ephemeral_domains):
        return "ephemeral"

    return "skip"
```

### B.3 Prefetch Detection

```python
def _should_register_prefetch(self, url: str, content: str, query: str) -> Optional[dict]:
    """
    Determine if content is worth registering for scheduled prefetch.
    """
    # Weather patterns
    weather_match = re.search(r"weather.*(?:in|for)\s+(\w+)", query, re.IGNORECASE)
    if weather_match and any(d in url for d in ["weather.com", "open-meteo.com", "buienradar"]):
        return {
            "namespace": "weather",
            "key": weather_match.group(1).lower(),
            "schedule": "0 6 * * *",
            "description": f"Weather for {weather_match.group(1)}",
        }

    # News patterns
    if "nos.nl" in url:
        return {
            "namespace": "news",
            "key": "nos",
            "schedule": "0 */6 * * *",
            "description": "Dutch news from NOS",
        }

    return None
```

### B.4 Scheduler Client

**New file:** `src/clients/scheduler_client.py`

```python
class SchedulerClient:
    """Client for external scheduler service."""

    def __init__(self, settings_client: SettingsClient):
        self.settings = settings_client

    async def _get_base_url(self) -> str:
        """Get scheduler URL from central settings."""
        return await self.settings.get("scheduler.base_url")

    async def register_task(self, task: SchedulerTask) -> bool:
        """Register a new scheduled task."""
        base_url = await self._get_base_url()
        # ... POST to scheduler API ...

    async def task_exists(self, task_name: str) -> bool:
        """Check if task already exists."""
        # ... GET from scheduler API ...
```

### B.5 Config Options

**Modify:** `src/models/hybrid_rag.py`

```python
class HybridRAGConfig(BaseModel):
    # ... existing fields ...

    # Memory auto-save options
    save_documents: bool = Field(default=False, description="Auto-upload PDFs to Paperless")
    save_volatile: bool = Field(default=True, description="Auto-cache ephemeral web results")
    register_prefetch: bool = Field(default=True, description="Auto-register scheduler tasks")
    volatile_ttl: int = Field(default=3600, description="TTL for reactive volatile cache")
```

---

## Phase C: Document Recall in HybridRAG

### C.1 Add Document Search

**Modify:** `src/services/hybrid_rag_service.py`

Add to `_retrieve_parallel()`:

```python
if config.enable_documents:
    async def document_search():
        results = await self.vector.search(
            query=query,
            user=user,
            limit=config.document_limit,
            doc_type="document"  # Filter to Paperless docs
        )
        return [{"paperless_id": r.metadata.get("paperless_id"), ...} for r in results]

    tasks["document"] = document_search()
```

### C.2 Config Options

```python
enable_documents: bool = Field(default=True)
document_limit: int = Field(default=5)
document_threshold: float = Field(default=0.6)
```

---

## Example Flow

1. **User searches:** "What's the weather in Amsterdam?"
2. **HybridRAG web search:** Returns open-meteo.com or weather site result
3. **Post-processor classifies:** Ephemeral weather content
4. **Immediate store:** `POST /volatile/store` (TTL: 1h)
5. **Prefetch detection:** Matches weather pattern
6. **Scheduler registration:** Creates task `volatile_weather_amsterdam_jpmschweitzer`
7. **Next day 6am:** Scheduler calls `/volatile/fetch/weather/amsterdam`
8. **Future searches:** Get cached weather from volatile

---

## Implementation Order

| Phase | Priority | Effort | Description |
|-------|----------|--------|-------------|
| **Settings DB** | High | Low | PostgreSQL schema + settings client |
| **B.4** | High | Low | Scheduler client |
| **B.1-B.3** | High | Medium | HybridRAG post-processor |
| **B.5** | High | Low | Config options |
| **C.1-C.2** | High | Low | Document recall in HybridRAG |
| **A.1** | Medium | Low | `/volatile/fetch` endpoint |
| **A.2** | Medium | Medium | Weather + News API clients |
| **A.3** | Medium | Low | Fetch service |

---

## Files Summary

### New Files

| Path | Purpose |
|------|---------|
| `src/clients/settings_client.py` | Read-only central settings access |
| `src/clients/scheduler_client.py` | Register/manage scheduler tasks |
| `src/clients/weather_client.py` | Open-Meteo API (with geocoding) |
| `src/clients/news_client.py` | NOS.nl RSS feeds |
| `src/services/volatile_fetch_service.py` | Orchestrates fetch + store |

### Modified Files

| Path | Changes |
|------|---------|
| `src/services/hybrid_rag_service.py` | Post-processor, prefetch detection, document search |
| `src/models/hybrid_rag.py` | Memory config options |
| `src/routers/volatile.py` | `/volatile/fetch/{namespace}/{key}` |
| `src/core/dependencies.py` | Settings client, scheduler client DI |
| `src/config.py` | `SYSTEM_SETTINGS_*` connection vars |

### Database

| Item | Details |
|------|---------|
| Database | `system_settings` (PostgreSQL) |
| Table | `settings (key, value JSONB, category, ...)` |
| Library-desk user | `library_desk_ro` (read-only) |
| Management | Direct psql commands |