Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4cfad2e8bc | ||
|
|
1e31e74ad6 | ||
|
|
72f515bf61 | ||
|
|
0c085d603e | ||
|
|
152b2f28c4 | ||
|
|
46b9bcd7a0 | ||
|
|
68eb1add3d | ||
|
|
d6b30570a0 | ||
|
|
6b0530ed79 | ||
|
|
0a8c2639a0 |
@@ -1,12 +1,25 @@
|
||||
name: Build and Push
|
||||
|
||||
on:
|
||||
release:
|
||||
types: [published]
|
||||
push:
|
||||
tags:
|
||||
- 'v*'
|
||||
|
||||
jobs:
|
||||
release:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Create Gitea Release
|
||||
run: |
|
||||
curl -sf -X POST \
|
||||
-H "Authorization: token ${{ secrets.GITHUB_TOKEN }}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"tag_name": "${{ github.ref_name }}", "name": "Release ${{ github.ref_name }}", "body": "Automated release for ${{ github.ref_name }}"}' \
|
||||
"${{ github.server_url }}/api/v1/repos/${{ github.repository }}/releases"
|
||||
|
||||
build:
|
||||
runs-on: ubuntu-latest
|
||||
needs: release
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
|
||||
@@ -5,6 +5,82 @@ All notable changes to Library Desk will be documented in this file.
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [1.7.2] - 2026-01-07
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Volatile TTL doubled** - TTL now 2x refresh interval to survive missed/delayed scheduler runs
|
||||
|
||||
## [1.7.1] - 2026-01-07
|
||||
|
||||
### Fixed
|
||||
|
||||
- **CI workflow** - Updated Gitea Actions to trigger on tag push (matching core-api)
|
||||
|
||||
## [1.7.0] - 2026-01-07
|
||||
|
||||
### Added
|
||||
|
||||
- **Combined Environment Endpoint** - `POST /volatile/fetch/environment/{city}`
|
||||
- Fetches weather and air quality concurrently with `asyncio.gather()`
|
||||
- Single geocode lookup shared between both API calls
|
||||
- More efficient than calling weather and air_quality separately
|
||||
- Reduces wall-clock time and eliminates redundant geocoding
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Scheduler executor name** - Fixed `rest_api` → `rest_api_executor` in SchedulerTask model and register_volatile_fetch() to prevent "Executor module not found" errors
|
||||
|
||||
## [1.6.2] - 2025-12-30
|
||||
|
||||
### Added
|
||||
|
||||
- **System Statistics Endpoint** - `GET /stats`
|
||||
- Neo4j: node counts by type (Document, Entity, Collection, Search)
|
||||
- Qdrant: collection counts, total vectors, per-collection breakdown
|
||||
- Wiki.js: total page count
|
||||
- Paperless: documents, tags, correspondents, document types
|
||||
|
||||
- **Weather/Forecast Separation** - Split weather into two distinct namespaces
|
||||
- `POST /volatile/fetch/weather/{city}` - Current conditions only (1hr TTL)
|
||||
- `POST /volatile/fetch/forecast/{city}` - 7-day outlook (12hr TTL)
|
||||
- Different update frequencies for efficient caching
|
||||
- `FORECAST` namespace added to volatile namespaces
|
||||
|
||||
### Changed
|
||||
|
||||
- Weather namespace TTL changed from 30 minutes to 1 hour (current conditions)
|
||||
- Forecast data now stored separately with 12 hour TTL
|
||||
|
||||
## [1.6.1] - 2025-12-30
|
||||
|
||||
### Added
|
||||
|
||||
- **Weather Forecast Support** - Enhanced weather fetch with 7-day daily forecasts
|
||||
- Current conditions now include UV index
|
||||
- Daily forecasts with high/low temps, conditions, precipitation chance, UV max
|
||||
- Natural language text summary with multi-day outlook
|
||||
- **Sun Times Endpoint** - `POST /volatile/fetch/sun/{city}`
|
||||
- Sunrise and sunset times (HH:MM and ISO formats)
|
||||
- Daylight duration in seconds and hours
|
||||
- Separate volatile namespace with 24hr TTL
|
||||
- Useful for home automation light triggers
|
||||
- **Air Quality Endpoint** - `POST /volatile/fetch/air_quality/{city}`
|
||||
- European and US AQI indices
|
||||
- Pollutants: PM2.5, PM10, ozone, nitrogen dioxide, sulphur dioxide, carbon monoxide
|
||||
- Pollen data (grass, birch, alder) for European locations (seasonal)
|
||||
- Hourly refresh (1hr TTL)
|
||||
- **New Base Models**
|
||||
- `SunTimes` dataclass for sunrise/sunset data
|
||||
- `AirQuality` dataclass with AQI and pollutants
|
||||
- `AirQualityProvider` abstract interface
|
||||
- **New Volatile Namespace** - `SUN` for sunrise/sunset times (86400s default TTL)
|
||||
|
||||
### Changed
|
||||
|
||||
- Weather fetch now uses `get_forecast()` instead of `get_current()` for richer data
|
||||
- `OpenMeteoProvider` now implements both `WeatherProvider` and `AirQualityProvider`
|
||||
|
||||
## [1.6.0] - 2025-12-29
|
||||
|
||||
### Added
|
||||
|
||||
@@ -41,12 +41,3 @@ Check for duplicate or highly similar documents using vector similarity and grap
|
||||
3. Check graph relationships
|
||||
4. Return candidates with similarity scores
|
||||
|
||||
## System Statistics
|
||||
|
||||
#### `GET /stats`
|
||||
Get system statistics (wiki pages, neo4j nodes, qdrant vectors).
|
||||
|
||||
**Implementation needed:**
|
||||
- Query Neo4j for node count
|
||||
- Query Qdrant for vector count
|
||||
- Query Wiki.js for page count
|
||||
|
||||
@@ -0,0 +1,102 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Quick script to check environmental data in Qdrant volatile cache."""
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import time
|
||||
from dotenv import load_dotenv
|
||||
|
||||
load_dotenv()
|
||||
|
||||
async def main():
|
||||
from qdrant_client import QdrantClient
|
||||
from qdrant_client.models import Filter, FieldCondition, MatchValue
|
||||
from src.config import get_settings
|
||||
|
||||
settings = get_settings()
|
||||
qdrant = QdrantClient(url=settings.qdrant_url)
|
||||
|
||||
user = "jpmschweitzer"
|
||||
collection = f"volatile_{user}"
|
||||
|
||||
print(f"\n{'='*60}")
|
||||
print(f"Volatile Data in Qdrant ({collection})")
|
||||
print(f"{'='*60}\n")
|
||||
|
||||
namespaces = ["weather", "air_quality", "forecast", "sun", "news"]
|
||||
|
||||
for ns in namespaces:
|
||||
try:
|
||||
results = qdrant.scroll(
|
||||
collection_name=collection,
|
||||
scroll_filter=Filter(
|
||||
must=[FieldCondition(key="namespace", match=MatchValue(value=ns))]
|
||||
),
|
||||
limit=10,
|
||||
with_payload=True,
|
||||
with_vectors=False,
|
||||
)
|
||||
|
||||
points = results[0]
|
||||
print(f"=== {ns.upper()} ({len(points)} records) ===")
|
||||
|
||||
if not points:
|
||||
print(" (no data)")
|
||||
print()
|
||||
continue
|
||||
|
||||
for point in points:
|
||||
payload = point.payload
|
||||
raw_data = payload.get("raw_data", {})
|
||||
|
||||
if ns == "weather":
|
||||
print(f" Temperature: {raw_data.get('temperature')}°C (feels like {raw_data.get('feels_like')}°C)")
|
||||
print(f" Conditions: {raw_data.get('conditions')}")
|
||||
print(f" Humidity: {raw_data.get('humidity')}%")
|
||||
print(f" Wind: {raw_data.get('wind_speed')} km/h")
|
||||
print(f" UV Index: {raw_data.get('uv_index')}")
|
||||
|
||||
elif ns == "air_quality":
|
||||
print(f" European AQI: {raw_data.get('aqi_european')}")
|
||||
print(f" US AQI: {raw_data.get('aqi_us')}")
|
||||
print(f" PM2.5: {raw_data.get('pm2_5')} µg/m³")
|
||||
print(f" PM10: {raw_data.get('pm10')} µg/m³")
|
||||
print(f" Ozone: {raw_data.get('ozone')} µg/m³")
|
||||
print(f" NO₂: {raw_data.get('nitrogen_dioxide')} µg/m³")
|
||||
|
||||
elif ns == "forecast":
|
||||
daily = raw_data.get("daily", [])
|
||||
for day in daily[:5]:
|
||||
print(f" {day.get('day_name', 'N/A')[:3]}: {day.get('temp_low'):.0f}-{day.get('temp_high'):.0f}°C, {day.get('conditions')}")
|
||||
|
||||
elif ns == "sun":
|
||||
print(f" Sunrise: {raw_data.get('sunrise')}")
|
||||
print(f" Sunset: {raw_data.get('sunset')}")
|
||||
print(f" Daylight: {raw_data.get('daylight_hours', 0):.1f} hours")
|
||||
|
||||
elif ns == "news":
|
||||
headlines = raw_data.get("headlines", [])
|
||||
print(f" Category: {raw_data.get('category', 'general')}")
|
||||
print(f" Headlines ({len(headlines)}):")
|
||||
for item in headlines[:5]:
|
||||
title = item.get("title", "")[:60]
|
||||
source = item.get("source", "")
|
||||
print(f" - [{source}] {title}...")
|
||||
|
||||
# Show TTL info
|
||||
ttl_expiry = payload.get("ttl_expiry")
|
||||
if ttl_expiry:
|
||||
remaining = (ttl_expiry / 1000) - time.time()
|
||||
if remaining > 0:
|
||||
print(f" TTL remaining: {int(remaining)}s ({int(remaining/60)} min)")
|
||||
else:
|
||||
print(f" TTL: EXPIRED")
|
||||
|
||||
print()
|
||||
|
||||
except Exception as e:
|
||||
print(f" Error fetching {ns}: {e}")
|
||||
print()
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -16,8 +16,16 @@ This document outlines the implementation of "remember" triggers for the memory
|
||||
| Memory Tier | Remember Trigger | Recall | Status |
|
||||
|-------------|------------------|--------|--------|
|
||||
| Wiki | Wiki.js webhook, Consolidation | HybridRAG vector+graph | ✅ Complete |
|
||||
| Documents | Paperless webhook | Manual `/documents/search` | ⚠️ Partial (no HybridRAG recall) |
|
||||
| Volatile | Manual `/volatile/store` only | HybridRAG volatile search | ⚠️ Partial (no auto-triggers) |
|
||||
| Documents | Paperless webhook | HybridRAG document search | ✅ Complete (v1.6.0) |
|
||||
| Volatile | Scheduler prefetch, HybridRAG post-processor | HybridRAG volatile search | ✅ Complete (v1.6.0) |
|
||||
|
||||
### Implementation Summary (v1.6.0)
|
||||
|
||||
- **Settings DB**: Central `system_settings` PostgreSQL database with `SettingsClient`
|
||||
- **Phase A**: Volatile fetch endpoints (`/volatile/fetch/{namespace}/{key}`) with weather, news, financial providers
|
||||
- **Phase B**: Unified memory routing in consolidation service (wiki/volatile/file/prefetch/skip classification)
|
||||
- **Phase C**: Document recall in HybridRAG (4-source parallel retrieval)
|
||||
- **Scheduler Integration**: `SchedulerClient` for external scheduler task registration
|
||||
|
||||
---
|
||||
|
||||
@@ -662,46 +670,60 @@ document_threshold: float = Field(default=0.6)
|
||||
|
||||
## Implementation Order
|
||||
|
||||
| Phase | Priority | Effort | Description |
|
||||
|-------|----------|--------|-------------|
|
||||
| **Settings DB** | High | Low | PostgreSQL schema + settings client |
|
||||
| **B.4** | High | Low | Scheduler client |
|
||||
| **B.1-B.3** | High | Medium | HybridRAG post-processor |
|
||||
| **B.5** | High | Low | Config options |
|
||||
| **C.1-C.2** | High | Low | Document recall in HybridRAG |
|
||||
| **A.1** | Medium | Low | `/volatile/fetch` endpoint |
|
||||
| **A.2** | Medium | Medium | Weather + News API clients |
|
||||
| **A.3** | Medium | Low | Fetch service |
|
||||
| Phase | Priority | Effort | Description | Status |
|
||||
|-------|----------|--------|-------------|--------|
|
||||
| **Settings DB** | High | Low | PostgreSQL schema + settings client | ✅ v1.5.0 |
|
||||
| **B.4** | High | Low | Scheduler client | ✅ v1.6.0 |
|
||||
| **B.1-B.3** | High | Medium | HybridRAG post-processor | ✅ v1.6.0 |
|
||||
| **B.5** | High | Low | Config options | ✅ v1.6.0 |
|
||||
| **C.1-C.2** | High | Low | Document recall in HybridRAG | ✅ v1.6.0 |
|
||||
| **A.1** | Medium | Low | `/volatile/fetch` endpoint | ✅ v1.6.0 |
|
||||
| **A.2** | Medium | Medium | Weather + News API clients | ✅ v1.5.0 |
|
||||
| **A.3** | Medium | Low | Fetch service | ✅ v1.6.0 |
|
||||
|
||||
### Remaining Work
|
||||
|
||||
| Item | Description | Status |
|
||||
|------|-------------|--------|
|
||||
| File upload | Download PDFs and upload to Paperless | ⚠️ Placeholder (logs only) |
|
||||
| Prefetch patterns | More sophisticated pattern detection | Optional enhancement |
|
||||
|
||||
---
|
||||
|
||||
## Files Summary
|
||||
|
||||
### New Files
|
||||
### New Files (Implemented)
|
||||
|
||||
| Path | Purpose |
|
||||
|------|---------|
|
||||
| `src/clients/settings_client.py` | Read-only central settings access |
|
||||
| `src/clients/scheduler_client.py` | Register/manage scheduler tasks |
|
||||
| `src/clients/weather_client.py` | Open-Meteo API (with geocoding) |
|
||||
| `src/clients/news_client.py` | NOS.nl RSS feeds |
|
||||
| `src/services/volatile_fetch_service.py` | Orchestrates fetch + store |
|
||||
| Path | Purpose | Version |
|
||||
|------|---------|---------|
|
||||
| `src/clients/settings_client.py` | Central settings database access | v1.5.0 |
|
||||
| `src/clients/scheduler_client.py` | External scheduler task management | v1.6.0 |
|
||||
| `src/apis/__init__.py` | External API providers package | v1.5.0 |
|
||||
| `src/apis/base.py` | Abstract base classes for providers | v1.5.0 |
|
||||
| `src/apis/weather.py` | OpenMeteoProvider (geocoding + forecast) | v1.5.0 |
|
||||
| `src/apis/news.py` | AggregatedNewsProvider | v1.5.0 |
|
||||
| `src/apis/nos.py` | NOSProvider (Dutch news RSS) | v1.5.0 |
|
||||
| `src/apis/bbc.py` | BBCProvider (English news RSS) | v1.5.0 |
|
||||
| `src/apis/financial.py` | AlphaVantageProvider (stocks/crypto) | v1.5.0 |
|
||||
| `src/services/volatile_fetch_service.py` | Orchestrates fetch + store | v1.6.0 |
|
||||
|
||||
### Modified Files
|
||||
|
||||
| Path | Changes |
|
||||
|------|---------|
|
||||
| `src/services/hybrid_rag_service.py` | Post-processor, prefetch detection, document search |
|
||||
| `src/models/hybrid_rag.py` | Memory config options |
|
||||
| `src/routers/volatile.py` | `/volatile/fetch/{namespace}/{key}` |
|
||||
| `src/core/dependencies.py` | Settings client, scheduler client DI |
|
||||
| `src/config.py` | `SYSTEM_SETTINGS_*` connection vars |
|
||||
| Path | Changes | Version |
|
||||
|------|---------|---------|
|
||||
| `src/services/hybrid_rag_service.py` | Document search (4-source parallel retrieval) | v1.6.0 |
|
||||
| `src/services/consolidation_service.py` | Unified memory routing, scheduler integration | v1.6.0 |
|
||||
| `src/models/hybrid_rag.py` | Document config options (`enable_documents`, `document_limit`) | v1.6.0 |
|
||||
| `src/models/consolidation.py` | Memory routing models | v1.6.0 |
|
||||
| `src/routers/volatile.py` | `/volatile/fetch/{namespace}/{key}` endpoints | v1.6.0 |
|
||||
| `src/core/dependencies.py` | Settings, scheduler, provider DI | v1.5.0-v1.6.0 |
|
||||
| `src/config.py` | `SYSTEM_SETTINGS_*`, `SCHEDULER_URL` vars | v1.5.0-v1.6.0 |
|
||||
|
||||
### Database
|
||||
|
||||
| Item | Details |
|
||||
|------|---------|
|
||||
| Database | `system_settings` (PostgreSQL) |
|
||||
| Table | `settings (key, value JSONB, category, ...)` |
|
||||
| Library-desk user | `library_desk_ro` (read-only) |
|
||||
| Management | Direct psql commands |
|
||||
| Database | `system_settings` (PostgreSQL on postgres-shared) |
|
||||
| Table | `settings (key, user_scope, value JSONB, schema JSONB, ...)` |
|
||||
| Library-desk access | Read-only via `SettingsClient` |
|
||||
| Management | Direct psql commands (future: CRUD manager UI) |
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "library-desk"
|
||||
version = "1.6.0"
|
||||
version = "1.7.2"
|
||||
description = "Coordination service for The Library system - HybridRAG queries, document ingestion, entity extraction, and knowledge consolidation"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12"
|
||||
|
||||
@@ -27,6 +27,9 @@ from .base import (
|
||||
DayForecast,
|
||||
WeatherForecast,
|
||||
GeoLocation,
|
||||
SunTimes,
|
||||
# Air quality models
|
||||
AirQuality,
|
||||
# News models
|
||||
NewsItem,
|
||||
NewsFeed,
|
||||
@@ -34,6 +37,7 @@ from .base import (
|
||||
StockQuote,
|
||||
# Abstract providers
|
||||
WeatherProvider,
|
||||
AirQualityProvider,
|
||||
NewsProvider,
|
||||
FinancialProvider,
|
||||
)
|
||||
@@ -53,8 +57,12 @@ __all__ = [
|
||||
"DayForecast",
|
||||
"WeatherForecast",
|
||||
"GeoLocation",
|
||||
"SunTimes",
|
||||
"WeatherProvider",
|
||||
"OpenMeteoProvider",
|
||||
# Air quality
|
||||
"AirQuality",
|
||||
"AirQualityProvider",
|
||||
# News
|
||||
"NewsItem",
|
||||
"NewsFeed",
|
||||
|
||||
+102
-4
@@ -44,14 +44,18 @@ class CurrentWeather:
|
||||
condition_text: str # Human-readable description
|
||||
timestamp: datetime
|
||||
location: str # City/location name
|
||||
uv_index: Optional[float] = None # UV index 0-11+
|
||||
|
||||
def to_text(self) -> str:
|
||||
"""Generate natural language description."""
|
||||
return (
|
||||
f"Currently {self.temperature:.1f}°C "
|
||||
f"({self.condition_text}) in {self.location}. "
|
||||
parts = [
|
||||
f"Currently {self.temperature:.1f}°C",
|
||||
f"({self.condition_text}) in {self.location}.",
|
||||
f"Humidity {self.humidity}%, wind {self.wind_speed:.0f} km/h."
|
||||
)
|
||||
]
|
||||
if self.uv_index is not None:
|
||||
parts.append(f"UV index: {self.uv_index:.0f}.")
|
||||
return " ".join(parts)
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -64,6 +68,14 @@ class DayForecast:
|
||||
condition_text: str
|
||||
precipitation_chance: Optional[int] # Percentage 0-100
|
||||
precipitation_mm: Optional[float]
|
||||
uv_index_max: Optional[float] = None # Max UV index for the day
|
||||
|
||||
def to_text(self) -> str:
|
||||
"""Generate natural language description."""
|
||||
date_str = self.date.strftime("%A") # Day name
|
||||
precip = f", {self.precipitation_chance}% rain" if self.precipitation_chance else ""
|
||||
uv = f", UV {self.uv_index_max:.0f}" if self.uv_index_max else ""
|
||||
return f"{date_str}: {self.temp_high:.0f}°/{self.temp_low:.0f}°C, {self.condition_text}{precip}{uv}"
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -84,6 +96,78 @@ class GeoLocation:
|
||||
admin_area: Optional[str] = None # State/province
|
||||
|
||||
|
||||
@dataclass
|
||||
class SunTimes:
|
||||
"""Sunrise/sunset times for a location."""
|
||||
location: str
|
||||
date: datetime
|
||||
sunrise: datetime
|
||||
sunset: datetime
|
||||
daylight_duration: int # seconds
|
||||
solar_noon: Optional[datetime] = None
|
||||
|
||||
def to_text(self) -> str:
|
||||
"""Generate natural language description."""
|
||||
sunrise_str = self.sunrise.strftime("%H:%M")
|
||||
sunset_str = self.sunset.strftime("%H:%M")
|
||||
hours = self.daylight_duration // 3600
|
||||
minutes = (self.daylight_duration % 3600) // 60
|
||||
return (
|
||||
f"Sun times for {self.location} on {self.date.strftime('%A %d %B')}: "
|
||||
f"Sunrise at {sunrise_str}, sunset at {sunset_str}. "
|
||||
f"Daylight duration: {hours}h {minutes}m."
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class AirQuality:
|
||||
"""Air quality measurements for a location."""
|
||||
location: str
|
||||
timestamp: datetime
|
||||
aqi_european: Optional[int] # European AQI 0-500+
|
||||
aqi_us: Optional[int] # US AQI 0-500+
|
||||
pm2_5: Optional[float] # µg/m³
|
||||
pm10: Optional[float] # µg/m³
|
||||
ozone: Optional[float] # µg/m³
|
||||
nitrogen_dioxide: Optional[float] # µg/m³
|
||||
sulphur_dioxide: Optional[float] # µg/m³
|
||||
carbon_monoxide: Optional[float] # µg/m³
|
||||
# Pollen (European data only, seasonal)
|
||||
pollen_grass: Optional[float] = None
|
||||
pollen_birch: Optional[float] = None
|
||||
pollen_alder: Optional[float] = None
|
||||
|
||||
def to_text(self) -> str:
|
||||
"""Generate natural language description."""
|
||||
parts = [f"Air quality in {self.location}:"]
|
||||
if self.aqi_european is not None:
|
||||
level = self._aqi_level(self.aqi_european)
|
||||
parts.append(f"European AQI {self.aqi_european} ({level}).")
|
||||
if self.pm2_5 is not None:
|
||||
parts.append(f"PM2.5: {self.pm2_5:.1f} µg/m³.")
|
||||
if self.pm10 is not None:
|
||||
parts.append(f"PM10: {self.pm10:.1f} µg/m³.")
|
||||
if self.ozone is not None:
|
||||
parts.append(f"Ozone: {self.ozone:.1f} µg/m³.")
|
||||
return " ".join(parts)
|
||||
|
||||
@staticmethod
|
||||
def _aqi_level(aqi: int) -> str:
|
||||
"""Convert AQI to human-readable level."""
|
||||
if aqi <= 20:
|
||||
return "good"
|
||||
elif aqi <= 40:
|
||||
return "fair"
|
||||
elif aqi <= 60:
|
||||
return "moderate"
|
||||
elif aqi <= 80:
|
||||
return "poor"
|
||||
elif aqi <= 100:
|
||||
return "very poor"
|
||||
else:
|
||||
return "hazardous"
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# News Models
|
||||
# =============================================================================
|
||||
@@ -163,6 +247,11 @@ class WeatherProvider(ABC):
|
||||
"""Get weather forecast for a location."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
async def get_sun_times(self, location: GeoLocation) -> SunTimes:
|
||||
"""Get sunrise/sunset times for today."""
|
||||
pass
|
||||
|
||||
async def get_weather_for_city(self, city: str) -> CurrentWeather:
|
||||
"""Convenience method: geocode and get current weather."""
|
||||
location = await self.geocode(city)
|
||||
@@ -171,6 +260,15 @@ class WeatherProvider(ABC):
|
||||
return await self.get_current(location)
|
||||
|
||||
|
||||
class AirQualityProvider(ABC):
|
||||
"""Abstract base class for air quality API providers."""
|
||||
|
||||
@abstractmethod
|
||||
async def get_air_quality(self, location: GeoLocation) -> AirQuality:
|
||||
"""Get current air quality for a location."""
|
||||
pass
|
||||
|
||||
|
||||
class NewsProvider(ABC):
|
||||
"""Abstract base class for news API providers."""
|
||||
|
||||
|
||||
+138
-8
@@ -14,11 +14,14 @@ from typing import Optional
|
||||
|
||||
from .base import (
|
||||
WeatherProvider,
|
||||
AirQualityProvider,
|
||||
WeatherCondition,
|
||||
CurrentWeather,
|
||||
DayForecast,
|
||||
WeatherForecast,
|
||||
GeoLocation,
|
||||
SunTimes,
|
||||
AirQuality,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -89,11 +92,12 @@ WMO_DESCRIPTIONS: dict[int, str] = {
|
||||
}
|
||||
|
||||
|
||||
class OpenMeteoProvider(WeatherProvider):
|
||||
"""Open-Meteo weather API implementation."""
|
||||
class OpenMeteoProvider(WeatherProvider, AirQualityProvider):
|
||||
"""Open-Meteo weather and air quality API implementation."""
|
||||
|
||||
GEOCODING_URL = "https://geocoding-api.open-meteo.com/v1/search"
|
||||
WEATHER_URL = "https://api.open-meteo.com/v1/forecast"
|
||||
AIR_QUALITY_URL = "https://air-quality-api.open-meteo.com/v1/air-quality"
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
@@ -194,9 +198,11 @@ class OpenMeteoProvider(WeatherProvider):
|
||||
"wind_speed_10m",
|
||||
"wind_direction_10m"
|
||||
],
|
||||
"daily": ["uv_index_max"],
|
||||
"timezone": self.timezone,
|
||||
"temperature_unit": "celsius",
|
||||
"wind_speed_unit": "kmh"
|
||||
"wind_speed_unit": "kmh",
|
||||
"forecast_days": 1
|
||||
}
|
||||
)
|
||||
response.raise_for_status()
|
||||
@@ -205,6 +211,12 @@ class OpenMeteoProvider(WeatherProvider):
|
||||
current = data.get("current", {})
|
||||
weather_code = current.get("weather_code", 0)
|
||||
|
||||
# Get today's UV index from daily data
|
||||
daily = data.get("daily", {})
|
||||
uv_index = None
|
||||
if daily.get("uv_index_max"):
|
||||
uv_index = daily["uv_index_max"][0]
|
||||
|
||||
return CurrentWeather(
|
||||
temperature=current.get("temperature_2m", 0.0),
|
||||
feels_like=current.get("apparent_temperature"),
|
||||
@@ -214,7 +226,8 @@ class OpenMeteoProvider(WeatherProvider):
|
||||
condition=WMO_CODE_MAP.get(weather_code, WeatherCondition.UNKNOWN),
|
||||
condition_text=WMO_DESCRIPTIONS.get(weather_code, "Unknown"),
|
||||
timestamp=datetime.now(),
|
||||
location=location.name
|
||||
location=location.name,
|
||||
uv_index=uv_index
|
||||
)
|
||||
except httpx.HTTPError as e:
|
||||
logger.error(f"Weather request failed for {location.name}: {e}")
|
||||
@@ -259,7 +272,8 @@ class OpenMeteoProvider(WeatherProvider):
|
||||
"temperature_2m_max",
|
||||
"temperature_2m_min",
|
||||
"precipitation_sum",
|
||||
"precipitation_probability_max"
|
||||
"precipitation_probability_max",
|
||||
"uv_index_max"
|
||||
],
|
||||
"timezone": self.timezone,
|
||||
"temperature_unit": "celsius",
|
||||
@@ -272,7 +286,14 @@ class OpenMeteoProvider(WeatherProvider):
|
||||
|
||||
# Parse current weather
|
||||
current_data = data.get("current", {})
|
||||
daily_data = data.get("daily", {})
|
||||
weather_code = current_data.get("weather_code", 0)
|
||||
|
||||
# Get today's UV from daily data
|
||||
uv_index = None
|
||||
if daily_data.get("uv_index_max"):
|
||||
uv_index = daily_data["uv_index_max"][0]
|
||||
|
||||
current = CurrentWeather(
|
||||
temperature=current_data.get("temperature_2m", 0.0),
|
||||
feels_like=current_data.get("apparent_temperature"),
|
||||
@@ -282,15 +303,16 @@ class OpenMeteoProvider(WeatherProvider):
|
||||
condition=WMO_CODE_MAP.get(weather_code, WeatherCondition.UNKNOWN),
|
||||
condition_text=WMO_DESCRIPTIONS.get(weather_code, "Unknown"),
|
||||
timestamp=datetime.now(),
|
||||
location=location.name
|
||||
location=location.name,
|
||||
uv_index=uv_index
|
||||
)
|
||||
|
||||
# Parse daily forecast
|
||||
daily_data = data.get("daily", {})
|
||||
daily = []
|
||||
dates = daily_data.get("time", [])
|
||||
for i, date_str in enumerate(dates):
|
||||
code = daily_data.get("weather_code", [])[i] if i < len(daily_data.get("weather_code", [])) else 0
|
||||
uv_max = daily_data.get("uv_index_max", [])[i] if i < len(daily_data.get("uv_index_max", [])) else None
|
||||
daily.append(DayForecast(
|
||||
date=datetime.fromisoformat(date_str),
|
||||
temp_high=daily_data.get("temperature_2m_max", [])[i] if i < len(daily_data.get("temperature_2m_max", [])) else 0.0,
|
||||
@@ -298,7 +320,8 @@ class OpenMeteoProvider(WeatherProvider):
|
||||
condition=WMO_CODE_MAP.get(code, WeatherCondition.UNKNOWN),
|
||||
condition_text=WMO_DESCRIPTIONS.get(code, "Unknown"),
|
||||
precipitation_chance=daily_data.get("precipitation_probability_max", [])[i] if i < len(daily_data.get("precipitation_probability_max", [])) else None,
|
||||
precipitation_mm=daily_data.get("precipitation_sum", [])[i] if i < len(daily_data.get("precipitation_sum", [])) else None
|
||||
precipitation_mm=daily_data.get("precipitation_sum", [])[i] if i < len(daily_data.get("precipitation_sum", [])) else None,
|
||||
uv_index_max=uv_max
|
||||
))
|
||||
|
||||
return WeatherForecast(
|
||||
@@ -309,3 +332,110 @@ class OpenMeteoProvider(WeatherProvider):
|
||||
except httpx.HTTPError as e:
|
||||
logger.error(f"Forecast request failed for {location.name}: {e}")
|
||||
raise ValueError(f"Failed to get forecast: {e}")
|
||||
|
||||
async def get_sun_times(self, location: GeoLocation) -> SunTimes:
|
||||
"""
|
||||
Get sunrise/sunset times for today.
|
||||
|
||||
Args:
|
||||
location: GeoLocation with lat/long
|
||||
|
||||
Returns:
|
||||
SunTimes with sunrise, sunset, and daylight duration
|
||||
|
||||
Raises:
|
||||
ValueError: If API request fails
|
||||
"""
|
||||
try:
|
||||
response = await self.client.get(
|
||||
self.WEATHER_URL,
|
||||
params={
|
||||
"latitude": location.latitude,
|
||||
"longitude": location.longitude,
|
||||
"daily": [
|
||||
"sunrise",
|
||||
"sunset",
|
||||
"daylight_duration"
|
||||
],
|
||||
"timezone": self.timezone,
|
||||
"forecast_days": 1
|
||||
}
|
||||
)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
|
||||
daily = data.get("daily", {})
|
||||
date_str = daily.get("time", [""])[0]
|
||||
sunrise_str = daily.get("sunrise", [""])[0]
|
||||
sunset_str = daily.get("sunset", [""])[0]
|
||||
daylight = daily.get("daylight_duration", [0])[0]
|
||||
|
||||
return SunTimes(
|
||||
location=location.name,
|
||||
date=datetime.fromisoformat(date_str) if date_str else datetime.now(),
|
||||
sunrise=datetime.fromisoformat(sunrise_str) if sunrise_str else datetime.now(),
|
||||
sunset=datetime.fromisoformat(sunset_str) if sunset_str else datetime.now(),
|
||||
daylight_duration=int(daylight) if daylight else 0
|
||||
)
|
||||
except httpx.HTTPError as e:
|
||||
logger.error(f"Sun times request failed for {location.name}: {e}")
|
||||
raise ValueError(f"Failed to get sun times: {e}")
|
||||
|
||||
async def get_air_quality(self, location: GeoLocation) -> AirQuality:
|
||||
"""
|
||||
Get current air quality for a location.
|
||||
|
||||
Args:
|
||||
location: GeoLocation with lat/long
|
||||
|
||||
Returns:
|
||||
AirQuality with pollutant measurements and AQI
|
||||
|
||||
Raises:
|
||||
ValueError: If API request fails
|
||||
"""
|
||||
try:
|
||||
response = await self.client.get(
|
||||
self.AIR_QUALITY_URL,
|
||||
params={
|
||||
"latitude": location.latitude,
|
||||
"longitude": location.longitude,
|
||||
"current": [
|
||||
"european_aqi",
|
||||
"us_aqi",
|
||||
"pm2_5",
|
||||
"pm10",
|
||||
"ozone",
|
||||
"nitrogen_dioxide",
|
||||
"sulphur_dioxide",
|
||||
"carbon_monoxide",
|
||||
"grass_pollen",
|
||||
"birch_pollen",
|
||||
"alder_pollen"
|
||||
],
|
||||
"timezone": self.timezone
|
||||
}
|
||||
)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
|
||||
current = data.get("current", {})
|
||||
|
||||
return AirQuality(
|
||||
location=location.name,
|
||||
timestamp=datetime.now(),
|
||||
aqi_european=current.get("european_aqi"),
|
||||
aqi_us=current.get("us_aqi"),
|
||||
pm2_5=current.get("pm2_5"),
|
||||
pm10=current.get("pm10"),
|
||||
ozone=current.get("ozone"),
|
||||
nitrogen_dioxide=current.get("nitrogen_dioxide"),
|
||||
sulphur_dioxide=current.get("sulphur_dioxide"),
|
||||
carbon_monoxide=current.get("carbon_monoxide"),
|
||||
pollen_grass=current.get("grass_pollen"),
|
||||
pollen_birch=current.get("birch_pollen"),
|
||||
pollen_alder=current.get("alder_pollen")
|
||||
)
|
||||
except httpx.HTTPError as e:
|
||||
logger.error(f"Air quality request failed for {location.name}: {e}")
|
||||
raise ValueError(f"Failed to get air quality: {e}")
|
||||
|
||||
@@ -18,7 +18,7 @@ class SchedulerTask(BaseModel):
|
||||
|
||||
task_name: str = Field(..., description="Unique task identifier")
|
||||
service: str = Field(default="library-desk", description="Service that owns this task")
|
||||
executor: str = Field(default="rest_api", description="Executor type")
|
||||
executor: str = Field(default="rest_api_executor", description="Executor type")
|
||||
priority: int = Field(default=50, ge=1, le=100, description="Priority (lower = higher)")
|
||||
description: Optional[str] = Field(None, description="Human-readable description")
|
||||
enabled: bool = Field(default=True, description="Whether task is enabled")
|
||||
@@ -291,7 +291,7 @@ class SchedulerClient:
|
||||
task = SchedulerTask(
|
||||
task_name=task_name,
|
||||
service="library-desk",
|
||||
executor="rest_api",
|
||||
executor="rest_api_executor",
|
||||
priority=60, # Background maintenance priority
|
||||
description=description or f"Prefetch {namespace}/{key} for {user}",
|
||||
minute=schedule.get("minute", -1),
|
||||
|
||||
+88
-1
@@ -18,7 +18,7 @@ from pathlib import Path
|
||||
|
||||
from src.config import Settings, get_settings, __version__
|
||||
from src.core.dependencies import (
|
||||
verify_api_key, QdrantDep, WikiJSDep, OllamaDep, Neo4jDep
|
||||
verify_api_key, QdrantDep, WikiJSDep, OllamaDep, Neo4jDep, PaperlessDep
|
||||
)
|
||||
from src.core.multi_tenancy import DEFAULT_USER
|
||||
|
||||
@@ -85,6 +85,14 @@ class HealthResponse(BaseModel):
|
||||
services: Dict[str, Any]
|
||||
|
||||
|
||||
class StatsResponse(BaseModel):
|
||||
"""System statistics response model."""
|
||||
neo4j: Dict[str, int]
|
||||
qdrant: Dict[str, Any]
|
||||
wiki_pages: int
|
||||
paperless: Dict[str, Any]
|
||||
|
||||
|
||||
# Routes
|
||||
@app.get("/", tags=["Root"])
|
||||
async def root() -> Dict[str, str]:
|
||||
@@ -141,6 +149,85 @@ async def health(settings: Settings = Depends(get_settings)) -> HealthResponse:
|
||||
)
|
||||
|
||||
|
||||
@app.get("/stats", response_model=StatsResponse, tags=["System"])
|
||||
async def stats(
|
||||
user: str = Query(default=DEFAULT_USER, description="User identifier"),
|
||||
neo4j: Neo4jDep = None,
|
||||
qdrant: QdrantDep = None,
|
||||
wikijs: WikiJSDep = None,
|
||||
paperless: PaperlessDep = None,
|
||||
api_key: str = Depends(verify_api_key)
|
||||
) -> StatsResponse:
|
||||
"""
|
||||
Get system statistics.
|
||||
|
||||
Returns counts for:
|
||||
- Neo4j: nodes by type (Document, Entity, Collection, Search)
|
||||
- Qdrant: vectors per collection
|
||||
- Wiki.js: total page count
|
||||
- Paperless: documents, tags, correspondents, document types
|
||||
"""
|
||||
# Neo4j node counts by label
|
||||
neo4j_stats = {}
|
||||
try:
|
||||
for label in ["Document", "Entity", "Collection", "Search"]:
|
||||
result = await neo4j.execute_query(
|
||||
f"MATCH (n:{label}) RETURN count(n) as count"
|
||||
)
|
||||
neo4j_stats[label.lower() + "_nodes"] = result[0]["count"] if result else 0
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to get Neo4j stats: {e}")
|
||||
neo4j_stats = {"error": str(e)}
|
||||
|
||||
# Qdrant collection stats
|
||||
qdrant_stats = {}
|
||||
try:
|
||||
collections = await qdrant.list_collections()
|
||||
qdrant_stats["collections"] = len(collections)
|
||||
qdrant_stats["total_vectors"] = sum(c.get("vectors_count", 0) for c in collections)
|
||||
qdrant_stats["by_collection"] = {
|
||||
c["name"]: c["vectors_count"] for c in collections
|
||||
}
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to get Qdrant stats: {e}")
|
||||
qdrant_stats = {"error": str(e)}
|
||||
|
||||
# Wiki.js page count
|
||||
wiki_pages = 0
|
||||
try:
|
||||
pages = await wikijs.list_all_pages(user)
|
||||
wiki_pages = len(pages)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to get Wiki.js stats: {e}")
|
||||
|
||||
# Paperless-ngx document stats
|
||||
paperless_stats = {}
|
||||
try:
|
||||
# Get document count (page_size=1 for efficiency, we just need the count)
|
||||
docs_result = await paperless.list_documents(page_size=1)
|
||||
paperless_stats["documents"] = docs_result.get("count", 0)
|
||||
|
||||
# Get metadata counts
|
||||
tags = await paperless.list_tags()
|
||||
paperless_stats["tags"] = len(tags)
|
||||
|
||||
correspondents = await paperless.list_correspondents()
|
||||
paperless_stats["correspondents"] = len(correspondents)
|
||||
|
||||
doc_types = await paperless.list_document_types()
|
||||
paperless_stats["document_types"] = len(doc_types)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to get Paperless stats: {e}")
|
||||
paperless_stats = {"error": str(e)}
|
||||
|
||||
return StatsResponse(
|
||||
neo4j=neo4j_stats,
|
||||
qdrant=qdrant_stats,
|
||||
wiki_pages=wiki_pages,
|
||||
paperless=paperless_stats
|
||||
)
|
||||
|
||||
|
||||
@app.post("/ingest/check-updates", tags=["Ingestion"])
|
||||
async def check_updates(
|
||||
documents: Dict[str, Any],
|
||||
|
||||
+17
-12
@@ -18,7 +18,9 @@ class VolatileNamespace(str, Enum):
|
||||
Each namespace can have different default TTLs and refresh schedules.
|
||||
"""
|
||||
# Real-time external data
|
||||
WEATHER = "weather" # Current conditions, forecasts
|
||||
WEATHER = "weather" # Current conditions (temperature, humidity, wind)
|
||||
FORECAST = "forecast" # Multi-day weather outlook
|
||||
SUN = "sun" # Sunrise, sunset, daylight duration
|
||||
NEWS = "news" # Headlines, breaking news
|
||||
FINANCIAL = "financial" # Stock prices, exchange rates, crypto
|
||||
TRANSIT = "transit" # Train/bus schedules, delays, disruptions
|
||||
@@ -36,18 +38,21 @@ class VolatileNamespace(str, Enum):
|
||||
|
||||
|
||||
# Default TTLs per namespace (in seconds)
|
||||
# TTL = 2x refresh interval to ensure data survives missed/delayed refreshes
|
||||
NAMESPACE_DEFAULT_TTL: Dict[str, int] = {
|
||||
VolatileNamespace.WEATHER: 1800, # 30 min - weather changes slowly
|
||||
VolatileNamespace.NEWS: 3600, # 1 hour - news cycles
|
||||
VolatileNamespace.FINANCIAL: 300, # 5 min - markets move fast
|
||||
VolatileNamespace.TRANSIT: 300, # 5 min - schedules update frequently
|
||||
VolatileNamespace.TRAFFIC: 600, # 10 min - traffic patterns
|
||||
VolatileNamespace.AIR_QUALITY: 3600, # 1 hour - air quality stable
|
||||
VolatileNamespace.SPORTS: 60, # 1 min - live scores
|
||||
VolatileNamespace.SOCIAL: 600, # 10 min - social notifications
|
||||
VolatileNamespace.SYSTEM: 60, # 1 min - system health
|
||||
VolatileNamespace.CONTEXT: 3600, # 1 hour - session context
|
||||
VolatileNamespace.CUSTOM: 3600, # 1 hour - default for custom
|
||||
VolatileNamespace.WEATHER: 7200, # 2 hours (hourly refresh)
|
||||
VolatileNamespace.FORECAST: 86400, # 24 hours (12hr refresh)
|
||||
VolatileNamespace.SUN: 172800, # 48 hours (daily refresh)
|
||||
VolatileNamespace.NEWS: 7200, # 2 hours (hourly refresh)
|
||||
VolatileNamespace.FINANCIAL: 600, # 10 min (5 min refresh)
|
||||
VolatileNamespace.TRANSIT: 600, # 10 min (5 min refresh)
|
||||
VolatileNamespace.TRAFFIC: 1200, # 20 min (10 min refresh)
|
||||
VolatileNamespace.AIR_QUALITY: 7200, # 2 hours (hourly refresh)
|
||||
VolatileNamespace.SPORTS: 120, # 2 min (1 min refresh)
|
||||
VolatileNamespace.SOCIAL: 1200, # 20 min (10 min refresh)
|
||||
VolatileNamespace.SYSTEM: 120, # 2 min (1 min refresh)
|
||||
VolatileNamespace.CONTEXT: 7200, # 2 hours - session context
|
||||
VolatileNamespace.CUSTOM: 7200, # 2 hours - default for custom
|
||||
}
|
||||
|
||||
|
||||
|
||||
+197
-5
@@ -233,16 +233,16 @@ async def store_volatile(
|
||||
async def fetch_weather(
|
||||
city: str,
|
||||
user: str = Query(default=DEFAULT_USER, description="User identifier"),
|
||||
ttl: int = Query(default=86400, ge=60, le=604800, description="TTL in seconds"),
|
||||
ttl: int = Query(default=3600, ge=60, le=86400, description="TTL in seconds (default 1 hour)"),
|
||||
qdrant: QdrantDep = None,
|
||||
ollama: OllamaDep = None,
|
||||
api_key: str = Depends(verify_api_key)
|
||||
):
|
||||
"""
|
||||
Fetch current weather for a city and store in volatile cache.
|
||||
Fetch current weather conditions for a city and store in volatile cache.
|
||||
|
||||
Called by scheduler for prefetch or on-demand. Geocodes city name
|
||||
and fetches weather from Open-Meteo API.
|
||||
Stores temperature, humidity, wind, UV index. For forecasts use /fetch/forecast.
|
||||
Called by scheduler for hourly prefetch or on-demand.
|
||||
|
||||
**Example:**
|
||||
```
|
||||
@@ -257,7 +257,49 @@ async def fetch_weather(
|
||||
weather_provider=weather_provider,
|
||||
)
|
||||
|
||||
result = await fetch_service.fetch_weather(user, city, ttl=ttl)
|
||||
result = await fetch_service.fetch_current_weather(user, city, ttl=ttl)
|
||||
|
||||
if not result.success:
|
||||
raise HTTPException(status_code=500, detail=result.error)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"namespace": result.namespace,
|
||||
"key": result.key,
|
||||
"record": result.record,
|
||||
}
|
||||
|
||||
|
||||
@router.post("/fetch/forecast/{city}")
|
||||
async def fetch_forecast(
|
||||
city: str,
|
||||
user: str = Query(default=DEFAULT_USER, description="User identifier"),
|
||||
days: int = Query(default=7, ge=1, le=16, description="Forecast days (1-16)"),
|
||||
ttl: int = Query(default=43200, ge=60, le=604800, description="TTL in seconds (default 12 hours)"),
|
||||
qdrant: QdrantDep = None,
|
||||
ollama: OllamaDep = None,
|
||||
api_key: str = Depends(verify_api_key)
|
||||
):
|
||||
"""
|
||||
Fetch weather forecast for a city and store in volatile cache.
|
||||
|
||||
Stores multi-day outlook with highs/lows, precipitation, UV.
|
||||
For current conditions use /fetch/weather.
|
||||
|
||||
**Example:**
|
||||
```
|
||||
POST /volatile/fetch/forecast/amsterdam?user=jpmschweitzer&days=7
|
||||
```
|
||||
"""
|
||||
volatile_service = get_volatile_service(qdrant, ollama)
|
||||
weather_provider = get_weather_provider()
|
||||
|
||||
fetch_service = VolatileFetchService(
|
||||
volatile_service=volatile_service,
|
||||
weather_provider=weather_provider,
|
||||
)
|
||||
|
||||
result = await fetch_service.fetch_forecast(user, city, days=days, ttl=ttl)
|
||||
|
||||
if not result.success:
|
||||
raise HTTPException(status_code=500, detail=result.error)
|
||||
@@ -411,6 +453,156 @@ async def fetch_crypto(
|
||||
}
|
||||
|
||||
|
||||
@router.post("/fetch/sun/{city}")
|
||||
async def fetch_sun_times(
|
||||
city: str,
|
||||
user: str = Query(default=DEFAULT_USER, description="User identifier"),
|
||||
ttl: int = Query(default=86400, ge=60, le=604800, description="TTL in seconds"),
|
||||
qdrant: QdrantDep = None,
|
||||
ollama: OllamaDep = None,
|
||||
api_key: str = Depends(verify_api_key)
|
||||
):
|
||||
"""
|
||||
Fetch sunrise/sunset times for a city and store in volatile cache.
|
||||
|
||||
Fetches from Open-Meteo API. Useful for home automation triggers.
|
||||
|
||||
**Example:**
|
||||
```
|
||||
POST /volatile/fetch/sun/rotterdam?user=jpmschweitzer
|
||||
```
|
||||
|
||||
**Response data includes:**
|
||||
- sunrise/sunset times (both HH:MM and ISO formats)
|
||||
- daylight_duration_seconds
|
||||
- daylight_hours
|
||||
- Natural language text summary
|
||||
"""
|
||||
volatile_service = get_volatile_service(qdrant, ollama)
|
||||
weather_provider = get_weather_provider()
|
||||
|
||||
fetch_service = VolatileFetchService(
|
||||
volatile_service=volatile_service,
|
||||
weather_provider=weather_provider,
|
||||
)
|
||||
|
||||
result = await fetch_service.fetch_sun_times(user, city, ttl=ttl)
|
||||
|
||||
if not result.success:
|
||||
raise HTTPException(status_code=500, detail=result.error)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"namespace": result.namespace,
|
||||
"key": result.key,
|
||||
"record": result.record,
|
||||
}
|
||||
|
||||
|
||||
@router.post("/fetch/air_quality/{city}")
|
||||
async def fetch_air_quality(
|
||||
city: str,
|
||||
user: str = Query(default=DEFAULT_USER, description="User identifier"),
|
||||
ttl: int = Query(default=3600, ge=60, le=86400, description="TTL in seconds"),
|
||||
qdrant: QdrantDep = None,
|
||||
ollama: OllamaDep = None,
|
||||
api_key: str = Depends(verify_api_key)
|
||||
):
|
||||
"""
|
||||
Fetch air quality data for a city and store in volatile cache.
|
||||
|
||||
Fetches from Open-Meteo Air Quality API.
|
||||
|
||||
**Example:**
|
||||
```
|
||||
POST /volatile/fetch/air_quality/rotterdam?user=jpmschweitzer
|
||||
```
|
||||
|
||||
**Response data includes:**
|
||||
- European and US AQI indices
|
||||
- Pollutants: PM2.5, PM10, ozone, nitrogen dioxide, etc.
|
||||
- Pollen data (European locations, seasonal)
|
||||
- Natural language text summary
|
||||
"""
|
||||
volatile_service = get_volatile_service(qdrant, ollama)
|
||||
weather_provider = get_weather_provider()
|
||||
|
||||
fetch_service = VolatileFetchService(
|
||||
volatile_service=volatile_service,
|
||||
weather_provider=weather_provider,
|
||||
)
|
||||
|
||||
result = await fetch_service.fetch_air_quality(user, city, ttl=ttl)
|
||||
|
||||
if not result.success:
|
||||
raise HTTPException(status_code=500, detail=result.error)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"namespace": result.namespace,
|
||||
"key": result.key,
|
||||
"record": result.record,
|
||||
}
|
||||
|
||||
|
||||
@router.post("/fetch/environment/{city}")
|
||||
async def fetch_environment(
|
||||
city: str,
|
||||
user: str = Query(default=DEFAULT_USER, description="User identifier"),
|
||||
weather_ttl: int = Query(default=3600, ge=60, le=86400, description="Weather TTL in seconds"),
|
||||
air_quality_ttl: int = Query(default=3600, ge=60, le=86400, description="Air quality TTL in seconds"),
|
||||
qdrant: QdrantDep = None,
|
||||
ollama: OllamaDep = None,
|
||||
api_key: str = Depends(verify_api_key)
|
||||
):
|
||||
"""
|
||||
Fetch weather and air quality concurrently for a city.
|
||||
|
||||
Performs a single geocode lookup and fetches both weather and air quality
|
||||
data in parallel, storing both in volatile cache. More efficient than
|
||||
calling /fetch/weather and /fetch/air_quality separately.
|
||||
|
||||
**Example:**
|
||||
```
|
||||
POST /volatile/fetch/environment/rotterdam?user=jpmschweitzer
|
||||
```
|
||||
|
||||
**Response includes:**
|
||||
- weather: Current conditions (temperature, humidity, wind, UV)
|
||||
- air_quality: AQI indices, pollutants, pollen data
|
||||
"""
|
||||
volatile_service = get_volatile_service(qdrant, ollama)
|
||||
weather_provider = get_weather_provider()
|
||||
|
||||
fetch_service = VolatileFetchService(
|
||||
volatile_service=volatile_service,
|
||||
weather_provider=weather_provider,
|
||||
)
|
||||
|
||||
result = await fetch_service.fetch_environment(
|
||||
user, city, weather_ttl=weather_ttl, air_quality_ttl=air_quality_ttl
|
||||
)
|
||||
|
||||
if not result.success:
|
||||
raise HTTPException(status_code=500, detail="; ".join(result.errors))
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"key": result.key,
|
||||
"weather": {
|
||||
"success": result.weather.success if result.weather else False,
|
||||
"record": result.weather.record if result.weather else None,
|
||||
"error": result.weather.error if result.weather else None,
|
||||
},
|
||||
"air_quality": {
|
||||
"success": result.air_quality.success if result.air_quality else False,
|
||||
"record": result.air_quality.record if result.air_quality else None,
|
||||
"error": result.air_quality.error if result.air_quality else None,
|
||||
},
|
||||
"errors": result.errors,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/{namespace}/{key}", response_model=VolatileRecordResponse)
|
||||
async def get_record(
|
||||
namespace: str,
|
||||
|
||||
@@ -5,15 +5,19 @@ Orchestrates fetching data from external APIs and storing in volatile cache.
|
||||
Called by scheduler for prefetch or by HybridRAG for reactive caching.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Optional
|
||||
from dataclasses import dataclass
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from src.apis import (
|
||||
OpenMeteoProvider,
|
||||
AggregatedNewsProvider,
|
||||
AlphaVantageProvider,
|
||||
CurrentWeather,
|
||||
WeatherForecast,
|
||||
SunTimes,
|
||||
AirQuality,
|
||||
NewsFeed,
|
||||
StockQuote,
|
||||
)
|
||||
@@ -33,6 +37,16 @@ class FetchResult:
|
||||
error: Optional[str] = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class EnvironmentFetchResult:
|
||||
"""Result of combined environment fetch (weather + air quality)."""
|
||||
success: bool
|
||||
key: str
|
||||
weather: Optional[FetchResult] = None
|
||||
air_quality: Optional[FetchResult] = None
|
||||
errors: list[str] = field(default_factory=list)
|
||||
|
||||
|
||||
class VolatileFetchService:
|
||||
"""
|
||||
Service to fetch external data and store in volatile cache.
|
||||
@@ -64,25 +78,25 @@ class VolatileFetchService:
|
||||
self.news = news_provider
|
||||
self.financial = financial_provider
|
||||
|
||||
async def fetch_weather(
|
||||
async def fetch_current_weather(
|
||||
self,
|
||||
user: str,
|
||||
city: str,
|
||||
ttl: int = 86400, # 24 hours
|
||||
ttl: int = 3600, # 1 hour
|
||||
) -> FetchResult:
|
||||
"""
|
||||
Fetch current weather for a city and store in volatile cache.
|
||||
Fetch current weather conditions for a city and store in volatile cache.
|
||||
|
||||
Args:
|
||||
user: User identifier
|
||||
city: City name (will be geocoded)
|
||||
ttl: Time-to-live in seconds
|
||||
ttl: Time-to-live in seconds (default 1 hour)
|
||||
|
||||
Returns:
|
||||
FetchResult with success status and stored record
|
||||
"""
|
||||
try:
|
||||
# Geocode city and get weather
|
||||
# Geocode city and get current conditions
|
||||
location = await self.weather.geocode(city)
|
||||
if not location:
|
||||
return FetchResult(
|
||||
@@ -92,19 +106,23 @@ class VolatileFetchService:
|
||||
error=f"Could not geocode city: {city}"
|
||||
)
|
||||
|
||||
weather = await self.weather.get_current(location)
|
||||
current = await self.weather.get_current(location)
|
||||
|
||||
# Generate natural language summary
|
||||
text = current.to_text()
|
||||
|
||||
# Convert to storage format
|
||||
data = {
|
||||
"temperature": weather.temperature,
|
||||
"feels_like": weather.feels_like,
|
||||
"humidity": weather.humidity,
|
||||
"wind_speed": weather.wind_speed,
|
||||
"wind_direction": weather.wind_direction,
|
||||
"conditions": weather.condition_text,
|
||||
"condition_code": weather.condition.value,
|
||||
"location": weather.location,
|
||||
"text": weather.to_text(),
|
||||
"temperature": current.temperature,
|
||||
"feels_like": current.feels_like,
|
||||
"humidity": current.humidity,
|
||||
"wind_speed": current.wind_speed,
|
||||
"wind_direction": current.wind_direction,
|
||||
"conditions": current.condition_text,
|
||||
"condition_code": current.condition.value,
|
||||
"uv_index": current.uv_index,
|
||||
"location": current.location,
|
||||
"text": text,
|
||||
}
|
||||
|
||||
# Store in volatile cache
|
||||
@@ -117,7 +135,7 @@ class VolatileFetchService:
|
||||
ttl=ttl,
|
||||
)
|
||||
|
||||
logger.info(f"Stored weather for {city} (user={user})")
|
||||
logger.info(f"Stored current weather for {city} (user={user})")
|
||||
return FetchResult(
|
||||
success=True,
|
||||
namespace="weather",
|
||||
@@ -126,7 +144,7 @@ class VolatileFetchService:
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to fetch weather for {city}: {e}")
|
||||
logger.error(f"Failed to fetch current weather for {city}: {e}")
|
||||
return FetchResult(
|
||||
success=False,
|
||||
namespace="weather",
|
||||
@@ -134,6 +152,94 @@ class VolatileFetchService:
|
||||
error=str(e)
|
||||
)
|
||||
|
||||
async def fetch_forecast(
|
||||
self,
|
||||
user: str,
|
||||
city: str,
|
||||
days: int = 7,
|
||||
ttl: int = 43200, # 12 hours
|
||||
) -> FetchResult:
|
||||
"""
|
||||
Fetch weather forecast for a city and store in volatile cache.
|
||||
|
||||
Args:
|
||||
user: User identifier
|
||||
city: City name (will be geocoded)
|
||||
days: Number of forecast days (1-16)
|
||||
ttl: Time-to-live in seconds (default 12 hours)
|
||||
|
||||
Returns:
|
||||
FetchResult with success status and stored record
|
||||
"""
|
||||
try:
|
||||
# Geocode city and get forecast
|
||||
location = await self.weather.geocode(city)
|
||||
if not location:
|
||||
return FetchResult(
|
||||
success=False,
|
||||
namespace="forecast",
|
||||
key=city.lower(),
|
||||
error=f"Could not geocode city: {city}"
|
||||
)
|
||||
|
||||
forecast = await self.weather.get_forecast(location, days=days)
|
||||
|
||||
# Build daily forecast array
|
||||
daily_forecasts = []
|
||||
for day in forecast.daily:
|
||||
daily_forecasts.append({
|
||||
"date": day.date.isoformat(),
|
||||
"day_name": day.date.strftime("%A"),
|
||||
"temp_high": day.temp_high,
|
||||
"temp_low": day.temp_low,
|
||||
"conditions": day.condition_text,
|
||||
"condition_code": day.condition.value,
|
||||
"precipitation_chance": day.precipitation_chance,
|
||||
"precipitation_mm": day.precipitation_mm,
|
||||
"uv_index_max": day.uv_index_max,
|
||||
})
|
||||
|
||||
# Generate natural language summary
|
||||
forecast_lines = [f"{city} {days}-day forecast:"]
|
||||
for day in forecast.daily:
|
||||
forecast_lines.append(day.to_text())
|
||||
text = "\n".join(forecast_lines)
|
||||
|
||||
# Convert to storage format
|
||||
data = {
|
||||
"days": days,
|
||||
"daily": daily_forecasts,
|
||||
"location": forecast.current.location,
|
||||
"text": text,
|
||||
}
|
||||
|
||||
# Store in volatile cache
|
||||
record = await self.volatile.store(
|
||||
user=user,
|
||||
namespace=VolatileNamespace.FORECAST,
|
||||
key=city.lower(),
|
||||
data=data,
|
||||
source="openmeteo",
|
||||
ttl=ttl,
|
||||
)
|
||||
|
||||
logger.info(f"Stored {days}-day forecast for {city} (user={user})")
|
||||
return FetchResult(
|
||||
success=True,
|
||||
namespace="forecast",
|
||||
key=city.lower(),
|
||||
record=record
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to fetch forecast for {city}: {e}")
|
||||
return FetchResult(
|
||||
success=False,
|
||||
namespace="forecast",
|
||||
key=city.lower(),
|
||||
error=str(e)
|
||||
)
|
||||
|
||||
async def fetch_news(
|
||||
self,
|
||||
user: str,
|
||||
@@ -357,3 +463,274 @@ class VolatileFetchService:
|
||||
key=f"{symbol.lower()}_{market.lower()}",
|
||||
error=str(e)
|
||||
)
|
||||
|
||||
async def fetch_sun_times(
|
||||
self,
|
||||
user: str,
|
||||
city: str,
|
||||
ttl: int = 86400, # 24 hours
|
||||
) -> FetchResult:
|
||||
"""
|
||||
Fetch sunrise/sunset times for a city and store in volatile cache.
|
||||
|
||||
Args:
|
||||
user: User identifier
|
||||
city: City name (will be geocoded)
|
||||
ttl: Time-to-live in seconds
|
||||
|
||||
Returns:
|
||||
FetchResult with success status and stored record
|
||||
"""
|
||||
try:
|
||||
# Geocode city and get sun times
|
||||
location = await self.weather.geocode(city)
|
||||
if not location:
|
||||
return FetchResult(
|
||||
success=False,
|
||||
namespace="sun",
|
||||
key=city.lower(),
|
||||
error=f"Could not geocode city: {city}"
|
||||
)
|
||||
|
||||
sun_times = await self.weather.get_sun_times(location)
|
||||
|
||||
# Convert to storage format
|
||||
data = {
|
||||
"location": sun_times.location,
|
||||
"date": sun_times.date.isoformat(),
|
||||
"sunrise": sun_times.sunrise.strftime("%H:%M"),
|
||||
"sunset": sun_times.sunset.strftime("%H:%M"),
|
||||
"sunrise_iso": sun_times.sunrise.isoformat(),
|
||||
"sunset_iso": sun_times.sunset.isoformat(),
|
||||
"daylight_duration_seconds": sun_times.daylight_duration,
|
||||
"daylight_hours": sun_times.daylight_duration / 3600,
|
||||
"text": sun_times.to_text(),
|
||||
}
|
||||
|
||||
# Store in volatile cache
|
||||
record = await self.volatile.store(
|
||||
user=user,
|
||||
namespace=VolatileNamespace.SUN,
|
||||
key=city.lower(),
|
||||
data=data,
|
||||
source="openmeteo",
|
||||
ttl=ttl,
|
||||
)
|
||||
|
||||
logger.info(f"Stored sun times for {city} (user={user})")
|
||||
return FetchResult(
|
||||
success=True,
|
||||
namespace="sun",
|
||||
key=city.lower(),
|
||||
record=record
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to fetch sun times for {city}: {e}")
|
||||
return FetchResult(
|
||||
success=False,
|
||||
namespace="sun",
|
||||
key=city.lower(),
|
||||
error=str(e)
|
||||
)
|
||||
|
||||
async def fetch_air_quality(
|
||||
self,
|
||||
user: str,
|
||||
city: str,
|
||||
ttl: int = 3600, # 1 hour
|
||||
) -> FetchResult:
|
||||
"""
|
||||
Fetch air quality data for a city and store in volatile cache.
|
||||
|
||||
Args:
|
||||
user: User identifier
|
||||
city: City name (will be geocoded)
|
||||
ttl: Time-to-live in seconds
|
||||
|
||||
Returns:
|
||||
FetchResult with success status and stored record
|
||||
"""
|
||||
try:
|
||||
# Geocode city and get air quality
|
||||
location = await self.weather.geocode(city)
|
||||
if not location:
|
||||
return FetchResult(
|
||||
success=False,
|
||||
namespace="air_quality",
|
||||
key=city.lower(),
|
||||
error=f"Could not geocode city: {city}"
|
||||
)
|
||||
|
||||
air_quality = await self.weather.get_air_quality(location)
|
||||
|
||||
# Convert to storage format
|
||||
data = {
|
||||
"location": air_quality.location,
|
||||
"aqi_european": air_quality.aqi_european,
|
||||
"aqi_us": air_quality.aqi_us,
|
||||
"pm2_5": air_quality.pm2_5,
|
||||
"pm10": air_quality.pm10,
|
||||
"ozone": air_quality.ozone,
|
||||
"nitrogen_dioxide": air_quality.nitrogen_dioxide,
|
||||
"sulphur_dioxide": air_quality.sulphur_dioxide,
|
||||
"carbon_monoxide": air_quality.carbon_monoxide,
|
||||
"pollen_grass": air_quality.pollen_grass,
|
||||
"pollen_birch": air_quality.pollen_birch,
|
||||
"pollen_alder": air_quality.pollen_alder,
|
||||
"text": air_quality.to_text(),
|
||||
}
|
||||
|
||||
# Store in volatile cache
|
||||
record = await self.volatile.store(
|
||||
user=user,
|
||||
namespace=VolatileNamespace.AIR_QUALITY,
|
||||
key=city.lower(),
|
||||
data=data,
|
||||
source="openmeteo",
|
||||
ttl=ttl,
|
||||
)
|
||||
|
||||
logger.info(f"Stored air quality for {city} (user={user})")
|
||||
return FetchResult(
|
||||
success=True,
|
||||
namespace="air_quality",
|
||||
key=city.lower(),
|
||||
record=record
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to fetch air quality for {city}: {e}")
|
||||
return FetchResult(
|
||||
success=False,
|
||||
namespace="air_quality",
|
||||
key=city.lower(),
|
||||
error=str(e)
|
||||
)
|
||||
|
||||
async def fetch_environment(
|
||||
self,
|
||||
user: str,
|
||||
city: str,
|
||||
weather_ttl: int = 3600,
|
||||
air_quality_ttl: int = 3600,
|
||||
) -> EnvironmentFetchResult:
|
||||
"""
|
||||
Fetch weather and air quality concurrently for a city.
|
||||
|
||||
Performs a single geocode lookup and fetches both weather and air quality
|
||||
data in parallel, storing both in volatile cache.
|
||||
|
||||
Args:
|
||||
user: User identifier
|
||||
city: City name (will be geocoded once)
|
||||
weather_ttl: TTL for weather data (default 1 hour)
|
||||
air_quality_ttl: TTL for air quality data (default 1 hour)
|
||||
|
||||
Returns:
|
||||
EnvironmentFetchResult with both weather and air quality results
|
||||
"""
|
||||
errors: list[str] = []
|
||||
key = city.lower()
|
||||
|
||||
# Single geocode lookup (shared by both fetches)
|
||||
try:
|
||||
location = await self.weather.geocode(city)
|
||||
if not location:
|
||||
return EnvironmentFetchResult(
|
||||
success=False,
|
||||
key=key,
|
||||
errors=[f"Could not geocode city: {city}"]
|
||||
)
|
||||
except Exception as e:
|
||||
return EnvironmentFetchResult(
|
||||
success=False,
|
||||
key=key,
|
||||
errors=[f"Geocoding failed: {e}"]
|
||||
)
|
||||
|
||||
# Fetch weather and air quality concurrently
|
||||
async def fetch_weather_data() -> FetchResult:
|
||||
try:
|
||||
current = await self.weather.get_current(location)
|
||||
text = current.to_text()
|
||||
data = {
|
||||
"temperature": current.temperature,
|
||||
"feels_like": current.feels_like,
|
||||
"humidity": current.humidity,
|
||||
"wind_speed": current.wind_speed,
|
||||
"wind_direction": current.wind_direction,
|
||||
"conditions": current.condition_text,
|
||||
"condition_code": current.condition.value,
|
||||
"uv_index": current.uv_index,
|
||||
"location": current.location,
|
||||
"text": text,
|
||||
}
|
||||
record = await self.volatile.store(
|
||||
user=user,
|
||||
namespace=VolatileNamespace.WEATHER,
|
||||
key=key,
|
||||
data=data,
|
||||
source="openmeteo",
|
||||
ttl=weather_ttl,
|
||||
)
|
||||
return FetchResult(success=True, namespace="weather", key=key, record=record)
|
||||
except Exception as e:
|
||||
return FetchResult(success=False, namespace="weather", key=key, error=str(e))
|
||||
|
||||
async def fetch_air_quality_data() -> FetchResult:
|
||||
try:
|
||||
air_quality = await self.weather.get_air_quality(location)
|
||||
data = {
|
||||
"location": air_quality.location,
|
||||
"aqi_european": air_quality.aqi_european,
|
||||
"aqi_us": air_quality.aqi_us,
|
||||
"pm2_5": air_quality.pm2_5,
|
||||
"pm10": air_quality.pm10,
|
||||
"ozone": air_quality.ozone,
|
||||
"nitrogen_dioxide": air_quality.nitrogen_dioxide,
|
||||
"sulphur_dioxide": air_quality.sulphur_dioxide,
|
||||
"carbon_monoxide": air_quality.carbon_monoxide,
|
||||
"pollen_grass": air_quality.pollen_grass,
|
||||
"pollen_birch": air_quality.pollen_birch,
|
||||
"pollen_alder": air_quality.pollen_alder,
|
||||
"text": air_quality.to_text(),
|
||||
}
|
||||
record = await self.volatile.store(
|
||||
user=user,
|
||||
namespace=VolatileNamespace.AIR_QUALITY,
|
||||
key=key,
|
||||
data=data,
|
||||
source="openmeteo",
|
||||
ttl=air_quality_ttl,
|
||||
)
|
||||
return FetchResult(success=True, namespace="air_quality", key=key, record=record)
|
||||
except Exception as e:
|
||||
return FetchResult(success=False, namespace="air_quality", key=key, error=str(e))
|
||||
|
||||
# Run both fetches concurrently
|
||||
weather_result, air_quality_result = await asyncio.gather(
|
||||
fetch_weather_data(),
|
||||
fetch_air_quality_data(),
|
||||
)
|
||||
|
||||
# Collect any errors
|
||||
if not weather_result.success:
|
||||
errors.append(f"Weather: {weather_result.error}")
|
||||
if not air_quality_result.success:
|
||||
errors.append(f"Air quality: {air_quality_result.error}")
|
||||
|
||||
success = weather_result.success or air_quality_result.success
|
||||
logger.info(
|
||||
f"Environment fetch for {city} (user={user}): "
|
||||
f"weather={'ok' if weather_result.success else 'failed'}, "
|
||||
f"air_quality={'ok' if air_quality_result.success else 'failed'}"
|
||||
)
|
||||
|
||||
return EnvironmentFetchResult(
|
||||
success=success,
|
||||
key=key,
|
||||
weather=weather_result,
|
||||
air_quality=air_quality_result,
|
||||
errors=errors,
|
||||
)
|
||||
|
||||
@@ -116,24 +116,6 @@ class WikiChangeListener:
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to handle notification: {e}", exc_info=True)
|
||||
|
||||
def _is_automated_user(self, email: str) -> bool:
|
||||
"""
|
||||
Check if email belongs to an automated system user.
|
||||
|
||||
These are edits made by library-desk via Wiki.js API (entity linking).
|
||||
We skip processing these to prevent loops.
|
||||
|
||||
Customize this list based on your Wiki.js username for library-desk.
|
||||
"""
|
||||
automated_users = [
|
||||
self.settings.wikijs_username, # Library-desk's Wiki.js API user
|
||||
"library-desk@system",
|
||||
"automation@system",
|
||||
"bot@system"
|
||||
]
|
||||
|
||||
return email.lower() in [u.lower() for u in automated_users]
|
||||
|
||||
def _is_recently_processed(self, page_id: int) -> bool:
|
||||
"""Check if page was processed recently (debouncing)."""
|
||||
if page_id not in self._recent_notifications:
|
||||
|
||||
@@ -98,7 +98,7 @@ class TestVolatileNamespaces:
|
||||
|
||||
def test_weather_default_ttl(self):
|
||||
"""Test weather namespace default TTL."""
|
||||
assert NAMESPACE_DEFAULT_TTL[VolatileNamespace.WEATHER] == 1800 # 30 min
|
||||
assert NAMESPACE_DEFAULT_TTL[VolatileNamespace.WEATHER] == 3600 # 1 hour (current conditions)
|
||||
|
||||
def test_financial_default_ttl(self):
|
||||
"""Test financial namespace default TTL."""
|
||||
@@ -110,7 +110,7 @@ class TestVolatileNamespaces:
|
||||
|
||||
def test_namespace_count(self):
|
||||
"""Test we have the expected number of namespaces."""
|
||||
assert len(VolatileNamespace) == 11
|
||||
assert len(VolatileNamespace) == 13 # Including SUN, FORECAST
|
||||
|
||||
|
||||
class TestVolatileListResponse:
|
||||
@@ -274,7 +274,7 @@ class TestVolatileService:
|
||||
def test_get_default_ttl_known_namespace(self, volatile_service):
|
||||
"""Test default TTL for known namespace."""
|
||||
ttl = volatile_service._get_default_ttl("weather")
|
||||
assert ttl == 1800 # Weather namespace default
|
||||
assert ttl == 3600 # Weather namespace default (1 hour)
|
||||
|
||||
def test_get_default_ttl_unknown_namespace(self, volatile_service):
|
||||
"""Test default TTL for unknown namespace."""
|
||||
@@ -366,7 +366,7 @@ class TestVolatileService:
|
||||
ttl=None # Not specified
|
||||
)
|
||||
|
||||
assert result.ttl == 1800 # Weather default
|
||||
assert result.ttl == 3600 # Weather default (1 hour)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_empty_collection(self, volatile_service, mock_qdrant, mock_ollama):
|
||||
|
||||
@@ -50,22 +50,6 @@ class TestWikiChangeListener:
|
||||
assert listener._debounce_seconds == 5
|
||||
assert len(listener._recent_notifications) == 0
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_automated_user_filtering(self, listener):
|
||||
"""Test that automated users are correctly identified."""
|
||||
# Automated users should be filtered
|
||||
assert listener._is_automated_user("librarian@schweitz.net") is True
|
||||
assert listener._is_automated_user("library-desk@system") is True
|
||||
assert listener._is_automated_user("automation@system") is True
|
||||
assert listener._is_automated_user("bot@system") is True
|
||||
|
||||
# Case insensitive
|
||||
assert listener._is_automated_user("LIBRARIAN@SCHWEITZ.NET") is True
|
||||
|
||||
# Regular users should not be filtered
|
||||
assert listener._is_automated_user("user@example.com") is False
|
||||
assert listener._is_automated_user("john@example.com") is False
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_debouncing_prevents_duplicates(self, listener):
|
||||
"""Test that debouncing prevents duplicate processing."""
|
||||
@@ -163,19 +147,29 @@ class TestWikiChangeListener:
|
||||
assert mock_process.call_args[1]['event'] == 'page.delete'
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_automated_user_notification_filtered(self, listener):
|
||||
"""Test that notifications from automated users are filtered out."""
|
||||
async def test_any_user_notification_processed(self, listener):
|
||||
"""Test that notifications are processed regardless of user email.
|
||||
|
||||
Note: The user_email in PostgreSQL notifications is the page CREATOR,
|
||||
not the editor. We cannot filter by user email because:
|
||||
- A page created by 'librarian' but edited by a human should be processed
|
||||
- Filtering by creator would break legitimate page ingestion
|
||||
Loop prevention is handled by debouncing instead.
|
||||
"""
|
||||
mock_connection = AsyncMock()
|
||||
|
||||
with patch.object(listener, '_process_page_change', new_callable=AsyncMock) as mock_process:
|
||||
# Notification from automated user should be skipped
|
||||
# Even system user notifications should be processed
|
||||
# (debouncing handles loop prevention, not user filtering)
|
||||
await listener._handle_notification(
|
||||
mock_connection, 1234, 'wiki_page_changes',
|
||||
'UPDATE:123:librarian@schweitz.net'
|
||||
)
|
||||
|
||||
# Process should NOT be called
|
||||
mock_process.assert_not_called()
|
||||
# Process SHOULD be called (user filtering is not used)
|
||||
mock_process.assert_called_once()
|
||||
assert mock_process.call_args[1]['page_id'] == 123
|
||||
assert mock_process.call_args[1]['event'] == 'page.update'
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_duplicate_notification_filtered(self, listener):
|
||||
|
||||
Reference in New Issue
Block a user