Compare commits

...
6 Commits
Author SHA1 Message Date
jpmschweitzerandClaude Opus 4.5 152b2f28c4 release: v1.6.2 - Stats endpoint, weather/forecast separation
Build and Push / build (release) Successful in 30s
- GET /stats endpoint with Neo4j, Qdrant, Wiki.js, Paperless stats
- Split weather into current (1hr TTL) and forecast (12hr TTL)
- New FORECAST namespace for multi-day outlook

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-30 12:50:44 +01:00
jpmschweitzerandClaude Opus 4.5 46b9bcd7a0 feat: separate current weather from forecast into distinct namespaces
- Add FORECAST namespace for multi-day outlook (12hr TTL)
- WEATHER namespace now stores only current conditions (1hr TTL)
- Split fetch_weather into fetch_current_weather + fetch_forecast
- Add POST /volatile/fetch/forecast/{city} endpoint
- Different update frequencies for efficient caching

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-30 12:44:16 +01:00
jpmschweitzerandClaude Opus 4.5 68eb1add3d feat: add GET /stats endpoint with system statistics
Returns counts for:
- Neo4j: nodes by type (Document, Entity, Collection, Search)
- Qdrant: vectors per collection
- Wiki.js: total page count
- Paperless: documents, tags, correspondents, document types

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-30 12:07:02 +01:00
jpmschweitzerandClaude Opus 4.5 d6b30570a0 release: v1.6.1 - Weather forecasts, sun times, air quality
Build and Push / build (release) Successful in 52s
- Weather fetch now returns 7-day forecasts with UV index
- New /volatile/fetch/sun/{city} endpoint for sunrise/sunset
- New /volatile/fetch/air_quality/{city} endpoint for AQI and pollutants
- OpenMeteoProvider now implements AirQualityProvider interface

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-30 10:13:06 +01:00
jpmschweitzerandClaude Opus 4.5 6b0530ed79 fix: remove dead automated user filtering code
The _is_automated_user method was never called - loop prevention is
handled by debouncing instead. User email filtering was intentionally
removed because the notification email is the page CREATOR, not editor.

- Remove unused _is_automated_user method
- Update test to verify notifications are processed regardless of user
- Remove obsolete test_automated_user_filtering test

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-29 21:47:55 +01:00
jpmschweitzerandClaude Opus 4.5 0a8c2639a0 docs: update MEMORY_REMEMBER_PLAN with implementation status
Mark all phases as complete (v1.5.0-v1.6.0):
- Settings DB, Phase A, B, C all implemented
- Updated files summary with actual implementations
- Added remaining work section for file upload placeholder

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-29 21:33:10 +01:00
14 changed files with 860 additions and 121 deletions
+50
View File
@@ -5,6 +5,56 @@ All notable changes to Library Desk will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [1.6.2] - 2025-12-30
### Added
- **System Statistics Endpoint** - `GET /stats`
- Neo4j: node counts by type (Document, Entity, Collection, Search)
- Qdrant: collection counts, total vectors, per-collection breakdown
- Wiki.js: total page count
- Paperless: documents, tags, correspondents, document types
- **Weather/Forecast Separation** - Split weather into two distinct namespaces
- `POST /volatile/fetch/weather/{city}` - Current conditions only (1hr TTL)
- `POST /volatile/fetch/forecast/{city}` - 7-day outlook (12hr TTL)
- Different update frequencies for efficient caching
- `FORECAST` namespace added to volatile namespaces
### Changed
- Weather namespace TTL changed from 30 minutes to 1 hour (current conditions)
- Forecast data now stored separately with 12 hour TTL
## [1.6.1] - 2025-12-30
### Added
- **Weather Forecast Support** - Enhanced weather fetch with 7-day daily forecasts
- Current conditions now include UV index
- Daily forecasts with high/low temps, conditions, precipitation chance, UV max
- Natural language text summary with multi-day outlook
- **Sun Times Endpoint** - `POST /volatile/fetch/sun/{city}`
- Sunrise and sunset times (HH:MM and ISO formats)
- Daylight duration in seconds and hours
- Separate volatile namespace with 24hr TTL
- Useful for home automation light triggers
- **Air Quality Endpoint** - `POST /volatile/fetch/air_quality/{city}`
- European and US AQI indices
- Pollutants: PM2.5, PM10, ozone, nitrogen dioxide, sulphur dioxide, carbon monoxide
- Pollen data (grass, birch, alder) for European locations (seasonal)
- Hourly refresh (1hr TTL)
- **New Base Models**
- `SunTimes` dataclass for sunrise/sunset data
- `AirQuality` dataclass with AQI and pollutants
- `AirQualityProvider` abstract interface
- **New Volatile Namespace** - `SUN` for sunrise/sunset times (86400s default TTL)
### Changed
- Weather fetch now uses `get_forecast()` instead of `get_current()` for richer data
- `OpenMeteoProvider` now implements both `WeatherProvider` and `AirQualityProvider`
## [1.6.0] - 2025-12-29 ## [1.6.0] - 2025-12-29
### Added ### Added
-9
View File
@@ -41,12 +41,3 @@ Check for duplicate or highly similar documents using vector similarity and grap
3. Check graph relationships 3. Check graph relationships
4. Return candidates with similarity scores 4. Return candidates with similarity scores
## System Statistics
#### `GET /stats`
Get system statistics (wiki pages, neo4j nodes, qdrant vectors).
**Implementation needed:**
- Query Neo4j for node count
- Query Qdrant for vector count
- Query Wiki.js for page count
+53 -31
View File
@@ -16,8 +16,16 @@ This document outlines the implementation of "remember" triggers for the memory
| Memory Tier | Remember Trigger | Recall | Status | | Memory Tier | Remember Trigger | Recall | Status |
|-------------|------------------|--------|--------| |-------------|------------------|--------|--------|
| Wiki | Wiki.js webhook, Consolidation | HybridRAG vector+graph | ✅ Complete | | Wiki | Wiki.js webhook, Consolidation | HybridRAG vector+graph | ✅ Complete |
| Documents | Paperless webhook | Manual `/documents/search` | ⚠️ Partial (no HybridRAG recall) | | Documents | Paperless webhook | HybridRAG document search | ✅ Complete (v1.6.0) |
| Volatile | Manual `/volatile/store` only | HybridRAG volatile search | ⚠️ Partial (no auto-triggers) | | Volatile | Scheduler prefetch, HybridRAG post-processor | HybridRAG volatile search | ✅ Complete (v1.6.0) |
### Implementation Summary (v1.6.0)
- **Settings DB**: Central `system_settings` PostgreSQL database with `SettingsClient`
- **Phase A**: Volatile fetch endpoints (`/volatile/fetch/{namespace}/{key}`) with weather, news, financial providers
- **Phase B**: Unified memory routing in consolidation service (wiki/volatile/file/prefetch/skip classification)
- **Phase C**: Document recall in HybridRAG (4-source parallel retrieval)
- **Scheduler Integration**: `SchedulerClient` for external scheduler task registration
--- ---
@@ -662,46 +670,60 @@ document_threshold: float = Field(default=0.6)
## Implementation Order ## Implementation Order
| Phase | Priority | Effort | Description | | Phase | Priority | Effort | Description | Status |
|-------|----------|--------|-------------| |-------|----------|--------|-------------|--------|
| **Settings DB** | High | Low | PostgreSQL schema + settings client | | **Settings DB** | High | Low | PostgreSQL schema + settings client | ✅ v1.5.0 |
| **B.4** | High | Low | Scheduler client | | **B.4** | High | Low | Scheduler client | ✅ v1.6.0 |
| **B.1-B.3** | High | Medium | HybridRAG post-processor | | **B.1-B.3** | High | Medium | HybridRAG post-processor | ✅ v1.6.0 |
| **B.5** | High | Low | Config options | | **B.5** | High | Low | Config options | ✅ v1.6.0 |
| **C.1-C.2** | High | Low | Document recall in HybridRAG | | **C.1-C.2** | High | Low | Document recall in HybridRAG | ✅ v1.6.0 |
| **A.1** | Medium | Low | `/volatile/fetch` endpoint | | **A.1** | Medium | Low | `/volatile/fetch` endpoint | ✅ v1.6.0 |
| **A.2** | Medium | Medium | Weather + News API clients | | **A.2** | Medium | Medium | Weather + News API clients | ✅ v1.5.0 |
| **A.3** | Medium | Low | Fetch service | | **A.3** | Medium | Low | Fetch service | ✅ v1.6.0 |
### Remaining Work
| Item | Description | Status |
|------|-------------|--------|
| File upload | Download PDFs and upload to Paperless | ⚠️ Placeholder (logs only) |
| Prefetch patterns | More sophisticated pattern detection | Optional enhancement |
--- ---
## Files Summary ## Files Summary
### New Files ### New Files (Implemented)
| Path | Purpose | | Path | Purpose | Version |
|------|---------| |------|---------|---------|
| `src/clients/settings_client.py` | Read-only central settings access | | `src/clients/settings_client.py` | Central settings database access | v1.5.0 |
| `src/clients/scheduler_client.py` | Register/manage scheduler tasks | | `src/clients/scheduler_client.py` | External scheduler task management | v1.6.0 |
| `src/clients/weather_client.py` | Open-Meteo API (with geocoding) | | `src/apis/__init__.py` | External API providers package | v1.5.0 |
| `src/clients/news_client.py` | NOS.nl RSS feeds | | `src/apis/base.py` | Abstract base classes for providers | v1.5.0 |
| `src/services/volatile_fetch_service.py` | Orchestrates fetch + store | | `src/apis/weather.py` | OpenMeteoProvider (geocoding + forecast) | v1.5.0 |
| `src/apis/news.py` | AggregatedNewsProvider | v1.5.0 |
| `src/apis/nos.py` | NOSProvider (Dutch news RSS) | v1.5.0 |
| `src/apis/bbc.py` | BBCProvider (English news RSS) | v1.5.0 |
| `src/apis/financial.py` | AlphaVantageProvider (stocks/crypto) | v1.5.0 |
| `src/services/volatile_fetch_service.py` | Orchestrates fetch + store | v1.6.0 |
### Modified Files ### Modified Files
| Path | Changes | | Path | Changes | Version |
|------|---------| |------|---------|---------|
| `src/services/hybrid_rag_service.py` | Post-processor, prefetch detection, document search | | `src/services/hybrid_rag_service.py` | Document search (4-source parallel retrieval) | v1.6.0 |
| `src/models/hybrid_rag.py` | Memory config options | | `src/services/consolidation_service.py` | Unified memory routing, scheduler integration | v1.6.0 |
| `src/routers/volatile.py` | `/volatile/fetch/{namespace}/{key}` | | `src/models/hybrid_rag.py` | Document config options (`enable_documents`, `document_limit`) | v1.6.0 |
| `src/core/dependencies.py` | Settings client, scheduler client DI | | `src/models/consolidation.py` | Memory routing models | v1.6.0 |
| `src/config.py` | `SYSTEM_SETTINGS_*` connection vars | | `src/routers/volatile.py` | `/volatile/fetch/{namespace}/{key}` endpoints | v1.6.0 |
| `src/core/dependencies.py` | Settings, scheduler, provider DI | v1.5.0-v1.6.0 |
| `src/config.py` | `SYSTEM_SETTINGS_*`, `SCHEDULER_URL` vars | v1.5.0-v1.6.0 |
### Database ### Database
| Item | Details | | Item | Details |
|------|---------| |------|---------|
| Database | `system_settings` (PostgreSQL) | | Database | `system_settings` (PostgreSQL on postgres-shared) |
| Table | `settings (key, value JSONB, category, ...)` | | Table | `settings (key, user_scope, value JSONB, schema JSONB, ...)` |
| Library-desk user | `library_desk_ro` (read-only) | | Library-desk access | Read-only via `SettingsClient` |
| Management | Direct psql commands | | Management | Direct psql commands (future: CRUD manager UI) |
+1 -1
View File
@@ -1,6 +1,6 @@
[project] [project]
name = "library-desk" name = "library-desk"
version = "1.6.0" version = "1.6.2"
description = "Coordination service for The Library system - HybridRAG queries, document ingestion, entity extraction, and knowledge consolidation" description = "Coordination service for The Library system - HybridRAG queries, document ingestion, entity extraction, and knowledge consolidation"
readme = "README.md" readme = "README.md"
requires-python = ">=3.12" requires-python = ">=3.12"
+8
View File
@@ -27,6 +27,9 @@ from .base import (
DayForecast, DayForecast,
WeatherForecast, WeatherForecast,
GeoLocation, GeoLocation,
SunTimes,
# Air quality models
AirQuality,
# News models # News models
NewsItem, NewsItem,
NewsFeed, NewsFeed,
@@ -34,6 +37,7 @@ from .base import (
StockQuote, StockQuote,
# Abstract providers # Abstract providers
WeatherProvider, WeatherProvider,
AirQualityProvider,
NewsProvider, NewsProvider,
FinancialProvider, FinancialProvider,
) )
@@ -53,8 +57,12 @@ __all__ = [
"DayForecast", "DayForecast",
"WeatherForecast", "WeatherForecast",
"GeoLocation", "GeoLocation",
"SunTimes",
"WeatherProvider", "WeatherProvider",
"OpenMeteoProvider", "OpenMeteoProvider",
# Air quality
"AirQuality",
"AirQualityProvider",
# News # News
"NewsItem", "NewsItem",
"NewsFeed", "NewsFeed",
+102 -4
View File
@@ -44,14 +44,18 @@ class CurrentWeather:
condition_text: str # Human-readable description condition_text: str # Human-readable description
timestamp: datetime timestamp: datetime
location: str # City/location name location: str # City/location name
uv_index: Optional[float] = None # UV index 0-11+
def to_text(self) -> str: def to_text(self) -> str:
"""Generate natural language description.""" """Generate natural language description."""
return ( parts = [
f"Currently {self.temperature:.1f}°C " f"Currently {self.temperature:.1f}°C",
f"({self.condition_text}) in {self.location}. " f"({self.condition_text}) in {self.location}.",
f"Humidity {self.humidity}%, wind {self.wind_speed:.0f} km/h." f"Humidity {self.humidity}%, wind {self.wind_speed:.0f} km/h."
) ]
if self.uv_index is not None:
parts.append(f"UV index: {self.uv_index:.0f}.")
return " ".join(parts)
@dataclass @dataclass
@@ -64,6 +68,14 @@ class DayForecast:
condition_text: str condition_text: str
precipitation_chance: Optional[int] # Percentage 0-100 precipitation_chance: Optional[int] # Percentage 0-100
precipitation_mm: Optional[float] precipitation_mm: Optional[float]
uv_index_max: Optional[float] = None # Max UV index for the day
def to_text(self) -> str:
"""Generate natural language description."""
date_str = self.date.strftime("%A") # Day name
precip = f", {self.precipitation_chance}% rain" if self.precipitation_chance else ""
uv = f", UV {self.uv_index_max:.0f}" if self.uv_index_max else ""
return f"{date_str}: {self.temp_high:.0f}°/{self.temp_low:.0f}°C, {self.condition_text}{precip}{uv}"
@dataclass @dataclass
@@ -84,6 +96,78 @@ class GeoLocation:
admin_area: Optional[str] = None # State/province admin_area: Optional[str] = None # State/province
@dataclass
class SunTimes:
"""Sunrise/sunset times for a location."""
location: str
date: datetime
sunrise: datetime
sunset: datetime
daylight_duration: int # seconds
solar_noon: Optional[datetime] = None
def to_text(self) -> str:
"""Generate natural language description."""
sunrise_str = self.sunrise.strftime("%H:%M")
sunset_str = self.sunset.strftime("%H:%M")
hours = self.daylight_duration // 3600
minutes = (self.daylight_duration % 3600) // 60
return (
f"Sun times for {self.location} on {self.date.strftime('%A %d %B')}: "
f"Sunrise at {sunrise_str}, sunset at {sunset_str}. "
f"Daylight duration: {hours}h {minutes}m."
)
@dataclass
class AirQuality:
"""Air quality measurements for a location."""
location: str
timestamp: datetime
aqi_european: Optional[int] # European AQI 0-500+
aqi_us: Optional[int] # US AQI 0-500+
pm2_5: Optional[float] # µg/m³
pm10: Optional[float] # µg/m³
ozone: Optional[float] # µg/m³
nitrogen_dioxide: Optional[float] # µg/m³
sulphur_dioxide: Optional[float] # µg/m³
carbon_monoxide: Optional[float] # µg/m³
# Pollen (European data only, seasonal)
pollen_grass: Optional[float] = None
pollen_birch: Optional[float] = None
pollen_alder: Optional[float] = None
def to_text(self) -> str:
"""Generate natural language description."""
parts = [f"Air quality in {self.location}:"]
if self.aqi_european is not None:
level = self._aqi_level(self.aqi_european)
parts.append(f"European AQI {self.aqi_european} ({level}).")
if self.pm2_5 is not None:
parts.append(f"PM2.5: {self.pm2_5:.1f} µg/m³.")
if self.pm10 is not None:
parts.append(f"PM10: {self.pm10:.1f} µg/m³.")
if self.ozone is not None:
parts.append(f"Ozone: {self.ozone:.1f} µg/m³.")
return " ".join(parts)
@staticmethod
def _aqi_level(aqi: int) -> str:
"""Convert AQI to human-readable level."""
if aqi <= 20:
return "good"
elif aqi <= 40:
return "fair"
elif aqi <= 60:
return "moderate"
elif aqi <= 80:
return "poor"
elif aqi <= 100:
return "very poor"
else:
return "hazardous"
# ============================================================================= # =============================================================================
# News Models # News Models
# ============================================================================= # =============================================================================
@@ -163,6 +247,11 @@ class WeatherProvider(ABC):
"""Get weather forecast for a location.""" """Get weather forecast for a location."""
pass pass
@abstractmethod
async def get_sun_times(self, location: GeoLocation) -> SunTimes:
"""Get sunrise/sunset times for today."""
pass
async def get_weather_for_city(self, city: str) -> CurrentWeather: async def get_weather_for_city(self, city: str) -> CurrentWeather:
"""Convenience method: geocode and get current weather.""" """Convenience method: geocode and get current weather."""
location = await self.geocode(city) location = await self.geocode(city)
@@ -171,6 +260,15 @@ class WeatherProvider(ABC):
return await self.get_current(location) return await self.get_current(location)
class AirQualityProvider(ABC):
"""Abstract base class for air quality API providers."""
@abstractmethod
async def get_air_quality(self, location: GeoLocation) -> AirQuality:
"""Get current air quality for a location."""
pass
class NewsProvider(ABC): class NewsProvider(ABC):
"""Abstract base class for news API providers.""" """Abstract base class for news API providers."""
+138 -8
View File
@@ -14,11 +14,14 @@ from typing import Optional
from .base import ( from .base import (
WeatherProvider, WeatherProvider,
AirQualityProvider,
WeatherCondition, WeatherCondition,
CurrentWeather, CurrentWeather,
DayForecast, DayForecast,
WeatherForecast, WeatherForecast,
GeoLocation, GeoLocation,
SunTimes,
AirQuality,
) )
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -89,11 +92,12 @@ WMO_DESCRIPTIONS: dict[int, str] = {
} }
class OpenMeteoProvider(WeatherProvider): class OpenMeteoProvider(WeatherProvider, AirQualityProvider):
"""Open-Meteo weather API implementation.""" """Open-Meteo weather and air quality API implementation."""
GEOCODING_URL = "https://geocoding-api.open-meteo.com/v1/search" GEOCODING_URL = "https://geocoding-api.open-meteo.com/v1/search"
WEATHER_URL = "https://api.open-meteo.com/v1/forecast" WEATHER_URL = "https://api.open-meteo.com/v1/forecast"
AIR_QUALITY_URL = "https://air-quality-api.open-meteo.com/v1/air-quality"
def __init__( def __init__(
self, self,
@@ -194,9 +198,11 @@ class OpenMeteoProvider(WeatherProvider):
"wind_speed_10m", "wind_speed_10m",
"wind_direction_10m" "wind_direction_10m"
], ],
"daily": ["uv_index_max"],
"timezone": self.timezone, "timezone": self.timezone,
"temperature_unit": "celsius", "temperature_unit": "celsius",
"wind_speed_unit": "kmh" "wind_speed_unit": "kmh",
"forecast_days": 1
} }
) )
response.raise_for_status() response.raise_for_status()
@@ -205,6 +211,12 @@ class OpenMeteoProvider(WeatherProvider):
current = data.get("current", {}) current = data.get("current", {})
weather_code = current.get("weather_code", 0) weather_code = current.get("weather_code", 0)
# Get today's UV index from daily data
daily = data.get("daily", {})
uv_index = None
if daily.get("uv_index_max"):
uv_index = daily["uv_index_max"][0]
return CurrentWeather( return CurrentWeather(
temperature=current.get("temperature_2m", 0.0), temperature=current.get("temperature_2m", 0.0),
feels_like=current.get("apparent_temperature"), feels_like=current.get("apparent_temperature"),
@@ -214,7 +226,8 @@ class OpenMeteoProvider(WeatherProvider):
condition=WMO_CODE_MAP.get(weather_code, WeatherCondition.UNKNOWN), condition=WMO_CODE_MAP.get(weather_code, WeatherCondition.UNKNOWN),
condition_text=WMO_DESCRIPTIONS.get(weather_code, "Unknown"), condition_text=WMO_DESCRIPTIONS.get(weather_code, "Unknown"),
timestamp=datetime.now(), timestamp=datetime.now(),
location=location.name location=location.name,
uv_index=uv_index
) )
except httpx.HTTPError as e: except httpx.HTTPError as e:
logger.error(f"Weather request failed for {location.name}: {e}") logger.error(f"Weather request failed for {location.name}: {e}")
@@ -259,7 +272,8 @@ class OpenMeteoProvider(WeatherProvider):
"temperature_2m_max", "temperature_2m_max",
"temperature_2m_min", "temperature_2m_min",
"precipitation_sum", "precipitation_sum",
"precipitation_probability_max" "precipitation_probability_max",
"uv_index_max"
], ],
"timezone": self.timezone, "timezone": self.timezone,
"temperature_unit": "celsius", "temperature_unit": "celsius",
@@ -272,7 +286,14 @@ class OpenMeteoProvider(WeatherProvider):
# Parse current weather # Parse current weather
current_data = data.get("current", {}) current_data = data.get("current", {})
daily_data = data.get("daily", {})
weather_code = current_data.get("weather_code", 0) weather_code = current_data.get("weather_code", 0)
# Get today's UV from daily data
uv_index = None
if daily_data.get("uv_index_max"):
uv_index = daily_data["uv_index_max"][0]
current = CurrentWeather( current = CurrentWeather(
temperature=current_data.get("temperature_2m", 0.0), temperature=current_data.get("temperature_2m", 0.0),
feels_like=current_data.get("apparent_temperature"), feels_like=current_data.get("apparent_temperature"),
@@ -282,15 +303,16 @@ class OpenMeteoProvider(WeatherProvider):
condition=WMO_CODE_MAP.get(weather_code, WeatherCondition.UNKNOWN), condition=WMO_CODE_MAP.get(weather_code, WeatherCondition.UNKNOWN),
condition_text=WMO_DESCRIPTIONS.get(weather_code, "Unknown"), condition_text=WMO_DESCRIPTIONS.get(weather_code, "Unknown"),
timestamp=datetime.now(), timestamp=datetime.now(),
location=location.name location=location.name,
uv_index=uv_index
) )
# Parse daily forecast # Parse daily forecast
daily_data = data.get("daily", {})
daily = [] daily = []
dates = daily_data.get("time", []) dates = daily_data.get("time", [])
for i, date_str in enumerate(dates): for i, date_str in enumerate(dates):
code = daily_data.get("weather_code", [])[i] if i < len(daily_data.get("weather_code", [])) else 0 code = daily_data.get("weather_code", [])[i] if i < len(daily_data.get("weather_code", [])) else 0
uv_max = daily_data.get("uv_index_max", [])[i] if i < len(daily_data.get("uv_index_max", [])) else None
daily.append(DayForecast( daily.append(DayForecast(
date=datetime.fromisoformat(date_str), date=datetime.fromisoformat(date_str),
temp_high=daily_data.get("temperature_2m_max", [])[i] if i < len(daily_data.get("temperature_2m_max", [])) else 0.0, temp_high=daily_data.get("temperature_2m_max", [])[i] if i < len(daily_data.get("temperature_2m_max", [])) else 0.0,
@@ -298,7 +320,8 @@ class OpenMeteoProvider(WeatherProvider):
condition=WMO_CODE_MAP.get(code, WeatherCondition.UNKNOWN), condition=WMO_CODE_MAP.get(code, WeatherCondition.UNKNOWN),
condition_text=WMO_DESCRIPTIONS.get(code, "Unknown"), condition_text=WMO_DESCRIPTIONS.get(code, "Unknown"),
precipitation_chance=daily_data.get("precipitation_probability_max", [])[i] if i < len(daily_data.get("precipitation_probability_max", [])) else None, precipitation_chance=daily_data.get("precipitation_probability_max", [])[i] if i < len(daily_data.get("precipitation_probability_max", [])) else None,
precipitation_mm=daily_data.get("precipitation_sum", [])[i] if i < len(daily_data.get("precipitation_sum", [])) else None precipitation_mm=daily_data.get("precipitation_sum", [])[i] if i < len(daily_data.get("precipitation_sum", [])) else None,
uv_index_max=uv_max
)) ))
return WeatherForecast( return WeatherForecast(
@@ -309,3 +332,110 @@ class OpenMeteoProvider(WeatherProvider):
except httpx.HTTPError as e: except httpx.HTTPError as e:
logger.error(f"Forecast request failed for {location.name}: {e}") logger.error(f"Forecast request failed for {location.name}: {e}")
raise ValueError(f"Failed to get forecast: {e}") raise ValueError(f"Failed to get forecast: {e}")
async def get_sun_times(self, location: GeoLocation) -> SunTimes:
"""
Get sunrise/sunset times for today.
Args:
location: GeoLocation with lat/long
Returns:
SunTimes with sunrise, sunset, and daylight duration
Raises:
ValueError: If API request fails
"""
try:
response = await self.client.get(
self.WEATHER_URL,
params={
"latitude": location.latitude,
"longitude": location.longitude,
"daily": [
"sunrise",
"sunset",
"daylight_duration"
],
"timezone": self.timezone,
"forecast_days": 1
}
)
response.raise_for_status()
data = response.json()
daily = data.get("daily", {})
date_str = daily.get("time", [""])[0]
sunrise_str = daily.get("sunrise", [""])[0]
sunset_str = daily.get("sunset", [""])[0]
daylight = daily.get("daylight_duration", [0])[0]
return SunTimes(
location=location.name,
date=datetime.fromisoformat(date_str) if date_str else datetime.now(),
sunrise=datetime.fromisoformat(sunrise_str) if sunrise_str else datetime.now(),
sunset=datetime.fromisoformat(sunset_str) if sunset_str else datetime.now(),
daylight_duration=int(daylight) if daylight else 0
)
except httpx.HTTPError as e:
logger.error(f"Sun times request failed for {location.name}: {e}")
raise ValueError(f"Failed to get sun times: {e}")
async def get_air_quality(self, location: GeoLocation) -> AirQuality:
"""
Get current air quality for a location.
Args:
location: GeoLocation with lat/long
Returns:
AirQuality with pollutant measurements and AQI
Raises:
ValueError: If API request fails
"""
try:
response = await self.client.get(
self.AIR_QUALITY_URL,
params={
"latitude": location.latitude,
"longitude": location.longitude,
"current": [
"european_aqi",
"us_aqi",
"pm2_5",
"pm10",
"ozone",
"nitrogen_dioxide",
"sulphur_dioxide",
"carbon_monoxide",
"grass_pollen",
"birch_pollen",
"alder_pollen"
],
"timezone": self.timezone
}
)
response.raise_for_status()
data = response.json()
current = data.get("current", {})
return AirQuality(
location=location.name,
timestamp=datetime.now(),
aqi_european=current.get("european_aqi"),
aqi_us=current.get("us_aqi"),
pm2_5=current.get("pm2_5"),
pm10=current.get("pm10"),
ozone=current.get("ozone"),
nitrogen_dioxide=current.get("nitrogen_dioxide"),
sulphur_dioxide=current.get("sulphur_dioxide"),
carbon_monoxide=current.get("carbon_monoxide"),
pollen_grass=current.get("grass_pollen"),
pollen_birch=current.get("birch_pollen"),
pollen_alder=current.get("alder_pollen")
)
except httpx.HTTPError as e:
logger.error(f"Air quality request failed for {location.name}: {e}")
raise ValueError(f"Failed to get air quality: {e}")
+88 -1
View File
@@ -18,7 +18,7 @@ from pathlib import Path
from src.config import Settings, get_settings, __version__ from src.config import Settings, get_settings, __version__
from src.core.dependencies import ( from src.core.dependencies import (
verify_api_key, QdrantDep, WikiJSDep, OllamaDep, Neo4jDep verify_api_key, QdrantDep, WikiJSDep, OllamaDep, Neo4jDep, PaperlessDep
) )
from src.core.multi_tenancy import DEFAULT_USER from src.core.multi_tenancy import DEFAULT_USER
@@ -85,6 +85,14 @@ class HealthResponse(BaseModel):
services: Dict[str, Any] services: Dict[str, Any]
class StatsResponse(BaseModel):
"""System statistics response model."""
neo4j: Dict[str, int]
qdrant: Dict[str, Any]
wiki_pages: int
paperless: Dict[str, Any]
# Routes # Routes
@app.get("/", tags=["Root"]) @app.get("/", tags=["Root"])
async def root() -> Dict[str, str]: async def root() -> Dict[str, str]:
@@ -141,6 +149,85 @@ async def health(settings: Settings = Depends(get_settings)) -> HealthResponse:
) )
@app.get("/stats", response_model=StatsResponse, tags=["System"])
async def stats(
user: str = Query(default=DEFAULT_USER, description="User identifier"),
neo4j: Neo4jDep = None,
qdrant: QdrantDep = None,
wikijs: WikiJSDep = None,
paperless: PaperlessDep = None,
api_key: str = Depends(verify_api_key)
) -> StatsResponse:
"""
Get system statistics.
Returns counts for:
- Neo4j: nodes by type (Document, Entity, Collection, Search)
- Qdrant: vectors per collection
- Wiki.js: total page count
- Paperless: documents, tags, correspondents, document types
"""
# Neo4j node counts by label
neo4j_stats = {}
try:
for label in ["Document", "Entity", "Collection", "Search"]:
result = await neo4j.execute_query(
f"MATCH (n:{label}) RETURN count(n) as count"
)
neo4j_stats[label.lower() + "_nodes"] = result[0]["count"] if result else 0
except Exception as e:
logger.error(f"Failed to get Neo4j stats: {e}")
neo4j_stats = {"error": str(e)}
# Qdrant collection stats
qdrant_stats = {}
try:
collections = await qdrant.list_collections()
qdrant_stats["collections"] = len(collections)
qdrant_stats["total_vectors"] = sum(c.get("vectors_count", 0) for c in collections)
qdrant_stats["by_collection"] = {
c["name"]: c["vectors_count"] for c in collections
}
except Exception as e:
logger.error(f"Failed to get Qdrant stats: {e}")
qdrant_stats = {"error": str(e)}
# Wiki.js page count
wiki_pages = 0
try:
pages = await wikijs.list_all_pages(user)
wiki_pages = len(pages)
except Exception as e:
logger.warning(f"Failed to get Wiki.js stats: {e}")
# Paperless-ngx document stats
paperless_stats = {}
try:
# Get document count (page_size=1 for efficiency, we just need the count)
docs_result = await paperless.list_documents(page_size=1)
paperless_stats["documents"] = docs_result.get("count", 0)
# Get metadata counts
tags = await paperless.list_tags()
paperless_stats["tags"] = len(tags)
correspondents = await paperless.list_correspondents()
paperless_stats["correspondents"] = len(correspondents)
doc_types = await paperless.list_document_types()
paperless_stats["document_types"] = len(doc_types)
except Exception as e:
logger.warning(f"Failed to get Paperless stats: {e}")
paperless_stats = {"error": str(e)}
return StatsResponse(
neo4j=neo4j_stats,
qdrant=qdrant_stats,
wiki_pages=wiki_pages,
paperless=paperless_stats
)
@app.post("/ingest/check-updates", tags=["Ingestion"]) @app.post("/ingest/check-updates", tags=["Ingestion"])
async def check_updates( async def check_updates(
documents: Dict[str, Any], documents: Dict[str, Any],
+6 -2
View File
@@ -18,7 +18,9 @@ class VolatileNamespace(str, Enum):
Each namespace can have different default TTLs and refresh schedules. Each namespace can have different default TTLs and refresh schedules.
""" """
# Real-time external data # Real-time external data
WEATHER = "weather" # Current conditions, forecasts WEATHER = "weather" # Current conditions (temperature, humidity, wind)
FORECAST = "forecast" # Multi-day weather outlook
SUN = "sun" # Sunrise, sunset, daylight duration
NEWS = "news" # Headlines, breaking news NEWS = "news" # Headlines, breaking news
FINANCIAL = "financial" # Stock prices, exchange rates, crypto FINANCIAL = "financial" # Stock prices, exchange rates, crypto
TRANSIT = "transit" # Train/bus schedules, delays, disruptions TRANSIT = "transit" # Train/bus schedules, delays, disruptions
@@ -37,7 +39,9 @@ class VolatileNamespace(str, Enum):
# Default TTLs per namespace (in seconds) # Default TTLs per namespace (in seconds)
NAMESPACE_DEFAULT_TTL: Dict[str, int] = { NAMESPACE_DEFAULT_TTL: Dict[str, int] = {
VolatileNamespace.WEATHER: 1800, # 30 min - weather changes slowly VolatileNamespace.WEATHER: 3600, # 1 hour - current conditions
VolatileNamespace.FORECAST: 43200, # 12 hours - forecast stable longer
VolatileNamespace.SUN: 86400, # 24 hours - sun times change daily
VolatileNamespace.NEWS: 3600, # 1 hour - news cycles VolatileNamespace.NEWS: 3600, # 1 hour - news cycles
VolatileNamespace.FINANCIAL: 300, # 5 min - markets move fast VolatileNamespace.FINANCIAL: 300, # 5 min - markets move fast
VolatileNamespace.TRANSIT: 300, # 5 min - schedules update frequently VolatileNamespace.TRANSIT: 300, # 5 min - schedules update frequently
+139 -5
View File
@@ -233,16 +233,16 @@ async def store_volatile(
async def fetch_weather( async def fetch_weather(
city: str, city: str,
user: str = Query(default=DEFAULT_USER, description="User identifier"), user: str = Query(default=DEFAULT_USER, description="User identifier"),
ttl: int = Query(default=86400, ge=60, le=604800, description="TTL in seconds"), ttl: int = Query(default=3600, ge=60, le=86400, description="TTL in seconds (default 1 hour)"),
qdrant: QdrantDep = None, qdrant: QdrantDep = None,
ollama: OllamaDep = None, ollama: OllamaDep = None,
api_key: str = Depends(verify_api_key) api_key: str = Depends(verify_api_key)
): ):
""" """
Fetch current weather for a city and store in volatile cache. Fetch current weather conditions for a city and store in volatile cache.
Called by scheduler for prefetch or on-demand. Geocodes city name Stores temperature, humidity, wind, UV index. For forecasts use /fetch/forecast.
and fetches weather from Open-Meteo API. Called by scheduler for hourly prefetch or on-demand.
**Example:** **Example:**
``` ```
@@ -257,7 +257,49 @@ async def fetch_weather(
weather_provider=weather_provider, weather_provider=weather_provider,
) )
result = await fetch_service.fetch_weather(user, city, ttl=ttl) result = await fetch_service.fetch_current_weather(user, city, ttl=ttl)
if not result.success:
raise HTTPException(status_code=500, detail=result.error)
return {
"success": True,
"namespace": result.namespace,
"key": result.key,
"record": result.record,
}
@router.post("/fetch/forecast/{city}")
async def fetch_forecast(
city: str,
user: str = Query(default=DEFAULT_USER, description="User identifier"),
days: int = Query(default=7, ge=1, le=16, description="Forecast days (1-16)"),
ttl: int = Query(default=43200, ge=60, le=604800, description="TTL in seconds (default 12 hours)"),
qdrant: QdrantDep = None,
ollama: OllamaDep = None,
api_key: str = Depends(verify_api_key)
):
"""
Fetch weather forecast for a city and store in volatile cache.
Stores multi-day outlook with highs/lows, precipitation, UV.
For current conditions use /fetch/weather.
**Example:**
```
POST /volatile/fetch/forecast/amsterdam?user=jpmschweitzer&days=7
```
"""
volatile_service = get_volatile_service(qdrant, ollama)
weather_provider = get_weather_provider()
fetch_service = VolatileFetchService(
volatile_service=volatile_service,
weather_provider=weather_provider,
)
result = await fetch_service.fetch_forecast(user, city, days=days, ttl=ttl)
if not result.success: if not result.success:
raise HTTPException(status_code=500, detail=result.error) raise HTTPException(status_code=500, detail=result.error)
@@ -411,6 +453,98 @@ async def fetch_crypto(
} }
@router.post("/fetch/sun/{city}")
async def fetch_sun_times(
city: str,
user: str = Query(default=DEFAULT_USER, description="User identifier"),
ttl: int = Query(default=86400, ge=60, le=604800, description="TTL in seconds"),
qdrant: QdrantDep = None,
ollama: OllamaDep = None,
api_key: str = Depends(verify_api_key)
):
"""
Fetch sunrise/sunset times for a city and store in volatile cache.
Fetches from Open-Meteo API. Useful for home automation triggers.
**Example:**
```
POST /volatile/fetch/sun/rotterdam?user=jpmschweitzer
```
**Response data includes:**
- sunrise/sunset times (both HH:MM and ISO formats)
- daylight_duration_seconds
- daylight_hours
- Natural language text summary
"""
volatile_service = get_volatile_service(qdrant, ollama)
weather_provider = get_weather_provider()
fetch_service = VolatileFetchService(
volatile_service=volatile_service,
weather_provider=weather_provider,
)
result = await fetch_service.fetch_sun_times(user, city, ttl=ttl)
if not result.success:
raise HTTPException(status_code=500, detail=result.error)
return {
"success": True,
"namespace": result.namespace,
"key": result.key,
"record": result.record,
}
@router.post("/fetch/air_quality/{city}")
async def fetch_air_quality(
city: str,
user: str = Query(default=DEFAULT_USER, description="User identifier"),
ttl: int = Query(default=3600, ge=60, le=86400, description="TTL in seconds"),
qdrant: QdrantDep = None,
ollama: OllamaDep = None,
api_key: str = Depends(verify_api_key)
):
"""
Fetch air quality data for a city and store in volatile cache.
Fetches from Open-Meteo Air Quality API.
**Example:**
```
POST /volatile/fetch/air_quality/rotterdam?user=jpmschweitzer
```
**Response data includes:**
- European and US AQI indices
- Pollutants: PM2.5, PM10, ozone, nitrogen dioxide, etc.
- Pollen data (European locations, seasonal)
- Natural language text summary
"""
volatile_service = get_volatile_service(qdrant, ollama)
weather_provider = get_weather_provider()
fetch_service = VolatileFetchService(
volatile_service=volatile_service,
weather_provider=weather_provider,
)
result = await fetch_service.fetch_air_quality(user, city, ttl=ttl)
if not result.success:
raise HTTPException(status_code=500, detail=result.error)
return {
"success": True,
"namespace": result.namespace,
"key": result.key,
"record": result.record,
}
@router.get("/{namespace}/{key}", response_model=VolatileRecordResponse) @router.get("/{namespace}/{key}", response_model=VolatileRecordResponse)
async def get_record( async def get_record(
namespace: str, namespace: str,
+256 -17
View File
@@ -14,6 +14,9 @@ from src.apis import (
AggregatedNewsProvider, AggregatedNewsProvider,
AlphaVantageProvider, AlphaVantageProvider,
CurrentWeather, CurrentWeather,
WeatherForecast,
SunTimes,
AirQuality,
NewsFeed, NewsFeed,
StockQuote, StockQuote,
) )
@@ -64,25 +67,25 @@ class VolatileFetchService:
self.news = news_provider self.news = news_provider
self.financial = financial_provider self.financial = financial_provider
async def fetch_weather( async def fetch_current_weather(
self, self,
user: str, user: str,
city: str, city: str,
ttl: int = 86400, # 24 hours ttl: int = 3600, # 1 hour
) -> FetchResult: ) -> FetchResult:
""" """
Fetch current weather for a city and store in volatile cache. Fetch current weather conditions for a city and store in volatile cache.
Args: Args:
user: User identifier user: User identifier
city: City name (will be geocoded) city: City name (will be geocoded)
ttl: Time-to-live in seconds ttl: Time-to-live in seconds (default 1 hour)
Returns: Returns:
FetchResult with success status and stored record FetchResult with success status and stored record
""" """
try: try:
# Geocode city and get weather # Geocode city and get current conditions
location = await self.weather.geocode(city) location = await self.weather.geocode(city)
if not location: if not location:
return FetchResult( return FetchResult(
@@ -92,19 +95,23 @@ class VolatileFetchService:
error=f"Could not geocode city: {city}" error=f"Could not geocode city: {city}"
) )
weather = await self.weather.get_current(location) current = await self.weather.get_current(location)
# Generate natural language summary
text = current.to_text()
# Convert to storage format # Convert to storage format
data = { data = {
"temperature": weather.temperature, "temperature": current.temperature,
"feels_like": weather.feels_like, "feels_like": current.feels_like,
"humidity": weather.humidity, "humidity": current.humidity,
"wind_speed": weather.wind_speed, "wind_speed": current.wind_speed,
"wind_direction": weather.wind_direction, "wind_direction": current.wind_direction,
"conditions": weather.condition_text, "conditions": current.condition_text,
"condition_code": weather.condition.value, "condition_code": current.condition.value,
"location": weather.location, "uv_index": current.uv_index,
"text": weather.to_text(), "location": current.location,
"text": text,
} }
# Store in volatile cache # Store in volatile cache
@@ -117,7 +124,7 @@ class VolatileFetchService:
ttl=ttl, ttl=ttl,
) )
logger.info(f"Stored weather for {city} (user={user})") logger.info(f"Stored current weather for {city} (user={user})")
return FetchResult( return FetchResult(
success=True, success=True,
namespace="weather", namespace="weather",
@@ -126,7 +133,7 @@ class VolatileFetchService:
) )
except Exception as e: except Exception as e:
logger.error(f"Failed to fetch weather for {city}: {e}") logger.error(f"Failed to fetch current weather for {city}: {e}")
return FetchResult( return FetchResult(
success=False, success=False,
namespace="weather", namespace="weather",
@@ -134,6 +141,94 @@ class VolatileFetchService:
error=str(e) error=str(e)
) )
async def fetch_forecast(
self,
user: str,
city: str,
days: int = 7,
ttl: int = 43200, # 12 hours
) -> FetchResult:
"""
Fetch weather forecast for a city and store in volatile cache.
Args:
user: User identifier
city: City name (will be geocoded)
days: Number of forecast days (1-16)
ttl: Time-to-live in seconds (default 12 hours)
Returns:
FetchResult with success status and stored record
"""
try:
# Geocode city and get forecast
location = await self.weather.geocode(city)
if not location:
return FetchResult(
success=False,
namespace="forecast",
key=city.lower(),
error=f"Could not geocode city: {city}"
)
forecast = await self.weather.get_forecast(location, days=days)
# Build daily forecast array
daily_forecasts = []
for day in forecast.daily:
daily_forecasts.append({
"date": day.date.isoformat(),
"day_name": day.date.strftime("%A"),
"temp_high": day.temp_high,
"temp_low": day.temp_low,
"conditions": day.condition_text,
"condition_code": day.condition.value,
"precipitation_chance": day.precipitation_chance,
"precipitation_mm": day.precipitation_mm,
"uv_index_max": day.uv_index_max,
})
# Generate natural language summary
forecast_lines = [f"{city} {days}-day forecast:"]
for day in forecast.daily:
forecast_lines.append(day.to_text())
text = "\n".join(forecast_lines)
# Convert to storage format
data = {
"days": days,
"daily": daily_forecasts,
"location": forecast.current.location,
"text": text,
}
# Store in volatile cache
record = await self.volatile.store(
user=user,
namespace=VolatileNamespace.FORECAST,
key=city.lower(),
data=data,
source="openmeteo",
ttl=ttl,
)
logger.info(f"Stored {days}-day forecast for {city} (user={user})")
return FetchResult(
success=True,
namespace="forecast",
key=city.lower(),
record=record
)
except Exception as e:
logger.error(f"Failed to fetch forecast for {city}: {e}")
return FetchResult(
success=False,
namespace="forecast",
key=city.lower(),
error=str(e)
)
async def fetch_news( async def fetch_news(
self, self,
user: str, user: str,
@@ -357,3 +452,147 @@ class VolatileFetchService:
key=f"{symbol.lower()}_{market.lower()}", key=f"{symbol.lower()}_{market.lower()}",
error=str(e) error=str(e)
) )
async def fetch_sun_times(
self,
user: str,
city: str,
ttl: int = 86400, # 24 hours
) -> FetchResult:
"""
Fetch sunrise/sunset times for a city and store in volatile cache.
Args:
user: User identifier
city: City name (will be geocoded)
ttl: Time-to-live in seconds
Returns:
FetchResult with success status and stored record
"""
try:
# Geocode city and get sun times
location = await self.weather.geocode(city)
if not location:
return FetchResult(
success=False,
namespace="sun",
key=city.lower(),
error=f"Could not geocode city: {city}"
)
sun_times = await self.weather.get_sun_times(location)
# Convert to storage format
data = {
"location": sun_times.location,
"date": sun_times.date.isoformat(),
"sunrise": sun_times.sunrise.strftime("%H:%M"),
"sunset": sun_times.sunset.strftime("%H:%M"),
"sunrise_iso": sun_times.sunrise.isoformat(),
"sunset_iso": sun_times.sunset.isoformat(),
"daylight_duration_seconds": sun_times.daylight_duration,
"daylight_hours": sun_times.daylight_duration / 3600,
"text": sun_times.to_text(),
}
# Store in volatile cache
record = await self.volatile.store(
user=user,
namespace=VolatileNamespace.SUN,
key=city.lower(),
data=data,
source="openmeteo",
ttl=ttl,
)
logger.info(f"Stored sun times for {city} (user={user})")
return FetchResult(
success=True,
namespace="sun",
key=city.lower(),
record=record
)
except Exception as e:
logger.error(f"Failed to fetch sun times for {city}: {e}")
return FetchResult(
success=False,
namespace="sun",
key=city.lower(),
error=str(e)
)
async def fetch_air_quality(
self,
user: str,
city: str,
ttl: int = 3600, # 1 hour
) -> FetchResult:
"""
Fetch air quality data for a city and store in volatile cache.
Args:
user: User identifier
city: City name (will be geocoded)
ttl: Time-to-live in seconds
Returns:
FetchResult with success status and stored record
"""
try:
# Geocode city and get air quality
location = await self.weather.geocode(city)
if not location:
return FetchResult(
success=False,
namespace="air_quality",
key=city.lower(),
error=f"Could not geocode city: {city}"
)
air_quality = await self.weather.get_air_quality(location)
# Convert to storage format
data = {
"location": air_quality.location,
"aqi_european": air_quality.aqi_european,
"aqi_us": air_quality.aqi_us,
"pm2_5": air_quality.pm2_5,
"pm10": air_quality.pm10,
"ozone": air_quality.ozone,
"nitrogen_dioxide": air_quality.nitrogen_dioxide,
"sulphur_dioxide": air_quality.sulphur_dioxide,
"carbon_monoxide": air_quality.carbon_monoxide,
"pollen_grass": air_quality.pollen_grass,
"pollen_birch": air_quality.pollen_birch,
"pollen_alder": air_quality.pollen_alder,
"text": air_quality.to_text(),
}
# Store in volatile cache
record = await self.volatile.store(
user=user,
namespace=VolatileNamespace.AIR_QUALITY,
key=city.lower(),
data=data,
source="openmeteo",
ttl=ttl,
)
logger.info(f"Stored air quality for {city} (user={user})")
return FetchResult(
success=True,
namespace="air_quality",
key=city.lower(),
record=record
)
except Exception as e:
logger.error(f"Failed to fetch air quality for {city}: {e}")
return FetchResult(
success=False,
namespace="air_quality",
key=city.lower(),
error=str(e)
)
-18
View File
@@ -116,24 +116,6 @@ class WikiChangeListener:
except Exception as e: except Exception as e:
logger.error(f"Failed to handle notification: {e}", exc_info=True) logger.error(f"Failed to handle notification: {e}", exc_info=True)
def _is_automated_user(self, email: str) -> bool:
"""
Check if email belongs to an automated system user.
These are edits made by library-desk via Wiki.js API (entity linking).
We skip processing these to prevent loops.
Customize this list based on your Wiki.js username for library-desk.
"""
automated_users = [
self.settings.wikijs_username, # Library-desk's Wiki.js API user
"library-desk@system",
"automation@system",
"bot@system"
]
return email.lower() in [u.lower() for u in automated_users]
def _is_recently_processed(self, page_id: int) -> bool: def _is_recently_processed(self, page_id: int) -> bool:
"""Check if page was processed recently (debouncing).""" """Check if page was processed recently (debouncing)."""
if page_id not in self._recent_notifications: if page_id not in self._recent_notifications:
+4 -4
View File
@@ -98,7 +98,7 @@ class TestVolatileNamespaces:
def test_weather_default_ttl(self): def test_weather_default_ttl(self):
"""Test weather namespace default TTL.""" """Test weather namespace default TTL."""
assert NAMESPACE_DEFAULT_TTL[VolatileNamespace.WEATHER] == 1800 # 30 min assert NAMESPACE_DEFAULT_TTL[VolatileNamespace.WEATHER] == 3600 # 1 hour (current conditions)
def test_financial_default_ttl(self): def test_financial_default_ttl(self):
"""Test financial namespace default TTL.""" """Test financial namespace default TTL."""
@@ -110,7 +110,7 @@ class TestVolatileNamespaces:
def test_namespace_count(self): def test_namespace_count(self):
"""Test we have the expected number of namespaces.""" """Test we have the expected number of namespaces."""
assert len(VolatileNamespace) == 11 assert len(VolatileNamespace) == 13 # Including SUN, FORECAST
class TestVolatileListResponse: class TestVolatileListResponse:
@@ -274,7 +274,7 @@ class TestVolatileService:
def test_get_default_ttl_known_namespace(self, volatile_service): def test_get_default_ttl_known_namespace(self, volatile_service):
"""Test default TTL for known namespace.""" """Test default TTL for known namespace."""
ttl = volatile_service._get_default_ttl("weather") ttl = volatile_service._get_default_ttl("weather")
assert ttl == 1800 # Weather namespace default assert ttl == 3600 # Weather namespace default (1 hour)
def test_get_default_ttl_unknown_namespace(self, volatile_service): def test_get_default_ttl_unknown_namespace(self, volatile_service):
"""Test default TTL for unknown namespace.""" """Test default TTL for unknown namespace."""
@@ -366,7 +366,7 @@ class TestVolatileService:
ttl=None # Not specified ttl=None # Not specified
) )
assert result.ttl == 1800 # Weather default assert result.ttl == 3600 # Weather default (1 hour)
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_search_empty_collection(self, volatile_service, mock_qdrant, mock_ollama): async def test_search_empty_collection(self, volatile_service, mock_qdrant, mock_ollama):
+15 -21
View File
@@ -50,22 +50,6 @@ class TestWikiChangeListener:
assert listener._debounce_seconds == 5 assert listener._debounce_seconds == 5
assert len(listener._recent_notifications) == 0 assert len(listener._recent_notifications) == 0
@pytest.mark.asyncio
async def test_automated_user_filtering(self, listener):
"""Test that automated users are correctly identified."""
# Automated users should be filtered
assert listener._is_automated_user("librarian@schweitz.net") is True
assert listener._is_automated_user("library-desk@system") is True
assert listener._is_automated_user("automation@system") is True
assert listener._is_automated_user("bot@system") is True
# Case insensitive
assert listener._is_automated_user("LIBRARIAN@SCHWEITZ.NET") is True
# Regular users should not be filtered
assert listener._is_automated_user("user@example.com") is False
assert listener._is_automated_user("john@example.com") is False
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_debouncing_prevents_duplicates(self, listener): async def test_debouncing_prevents_duplicates(self, listener):
"""Test that debouncing prevents duplicate processing.""" """Test that debouncing prevents duplicate processing."""
@@ -163,19 +147,29 @@ class TestWikiChangeListener:
assert mock_process.call_args[1]['event'] == 'page.delete' assert mock_process.call_args[1]['event'] == 'page.delete'
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_automated_user_notification_filtered(self, listener): async def test_any_user_notification_processed(self, listener):
"""Test that notifications from automated users are filtered out.""" """Test that notifications are processed regardless of user email.
Note: The user_email in PostgreSQL notifications is the page CREATOR,
not the editor. We cannot filter by user email because:
- A page created by 'librarian' but edited by a human should be processed
- Filtering by creator would break legitimate page ingestion
Loop prevention is handled by debouncing instead.
"""
mock_connection = AsyncMock() mock_connection = AsyncMock()
with patch.object(listener, '_process_page_change', new_callable=AsyncMock) as mock_process: with patch.object(listener, '_process_page_change', new_callable=AsyncMock) as mock_process:
# Notification from automated user should be skipped # Even system user notifications should be processed
# (debouncing handles loop prevention, not user filtering)
await listener._handle_notification( await listener._handle_notification(
mock_connection, 1234, 'wiki_page_changes', mock_connection, 1234, 'wiki_page_changes',
'UPDATE:123:librarian@schweitz.net' 'UPDATE:123:librarian@schweitz.net'
) )
# Process should NOT be called # Process SHOULD be called (user filtering is not used)
mock_process.assert_not_called() mock_process.assert_called_once()
assert mock_process.call_args[1]['page_id'] == 123
assert mock_process.call_args[1]['event'] == 'page.update'
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_duplicate_notification_filtered(self, listener): async def test_duplicate_notification_filtered(self, listener):