Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
  jpmschweitzerandClaude Opus 4.5
|
d6b30570a0 | ||
|
jpmschweitzerandClaude Opus 4.5
|
6b0530ed79 | ||
|
jpmschweitzerandClaude Opus 4.5
|
0a8c2639a0 |
@@ -5,6 +5,35 @@ All notable changes to Library Desk will be documented in this file.
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [1.6.1] - 2025-12-30
|
||||
|
||||
### Added
|
||||
|
||||
- **Weather Forecast Support** - Enhanced weather fetch with 7-day daily forecasts
|
||||
- Current conditions now include UV index
|
||||
- Daily forecasts with high/low temps, conditions, precipitation chance, UV max
|
||||
- Natural language text summary with multi-day outlook
|
||||
- **Sun Times Endpoint** - `POST /volatile/fetch/sun/{city}`
|
||||
- Sunrise and sunset times (HH:MM and ISO formats)
|
||||
- Daylight duration in seconds and hours
|
||||
- Separate volatile namespace with 24hr TTL
|
||||
- Useful for home automation light triggers
|
||||
- **Air Quality Endpoint** - `POST /volatile/fetch/air_quality/{city}`
|
||||
- European and US AQI indices
|
||||
- Pollutants: PM2.5, PM10, ozone, nitrogen dioxide, sulphur dioxide, carbon monoxide
|
||||
- Pollen data (grass, birch, alder) for European locations (seasonal)
|
||||
- Hourly refresh (1hr TTL)
|
||||
- **New Base Models**
|
||||
- `SunTimes` dataclass for sunrise/sunset data
|
||||
- `AirQuality` dataclass with AQI and pollutants
|
||||
- `AirQualityProvider` abstract interface
|
||||
- **New Volatile Namespace** - `SUN` for sunrise/sunset times (86400s default TTL)
|
||||
|
||||
### Changed
|
||||
|
||||
- Weather fetch now uses `get_forecast()` instead of `get_current()` for richer data
|
||||
- `OpenMeteoProvider` now implements both `WeatherProvider` and `AirQualityProvider`
|
||||
|
||||
## [1.6.0] - 2025-12-29
|
||||
|
||||
### Added
|
||||
|
||||
@@ -16,8 +16,16 @@ This document outlines the implementation of "remember" triggers for the memory
|
||||
| Memory Tier | Remember Trigger | Recall | Status |
|
||||
|-------------|------------------|--------|--------|
|
||||
| Wiki | Wiki.js webhook, Consolidation | HybridRAG vector+graph | ✅ Complete |
|
||||
| Documents | Paperless webhook | Manual `/documents/search` | ⚠️ Partial (no HybridRAG recall) |
|
||||
| Volatile | Manual `/volatile/store` only | HybridRAG volatile search | ⚠️ Partial (no auto-triggers) |
|
||||
| Documents | Paperless webhook | HybridRAG document search | ✅ Complete (v1.6.0) |
|
||||
| Volatile | Scheduler prefetch, HybridRAG post-processor | HybridRAG volatile search | ✅ Complete (v1.6.0) |
|
||||
|
||||
### Implementation Summary (v1.6.0)
|
||||
|
||||
- **Settings DB**: Central `system_settings` PostgreSQL database with `SettingsClient`
|
||||
- **Phase A**: Volatile fetch endpoints (`/volatile/fetch/{namespace}/{key}`) with weather, news, financial providers
|
||||
- **Phase B**: Unified memory routing in consolidation service (wiki/volatile/file/prefetch/skip classification)
|
||||
- **Phase C**: Document recall in HybridRAG (4-source parallel retrieval)
|
||||
- **Scheduler Integration**: `SchedulerClient` for external scheduler task registration
|
||||
|
||||
---
|
||||
|
||||
@@ -662,46 +670,60 @@ document_threshold: float = Field(default=0.6)
|
||||
|
||||
## Implementation Order
|
||||
|
||||
| Phase | Priority | Effort | Description |
|
||||
|-------|----------|--------|-------------|
|
||||
| **Settings DB** | High | Low | PostgreSQL schema + settings client |
|
||||
| **B.4** | High | Low | Scheduler client |
|
||||
| **B.1-B.3** | High | Medium | HybridRAG post-processor |
|
||||
| **B.5** | High | Low | Config options |
|
||||
| **C.1-C.2** | High | Low | Document recall in HybridRAG |
|
||||
| **A.1** | Medium | Low | `/volatile/fetch` endpoint |
|
||||
| **A.2** | Medium | Medium | Weather + News API clients |
|
||||
| **A.3** | Medium | Low | Fetch service |
|
||||
| Phase | Priority | Effort | Description | Status |
|
||||
|-------|----------|--------|-------------|--------|
|
||||
| **Settings DB** | High | Low | PostgreSQL schema + settings client | ✅ v1.5.0 |
|
||||
| **B.4** | High | Low | Scheduler client | ✅ v1.6.0 |
|
||||
| **B.1-B.3** | High | Medium | HybridRAG post-processor | ✅ v1.6.0 |
|
||||
| **B.5** | High | Low | Config options | ✅ v1.6.0 |
|
||||
| **C.1-C.2** | High | Low | Document recall in HybridRAG | ✅ v1.6.0 |
|
||||
| **A.1** | Medium | Low | `/volatile/fetch` endpoint | ✅ v1.6.0 |
|
||||
| **A.2** | Medium | Medium | Weather + News API clients | ✅ v1.5.0 |
|
||||
| **A.3** | Medium | Low | Fetch service | ✅ v1.6.0 |
|
||||
|
||||
### Remaining Work
|
||||
|
||||
| Item | Description | Status |
|
||||
|------|-------------|--------|
|
||||
| File upload | Download PDFs and upload to Paperless | ⚠️ Placeholder (logs only) |
|
||||
| Prefetch patterns | More sophisticated pattern detection | Optional enhancement |
|
||||
|
||||
---
|
||||
|
||||
## Files Summary
|
||||
|
||||
### New Files
|
||||
### New Files (Implemented)
|
||||
|
||||
| Path | Purpose |
|
||||
|------|---------|
|
||||
| `src/clients/settings_client.py` | Read-only central settings access |
|
||||
| `src/clients/scheduler_client.py` | Register/manage scheduler tasks |
|
||||
| `src/clients/weather_client.py` | Open-Meteo API (with geocoding) |
|
||||
| `src/clients/news_client.py` | NOS.nl RSS feeds |
|
||||
| `src/services/volatile_fetch_service.py` | Orchestrates fetch + store |
|
||||
| Path | Purpose | Version |
|
||||
|------|---------|---------|
|
||||
| `src/clients/settings_client.py` | Central settings database access | v1.5.0 |
|
||||
| `src/clients/scheduler_client.py` | External scheduler task management | v1.6.0 |
|
||||
| `src/apis/__init__.py` | External API providers package | v1.5.0 |
|
||||
| `src/apis/base.py` | Abstract base classes for providers | v1.5.0 |
|
||||
| `src/apis/weather.py` | OpenMeteoProvider (geocoding + forecast) | v1.5.0 |
|
||||
| `src/apis/news.py` | AggregatedNewsProvider | v1.5.0 |
|
||||
| `src/apis/nos.py` | NOSProvider (Dutch news RSS) | v1.5.0 |
|
||||
| `src/apis/bbc.py` | BBCProvider (English news RSS) | v1.5.0 |
|
||||
| `src/apis/financial.py` | AlphaVantageProvider (stocks/crypto) | v1.5.0 |
|
||||
| `src/services/volatile_fetch_service.py` | Orchestrates fetch + store | v1.6.0 |
|
||||
|
||||
### Modified Files
|
||||
|
||||
| Path | Changes |
|
||||
|------|---------|
|
||||
| `src/services/hybrid_rag_service.py` | Post-processor, prefetch detection, document search |
|
||||
| `src/models/hybrid_rag.py` | Memory config options |
|
||||
| `src/routers/volatile.py` | `/volatile/fetch/{namespace}/{key}` |
|
||||
| `src/core/dependencies.py` | Settings client, scheduler client DI |
|
||||
| `src/config.py` | `SYSTEM_SETTINGS_*` connection vars |
|
||||
| Path | Changes | Version |
|
||||
|------|---------|---------|
|
||||
| `src/services/hybrid_rag_service.py` | Document search (4-source parallel retrieval) | v1.6.0 |
|
||||
| `src/services/consolidation_service.py` | Unified memory routing, scheduler integration | v1.6.0 |
|
||||
| `src/models/hybrid_rag.py` | Document config options (`enable_documents`, `document_limit`) | v1.6.0 |
|
||||
| `src/models/consolidation.py` | Memory routing models | v1.6.0 |
|
||||
| `src/routers/volatile.py` | `/volatile/fetch/{namespace}/{key}` endpoints | v1.6.0 |
|
||||
| `src/core/dependencies.py` | Settings, scheduler, provider DI | v1.5.0-v1.6.0 |
|
||||
| `src/config.py` | `SYSTEM_SETTINGS_*`, `SCHEDULER_URL` vars | v1.5.0-v1.6.0 |
|
||||
|
||||
### Database
|
||||
|
||||
| Item | Details |
|
||||
|------|---------|
|
||||
| Database | `system_settings` (PostgreSQL) |
|
||||
| Table | `settings (key, value JSONB, category, ...)` |
|
||||
| Library-desk user | `library_desk_ro` (read-only) |
|
||||
| Management | Direct psql commands |
|
||||
| Database | `system_settings` (PostgreSQL on postgres-shared) |
|
||||
| Table | `settings (key, user_scope, value JSONB, schema JSONB, ...)` |
|
||||
| Library-desk access | Read-only via `SettingsClient` |
|
||||
| Management | Direct psql commands (future: CRUD manager UI) |
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "library-desk"
|
||||
version = "1.6.0"
|
||||
version = "1.6.1"
|
||||
description = "Coordination service for The Library system - HybridRAG queries, document ingestion, entity extraction, and knowledge consolidation"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12"
|
||||
|
||||
@@ -27,6 +27,9 @@ from .base import (
|
||||
DayForecast,
|
||||
WeatherForecast,
|
||||
GeoLocation,
|
||||
SunTimes,
|
||||
# Air quality models
|
||||
AirQuality,
|
||||
# News models
|
||||
NewsItem,
|
||||
NewsFeed,
|
||||
@@ -34,6 +37,7 @@ from .base import (
|
||||
StockQuote,
|
||||
# Abstract providers
|
||||
WeatherProvider,
|
||||
AirQualityProvider,
|
||||
NewsProvider,
|
||||
FinancialProvider,
|
||||
)
|
||||
@@ -53,8 +57,12 @@ __all__ = [
|
||||
"DayForecast",
|
||||
"WeatherForecast",
|
||||
"GeoLocation",
|
||||
"SunTimes",
|
||||
"WeatherProvider",
|
||||
"OpenMeteoProvider",
|
||||
# Air quality
|
||||
"AirQuality",
|
||||
"AirQualityProvider",
|
||||
# News
|
||||
"NewsItem",
|
||||
"NewsFeed",
|
||||
|
||||
+102
-4
@@ -44,14 +44,18 @@ class CurrentWeather:
|
||||
condition_text: str # Human-readable description
|
||||
timestamp: datetime
|
||||
location: str # City/location name
|
||||
uv_index: Optional[float] = None # UV index 0-11+
|
||||
|
||||
def to_text(self) -> str:
|
||||
"""Generate natural language description."""
|
||||
return (
|
||||
f"Currently {self.temperature:.1f}°C "
|
||||
f"({self.condition_text}) in {self.location}. "
|
||||
parts = [
|
||||
f"Currently {self.temperature:.1f}°C",
|
||||
f"({self.condition_text}) in {self.location}.",
|
||||
f"Humidity {self.humidity}%, wind {self.wind_speed:.0f} km/h."
|
||||
)
|
||||
]
|
||||
if self.uv_index is not None:
|
||||
parts.append(f"UV index: {self.uv_index:.0f}.")
|
||||
return " ".join(parts)
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -64,6 +68,14 @@ class DayForecast:
|
||||
condition_text: str
|
||||
precipitation_chance: Optional[int] # Percentage 0-100
|
||||
precipitation_mm: Optional[float]
|
||||
uv_index_max: Optional[float] = None # Max UV index for the day
|
||||
|
||||
def to_text(self) -> str:
|
||||
"""Generate natural language description."""
|
||||
date_str = self.date.strftime("%A") # Day name
|
||||
precip = f", {self.precipitation_chance}% rain" if self.precipitation_chance else ""
|
||||
uv = f", UV {self.uv_index_max:.0f}" if self.uv_index_max else ""
|
||||
return f"{date_str}: {self.temp_high:.0f}°/{self.temp_low:.0f}°C, {self.condition_text}{precip}{uv}"
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -84,6 +96,78 @@ class GeoLocation:
|
||||
admin_area: Optional[str] = None # State/province
|
||||
|
||||
|
||||
@dataclass
|
||||
class SunTimes:
|
||||
"""Sunrise/sunset times for a location."""
|
||||
location: str
|
||||
date: datetime
|
||||
sunrise: datetime
|
||||
sunset: datetime
|
||||
daylight_duration: int # seconds
|
||||
solar_noon: Optional[datetime] = None
|
||||
|
||||
def to_text(self) -> str:
|
||||
"""Generate natural language description."""
|
||||
sunrise_str = self.sunrise.strftime("%H:%M")
|
||||
sunset_str = self.sunset.strftime("%H:%M")
|
||||
hours = self.daylight_duration // 3600
|
||||
minutes = (self.daylight_duration % 3600) // 60
|
||||
return (
|
||||
f"Sun times for {self.location} on {self.date.strftime('%A %d %B')}: "
|
||||
f"Sunrise at {sunrise_str}, sunset at {sunset_str}. "
|
||||
f"Daylight duration: {hours}h {minutes}m."
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class AirQuality:
|
||||
"""Air quality measurements for a location."""
|
||||
location: str
|
||||
timestamp: datetime
|
||||
aqi_european: Optional[int] # European AQI 0-500+
|
||||
aqi_us: Optional[int] # US AQI 0-500+
|
||||
pm2_5: Optional[float] # µg/m³
|
||||
pm10: Optional[float] # µg/m³
|
||||
ozone: Optional[float] # µg/m³
|
||||
nitrogen_dioxide: Optional[float] # µg/m³
|
||||
sulphur_dioxide: Optional[float] # µg/m³
|
||||
carbon_monoxide: Optional[float] # µg/m³
|
||||
# Pollen (European data only, seasonal)
|
||||
pollen_grass: Optional[float] = None
|
||||
pollen_birch: Optional[float] = None
|
||||
pollen_alder: Optional[float] = None
|
||||
|
||||
def to_text(self) -> str:
|
||||
"""Generate natural language description."""
|
||||
parts = [f"Air quality in {self.location}:"]
|
||||
if self.aqi_european is not None:
|
||||
level = self._aqi_level(self.aqi_european)
|
||||
parts.append(f"European AQI {self.aqi_european} ({level}).")
|
||||
if self.pm2_5 is not None:
|
||||
parts.append(f"PM2.5: {self.pm2_5:.1f} µg/m³.")
|
||||
if self.pm10 is not None:
|
||||
parts.append(f"PM10: {self.pm10:.1f} µg/m³.")
|
||||
if self.ozone is not None:
|
||||
parts.append(f"Ozone: {self.ozone:.1f} µg/m³.")
|
||||
return " ".join(parts)
|
||||
|
||||
@staticmethod
|
||||
def _aqi_level(aqi: int) -> str:
|
||||
"""Convert AQI to human-readable level."""
|
||||
if aqi <= 20:
|
||||
return "good"
|
||||
elif aqi <= 40:
|
||||
return "fair"
|
||||
elif aqi <= 60:
|
||||
return "moderate"
|
||||
elif aqi <= 80:
|
||||
return "poor"
|
||||
elif aqi <= 100:
|
||||
return "very poor"
|
||||
else:
|
||||
return "hazardous"
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# News Models
|
||||
# =============================================================================
|
||||
@@ -163,6 +247,11 @@ class WeatherProvider(ABC):
|
||||
"""Get weather forecast for a location."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
async def get_sun_times(self, location: GeoLocation) -> SunTimes:
|
||||
"""Get sunrise/sunset times for today."""
|
||||
pass
|
||||
|
||||
async def get_weather_for_city(self, city: str) -> CurrentWeather:
|
||||
"""Convenience method: geocode and get current weather."""
|
||||
location = await self.geocode(city)
|
||||
@@ -171,6 +260,15 @@ class WeatherProvider(ABC):
|
||||
return await self.get_current(location)
|
||||
|
||||
|
||||
class AirQualityProvider(ABC):
|
||||
"""Abstract base class for air quality API providers."""
|
||||
|
||||
@abstractmethod
|
||||
async def get_air_quality(self, location: GeoLocation) -> AirQuality:
|
||||
"""Get current air quality for a location."""
|
||||
pass
|
||||
|
||||
|
||||
class NewsProvider(ABC):
|
||||
"""Abstract base class for news API providers."""
|
||||
|
||||
|
||||
+138
-8
@@ -14,11 +14,14 @@ from typing import Optional
|
||||
|
||||
from .base import (
|
||||
WeatherProvider,
|
||||
AirQualityProvider,
|
||||
WeatherCondition,
|
||||
CurrentWeather,
|
||||
DayForecast,
|
||||
WeatherForecast,
|
||||
GeoLocation,
|
||||
SunTimes,
|
||||
AirQuality,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -89,11 +92,12 @@ WMO_DESCRIPTIONS: dict[int, str] = {
|
||||
}
|
||||
|
||||
|
||||
class OpenMeteoProvider(WeatherProvider):
|
||||
"""Open-Meteo weather API implementation."""
|
||||
class OpenMeteoProvider(WeatherProvider, AirQualityProvider):
|
||||
"""Open-Meteo weather and air quality API implementation."""
|
||||
|
||||
GEOCODING_URL = "https://geocoding-api.open-meteo.com/v1/search"
|
||||
WEATHER_URL = "https://api.open-meteo.com/v1/forecast"
|
||||
AIR_QUALITY_URL = "https://air-quality-api.open-meteo.com/v1/air-quality"
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
@@ -194,9 +198,11 @@ class OpenMeteoProvider(WeatherProvider):
|
||||
"wind_speed_10m",
|
||||
"wind_direction_10m"
|
||||
],
|
||||
"daily": ["uv_index_max"],
|
||||
"timezone": self.timezone,
|
||||
"temperature_unit": "celsius",
|
||||
"wind_speed_unit": "kmh"
|
||||
"wind_speed_unit": "kmh",
|
||||
"forecast_days": 1
|
||||
}
|
||||
)
|
||||
response.raise_for_status()
|
||||
@@ -205,6 +211,12 @@ class OpenMeteoProvider(WeatherProvider):
|
||||
current = data.get("current", {})
|
||||
weather_code = current.get("weather_code", 0)
|
||||
|
||||
# Get today's UV index from daily data
|
||||
daily = data.get("daily", {})
|
||||
uv_index = None
|
||||
if daily.get("uv_index_max"):
|
||||
uv_index = daily["uv_index_max"][0]
|
||||
|
||||
return CurrentWeather(
|
||||
temperature=current.get("temperature_2m", 0.0),
|
||||
feels_like=current.get("apparent_temperature"),
|
||||
@@ -214,7 +226,8 @@ class OpenMeteoProvider(WeatherProvider):
|
||||
condition=WMO_CODE_MAP.get(weather_code, WeatherCondition.UNKNOWN),
|
||||
condition_text=WMO_DESCRIPTIONS.get(weather_code, "Unknown"),
|
||||
timestamp=datetime.now(),
|
||||
location=location.name
|
||||
location=location.name,
|
||||
uv_index=uv_index
|
||||
)
|
||||
except httpx.HTTPError as e:
|
||||
logger.error(f"Weather request failed for {location.name}: {e}")
|
||||
@@ -259,7 +272,8 @@ class OpenMeteoProvider(WeatherProvider):
|
||||
"temperature_2m_max",
|
||||
"temperature_2m_min",
|
||||
"precipitation_sum",
|
||||
"precipitation_probability_max"
|
||||
"precipitation_probability_max",
|
||||
"uv_index_max"
|
||||
],
|
||||
"timezone": self.timezone,
|
||||
"temperature_unit": "celsius",
|
||||
@@ -272,7 +286,14 @@ class OpenMeteoProvider(WeatherProvider):
|
||||
|
||||
# Parse current weather
|
||||
current_data = data.get("current", {})
|
||||
daily_data = data.get("daily", {})
|
||||
weather_code = current_data.get("weather_code", 0)
|
||||
|
||||
# Get today's UV from daily data
|
||||
uv_index = None
|
||||
if daily_data.get("uv_index_max"):
|
||||
uv_index = daily_data["uv_index_max"][0]
|
||||
|
||||
current = CurrentWeather(
|
||||
temperature=current_data.get("temperature_2m", 0.0),
|
||||
feels_like=current_data.get("apparent_temperature"),
|
||||
@@ -282,15 +303,16 @@ class OpenMeteoProvider(WeatherProvider):
|
||||
condition=WMO_CODE_MAP.get(weather_code, WeatherCondition.UNKNOWN),
|
||||
condition_text=WMO_DESCRIPTIONS.get(weather_code, "Unknown"),
|
||||
timestamp=datetime.now(),
|
||||
location=location.name
|
||||
location=location.name,
|
||||
uv_index=uv_index
|
||||
)
|
||||
|
||||
# Parse daily forecast
|
||||
daily_data = data.get("daily", {})
|
||||
daily = []
|
||||
dates = daily_data.get("time", [])
|
||||
for i, date_str in enumerate(dates):
|
||||
code = daily_data.get("weather_code", [])[i] if i < len(daily_data.get("weather_code", [])) else 0
|
||||
uv_max = daily_data.get("uv_index_max", [])[i] if i < len(daily_data.get("uv_index_max", [])) else None
|
||||
daily.append(DayForecast(
|
||||
date=datetime.fromisoformat(date_str),
|
||||
temp_high=daily_data.get("temperature_2m_max", [])[i] if i < len(daily_data.get("temperature_2m_max", [])) else 0.0,
|
||||
@@ -298,7 +320,8 @@ class OpenMeteoProvider(WeatherProvider):
|
||||
condition=WMO_CODE_MAP.get(code, WeatherCondition.UNKNOWN),
|
||||
condition_text=WMO_DESCRIPTIONS.get(code, "Unknown"),
|
||||
precipitation_chance=daily_data.get("precipitation_probability_max", [])[i] if i < len(daily_data.get("precipitation_probability_max", [])) else None,
|
||||
precipitation_mm=daily_data.get("precipitation_sum", [])[i] if i < len(daily_data.get("precipitation_sum", [])) else None
|
||||
precipitation_mm=daily_data.get("precipitation_sum", [])[i] if i < len(daily_data.get("precipitation_sum", [])) else None,
|
||||
uv_index_max=uv_max
|
||||
))
|
||||
|
||||
return WeatherForecast(
|
||||
@@ -309,3 +332,110 @@ class OpenMeteoProvider(WeatherProvider):
|
||||
except httpx.HTTPError as e:
|
||||
logger.error(f"Forecast request failed for {location.name}: {e}")
|
||||
raise ValueError(f"Failed to get forecast: {e}")
|
||||
|
||||
async def get_sun_times(self, location: GeoLocation) -> SunTimes:
|
||||
"""
|
||||
Get sunrise/sunset times for today.
|
||||
|
||||
Args:
|
||||
location: GeoLocation with lat/long
|
||||
|
||||
Returns:
|
||||
SunTimes with sunrise, sunset, and daylight duration
|
||||
|
||||
Raises:
|
||||
ValueError: If API request fails
|
||||
"""
|
||||
try:
|
||||
response = await self.client.get(
|
||||
self.WEATHER_URL,
|
||||
params={
|
||||
"latitude": location.latitude,
|
||||
"longitude": location.longitude,
|
||||
"daily": [
|
||||
"sunrise",
|
||||
"sunset",
|
||||
"daylight_duration"
|
||||
],
|
||||
"timezone": self.timezone,
|
||||
"forecast_days": 1
|
||||
}
|
||||
)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
|
||||
daily = data.get("daily", {})
|
||||
date_str = daily.get("time", [""])[0]
|
||||
sunrise_str = daily.get("sunrise", [""])[0]
|
||||
sunset_str = daily.get("sunset", [""])[0]
|
||||
daylight = daily.get("daylight_duration", [0])[0]
|
||||
|
||||
return SunTimes(
|
||||
location=location.name,
|
||||
date=datetime.fromisoformat(date_str) if date_str else datetime.now(),
|
||||
sunrise=datetime.fromisoformat(sunrise_str) if sunrise_str else datetime.now(),
|
||||
sunset=datetime.fromisoformat(sunset_str) if sunset_str else datetime.now(),
|
||||
daylight_duration=int(daylight) if daylight else 0
|
||||
)
|
||||
except httpx.HTTPError as e:
|
||||
logger.error(f"Sun times request failed for {location.name}: {e}")
|
||||
raise ValueError(f"Failed to get sun times: {e}")
|
||||
|
||||
async def get_air_quality(self, location: GeoLocation) -> AirQuality:
|
||||
"""
|
||||
Get current air quality for a location.
|
||||
|
||||
Args:
|
||||
location: GeoLocation with lat/long
|
||||
|
||||
Returns:
|
||||
AirQuality with pollutant measurements and AQI
|
||||
|
||||
Raises:
|
||||
ValueError: If API request fails
|
||||
"""
|
||||
try:
|
||||
response = await self.client.get(
|
||||
self.AIR_QUALITY_URL,
|
||||
params={
|
||||
"latitude": location.latitude,
|
||||
"longitude": location.longitude,
|
||||
"current": [
|
||||
"european_aqi",
|
||||
"us_aqi",
|
||||
"pm2_5",
|
||||
"pm10",
|
||||
"ozone",
|
||||
"nitrogen_dioxide",
|
||||
"sulphur_dioxide",
|
||||
"carbon_monoxide",
|
||||
"grass_pollen",
|
||||
"birch_pollen",
|
||||
"alder_pollen"
|
||||
],
|
||||
"timezone": self.timezone
|
||||
}
|
||||
)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
|
||||
current = data.get("current", {})
|
||||
|
||||
return AirQuality(
|
||||
location=location.name,
|
||||
timestamp=datetime.now(),
|
||||
aqi_european=current.get("european_aqi"),
|
||||
aqi_us=current.get("us_aqi"),
|
||||
pm2_5=current.get("pm2_5"),
|
||||
pm10=current.get("pm10"),
|
||||
ozone=current.get("ozone"),
|
||||
nitrogen_dioxide=current.get("nitrogen_dioxide"),
|
||||
sulphur_dioxide=current.get("sulphur_dioxide"),
|
||||
carbon_monoxide=current.get("carbon_monoxide"),
|
||||
pollen_grass=current.get("grass_pollen"),
|
||||
pollen_birch=current.get("birch_pollen"),
|
||||
pollen_alder=current.get("alder_pollen")
|
||||
)
|
||||
except httpx.HTTPError as e:
|
||||
logger.error(f"Air quality request failed for {location.name}: {e}")
|
||||
raise ValueError(f"Failed to get air quality: {e}")
|
||||
|
||||
@@ -19,6 +19,7 @@ class VolatileNamespace(str, Enum):
|
||||
"""
|
||||
# Real-time external data
|
||||
WEATHER = "weather" # Current conditions, forecasts
|
||||
SUN = "sun" # Sunrise, sunset, daylight duration
|
||||
NEWS = "news" # Headlines, breaking news
|
||||
FINANCIAL = "financial" # Stock prices, exchange rates, crypto
|
||||
TRANSIT = "transit" # Train/bus schedules, delays, disruptions
|
||||
@@ -38,6 +39,7 @@ class VolatileNamespace(str, Enum):
|
||||
# Default TTLs per namespace (in seconds)
|
||||
NAMESPACE_DEFAULT_TTL: Dict[str, int] = {
|
||||
VolatileNamespace.WEATHER: 1800, # 30 min - weather changes slowly
|
||||
VolatileNamespace.SUN: 86400, # 24 hours - sun times change daily
|
||||
VolatileNamespace.NEWS: 3600, # 1 hour - news cycles
|
||||
VolatileNamespace.FINANCIAL: 300, # 5 min - markets move fast
|
||||
VolatileNamespace.TRANSIT: 300, # 5 min - schedules update frequently
|
||||
|
||||
@@ -411,6 +411,98 @@ async def fetch_crypto(
|
||||
}
|
||||
|
||||
|
||||
@router.post("/fetch/sun/{city}")
|
||||
async def fetch_sun_times(
|
||||
city: str,
|
||||
user: str = Query(default=DEFAULT_USER, description="User identifier"),
|
||||
ttl: int = Query(default=86400, ge=60, le=604800, description="TTL in seconds"),
|
||||
qdrant: QdrantDep = None,
|
||||
ollama: OllamaDep = None,
|
||||
api_key: str = Depends(verify_api_key)
|
||||
):
|
||||
"""
|
||||
Fetch sunrise/sunset times for a city and store in volatile cache.
|
||||
|
||||
Fetches from Open-Meteo API. Useful for home automation triggers.
|
||||
|
||||
**Example:**
|
||||
```
|
||||
POST /volatile/fetch/sun/rotterdam?user=jpmschweitzer
|
||||
```
|
||||
|
||||
**Response data includes:**
|
||||
- sunrise/sunset times (both HH:MM and ISO formats)
|
||||
- daylight_duration_seconds
|
||||
- daylight_hours
|
||||
- Natural language text summary
|
||||
"""
|
||||
volatile_service = get_volatile_service(qdrant, ollama)
|
||||
weather_provider = get_weather_provider()
|
||||
|
||||
fetch_service = VolatileFetchService(
|
||||
volatile_service=volatile_service,
|
||||
weather_provider=weather_provider,
|
||||
)
|
||||
|
||||
result = await fetch_service.fetch_sun_times(user, city, ttl=ttl)
|
||||
|
||||
if not result.success:
|
||||
raise HTTPException(status_code=500, detail=result.error)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"namespace": result.namespace,
|
||||
"key": result.key,
|
||||
"record": result.record,
|
||||
}
|
||||
|
||||
|
||||
@router.post("/fetch/air_quality/{city}")
|
||||
async def fetch_air_quality(
|
||||
city: str,
|
||||
user: str = Query(default=DEFAULT_USER, description="User identifier"),
|
||||
ttl: int = Query(default=3600, ge=60, le=86400, description="TTL in seconds"),
|
||||
qdrant: QdrantDep = None,
|
||||
ollama: OllamaDep = None,
|
||||
api_key: str = Depends(verify_api_key)
|
||||
):
|
||||
"""
|
||||
Fetch air quality data for a city and store in volatile cache.
|
||||
|
||||
Fetches from Open-Meteo Air Quality API.
|
||||
|
||||
**Example:**
|
||||
```
|
||||
POST /volatile/fetch/air_quality/rotterdam?user=jpmschweitzer
|
||||
```
|
||||
|
||||
**Response data includes:**
|
||||
- European and US AQI indices
|
||||
- Pollutants: PM2.5, PM10, ozone, nitrogen dioxide, etc.
|
||||
- Pollen data (European locations, seasonal)
|
||||
- Natural language text summary
|
||||
"""
|
||||
volatile_service = get_volatile_service(qdrant, ollama)
|
||||
weather_provider = get_weather_provider()
|
||||
|
||||
fetch_service = VolatileFetchService(
|
||||
volatile_service=volatile_service,
|
||||
weather_provider=weather_provider,
|
||||
)
|
||||
|
||||
result = await fetch_service.fetch_air_quality(user, city, ttl=ttl)
|
||||
|
||||
if not result.success:
|
||||
raise HTTPException(status_code=500, detail=result.error)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"namespace": result.namespace,
|
||||
"key": result.key,
|
||||
"record": result.record,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/{namespace}/{key}", response_model=VolatileRecordResponse)
|
||||
async def get_record(
|
||||
namespace: str,
|
||||
|
||||
@@ -14,6 +14,9 @@ from src.apis import (
|
||||
AggregatedNewsProvider,
|
||||
AlphaVantageProvider,
|
||||
CurrentWeather,
|
||||
WeatherForecast,
|
||||
SunTimes,
|
||||
AirQuality,
|
||||
NewsFeed,
|
||||
StockQuote,
|
||||
)
|
||||
@@ -68,21 +71,23 @@ class VolatileFetchService:
|
||||
self,
|
||||
user: str,
|
||||
city: str,
|
||||
days: int = 7,
|
||||
ttl: int = 86400, # 24 hours
|
||||
) -> FetchResult:
|
||||
"""
|
||||
Fetch current weather for a city and store in volatile cache.
|
||||
Fetch weather forecast for a city and store in volatile cache.
|
||||
|
||||
Args:
|
||||
user: User identifier
|
||||
city: City name (will be geocoded)
|
||||
days: Number of forecast days (1-16)
|
||||
ttl: Time-to-live in seconds
|
||||
|
||||
Returns:
|
||||
FetchResult with success status and stored record
|
||||
"""
|
||||
try:
|
||||
# Geocode city and get weather
|
||||
# Geocode city and get forecast
|
||||
location = await self.weather.geocode(city)
|
||||
if not location:
|
||||
return FetchResult(
|
||||
@@ -92,19 +97,45 @@ class VolatileFetchService:
|
||||
error=f"Could not geocode city: {city}"
|
||||
)
|
||||
|
||||
weather = await self.weather.get_current(location)
|
||||
forecast = await self.weather.get_forecast(location, days=days)
|
||||
current = forecast.current
|
||||
|
||||
# Build daily forecast array
|
||||
daily_forecasts = []
|
||||
for day in forecast.daily:
|
||||
daily_forecasts.append({
|
||||
"date": day.date.isoformat(),
|
||||
"day_name": day.date.strftime("%A"),
|
||||
"temp_high": day.temp_high,
|
||||
"temp_low": day.temp_low,
|
||||
"conditions": day.condition_text,
|
||||
"condition_code": day.condition.value,
|
||||
"precipitation_chance": day.precipitation_chance,
|
||||
"precipitation_mm": day.precipitation_mm,
|
||||
"uv_index_max": day.uv_index_max,
|
||||
})
|
||||
|
||||
# Generate natural language summary
|
||||
forecast_lines = [current.to_text()]
|
||||
for day in forecast.daily[:5]: # First 5 days
|
||||
forecast_lines.append(day.to_text())
|
||||
text = "\n".join(forecast_lines)
|
||||
|
||||
# Convert to storage format
|
||||
data = {
|
||||
"temperature": weather.temperature,
|
||||
"feels_like": weather.feels_like,
|
||||
"humidity": weather.humidity,
|
||||
"wind_speed": weather.wind_speed,
|
||||
"wind_direction": weather.wind_direction,
|
||||
"conditions": weather.condition_text,
|
||||
"condition_code": weather.condition.value,
|
||||
"location": weather.location,
|
||||
"text": weather.to_text(),
|
||||
"current": {
|
||||
"temperature": current.temperature,
|
||||
"feels_like": current.feels_like,
|
||||
"humidity": current.humidity,
|
||||
"wind_speed": current.wind_speed,
|
||||
"wind_direction": current.wind_direction,
|
||||
"conditions": current.condition_text,
|
||||
"condition_code": current.condition.value,
|
||||
"uv_index": current.uv_index,
|
||||
},
|
||||
"daily": daily_forecasts,
|
||||
"location": current.location,
|
||||
"text": text,
|
||||
}
|
||||
|
||||
# Store in volatile cache
|
||||
@@ -117,7 +148,7 @@ class VolatileFetchService:
|
||||
ttl=ttl,
|
||||
)
|
||||
|
||||
logger.info(f"Stored weather for {city} (user={user})")
|
||||
logger.info(f"Stored {days}-day forecast for {city} (user={user})")
|
||||
return FetchResult(
|
||||
success=True,
|
||||
namespace="weather",
|
||||
@@ -357,3 +388,147 @@ class VolatileFetchService:
|
||||
key=f"{symbol.lower()}_{market.lower()}",
|
||||
error=str(e)
|
||||
)
|
||||
|
||||
async def fetch_sun_times(
|
||||
self,
|
||||
user: str,
|
||||
city: str,
|
||||
ttl: int = 86400, # 24 hours
|
||||
) -> FetchResult:
|
||||
"""
|
||||
Fetch sunrise/sunset times for a city and store in volatile cache.
|
||||
|
||||
Args:
|
||||
user: User identifier
|
||||
city: City name (will be geocoded)
|
||||
ttl: Time-to-live in seconds
|
||||
|
||||
Returns:
|
||||
FetchResult with success status and stored record
|
||||
"""
|
||||
try:
|
||||
# Geocode city and get sun times
|
||||
location = await self.weather.geocode(city)
|
||||
if not location:
|
||||
return FetchResult(
|
||||
success=False,
|
||||
namespace="sun",
|
||||
key=city.lower(),
|
||||
error=f"Could not geocode city: {city}"
|
||||
)
|
||||
|
||||
sun_times = await self.weather.get_sun_times(location)
|
||||
|
||||
# Convert to storage format
|
||||
data = {
|
||||
"location": sun_times.location,
|
||||
"date": sun_times.date.isoformat(),
|
||||
"sunrise": sun_times.sunrise.strftime("%H:%M"),
|
||||
"sunset": sun_times.sunset.strftime("%H:%M"),
|
||||
"sunrise_iso": sun_times.sunrise.isoformat(),
|
||||
"sunset_iso": sun_times.sunset.isoformat(),
|
||||
"daylight_duration_seconds": sun_times.daylight_duration,
|
||||
"daylight_hours": sun_times.daylight_duration / 3600,
|
||||
"text": sun_times.to_text(),
|
||||
}
|
||||
|
||||
# Store in volatile cache
|
||||
record = await self.volatile.store(
|
||||
user=user,
|
||||
namespace=VolatileNamespace.SUN,
|
||||
key=city.lower(),
|
||||
data=data,
|
||||
source="openmeteo",
|
||||
ttl=ttl,
|
||||
)
|
||||
|
||||
logger.info(f"Stored sun times for {city} (user={user})")
|
||||
return FetchResult(
|
||||
success=True,
|
||||
namespace="sun",
|
||||
key=city.lower(),
|
||||
record=record
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to fetch sun times for {city}: {e}")
|
||||
return FetchResult(
|
||||
success=False,
|
||||
namespace="sun",
|
||||
key=city.lower(),
|
||||
error=str(e)
|
||||
)
|
||||
|
||||
async def fetch_air_quality(
|
||||
self,
|
||||
user: str,
|
||||
city: str,
|
||||
ttl: int = 3600, # 1 hour
|
||||
) -> FetchResult:
|
||||
"""
|
||||
Fetch air quality data for a city and store in volatile cache.
|
||||
|
||||
Args:
|
||||
user: User identifier
|
||||
city: City name (will be geocoded)
|
||||
ttl: Time-to-live in seconds
|
||||
|
||||
Returns:
|
||||
FetchResult with success status and stored record
|
||||
"""
|
||||
try:
|
||||
# Geocode city and get air quality
|
||||
location = await self.weather.geocode(city)
|
||||
if not location:
|
||||
return FetchResult(
|
||||
success=False,
|
||||
namespace="air_quality",
|
||||
key=city.lower(),
|
||||
error=f"Could not geocode city: {city}"
|
||||
)
|
||||
|
||||
air_quality = await self.weather.get_air_quality(location)
|
||||
|
||||
# Convert to storage format
|
||||
data = {
|
||||
"location": air_quality.location,
|
||||
"aqi_european": air_quality.aqi_european,
|
||||
"aqi_us": air_quality.aqi_us,
|
||||
"pm2_5": air_quality.pm2_5,
|
||||
"pm10": air_quality.pm10,
|
||||
"ozone": air_quality.ozone,
|
||||
"nitrogen_dioxide": air_quality.nitrogen_dioxide,
|
||||
"sulphur_dioxide": air_quality.sulphur_dioxide,
|
||||
"carbon_monoxide": air_quality.carbon_monoxide,
|
||||
"pollen_grass": air_quality.pollen_grass,
|
||||
"pollen_birch": air_quality.pollen_birch,
|
||||
"pollen_alder": air_quality.pollen_alder,
|
||||
"text": air_quality.to_text(),
|
||||
}
|
||||
|
||||
# Store in volatile cache
|
||||
record = await self.volatile.store(
|
||||
user=user,
|
||||
namespace=VolatileNamespace.AIR_QUALITY,
|
||||
key=city.lower(),
|
||||
data=data,
|
||||
source="openmeteo",
|
||||
ttl=ttl,
|
||||
)
|
||||
|
||||
logger.info(f"Stored air quality for {city} (user={user})")
|
||||
return FetchResult(
|
||||
success=True,
|
||||
namespace="air_quality",
|
||||
key=city.lower(),
|
||||
record=record
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to fetch air quality for {city}: {e}")
|
||||
return FetchResult(
|
||||
success=False,
|
||||
namespace="air_quality",
|
||||
key=city.lower(),
|
||||
error=str(e)
|
||||
)
|
||||
|
||||
@@ -116,24 +116,6 @@ class WikiChangeListener:
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to handle notification: {e}", exc_info=True)
|
||||
|
||||
def _is_automated_user(self, email: str) -> bool:
|
||||
"""
|
||||
Check if email belongs to an automated system user.
|
||||
|
||||
These are edits made by library-desk via Wiki.js API (entity linking).
|
||||
We skip processing these to prevent loops.
|
||||
|
||||
Customize this list based on your Wiki.js username for library-desk.
|
||||
"""
|
||||
automated_users = [
|
||||
self.settings.wikijs_username, # Library-desk's Wiki.js API user
|
||||
"library-desk@system",
|
||||
"automation@system",
|
||||
"bot@system"
|
||||
]
|
||||
|
||||
return email.lower() in [u.lower() for u in automated_users]
|
||||
|
||||
def _is_recently_processed(self, page_id: int) -> bool:
|
||||
"""Check if page was processed recently (debouncing)."""
|
||||
if page_id not in self._recent_notifications:
|
||||
|
||||
@@ -110,7 +110,7 @@ class TestVolatileNamespaces:
|
||||
|
||||
def test_namespace_count(self):
|
||||
"""Test we have the expected number of namespaces."""
|
||||
assert len(VolatileNamespace) == 11
|
||||
assert len(VolatileNamespace) == 12 # Including SUN for sunrise/sunset
|
||||
|
||||
|
||||
class TestVolatileListResponse:
|
||||
|
||||
@@ -50,22 +50,6 @@ class TestWikiChangeListener:
|
||||
assert listener._debounce_seconds == 5
|
||||
assert len(listener._recent_notifications) == 0
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_automated_user_filtering(self, listener):
|
||||
"""Test that automated users are correctly identified."""
|
||||
# Automated users should be filtered
|
||||
assert listener._is_automated_user("librarian@schweitz.net") is True
|
||||
assert listener._is_automated_user("library-desk@system") is True
|
||||
assert listener._is_automated_user("automation@system") is True
|
||||
assert listener._is_automated_user("bot@system") is True
|
||||
|
||||
# Case insensitive
|
||||
assert listener._is_automated_user("LIBRARIAN@SCHWEITZ.NET") is True
|
||||
|
||||
# Regular users should not be filtered
|
||||
assert listener._is_automated_user("user@example.com") is False
|
||||
assert listener._is_automated_user("john@example.com") is False
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_debouncing_prevents_duplicates(self, listener):
|
||||
"""Test that debouncing prevents duplicate processing."""
|
||||
@@ -163,19 +147,29 @@ class TestWikiChangeListener:
|
||||
assert mock_process.call_args[1]['event'] == 'page.delete'
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_automated_user_notification_filtered(self, listener):
|
||||
"""Test that notifications from automated users are filtered out."""
|
||||
async def test_any_user_notification_processed(self, listener):
|
||||
"""Test that notifications are processed regardless of user email.
|
||||
|
||||
Note: The user_email in PostgreSQL notifications is the page CREATOR,
|
||||
not the editor. We cannot filter by user email because:
|
||||
- A page created by 'librarian' but edited by a human should be processed
|
||||
- Filtering by creator would break legitimate page ingestion
|
||||
Loop prevention is handled by debouncing instead.
|
||||
"""
|
||||
mock_connection = AsyncMock()
|
||||
|
||||
with patch.object(listener, '_process_page_change', new_callable=AsyncMock) as mock_process:
|
||||
# Notification from automated user should be skipped
|
||||
# Even system user notifications should be processed
|
||||
# (debouncing handles loop prevention, not user filtering)
|
||||
await listener._handle_notification(
|
||||
mock_connection, 1234, 'wiki_page_changes',
|
||||
'UPDATE:123:librarian@schweitz.net'
|
||||
)
|
||||
|
||||
# Process should NOT be called
|
||||
mock_process.assert_not_called()
|
||||
# Process SHOULD be called (user filtering is not used)
|
||||
mock_process.assert_called_once()
|
||||
assert mock_process.call_args[1]['page_id'] == 123
|
||||
assert mock_process.call_args[1]['event'] == 'page.update'
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_duplicate_notification_filtered(self, listener):
|
||||
|
||||
Reference in New Issue
Block a user