Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
152b2f28c4 | ||
|
|
46b9bcd7a0 | ||
|
|
68eb1add3d | ||
|
|
d6b30570a0 | ||
|
|
6b0530ed79 | ||
|
|
0a8c2639a0 |
@@ -5,6 +5,56 @@ All notable changes to Library Desk will be documented in this file.
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [1.6.2] - 2025-12-30
|
||||
|
||||
### Added
|
||||
|
||||
- **System Statistics Endpoint** - `GET /stats`
|
||||
- Neo4j: node counts by type (Document, Entity, Collection, Search)
|
||||
- Qdrant: collection counts, total vectors, per-collection breakdown
|
||||
- Wiki.js: total page count
|
||||
- Paperless: documents, tags, correspondents, document types
|
||||
|
||||
- **Weather/Forecast Separation** - Split weather into two distinct namespaces
|
||||
- `POST /volatile/fetch/weather/{city}` - Current conditions only (1hr TTL)
|
||||
- `POST /volatile/fetch/forecast/{city}` - 7-day outlook (12hr TTL)
|
||||
- Different update frequencies for efficient caching
|
||||
- `FORECAST` namespace added to volatile namespaces
|
||||
|
||||
### Changed
|
||||
|
||||
- Weather namespace TTL changed from 30 minutes to 1 hour (current conditions)
|
||||
- Forecast data now stored separately with 12 hour TTL
|
||||
|
||||
## [1.6.1] - 2025-12-30
|
||||
|
||||
### Added
|
||||
|
||||
- **Weather Forecast Support** - Enhanced weather fetch with 7-day daily forecasts
|
||||
- Current conditions now include UV index
|
||||
- Daily forecasts with high/low temps, conditions, precipitation chance, UV max
|
||||
- Natural language text summary with multi-day outlook
|
||||
- **Sun Times Endpoint** - `POST /volatile/fetch/sun/{city}`
|
||||
- Sunrise and sunset times (HH:MM and ISO formats)
|
||||
- Daylight duration in seconds and hours
|
||||
- Separate volatile namespace with 24hr TTL
|
||||
- Useful for home automation light triggers
|
||||
- **Air Quality Endpoint** - `POST /volatile/fetch/air_quality/{city}`
|
||||
- European and US AQI indices
|
||||
- Pollutants: PM2.5, PM10, ozone, nitrogen dioxide, sulphur dioxide, carbon monoxide
|
||||
- Pollen data (grass, birch, alder) for European locations (seasonal)
|
||||
- Hourly refresh (1hr TTL)
|
||||
- **New Base Models**
|
||||
- `SunTimes` dataclass for sunrise/sunset data
|
||||
- `AirQuality` dataclass with AQI and pollutants
|
||||
- `AirQualityProvider` abstract interface
|
||||
- **New Volatile Namespace** - `SUN` for sunrise/sunset times (86400s default TTL)
|
||||
|
||||
### Changed
|
||||
|
||||
- Weather fetch now uses `get_forecast()` instead of `get_current()` for richer data
|
||||
- `OpenMeteoProvider` now implements both `WeatherProvider` and `AirQualityProvider`
|
||||
|
||||
## [1.6.0] - 2025-12-29
|
||||
|
||||
### Added
|
||||
|
||||
@@ -41,12 +41,3 @@ Check for duplicate or highly similar documents using vector similarity and grap
|
||||
3. Check graph relationships
|
||||
4. Return candidates with similarity scores
|
||||
|
||||
## System Statistics
|
||||
|
||||
#### `GET /stats`
|
||||
Get system statistics (wiki pages, neo4j nodes, qdrant vectors).
|
||||
|
||||
**Implementation needed:**
|
||||
- Query Neo4j for node count
|
||||
- Query Qdrant for vector count
|
||||
- Query Wiki.js for page count
|
||||
|
||||
@@ -16,8 +16,16 @@ This document outlines the implementation of "remember" triggers for the memory
|
||||
| Memory Tier | Remember Trigger | Recall | Status |
|
||||
|-------------|------------------|--------|--------|
|
||||
| Wiki | Wiki.js webhook, Consolidation | HybridRAG vector+graph | ✅ Complete |
|
||||
| Documents | Paperless webhook | Manual `/documents/search` | ⚠️ Partial (no HybridRAG recall) |
|
||||
| Volatile | Manual `/volatile/store` only | HybridRAG volatile search | ⚠️ Partial (no auto-triggers) |
|
||||
| Documents | Paperless webhook | HybridRAG document search | ✅ Complete (v1.6.0) |
|
||||
| Volatile | Scheduler prefetch, HybridRAG post-processor | HybridRAG volatile search | ✅ Complete (v1.6.0) |
|
||||
|
||||
### Implementation Summary (v1.6.0)
|
||||
|
||||
- **Settings DB**: Central `system_settings` PostgreSQL database with `SettingsClient`
|
||||
- **Phase A**: Volatile fetch endpoints (`/volatile/fetch/{namespace}/{key}`) with weather, news, financial providers
|
||||
- **Phase B**: Unified memory routing in consolidation service (wiki/volatile/file/prefetch/skip classification)
|
||||
- **Phase C**: Document recall in HybridRAG (4-source parallel retrieval)
|
||||
- **Scheduler Integration**: `SchedulerClient` for external scheduler task registration
|
||||
|
||||
---
|
||||
|
||||
@@ -662,46 +670,60 @@ document_threshold: float = Field(default=0.6)
|
||||
|
||||
## Implementation Order
|
||||
|
||||
| Phase | Priority | Effort | Description |
|
||||
|-------|----------|--------|-------------|
|
||||
| **Settings DB** | High | Low | PostgreSQL schema + settings client |
|
||||
| **B.4** | High | Low | Scheduler client |
|
||||
| **B.1-B.3** | High | Medium | HybridRAG post-processor |
|
||||
| **B.5** | High | Low | Config options |
|
||||
| **C.1-C.2** | High | Low | Document recall in HybridRAG |
|
||||
| **A.1** | Medium | Low | `/volatile/fetch` endpoint |
|
||||
| **A.2** | Medium | Medium | Weather + News API clients |
|
||||
| **A.3** | Medium | Low | Fetch service |
|
||||
| Phase | Priority | Effort | Description | Status |
|
||||
|-------|----------|--------|-------------|--------|
|
||||
| **Settings DB** | High | Low | PostgreSQL schema + settings client | ✅ v1.5.0 |
|
||||
| **B.4** | High | Low | Scheduler client | ✅ v1.6.0 |
|
||||
| **B.1-B.3** | High | Medium | HybridRAG post-processor | ✅ v1.6.0 |
|
||||
| **B.5** | High | Low | Config options | ✅ v1.6.0 |
|
||||
| **C.1-C.2** | High | Low | Document recall in HybridRAG | ✅ v1.6.0 |
|
||||
| **A.1** | Medium | Low | `/volatile/fetch` endpoint | ✅ v1.6.0 |
|
||||
| **A.2** | Medium | Medium | Weather + News API clients | ✅ v1.5.0 |
|
||||
| **A.3** | Medium | Low | Fetch service | ✅ v1.6.0 |
|
||||
|
||||
### Remaining Work
|
||||
|
||||
| Item | Description | Status |
|
||||
|------|-------------|--------|
|
||||
| File upload | Download PDFs and upload to Paperless | ⚠️ Placeholder (logs only) |
|
||||
| Prefetch patterns | More sophisticated pattern detection | Optional enhancement |
|
||||
|
||||
---
|
||||
|
||||
## Files Summary
|
||||
|
||||
### New Files
|
||||
### New Files (Implemented)
|
||||
|
||||
| Path | Purpose |
|
||||
|------|---------|
|
||||
| `src/clients/settings_client.py` | Read-only central settings access |
|
||||
| `src/clients/scheduler_client.py` | Register/manage scheduler tasks |
|
||||
| `src/clients/weather_client.py` | Open-Meteo API (with geocoding) |
|
||||
| `src/clients/news_client.py` | NOS.nl RSS feeds |
|
||||
| `src/services/volatile_fetch_service.py` | Orchestrates fetch + store |
|
||||
| Path | Purpose | Version |
|
||||
|------|---------|---------|
|
||||
| `src/clients/settings_client.py` | Central settings database access | v1.5.0 |
|
||||
| `src/clients/scheduler_client.py` | External scheduler task management | v1.6.0 |
|
||||
| `src/apis/__init__.py` | External API providers package | v1.5.0 |
|
||||
| `src/apis/base.py` | Abstract base classes for providers | v1.5.0 |
|
||||
| `src/apis/weather.py` | OpenMeteoProvider (geocoding + forecast) | v1.5.0 |
|
||||
| `src/apis/news.py` | AggregatedNewsProvider | v1.5.0 |
|
||||
| `src/apis/nos.py` | NOSProvider (Dutch news RSS) | v1.5.0 |
|
||||
| `src/apis/bbc.py` | BBCProvider (English news RSS) | v1.5.0 |
|
||||
| `src/apis/financial.py` | AlphaVantageProvider (stocks/crypto) | v1.5.0 |
|
||||
| `src/services/volatile_fetch_service.py` | Orchestrates fetch + store | v1.6.0 |
|
||||
|
||||
### Modified Files
|
||||
|
||||
| Path | Changes |
|
||||
|------|---------|
|
||||
| `src/services/hybrid_rag_service.py` | Post-processor, prefetch detection, document search |
|
||||
| `src/models/hybrid_rag.py` | Memory config options |
|
||||
| `src/routers/volatile.py` | `/volatile/fetch/{namespace}/{key}` |
|
||||
| `src/core/dependencies.py` | Settings client, scheduler client DI |
|
||||
| `src/config.py` | `SYSTEM_SETTINGS_*` connection vars |
|
||||
| Path | Changes | Version |
|
||||
|------|---------|---------|
|
||||
| `src/services/hybrid_rag_service.py` | Document search (4-source parallel retrieval) | v1.6.0 |
|
||||
| `src/services/consolidation_service.py` | Unified memory routing, scheduler integration | v1.6.0 |
|
||||
| `src/models/hybrid_rag.py` | Document config options (`enable_documents`, `document_limit`) | v1.6.0 |
|
||||
| `src/models/consolidation.py` | Memory routing models | v1.6.0 |
|
||||
| `src/routers/volatile.py` | `/volatile/fetch/{namespace}/{key}` endpoints | v1.6.0 |
|
||||
| `src/core/dependencies.py` | Settings, scheduler, provider DI | v1.5.0-v1.6.0 |
|
||||
| `src/config.py` | `SYSTEM_SETTINGS_*`, `SCHEDULER_URL` vars | v1.5.0-v1.6.0 |
|
||||
|
||||
### Database
|
||||
|
||||
| Item | Details |
|
||||
|------|---------|
|
||||
| Database | `system_settings` (PostgreSQL) |
|
||||
| Table | `settings (key, value JSONB, category, ...)` |
|
||||
| Library-desk user | `library_desk_ro` (read-only) |
|
||||
| Management | Direct psql commands |
|
||||
| Database | `system_settings` (PostgreSQL on postgres-shared) |
|
||||
| Table | `settings (key, user_scope, value JSONB, schema JSONB, ...)` |
|
||||
| Library-desk access | Read-only via `SettingsClient` |
|
||||
| Management | Direct psql commands (future: CRUD manager UI) |
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "library-desk"
|
||||
version = "1.6.0"
|
||||
version = "1.6.2"
|
||||
description = "Coordination service for The Library system - HybridRAG queries, document ingestion, entity extraction, and knowledge consolidation"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12"
|
||||
|
||||
@@ -27,6 +27,9 @@ from .base import (
|
||||
DayForecast,
|
||||
WeatherForecast,
|
||||
GeoLocation,
|
||||
SunTimes,
|
||||
# Air quality models
|
||||
AirQuality,
|
||||
# News models
|
||||
NewsItem,
|
||||
NewsFeed,
|
||||
@@ -34,6 +37,7 @@ from .base import (
|
||||
StockQuote,
|
||||
# Abstract providers
|
||||
WeatherProvider,
|
||||
AirQualityProvider,
|
||||
NewsProvider,
|
||||
FinancialProvider,
|
||||
)
|
||||
@@ -53,8 +57,12 @@ __all__ = [
|
||||
"DayForecast",
|
||||
"WeatherForecast",
|
||||
"GeoLocation",
|
||||
"SunTimes",
|
||||
"WeatherProvider",
|
||||
"OpenMeteoProvider",
|
||||
# Air quality
|
||||
"AirQuality",
|
||||
"AirQualityProvider",
|
||||
# News
|
||||
"NewsItem",
|
||||
"NewsFeed",
|
||||
|
||||
+102
-4
@@ -44,14 +44,18 @@ class CurrentWeather:
|
||||
condition_text: str # Human-readable description
|
||||
timestamp: datetime
|
||||
location: str # City/location name
|
||||
uv_index: Optional[float] = None # UV index 0-11+
|
||||
|
||||
def to_text(self) -> str:
|
||||
"""Generate natural language description."""
|
||||
return (
|
||||
f"Currently {self.temperature:.1f}°C "
|
||||
f"({self.condition_text}) in {self.location}. "
|
||||
parts = [
|
||||
f"Currently {self.temperature:.1f}°C",
|
||||
f"({self.condition_text}) in {self.location}.",
|
||||
f"Humidity {self.humidity}%, wind {self.wind_speed:.0f} km/h."
|
||||
)
|
||||
]
|
||||
if self.uv_index is not None:
|
||||
parts.append(f"UV index: {self.uv_index:.0f}.")
|
||||
return " ".join(parts)
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -64,6 +68,14 @@ class DayForecast:
|
||||
condition_text: str
|
||||
precipitation_chance: Optional[int] # Percentage 0-100
|
||||
precipitation_mm: Optional[float]
|
||||
uv_index_max: Optional[float] = None # Max UV index for the day
|
||||
|
||||
def to_text(self) -> str:
|
||||
"""Generate natural language description."""
|
||||
date_str = self.date.strftime("%A") # Day name
|
||||
precip = f", {self.precipitation_chance}% rain" if self.precipitation_chance else ""
|
||||
uv = f", UV {self.uv_index_max:.0f}" if self.uv_index_max else ""
|
||||
return f"{date_str}: {self.temp_high:.0f}°/{self.temp_low:.0f}°C, {self.condition_text}{precip}{uv}"
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -84,6 +96,78 @@ class GeoLocation:
|
||||
admin_area: Optional[str] = None # State/province
|
||||
|
||||
|
||||
@dataclass
|
||||
class SunTimes:
|
||||
"""Sunrise/sunset times for a location."""
|
||||
location: str
|
||||
date: datetime
|
||||
sunrise: datetime
|
||||
sunset: datetime
|
||||
daylight_duration: int # seconds
|
||||
solar_noon: Optional[datetime] = None
|
||||
|
||||
def to_text(self) -> str:
|
||||
"""Generate natural language description."""
|
||||
sunrise_str = self.sunrise.strftime("%H:%M")
|
||||
sunset_str = self.sunset.strftime("%H:%M")
|
||||
hours = self.daylight_duration // 3600
|
||||
minutes = (self.daylight_duration % 3600) // 60
|
||||
return (
|
||||
f"Sun times for {self.location} on {self.date.strftime('%A %d %B')}: "
|
||||
f"Sunrise at {sunrise_str}, sunset at {sunset_str}. "
|
||||
f"Daylight duration: {hours}h {minutes}m."
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class AirQuality:
|
||||
"""Air quality measurements for a location."""
|
||||
location: str
|
||||
timestamp: datetime
|
||||
aqi_european: Optional[int] # European AQI 0-500+
|
||||
aqi_us: Optional[int] # US AQI 0-500+
|
||||
pm2_5: Optional[float] # µg/m³
|
||||
pm10: Optional[float] # µg/m³
|
||||
ozone: Optional[float] # µg/m³
|
||||
nitrogen_dioxide: Optional[float] # µg/m³
|
||||
sulphur_dioxide: Optional[float] # µg/m³
|
||||
carbon_monoxide: Optional[float] # µg/m³
|
||||
# Pollen (European data only, seasonal)
|
||||
pollen_grass: Optional[float] = None
|
||||
pollen_birch: Optional[float] = None
|
||||
pollen_alder: Optional[float] = None
|
||||
|
||||
def to_text(self) -> str:
|
||||
"""Generate natural language description."""
|
||||
parts = [f"Air quality in {self.location}:"]
|
||||
if self.aqi_european is not None:
|
||||
level = self._aqi_level(self.aqi_european)
|
||||
parts.append(f"European AQI {self.aqi_european} ({level}).")
|
||||
if self.pm2_5 is not None:
|
||||
parts.append(f"PM2.5: {self.pm2_5:.1f} µg/m³.")
|
||||
if self.pm10 is not None:
|
||||
parts.append(f"PM10: {self.pm10:.1f} µg/m³.")
|
||||
if self.ozone is not None:
|
||||
parts.append(f"Ozone: {self.ozone:.1f} µg/m³.")
|
||||
return " ".join(parts)
|
||||
|
||||
@staticmethod
|
||||
def _aqi_level(aqi: int) -> str:
|
||||
"""Convert AQI to human-readable level."""
|
||||
if aqi <= 20:
|
||||
return "good"
|
||||
elif aqi <= 40:
|
||||
return "fair"
|
||||
elif aqi <= 60:
|
||||
return "moderate"
|
||||
elif aqi <= 80:
|
||||
return "poor"
|
||||
elif aqi <= 100:
|
||||
return "very poor"
|
||||
else:
|
||||
return "hazardous"
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# News Models
|
||||
# =============================================================================
|
||||
@@ -163,6 +247,11 @@ class WeatherProvider(ABC):
|
||||
"""Get weather forecast for a location."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
async def get_sun_times(self, location: GeoLocation) -> SunTimes:
|
||||
"""Get sunrise/sunset times for today."""
|
||||
pass
|
||||
|
||||
async def get_weather_for_city(self, city: str) -> CurrentWeather:
|
||||
"""Convenience method: geocode and get current weather."""
|
||||
location = await self.geocode(city)
|
||||
@@ -171,6 +260,15 @@ class WeatherProvider(ABC):
|
||||
return await self.get_current(location)
|
||||
|
||||
|
||||
class AirQualityProvider(ABC):
|
||||
"""Abstract base class for air quality API providers."""
|
||||
|
||||
@abstractmethod
|
||||
async def get_air_quality(self, location: GeoLocation) -> AirQuality:
|
||||
"""Get current air quality for a location."""
|
||||
pass
|
||||
|
||||
|
||||
class NewsProvider(ABC):
|
||||
"""Abstract base class for news API providers."""
|
||||
|
||||
|
||||
+138
-8
@@ -14,11 +14,14 @@ from typing import Optional
|
||||
|
||||
from .base import (
|
||||
WeatherProvider,
|
||||
AirQualityProvider,
|
||||
WeatherCondition,
|
||||
CurrentWeather,
|
||||
DayForecast,
|
||||
WeatherForecast,
|
||||
GeoLocation,
|
||||
SunTimes,
|
||||
AirQuality,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -89,11 +92,12 @@ WMO_DESCRIPTIONS: dict[int, str] = {
|
||||
}
|
||||
|
||||
|
||||
class OpenMeteoProvider(WeatherProvider):
|
||||
"""Open-Meteo weather API implementation."""
|
||||
class OpenMeteoProvider(WeatherProvider, AirQualityProvider):
|
||||
"""Open-Meteo weather and air quality API implementation."""
|
||||
|
||||
GEOCODING_URL = "https://geocoding-api.open-meteo.com/v1/search"
|
||||
WEATHER_URL = "https://api.open-meteo.com/v1/forecast"
|
||||
AIR_QUALITY_URL = "https://air-quality-api.open-meteo.com/v1/air-quality"
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
@@ -194,9 +198,11 @@ class OpenMeteoProvider(WeatherProvider):
|
||||
"wind_speed_10m",
|
||||
"wind_direction_10m"
|
||||
],
|
||||
"daily": ["uv_index_max"],
|
||||
"timezone": self.timezone,
|
||||
"temperature_unit": "celsius",
|
||||
"wind_speed_unit": "kmh"
|
||||
"wind_speed_unit": "kmh",
|
||||
"forecast_days": 1
|
||||
}
|
||||
)
|
||||
response.raise_for_status()
|
||||
@@ -205,6 +211,12 @@ class OpenMeteoProvider(WeatherProvider):
|
||||
current = data.get("current", {})
|
||||
weather_code = current.get("weather_code", 0)
|
||||
|
||||
# Get today's UV index from daily data
|
||||
daily = data.get("daily", {})
|
||||
uv_index = None
|
||||
if daily.get("uv_index_max"):
|
||||
uv_index = daily["uv_index_max"][0]
|
||||
|
||||
return CurrentWeather(
|
||||
temperature=current.get("temperature_2m", 0.0),
|
||||
feels_like=current.get("apparent_temperature"),
|
||||
@@ -214,7 +226,8 @@ class OpenMeteoProvider(WeatherProvider):
|
||||
condition=WMO_CODE_MAP.get(weather_code, WeatherCondition.UNKNOWN),
|
||||
condition_text=WMO_DESCRIPTIONS.get(weather_code, "Unknown"),
|
||||
timestamp=datetime.now(),
|
||||
location=location.name
|
||||
location=location.name,
|
||||
uv_index=uv_index
|
||||
)
|
||||
except httpx.HTTPError as e:
|
||||
logger.error(f"Weather request failed for {location.name}: {e}")
|
||||
@@ -259,7 +272,8 @@ class OpenMeteoProvider(WeatherProvider):
|
||||
"temperature_2m_max",
|
||||
"temperature_2m_min",
|
||||
"precipitation_sum",
|
||||
"precipitation_probability_max"
|
||||
"precipitation_probability_max",
|
||||
"uv_index_max"
|
||||
],
|
||||
"timezone": self.timezone,
|
||||
"temperature_unit": "celsius",
|
||||
@@ -272,7 +286,14 @@ class OpenMeteoProvider(WeatherProvider):
|
||||
|
||||
# Parse current weather
|
||||
current_data = data.get("current", {})
|
||||
daily_data = data.get("daily", {})
|
||||
weather_code = current_data.get("weather_code", 0)
|
||||
|
||||
# Get today's UV from daily data
|
||||
uv_index = None
|
||||
if daily_data.get("uv_index_max"):
|
||||
uv_index = daily_data["uv_index_max"][0]
|
||||
|
||||
current = CurrentWeather(
|
||||
temperature=current_data.get("temperature_2m", 0.0),
|
||||
feels_like=current_data.get("apparent_temperature"),
|
||||
@@ -282,15 +303,16 @@ class OpenMeteoProvider(WeatherProvider):
|
||||
condition=WMO_CODE_MAP.get(weather_code, WeatherCondition.UNKNOWN),
|
||||
condition_text=WMO_DESCRIPTIONS.get(weather_code, "Unknown"),
|
||||
timestamp=datetime.now(),
|
||||
location=location.name
|
||||
location=location.name,
|
||||
uv_index=uv_index
|
||||
)
|
||||
|
||||
# Parse daily forecast
|
||||
daily_data = data.get("daily", {})
|
||||
daily = []
|
||||
dates = daily_data.get("time", [])
|
||||
for i, date_str in enumerate(dates):
|
||||
code = daily_data.get("weather_code", [])[i] if i < len(daily_data.get("weather_code", [])) else 0
|
||||
uv_max = daily_data.get("uv_index_max", [])[i] if i < len(daily_data.get("uv_index_max", [])) else None
|
||||
daily.append(DayForecast(
|
||||
date=datetime.fromisoformat(date_str),
|
||||
temp_high=daily_data.get("temperature_2m_max", [])[i] if i < len(daily_data.get("temperature_2m_max", [])) else 0.0,
|
||||
@@ -298,7 +320,8 @@ class OpenMeteoProvider(WeatherProvider):
|
||||
condition=WMO_CODE_MAP.get(code, WeatherCondition.UNKNOWN),
|
||||
condition_text=WMO_DESCRIPTIONS.get(code, "Unknown"),
|
||||
precipitation_chance=daily_data.get("precipitation_probability_max", [])[i] if i < len(daily_data.get("precipitation_probability_max", [])) else None,
|
||||
precipitation_mm=daily_data.get("precipitation_sum", [])[i] if i < len(daily_data.get("precipitation_sum", [])) else None
|
||||
precipitation_mm=daily_data.get("precipitation_sum", [])[i] if i < len(daily_data.get("precipitation_sum", [])) else None,
|
||||
uv_index_max=uv_max
|
||||
))
|
||||
|
||||
return WeatherForecast(
|
||||
@@ -309,3 +332,110 @@ class OpenMeteoProvider(WeatherProvider):
|
||||
except httpx.HTTPError as e:
|
||||
logger.error(f"Forecast request failed for {location.name}: {e}")
|
||||
raise ValueError(f"Failed to get forecast: {e}")
|
||||
|
||||
async def get_sun_times(self, location: GeoLocation) -> SunTimes:
|
||||
"""
|
||||
Get sunrise/sunset times for today.
|
||||
|
||||
Args:
|
||||
location: GeoLocation with lat/long
|
||||
|
||||
Returns:
|
||||
SunTimes with sunrise, sunset, and daylight duration
|
||||
|
||||
Raises:
|
||||
ValueError: If API request fails
|
||||
"""
|
||||
try:
|
||||
response = await self.client.get(
|
||||
self.WEATHER_URL,
|
||||
params={
|
||||
"latitude": location.latitude,
|
||||
"longitude": location.longitude,
|
||||
"daily": [
|
||||
"sunrise",
|
||||
"sunset",
|
||||
"daylight_duration"
|
||||
],
|
||||
"timezone": self.timezone,
|
||||
"forecast_days": 1
|
||||
}
|
||||
)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
|
||||
daily = data.get("daily", {})
|
||||
date_str = daily.get("time", [""])[0]
|
||||
sunrise_str = daily.get("sunrise", [""])[0]
|
||||
sunset_str = daily.get("sunset", [""])[0]
|
||||
daylight = daily.get("daylight_duration", [0])[0]
|
||||
|
||||
return SunTimes(
|
||||
location=location.name,
|
||||
date=datetime.fromisoformat(date_str) if date_str else datetime.now(),
|
||||
sunrise=datetime.fromisoformat(sunrise_str) if sunrise_str else datetime.now(),
|
||||
sunset=datetime.fromisoformat(sunset_str) if sunset_str else datetime.now(),
|
||||
daylight_duration=int(daylight) if daylight else 0
|
||||
)
|
||||
except httpx.HTTPError as e:
|
||||
logger.error(f"Sun times request failed for {location.name}: {e}")
|
||||
raise ValueError(f"Failed to get sun times: {e}")
|
||||
|
||||
async def get_air_quality(self, location: GeoLocation) -> AirQuality:
|
||||
"""
|
||||
Get current air quality for a location.
|
||||
|
||||
Args:
|
||||
location: GeoLocation with lat/long
|
||||
|
||||
Returns:
|
||||
AirQuality with pollutant measurements and AQI
|
||||
|
||||
Raises:
|
||||
ValueError: If API request fails
|
||||
"""
|
||||
try:
|
||||
response = await self.client.get(
|
||||
self.AIR_QUALITY_URL,
|
||||
params={
|
||||
"latitude": location.latitude,
|
||||
"longitude": location.longitude,
|
||||
"current": [
|
||||
"european_aqi",
|
||||
"us_aqi",
|
||||
"pm2_5",
|
||||
"pm10",
|
||||
"ozone",
|
||||
"nitrogen_dioxide",
|
||||
"sulphur_dioxide",
|
||||
"carbon_monoxide",
|
||||
"grass_pollen",
|
||||
"birch_pollen",
|
||||
"alder_pollen"
|
||||
],
|
||||
"timezone": self.timezone
|
||||
}
|
||||
)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
|
||||
current = data.get("current", {})
|
||||
|
||||
return AirQuality(
|
||||
location=location.name,
|
||||
timestamp=datetime.now(),
|
||||
aqi_european=current.get("european_aqi"),
|
||||
aqi_us=current.get("us_aqi"),
|
||||
pm2_5=current.get("pm2_5"),
|
||||
pm10=current.get("pm10"),
|
||||
ozone=current.get("ozone"),
|
||||
nitrogen_dioxide=current.get("nitrogen_dioxide"),
|
||||
sulphur_dioxide=current.get("sulphur_dioxide"),
|
||||
carbon_monoxide=current.get("carbon_monoxide"),
|
||||
pollen_grass=current.get("grass_pollen"),
|
||||
pollen_birch=current.get("birch_pollen"),
|
||||
pollen_alder=current.get("alder_pollen")
|
||||
)
|
||||
except httpx.HTTPError as e:
|
||||
logger.error(f"Air quality request failed for {location.name}: {e}")
|
||||
raise ValueError(f"Failed to get air quality: {e}")
|
||||
|
||||
+88
-1
@@ -18,7 +18,7 @@ from pathlib import Path
|
||||
|
||||
from src.config import Settings, get_settings, __version__
|
||||
from src.core.dependencies import (
|
||||
verify_api_key, QdrantDep, WikiJSDep, OllamaDep, Neo4jDep
|
||||
verify_api_key, QdrantDep, WikiJSDep, OllamaDep, Neo4jDep, PaperlessDep
|
||||
)
|
||||
from src.core.multi_tenancy import DEFAULT_USER
|
||||
|
||||
@@ -85,6 +85,14 @@ class HealthResponse(BaseModel):
|
||||
services: Dict[str, Any]
|
||||
|
||||
|
||||
class StatsResponse(BaseModel):
|
||||
"""System statistics response model."""
|
||||
neo4j: Dict[str, int]
|
||||
qdrant: Dict[str, Any]
|
||||
wiki_pages: int
|
||||
paperless: Dict[str, Any]
|
||||
|
||||
|
||||
# Routes
|
||||
@app.get("/", tags=["Root"])
|
||||
async def root() -> Dict[str, str]:
|
||||
@@ -141,6 +149,85 @@ async def health(settings: Settings = Depends(get_settings)) -> HealthResponse:
|
||||
)
|
||||
|
||||
|
||||
@app.get("/stats", response_model=StatsResponse, tags=["System"])
|
||||
async def stats(
|
||||
user: str = Query(default=DEFAULT_USER, description="User identifier"),
|
||||
neo4j: Neo4jDep = None,
|
||||
qdrant: QdrantDep = None,
|
||||
wikijs: WikiJSDep = None,
|
||||
paperless: PaperlessDep = None,
|
||||
api_key: str = Depends(verify_api_key)
|
||||
) -> StatsResponse:
|
||||
"""
|
||||
Get system statistics.
|
||||
|
||||
Returns counts for:
|
||||
- Neo4j: nodes by type (Document, Entity, Collection, Search)
|
||||
- Qdrant: vectors per collection
|
||||
- Wiki.js: total page count
|
||||
- Paperless: documents, tags, correspondents, document types
|
||||
"""
|
||||
# Neo4j node counts by label
|
||||
neo4j_stats = {}
|
||||
try:
|
||||
for label in ["Document", "Entity", "Collection", "Search"]:
|
||||
result = await neo4j.execute_query(
|
||||
f"MATCH (n:{label}) RETURN count(n) as count"
|
||||
)
|
||||
neo4j_stats[label.lower() + "_nodes"] = result[0]["count"] if result else 0
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to get Neo4j stats: {e}")
|
||||
neo4j_stats = {"error": str(e)}
|
||||
|
||||
# Qdrant collection stats
|
||||
qdrant_stats = {}
|
||||
try:
|
||||
collections = await qdrant.list_collections()
|
||||
qdrant_stats["collections"] = len(collections)
|
||||
qdrant_stats["total_vectors"] = sum(c.get("vectors_count", 0) for c in collections)
|
||||
qdrant_stats["by_collection"] = {
|
||||
c["name"]: c["vectors_count"] for c in collections
|
||||
}
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to get Qdrant stats: {e}")
|
||||
qdrant_stats = {"error": str(e)}
|
||||
|
||||
# Wiki.js page count
|
||||
wiki_pages = 0
|
||||
try:
|
||||
pages = await wikijs.list_all_pages(user)
|
||||
wiki_pages = len(pages)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to get Wiki.js stats: {e}")
|
||||
|
||||
# Paperless-ngx document stats
|
||||
paperless_stats = {}
|
||||
try:
|
||||
# Get document count (page_size=1 for efficiency, we just need the count)
|
||||
docs_result = await paperless.list_documents(page_size=1)
|
||||
paperless_stats["documents"] = docs_result.get("count", 0)
|
||||
|
||||
# Get metadata counts
|
||||
tags = await paperless.list_tags()
|
||||
paperless_stats["tags"] = len(tags)
|
||||
|
||||
correspondents = await paperless.list_correspondents()
|
||||
paperless_stats["correspondents"] = len(correspondents)
|
||||
|
||||
doc_types = await paperless.list_document_types()
|
||||
paperless_stats["document_types"] = len(doc_types)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to get Paperless stats: {e}")
|
||||
paperless_stats = {"error": str(e)}
|
||||
|
||||
return StatsResponse(
|
||||
neo4j=neo4j_stats,
|
||||
qdrant=qdrant_stats,
|
||||
wiki_pages=wiki_pages,
|
||||
paperless=paperless_stats
|
||||
)
|
||||
|
||||
|
||||
@app.post("/ingest/check-updates", tags=["Ingestion"])
|
||||
async def check_updates(
|
||||
documents: Dict[str, Any],
|
||||
|
||||
@@ -18,7 +18,9 @@ class VolatileNamespace(str, Enum):
|
||||
Each namespace can have different default TTLs and refresh schedules.
|
||||
"""
|
||||
# Real-time external data
|
||||
WEATHER = "weather" # Current conditions, forecasts
|
||||
WEATHER = "weather" # Current conditions (temperature, humidity, wind)
|
||||
FORECAST = "forecast" # Multi-day weather outlook
|
||||
SUN = "sun" # Sunrise, sunset, daylight duration
|
||||
NEWS = "news" # Headlines, breaking news
|
||||
FINANCIAL = "financial" # Stock prices, exchange rates, crypto
|
||||
TRANSIT = "transit" # Train/bus schedules, delays, disruptions
|
||||
@@ -37,7 +39,9 @@ class VolatileNamespace(str, Enum):
|
||||
|
||||
# Default TTLs per namespace (in seconds)
|
||||
NAMESPACE_DEFAULT_TTL: Dict[str, int] = {
|
||||
VolatileNamespace.WEATHER: 1800, # 30 min - weather changes slowly
|
||||
VolatileNamespace.WEATHER: 3600, # 1 hour - current conditions
|
||||
VolatileNamespace.FORECAST: 43200, # 12 hours - forecast stable longer
|
||||
VolatileNamespace.SUN: 86400, # 24 hours - sun times change daily
|
||||
VolatileNamespace.NEWS: 3600, # 1 hour - news cycles
|
||||
VolatileNamespace.FINANCIAL: 300, # 5 min - markets move fast
|
||||
VolatileNamespace.TRANSIT: 300, # 5 min - schedules update frequently
|
||||
|
||||
+139
-5
@@ -233,16 +233,16 @@ async def store_volatile(
|
||||
async def fetch_weather(
|
||||
city: str,
|
||||
user: str = Query(default=DEFAULT_USER, description="User identifier"),
|
||||
ttl: int = Query(default=86400, ge=60, le=604800, description="TTL in seconds"),
|
||||
ttl: int = Query(default=3600, ge=60, le=86400, description="TTL in seconds (default 1 hour)"),
|
||||
qdrant: QdrantDep = None,
|
||||
ollama: OllamaDep = None,
|
||||
api_key: str = Depends(verify_api_key)
|
||||
):
|
||||
"""
|
||||
Fetch current weather for a city and store in volatile cache.
|
||||
Fetch current weather conditions for a city and store in volatile cache.
|
||||
|
||||
Called by scheduler for prefetch or on-demand. Geocodes city name
|
||||
and fetches weather from Open-Meteo API.
|
||||
Stores temperature, humidity, wind, UV index. For forecasts use /fetch/forecast.
|
||||
Called by scheduler for hourly prefetch or on-demand.
|
||||
|
||||
**Example:**
|
||||
```
|
||||
@@ -257,7 +257,49 @@ async def fetch_weather(
|
||||
weather_provider=weather_provider,
|
||||
)
|
||||
|
||||
result = await fetch_service.fetch_weather(user, city, ttl=ttl)
|
||||
result = await fetch_service.fetch_current_weather(user, city, ttl=ttl)
|
||||
|
||||
if not result.success:
|
||||
raise HTTPException(status_code=500, detail=result.error)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"namespace": result.namespace,
|
||||
"key": result.key,
|
||||
"record": result.record,
|
||||
}
|
||||
|
||||
|
||||
@router.post("/fetch/forecast/{city}")
|
||||
async def fetch_forecast(
|
||||
city: str,
|
||||
user: str = Query(default=DEFAULT_USER, description="User identifier"),
|
||||
days: int = Query(default=7, ge=1, le=16, description="Forecast days (1-16)"),
|
||||
ttl: int = Query(default=43200, ge=60, le=604800, description="TTL in seconds (default 12 hours)"),
|
||||
qdrant: QdrantDep = None,
|
||||
ollama: OllamaDep = None,
|
||||
api_key: str = Depends(verify_api_key)
|
||||
):
|
||||
"""
|
||||
Fetch weather forecast for a city and store in volatile cache.
|
||||
|
||||
Stores multi-day outlook with highs/lows, precipitation, UV.
|
||||
For current conditions use /fetch/weather.
|
||||
|
||||
**Example:**
|
||||
```
|
||||
POST /volatile/fetch/forecast/amsterdam?user=jpmschweitzer&days=7
|
||||
```
|
||||
"""
|
||||
volatile_service = get_volatile_service(qdrant, ollama)
|
||||
weather_provider = get_weather_provider()
|
||||
|
||||
fetch_service = VolatileFetchService(
|
||||
volatile_service=volatile_service,
|
||||
weather_provider=weather_provider,
|
||||
)
|
||||
|
||||
result = await fetch_service.fetch_forecast(user, city, days=days, ttl=ttl)
|
||||
|
||||
if not result.success:
|
||||
raise HTTPException(status_code=500, detail=result.error)
|
||||
@@ -411,6 +453,98 @@ async def fetch_crypto(
|
||||
}
|
||||
|
||||
|
||||
@router.post("/fetch/sun/{city}")
|
||||
async def fetch_sun_times(
|
||||
city: str,
|
||||
user: str = Query(default=DEFAULT_USER, description="User identifier"),
|
||||
ttl: int = Query(default=86400, ge=60, le=604800, description="TTL in seconds"),
|
||||
qdrant: QdrantDep = None,
|
||||
ollama: OllamaDep = None,
|
||||
api_key: str = Depends(verify_api_key)
|
||||
):
|
||||
"""
|
||||
Fetch sunrise/sunset times for a city and store in volatile cache.
|
||||
|
||||
Fetches from Open-Meteo API. Useful for home automation triggers.
|
||||
|
||||
**Example:**
|
||||
```
|
||||
POST /volatile/fetch/sun/rotterdam?user=jpmschweitzer
|
||||
```
|
||||
|
||||
**Response data includes:**
|
||||
- sunrise/sunset times (both HH:MM and ISO formats)
|
||||
- daylight_duration_seconds
|
||||
- daylight_hours
|
||||
- Natural language text summary
|
||||
"""
|
||||
volatile_service = get_volatile_service(qdrant, ollama)
|
||||
weather_provider = get_weather_provider()
|
||||
|
||||
fetch_service = VolatileFetchService(
|
||||
volatile_service=volatile_service,
|
||||
weather_provider=weather_provider,
|
||||
)
|
||||
|
||||
result = await fetch_service.fetch_sun_times(user, city, ttl=ttl)
|
||||
|
||||
if not result.success:
|
||||
raise HTTPException(status_code=500, detail=result.error)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"namespace": result.namespace,
|
||||
"key": result.key,
|
||||
"record": result.record,
|
||||
}
|
||||
|
||||
|
||||
@router.post("/fetch/air_quality/{city}")
|
||||
async def fetch_air_quality(
|
||||
city: str,
|
||||
user: str = Query(default=DEFAULT_USER, description="User identifier"),
|
||||
ttl: int = Query(default=3600, ge=60, le=86400, description="TTL in seconds"),
|
||||
qdrant: QdrantDep = None,
|
||||
ollama: OllamaDep = None,
|
||||
api_key: str = Depends(verify_api_key)
|
||||
):
|
||||
"""
|
||||
Fetch air quality data for a city and store in volatile cache.
|
||||
|
||||
Fetches from Open-Meteo Air Quality API.
|
||||
|
||||
**Example:**
|
||||
```
|
||||
POST /volatile/fetch/air_quality/rotterdam?user=jpmschweitzer
|
||||
```
|
||||
|
||||
**Response data includes:**
|
||||
- European and US AQI indices
|
||||
- Pollutants: PM2.5, PM10, ozone, nitrogen dioxide, etc.
|
||||
- Pollen data (European locations, seasonal)
|
||||
- Natural language text summary
|
||||
"""
|
||||
volatile_service = get_volatile_service(qdrant, ollama)
|
||||
weather_provider = get_weather_provider()
|
||||
|
||||
fetch_service = VolatileFetchService(
|
||||
volatile_service=volatile_service,
|
||||
weather_provider=weather_provider,
|
||||
)
|
||||
|
||||
result = await fetch_service.fetch_air_quality(user, city, ttl=ttl)
|
||||
|
||||
if not result.success:
|
||||
raise HTTPException(status_code=500, detail=result.error)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"namespace": result.namespace,
|
||||
"key": result.key,
|
||||
"record": result.record,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/{namespace}/{key}", response_model=VolatileRecordResponse)
|
||||
async def get_record(
|
||||
namespace: str,
|
||||
|
||||
@@ -14,6 +14,9 @@ from src.apis import (
|
||||
AggregatedNewsProvider,
|
||||
AlphaVantageProvider,
|
||||
CurrentWeather,
|
||||
WeatherForecast,
|
||||
SunTimes,
|
||||
AirQuality,
|
||||
NewsFeed,
|
||||
StockQuote,
|
||||
)
|
||||
@@ -64,25 +67,25 @@ class VolatileFetchService:
|
||||
self.news = news_provider
|
||||
self.financial = financial_provider
|
||||
|
||||
async def fetch_weather(
|
||||
async def fetch_current_weather(
|
||||
self,
|
||||
user: str,
|
||||
city: str,
|
||||
ttl: int = 86400, # 24 hours
|
||||
ttl: int = 3600, # 1 hour
|
||||
) -> FetchResult:
|
||||
"""
|
||||
Fetch current weather for a city and store in volatile cache.
|
||||
Fetch current weather conditions for a city and store in volatile cache.
|
||||
|
||||
Args:
|
||||
user: User identifier
|
||||
city: City name (will be geocoded)
|
||||
ttl: Time-to-live in seconds
|
||||
ttl: Time-to-live in seconds (default 1 hour)
|
||||
|
||||
Returns:
|
||||
FetchResult with success status and stored record
|
||||
"""
|
||||
try:
|
||||
# Geocode city and get weather
|
||||
# Geocode city and get current conditions
|
||||
location = await self.weather.geocode(city)
|
||||
if not location:
|
||||
return FetchResult(
|
||||
@@ -92,19 +95,23 @@ class VolatileFetchService:
|
||||
error=f"Could not geocode city: {city}"
|
||||
)
|
||||
|
||||
weather = await self.weather.get_current(location)
|
||||
current = await self.weather.get_current(location)
|
||||
|
||||
# Generate natural language summary
|
||||
text = current.to_text()
|
||||
|
||||
# Convert to storage format
|
||||
data = {
|
||||
"temperature": weather.temperature,
|
||||
"feels_like": weather.feels_like,
|
||||
"humidity": weather.humidity,
|
||||
"wind_speed": weather.wind_speed,
|
||||
"wind_direction": weather.wind_direction,
|
||||
"conditions": weather.condition_text,
|
||||
"condition_code": weather.condition.value,
|
||||
"location": weather.location,
|
||||
"text": weather.to_text(),
|
||||
"temperature": current.temperature,
|
||||
"feels_like": current.feels_like,
|
||||
"humidity": current.humidity,
|
||||
"wind_speed": current.wind_speed,
|
||||
"wind_direction": current.wind_direction,
|
||||
"conditions": current.condition_text,
|
||||
"condition_code": current.condition.value,
|
||||
"uv_index": current.uv_index,
|
||||
"location": current.location,
|
||||
"text": text,
|
||||
}
|
||||
|
||||
# Store in volatile cache
|
||||
@@ -117,7 +124,7 @@ class VolatileFetchService:
|
||||
ttl=ttl,
|
||||
)
|
||||
|
||||
logger.info(f"Stored weather for {city} (user={user})")
|
||||
logger.info(f"Stored current weather for {city} (user={user})")
|
||||
return FetchResult(
|
||||
success=True,
|
||||
namespace="weather",
|
||||
@@ -126,7 +133,7 @@ class VolatileFetchService:
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to fetch weather for {city}: {e}")
|
||||
logger.error(f"Failed to fetch current weather for {city}: {e}")
|
||||
return FetchResult(
|
||||
success=False,
|
||||
namespace="weather",
|
||||
@@ -134,6 +141,94 @@ class VolatileFetchService:
|
||||
error=str(e)
|
||||
)
|
||||
|
||||
async def fetch_forecast(
|
||||
self,
|
||||
user: str,
|
||||
city: str,
|
||||
days: int = 7,
|
||||
ttl: int = 43200, # 12 hours
|
||||
) -> FetchResult:
|
||||
"""
|
||||
Fetch weather forecast for a city and store in volatile cache.
|
||||
|
||||
Args:
|
||||
user: User identifier
|
||||
city: City name (will be geocoded)
|
||||
days: Number of forecast days (1-16)
|
||||
ttl: Time-to-live in seconds (default 12 hours)
|
||||
|
||||
Returns:
|
||||
FetchResult with success status and stored record
|
||||
"""
|
||||
try:
|
||||
# Geocode city and get forecast
|
||||
location = await self.weather.geocode(city)
|
||||
if not location:
|
||||
return FetchResult(
|
||||
success=False,
|
||||
namespace="forecast",
|
||||
key=city.lower(),
|
||||
error=f"Could not geocode city: {city}"
|
||||
)
|
||||
|
||||
forecast = await self.weather.get_forecast(location, days=days)
|
||||
|
||||
# Build daily forecast array
|
||||
daily_forecasts = []
|
||||
for day in forecast.daily:
|
||||
daily_forecasts.append({
|
||||
"date": day.date.isoformat(),
|
||||
"day_name": day.date.strftime("%A"),
|
||||
"temp_high": day.temp_high,
|
||||
"temp_low": day.temp_low,
|
||||
"conditions": day.condition_text,
|
||||
"condition_code": day.condition.value,
|
||||
"precipitation_chance": day.precipitation_chance,
|
||||
"precipitation_mm": day.precipitation_mm,
|
||||
"uv_index_max": day.uv_index_max,
|
||||
})
|
||||
|
||||
# Generate natural language summary
|
||||
forecast_lines = [f"{city} {days}-day forecast:"]
|
||||
for day in forecast.daily:
|
||||
forecast_lines.append(day.to_text())
|
||||
text = "\n".join(forecast_lines)
|
||||
|
||||
# Convert to storage format
|
||||
data = {
|
||||
"days": days,
|
||||
"daily": daily_forecasts,
|
||||
"location": forecast.current.location,
|
||||
"text": text,
|
||||
}
|
||||
|
||||
# Store in volatile cache
|
||||
record = await self.volatile.store(
|
||||
user=user,
|
||||
namespace=VolatileNamespace.FORECAST,
|
||||
key=city.lower(),
|
||||
data=data,
|
||||
source="openmeteo",
|
||||
ttl=ttl,
|
||||
)
|
||||
|
||||
logger.info(f"Stored {days}-day forecast for {city} (user={user})")
|
||||
return FetchResult(
|
||||
success=True,
|
||||
namespace="forecast",
|
||||
key=city.lower(),
|
||||
record=record
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to fetch forecast for {city}: {e}")
|
||||
return FetchResult(
|
||||
success=False,
|
||||
namespace="forecast",
|
||||
key=city.lower(),
|
||||
error=str(e)
|
||||
)
|
||||
|
||||
async def fetch_news(
|
||||
self,
|
||||
user: str,
|
||||
@@ -357,3 +452,147 @@ class VolatileFetchService:
|
||||
key=f"{symbol.lower()}_{market.lower()}",
|
||||
error=str(e)
|
||||
)
|
||||
|
||||
async def fetch_sun_times(
|
||||
self,
|
||||
user: str,
|
||||
city: str,
|
||||
ttl: int = 86400, # 24 hours
|
||||
) -> FetchResult:
|
||||
"""
|
||||
Fetch sunrise/sunset times for a city and store in volatile cache.
|
||||
|
||||
Args:
|
||||
user: User identifier
|
||||
city: City name (will be geocoded)
|
||||
ttl: Time-to-live in seconds
|
||||
|
||||
Returns:
|
||||
FetchResult with success status and stored record
|
||||
"""
|
||||
try:
|
||||
# Geocode city and get sun times
|
||||
location = await self.weather.geocode(city)
|
||||
if not location:
|
||||
return FetchResult(
|
||||
success=False,
|
||||
namespace="sun",
|
||||
key=city.lower(),
|
||||
error=f"Could not geocode city: {city}"
|
||||
)
|
||||
|
||||
sun_times = await self.weather.get_sun_times(location)
|
||||
|
||||
# Convert to storage format
|
||||
data = {
|
||||
"location": sun_times.location,
|
||||
"date": sun_times.date.isoformat(),
|
||||
"sunrise": sun_times.sunrise.strftime("%H:%M"),
|
||||
"sunset": sun_times.sunset.strftime("%H:%M"),
|
||||
"sunrise_iso": sun_times.sunrise.isoformat(),
|
||||
"sunset_iso": sun_times.sunset.isoformat(),
|
||||
"daylight_duration_seconds": sun_times.daylight_duration,
|
||||
"daylight_hours": sun_times.daylight_duration / 3600,
|
||||
"text": sun_times.to_text(),
|
||||
}
|
||||
|
||||
# Store in volatile cache
|
||||
record = await self.volatile.store(
|
||||
user=user,
|
||||
namespace=VolatileNamespace.SUN,
|
||||
key=city.lower(),
|
||||
data=data,
|
||||
source="openmeteo",
|
||||
ttl=ttl,
|
||||
)
|
||||
|
||||
logger.info(f"Stored sun times for {city} (user={user})")
|
||||
return FetchResult(
|
||||
success=True,
|
||||
namespace="sun",
|
||||
key=city.lower(),
|
||||
record=record
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to fetch sun times for {city}: {e}")
|
||||
return FetchResult(
|
||||
success=False,
|
||||
namespace="sun",
|
||||
key=city.lower(),
|
||||
error=str(e)
|
||||
)
|
||||
|
||||
async def fetch_air_quality(
|
||||
self,
|
||||
user: str,
|
||||
city: str,
|
||||
ttl: int = 3600, # 1 hour
|
||||
) -> FetchResult:
|
||||
"""
|
||||
Fetch air quality data for a city and store in volatile cache.
|
||||
|
||||
Args:
|
||||
user: User identifier
|
||||
city: City name (will be geocoded)
|
||||
ttl: Time-to-live in seconds
|
||||
|
||||
Returns:
|
||||
FetchResult with success status and stored record
|
||||
"""
|
||||
try:
|
||||
# Geocode city and get air quality
|
||||
location = await self.weather.geocode(city)
|
||||
if not location:
|
||||
return FetchResult(
|
||||
success=False,
|
||||
namespace="air_quality",
|
||||
key=city.lower(),
|
||||
error=f"Could not geocode city: {city}"
|
||||
)
|
||||
|
||||
air_quality = await self.weather.get_air_quality(location)
|
||||
|
||||
# Convert to storage format
|
||||
data = {
|
||||
"location": air_quality.location,
|
||||
"aqi_european": air_quality.aqi_european,
|
||||
"aqi_us": air_quality.aqi_us,
|
||||
"pm2_5": air_quality.pm2_5,
|
||||
"pm10": air_quality.pm10,
|
||||
"ozone": air_quality.ozone,
|
||||
"nitrogen_dioxide": air_quality.nitrogen_dioxide,
|
||||
"sulphur_dioxide": air_quality.sulphur_dioxide,
|
||||
"carbon_monoxide": air_quality.carbon_monoxide,
|
||||
"pollen_grass": air_quality.pollen_grass,
|
||||
"pollen_birch": air_quality.pollen_birch,
|
||||
"pollen_alder": air_quality.pollen_alder,
|
||||
"text": air_quality.to_text(),
|
||||
}
|
||||
|
||||
# Store in volatile cache
|
||||
record = await self.volatile.store(
|
||||
user=user,
|
||||
namespace=VolatileNamespace.AIR_QUALITY,
|
||||
key=city.lower(),
|
||||
data=data,
|
||||
source="openmeteo",
|
||||
ttl=ttl,
|
||||
)
|
||||
|
||||
logger.info(f"Stored air quality for {city} (user={user})")
|
||||
return FetchResult(
|
||||
success=True,
|
||||
namespace="air_quality",
|
||||
key=city.lower(),
|
||||
record=record
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to fetch air quality for {city}: {e}")
|
||||
return FetchResult(
|
||||
success=False,
|
||||
namespace="air_quality",
|
||||
key=city.lower(),
|
||||
error=str(e)
|
||||
)
|
||||
|
||||
@@ -116,24 +116,6 @@ class WikiChangeListener:
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to handle notification: {e}", exc_info=True)
|
||||
|
||||
def _is_automated_user(self, email: str) -> bool:
|
||||
"""
|
||||
Check if email belongs to an automated system user.
|
||||
|
||||
These are edits made by library-desk via Wiki.js API (entity linking).
|
||||
We skip processing these to prevent loops.
|
||||
|
||||
Customize this list based on your Wiki.js username for library-desk.
|
||||
"""
|
||||
automated_users = [
|
||||
self.settings.wikijs_username, # Library-desk's Wiki.js API user
|
||||
"library-desk@system",
|
||||
"automation@system",
|
||||
"bot@system"
|
||||
]
|
||||
|
||||
return email.lower() in [u.lower() for u in automated_users]
|
||||
|
||||
def _is_recently_processed(self, page_id: int) -> bool:
|
||||
"""Check if page was processed recently (debouncing)."""
|
||||
if page_id not in self._recent_notifications:
|
||||
|
||||
@@ -98,7 +98,7 @@ class TestVolatileNamespaces:
|
||||
|
||||
def test_weather_default_ttl(self):
|
||||
"""Test weather namespace default TTL."""
|
||||
assert NAMESPACE_DEFAULT_TTL[VolatileNamespace.WEATHER] == 1800 # 30 min
|
||||
assert NAMESPACE_DEFAULT_TTL[VolatileNamespace.WEATHER] == 3600 # 1 hour (current conditions)
|
||||
|
||||
def test_financial_default_ttl(self):
|
||||
"""Test financial namespace default TTL."""
|
||||
@@ -110,7 +110,7 @@ class TestVolatileNamespaces:
|
||||
|
||||
def test_namespace_count(self):
|
||||
"""Test we have the expected number of namespaces."""
|
||||
assert len(VolatileNamespace) == 11
|
||||
assert len(VolatileNamespace) == 13 # Including SUN, FORECAST
|
||||
|
||||
|
||||
class TestVolatileListResponse:
|
||||
@@ -274,7 +274,7 @@ class TestVolatileService:
|
||||
def test_get_default_ttl_known_namespace(self, volatile_service):
|
||||
"""Test default TTL for known namespace."""
|
||||
ttl = volatile_service._get_default_ttl("weather")
|
||||
assert ttl == 1800 # Weather namespace default
|
||||
assert ttl == 3600 # Weather namespace default (1 hour)
|
||||
|
||||
def test_get_default_ttl_unknown_namespace(self, volatile_service):
|
||||
"""Test default TTL for unknown namespace."""
|
||||
@@ -366,7 +366,7 @@ class TestVolatileService:
|
||||
ttl=None # Not specified
|
||||
)
|
||||
|
||||
assert result.ttl == 1800 # Weather default
|
||||
assert result.ttl == 3600 # Weather default (1 hour)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_empty_collection(self, volatile_service, mock_qdrant, mock_ollama):
|
||||
|
||||
@@ -50,22 +50,6 @@ class TestWikiChangeListener:
|
||||
assert listener._debounce_seconds == 5
|
||||
assert len(listener._recent_notifications) == 0
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_automated_user_filtering(self, listener):
|
||||
"""Test that automated users are correctly identified."""
|
||||
# Automated users should be filtered
|
||||
assert listener._is_automated_user("librarian@schweitz.net") is True
|
||||
assert listener._is_automated_user("library-desk@system") is True
|
||||
assert listener._is_automated_user("automation@system") is True
|
||||
assert listener._is_automated_user("bot@system") is True
|
||||
|
||||
# Case insensitive
|
||||
assert listener._is_automated_user("LIBRARIAN@SCHWEITZ.NET") is True
|
||||
|
||||
# Regular users should not be filtered
|
||||
assert listener._is_automated_user("user@example.com") is False
|
||||
assert listener._is_automated_user("john@example.com") is False
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_debouncing_prevents_duplicates(self, listener):
|
||||
"""Test that debouncing prevents duplicate processing."""
|
||||
@@ -163,19 +147,29 @@ class TestWikiChangeListener:
|
||||
assert mock_process.call_args[1]['event'] == 'page.delete'
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_automated_user_notification_filtered(self, listener):
|
||||
"""Test that notifications from automated users are filtered out."""
|
||||
async def test_any_user_notification_processed(self, listener):
|
||||
"""Test that notifications are processed regardless of user email.
|
||||
|
||||
Note: The user_email in PostgreSQL notifications is the page CREATOR,
|
||||
not the editor. We cannot filter by user email because:
|
||||
- A page created by 'librarian' but edited by a human should be processed
|
||||
- Filtering by creator would break legitimate page ingestion
|
||||
Loop prevention is handled by debouncing instead.
|
||||
"""
|
||||
mock_connection = AsyncMock()
|
||||
|
||||
with patch.object(listener, '_process_page_change', new_callable=AsyncMock) as mock_process:
|
||||
# Notification from automated user should be skipped
|
||||
# Even system user notifications should be processed
|
||||
# (debouncing handles loop prevention, not user filtering)
|
||||
await listener._handle_notification(
|
||||
mock_connection, 1234, 'wiki_page_changes',
|
||||
'UPDATE:123:librarian@schweitz.net'
|
||||
)
|
||||
|
||||
# Process should NOT be called
|
||||
mock_process.assert_not_called()
|
||||
# Process SHOULD be called (user filtering is not used)
|
||||
mock_process.assert_called_once()
|
||||
assert mock_process.call_args[1]['page_id'] == 123
|
||||
assert mock_process.call_args[1]['event'] == 'page.update'
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_duplicate_notification_filtered(self, listener):
|
||||
|
||||
Reference in New Issue
Block a user