Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
152b2f28c4 | ||
|
|
46b9bcd7a0 | ||
|
|
68eb1add3d | ||
|
|
d6b30570a0 | ||
|
|
6b0530ed79 | ||
|
|
0a8c2639a0 |
@@ -5,6 +5,56 @@ All notable changes to Library Desk will be documented in this file.
|
|||||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||||
|
|
||||||
|
## [1.6.2] - 2025-12-30
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- **System Statistics Endpoint** - `GET /stats`
|
||||||
|
- Neo4j: node counts by type (Document, Entity, Collection, Search)
|
||||||
|
- Qdrant: collection counts, total vectors, per-collection breakdown
|
||||||
|
- Wiki.js: total page count
|
||||||
|
- Paperless: documents, tags, correspondents, document types
|
||||||
|
|
||||||
|
- **Weather/Forecast Separation** - Split weather into two distinct namespaces
|
||||||
|
- `POST /volatile/fetch/weather/{city}` - Current conditions only (1hr TTL)
|
||||||
|
- `POST /volatile/fetch/forecast/{city}` - 7-day outlook (12hr TTL)
|
||||||
|
- Different update frequencies for efficient caching
|
||||||
|
- `FORECAST` namespace added to volatile namespaces
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- Weather namespace TTL changed from 30 minutes to 1 hour (current conditions)
|
||||||
|
- Forecast data now stored separately with 12 hour TTL
|
||||||
|
|
||||||
|
## [1.6.1] - 2025-12-30
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- **Weather Forecast Support** - Enhanced weather fetch with 7-day daily forecasts
|
||||||
|
- Current conditions now include UV index
|
||||||
|
- Daily forecasts with high/low temps, conditions, precipitation chance, UV max
|
||||||
|
- Natural language text summary with multi-day outlook
|
||||||
|
- **Sun Times Endpoint** - `POST /volatile/fetch/sun/{city}`
|
||||||
|
- Sunrise and sunset times (HH:MM and ISO formats)
|
||||||
|
- Daylight duration in seconds and hours
|
||||||
|
- Separate volatile namespace with 24hr TTL
|
||||||
|
- Useful for home automation light triggers
|
||||||
|
- **Air Quality Endpoint** - `POST /volatile/fetch/air_quality/{city}`
|
||||||
|
- European and US AQI indices
|
||||||
|
- Pollutants: PM2.5, PM10, ozone, nitrogen dioxide, sulphur dioxide, carbon monoxide
|
||||||
|
- Pollen data (grass, birch, alder) for European locations (seasonal)
|
||||||
|
- Hourly refresh (1hr TTL)
|
||||||
|
- **New Base Models**
|
||||||
|
- `SunTimes` dataclass for sunrise/sunset data
|
||||||
|
- `AirQuality` dataclass with AQI and pollutants
|
||||||
|
- `AirQualityProvider` abstract interface
|
||||||
|
- **New Volatile Namespace** - `SUN` for sunrise/sunset times (86400s default TTL)
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- Weather fetch now uses `get_forecast()` instead of `get_current()` for richer data
|
||||||
|
- `OpenMeteoProvider` now implements both `WeatherProvider` and `AirQualityProvider`
|
||||||
|
|
||||||
## [1.6.0] - 2025-12-29
|
## [1.6.0] - 2025-12-29
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
|
|||||||
@@ -41,12 +41,3 @@ Check for duplicate or highly similar documents using vector similarity and grap
|
|||||||
3. Check graph relationships
|
3. Check graph relationships
|
||||||
4. Return candidates with similarity scores
|
4. Return candidates with similarity scores
|
||||||
|
|
||||||
## System Statistics
|
|
||||||
|
|
||||||
#### `GET /stats`
|
|
||||||
Get system statistics (wiki pages, neo4j nodes, qdrant vectors).
|
|
||||||
|
|
||||||
**Implementation needed:**
|
|
||||||
- Query Neo4j for node count
|
|
||||||
- Query Qdrant for vector count
|
|
||||||
- Query Wiki.js for page count
|
|
||||||
|
|||||||
@@ -16,8 +16,16 @@ This document outlines the implementation of "remember" triggers for the memory
|
|||||||
| Memory Tier | Remember Trigger | Recall | Status |
|
| Memory Tier | Remember Trigger | Recall | Status |
|
||||||
|-------------|------------------|--------|--------|
|
|-------------|------------------|--------|--------|
|
||||||
| Wiki | Wiki.js webhook, Consolidation | HybridRAG vector+graph | ✅ Complete |
|
| Wiki | Wiki.js webhook, Consolidation | HybridRAG vector+graph | ✅ Complete |
|
||||||
| Documents | Paperless webhook | Manual `/documents/search` | ⚠️ Partial (no HybridRAG recall) |
|
| Documents | Paperless webhook | HybridRAG document search | ✅ Complete (v1.6.0) |
|
||||||
| Volatile | Manual `/volatile/store` only | HybridRAG volatile search | ⚠️ Partial (no auto-triggers) |
|
| Volatile | Scheduler prefetch, HybridRAG post-processor | HybridRAG volatile search | ✅ Complete (v1.6.0) |
|
||||||
|
|
||||||
|
### Implementation Summary (v1.6.0)
|
||||||
|
|
||||||
|
- **Settings DB**: Central `system_settings` PostgreSQL database with `SettingsClient`
|
||||||
|
- **Phase A**: Volatile fetch endpoints (`/volatile/fetch/{namespace}/{key}`) with weather, news, financial providers
|
||||||
|
- **Phase B**: Unified memory routing in consolidation service (wiki/volatile/file/prefetch/skip classification)
|
||||||
|
- **Phase C**: Document recall in HybridRAG (4-source parallel retrieval)
|
||||||
|
- **Scheduler Integration**: `SchedulerClient` for external scheduler task registration
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -662,46 +670,60 @@ document_threshold: float = Field(default=0.6)
|
|||||||
|
|
||||||
## Implementation Order
|
## Implementation Order
|
||||||
|
|
||||||
| Phase | Priority | Effort | Description |
|
| Phase | Priority | Effort | Description | Status |
|
||||||
|-------|----------|--------|-------------|
|
|-------|----------|--------|-------------|--------|
|
||||||
| **Settings DB** | High | Low | PostgreSQL schema + settings client |
|
| **Settings DB** | High | Low | PostgreSQL schema + settings client | ✅ v1.5.0 |
|
||||||
| **B.4** | High | Low | Scheduler client |
|
| **B.4** | High | Low | Scheduler client | ✅ v1.6.0 |
|
||||||
| **B.1-B.3** | High | Medium | HybridRAG post-processor |
|
| **B.1-B.3** | High | Medium | HybridRAG post-processor | ✅ v1.6.0 |
|
||||||
| **B.5** | High | Low | Config options |
|
| **B.5** | High | Low | Config options | ✅ v1.6.0 |
|
||||||
| **C.1-C.2** | High | Low | Document recall in HybridRAG |
|
| **C.1-C.2** | High | Low | Document recall in HybridRAG | ✅ v1.6.0 |
|
||||||
| **A.1** | Medium | Low | `/volatile/fetch` endpoint |
|
| **A.1** | Medium | Low | `/volatile/fetch` endpoint | ✅ v1.6.0 |
|
||||||
| **A.2** | Medium | Medium | Weather + News API clients |
|
| **A.2** | Medium | Medium | Weather + News API clients | ✅ v1.5.0 |
|
||||||
| **A.3** | Medium | Low | Fetch service |
|
| **A.3** | Medium | Low | Fetch service | ✅ v1.6.0 |
|
||||||
|
|
||||||
|
### Remaining Work
|
||||||
|
|
||||||
|
| Item | Description | Status |
|
||||||
|
|------|-------------|--------|
|
||||||
|
| File upload | Download PDFs and upload to Paperless | ⚠️ Placeholder (logs only) |
|
||||||
|
| Prefetch patterns | More sophisticated pattern detection | Optional enhancement |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Files Summary
|
## Files Summary
|
||||||
|
|
||||||
### New Files
|
### New Files (Implemented)
|
||||||
|
|
||||||
| Path | Purpose |
|
| Path | Purpose | Version |
|
||||||
|------|---------|
|
|------|---------|---------|
|
||||||
| `src/clients/settings_client.py` | Read-only central settings access |
|
| `src/clients/settings_client.py` | Central settings database access | v1.5.0 |
|
||||||
| `src/clients/scheduler_client.py` | Register/manage scheduler tasks |
|
| `src/clients/scheduler_client.py` | External scheduler task management | v1.6.0 |
|
||||||
| `src/clients/weather_client.py` | Open-Meteo API (with geocoding) |
|
| `src/apis/__init__.py` | External API providers package | v1.5.0 |
|
||||||
| `src/clients/news_client.py` | NOS.nl RSS feeds |
|
| `src/apis/base.py` | Abstract base classes for providers | v1.5.0 |
|
||||||
| `src/services/volatile_fetch_service.py` | Orchestrates fetch + store |
|
| `src/apis/weather.py` | OpenMeteoProvider (geocoding + forecast) | v1.5.0 |
|
||||||
|
| `src/apis/news.py` | AggregatedNewsProvider | v1.5.0 |
|
||||||
|
| `src/apis/nos.py` | NOSProvider (Dutch news RSS) | v1.5.0 |
|
||||||
|
| `src/apis/bbc.py` | BBCProvider (English news RSS) | v1.5.0 |
|
||||||
|
| `src/apis/financial.py` | AlphaVantageProvider (stocks/crypto) | v1.5.0 |
|
||||||
|
| `src/services/volatile_fetch_service.py` | Orchestrates fetch + store | v1.6.0 |
|
||||||
|
|
||||||
### Modified Files
|
### Modified Files
|
||||||
|
|
||||||
| Path | Changes |
|
| Path | Changes | Version |
|
||||||
|------|---------|
|
|------|---------|---------|
|
||||||
| `src/services/hybrid_rag_service.py` | Post-processor, prefetch detection, document search |
|
| `src/services/hybrid_rag_service.py` | Document search (4-source parallel retrieval) | v1.6.0 |
|
||||||
| `src/models/hybrid_rag.py` | Memory config options |
|
| `src/services/consolidation_service.py` | Unified memory routing, scheduler integration | v1.6.0 |
|
||||||
| `src/routers/volatile.py` | `/volatile/fetch/{namespace}/{key}` |
|
| `src/models/hybrid_rag.py` | Document config options (`enable_documents`, `document_limit`) | v1.6.0 |
|
||||||
| `src/core/dependencies.py` | Settings client, scheduler client DI |
|
| `src/models/consolidation.py` | Memory routing models | v1.6.0 |
|
||||||
| `src/config.py` | `SYSTEM_SETTINGS_*` connection vars |
|
| `src/routers/volatile.py` | `/volatile/fetch/{namespace}/{key}` endpoints | v1.6.0 |
|
||||||
|
| `src/core/dependencies.py` | Settings, scheduler, provider DI | v1.5.0-v1.6.0 |
|
||||||
|
| `src/config.py` | `SYSTEM_SETTINGS_*`, `SCHEDULER_URL` vars | v1.5.0-v1.6.0 |
|
||||||
|
|
||||||
### Database
|
### Database
|
||||||
|
|
||||||
| Item | Details |
|
| Item | Details |
|
||||||
|------|---------|
|
|------|---------|
|
||||||
| Database | `system_settings` (PostgreSQL) |
|
| Database | `system_settings` (PostgreSQL on postgres-shared) |
|
||||||
| Table | `settings (key, value JSONB, category, ...)` |
|
| Table | `settings (key, user_scope, value JSONB, schema JSONB, ...)` |
|
||||||
| Library-desk user | `library_desk_ro` (read-only) |
|
| Library-desk access | Read-only via `SettingsClient` |
|
||||||
| Management | Direct psql commands |
|
| Management | Direct psql commands (future: CRUD manager UI) |
|
||||||
|
|||||||
+1
-1
@@ -1,6 +1,6 @@
|
|||||||
[project]
|
[project]
|
||||||
name = "library-desk"
|
name = "library-desk"
|
||||||
version = "1.6.0"
|
version = "1.6.2"
|
||||||
description = "Coordination service for The Library system - HybridRAG queries, document ingestion, entity extraction, and knowledge consolidation"
|
description = "Coordination service for The Library system - HybridRAG queries, document ingestion, entity extraction, and knowledge consolidation"
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
requires-python = ">=3.12"
|
requires-python = ">=3.12"
|
||||||
|
|||||||
@@ -27,6 +27,9 @@ from .base import (
|
|||||||
DayForecast,
|
DayForecast,
|
||||||
WeatherForecast,
|
WeatherForecast,
|
||||||
GeoLocation,
|
GeoLocation,
|
||||||
|
SunTimes,
|
||||||
|
# Air quality models
|
||||||
|
AirQuality,
|
||||||
# News models
|
# News models
|
||||||
NewsItem,
|
NewsItem,
|
||||||
NewsFeed,
|
NewsFeed,
|
||||||
@@ -34,6 +37,7 @@ from .base import (
|
|||||||
StockQuote,
|
StockQuote,
|
||||||
# Abstract providers
|
# Abstract providers
|
||||||
WeatherProvider,
|
WeatherProvider,
|
||||||
|
AirQualityProvider,
|
||||||
NewsProvider,
|
NewsProvider,
|
||||||
FinancialProvider,
|
FinancialProvider,
|
||||||
)
|
)
|
||||||
@@ -53,8 +57,12 @@ __all__ = [
|
|||||||
"DayForecast",
|
"DayForecast",
|
||||||
"WeatherForecast",
|
"WeatherForecast",
|
||||||
"GeoLocation",
|
"GeoLocation",
|
||||||
|
"SunTimes",
|
||||||
"WeatherProvider",
|
"WeatherProvider",
|
||||||
"OpenMeteoProvider",
|
"OpenMeteoProvider",
|
||||||
|
# Air quality
|
||||||
|
"AirQuality",
|
||||||
|
"AirQualityProvider",
|
||||||
# News
|
# News
|
||||||
"NewsItem",
|
"NewsItem",
|
||||||
"NewsFeed",
|
"NewsFeed",
|
||||||
|
|||||||
+102
-4
@@ -44,14 +44,18 @@ class CurrentWeather:
|
|||||||
condition_text: str # Human-readable description
|
condition_text: str # Human-readable description
|
||||||
timestamp: datetime
|
timestamp: datetime
|
||||||
location: str # City/location name
|
location: str # City/location name
|
||||||
|
uv_index: Optional[float] = None # UV index 0-11+
|
||||||
|
|
||||||
def to_text(self) -> str:
|
def to_text(self) -> str:
|
||||||
"""Generate natural language description."""
|
"""Generate natural language description."""
|
||||||
return (
|
parts = [
|
||||||
f"Currently {self.temperature:.1f}°C "
|
f"Currently {self.temperature:.1f}°C",
|
||||||
f"({self.condition_text}) in {self.location}. "
|
f"({self.condition_text}) in {self.location}.",
|
||||||
f"Humidity {self.humidity}%, wind {self.wind_speed:.0f} km/h."
|
f"Humidity {self.humidity}%, wind {self.wind_speed:.0f} km/h."
|
||||||
)
|
]
|
||||||
|
if self.uv_index is not None:
|
||||||
|
parts.append(f"UV index: {self.uv_index:.0f}.")
|
||||||
|
return " ".join(parts)
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
@@ -64,6 +68,14 @@ class DayForecast:
|
|||||||
condition_text: str
|
condition_text: str
|
||||||
precipitation_chance: Optional[int] # Percentage 0-100
|
precipitation_chance: Optional[int] # Percentage 0-100
|
||||||
precipitation_mm: Optional[float]
|
precipitation_mm: Optional[float]
|
||||||
|
uv_index_max: Optional[float] = None # Max UV index for the day
|
||||||
|
|
||||||
|
def to_text(self) -> str:
|
||||||
|
"""Generate natural language description."""
|
||||||
|
date_str = self.date.strftime("%A") # Day name
|
||||||
|
precip = f", {self.precipitation_chance}% rain" if self.precipitation_chance else ""
|
||||||
|
uv = f", UV {self.uv_index_max:.0f}" if self.uv_index_max else ""
|
||||||
|
return f"{date_str}: {self.temp_high:.0f}°/{self.temp_low:.0f}°C, {self.condition_text}{precip}{uv}"
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
@@ -84,6 +96,78 @@ class GeoLocation:
|
|||||||
admin_area: Optional[str] = None # State/province
|
admin_area: Optional[str] = None # State/province
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class SunTimes:
|
||||||
|
"""Sunrise/sunset times for a location."""
|
||||||
|
location: str
|
||||||
|
date: datetime
|
||||||
|
sunrise: datetime
|
||||||
|
sunset: datetime
|
||||||
|
daylight_duration: int # seconds
|
||||||
|
solar_noon: Optional[datetime] = None
|
||||||
|
|
||||||
|
def to_text(self) -> str:
|
||||||
|
"""Generate natural language description."""
|
||||||
|
sunrise_str = self.sunrise.strftime("%H:%M")
|
||||||
|
sunset_str = self.sunset.strftime("%H:%M")
|
||||||
|
hours = self.daylight_duration // 3600
|
||||||
|
minutes = (self.daylight_duration % 3600) // 60
|
||||||
|
return (
|
||||||
|
f"Sun times for {self.location} on {self.date.strftime('%A %d %B')}: "
|
||||||
|
f"Sunrise at {sunrise_str}, sunset at {sunset_str}. "
|
||||||
|
f"Daylight duration: {hours}h {minutes}m."
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class AirQuality:
|
||||||
|
"""Air quality measurements for a location."""
|
||||||
|
location: str
|
||||||
|
timestamp: datetime
|
||||||
|
aqi_european: Optional[int] # European AQI 0-500+
|
||||||
|
aqi_us: Optional[int] # US AQI 0-500+
|
||||||
|
pm2_5: Optional[float] # µg/m³
|
||||||
|
pm10: Optional[float] # µg/m³
|
||||||
|
ozone: Optional[float] # µg/m³
|
||||||
|
nitrogen_dioxide: Optional[float] # µg/m³
|
||||||
|
sulphur_dioxide: Optional[float] # µg/m³
|
||||||
|
carbon_monoxide: Optional[float] # µg/m³
|
||||||
|
# Pollen (European data only, seasonal)
|
||||||
|
pollen_grass: Optional[float] = None
|
||||||
|
pollen_birch: Optional[float] = None
|
||||||
|
pollen_alder: Optional[float] = None
|
||||||
|
|
||||||
|
def to_text(self) -> str:
|
||||||
|
"""Generate natural language description."""
|
||||||
|
parts = [f"Air quality in {self.location}:"]
|
||||||
|
if self.aqi_european is not None:
|
||||||
|
level = self._aqi_level(self.aqi_european)
|
||||||
|
parts.append(f"European AQI {self.aqi_european} ({level}).")
|
||||||
|
if self.pm2_5 is not None:
|
||||||
|
parts.append(f"PM2.5: {self.pm2_5:.1f} µg/m³.")
|
||||||
|
if self.pm10 is not None:
|
||||||
|
parts.append(f"PM10: {self.pm10:.1f} µg/m³.")
|
||||||
|
if self.ozone is not None:
|
||||||
|
parts.append(f"Ozone: {self.ozone:.1f} µg/m³.")
|
||||||
|
return " ".join(parts)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _aqi_level(aqi: int) -> str:
|
||||||
|
"""Convert AQI to human-readable level."""
|
||||||
|
if aqi <= 20:
|
||||||
|
return "good"
|
||||||
|
elif aqi <= 40:
|
||||||
|
return "fair"
|
||||||
|
elif aqi <= 60:
|
||||||
|
return "moderate"
|
||||||
|
elif aqi <= 80:
|
||||||
|
return "poor"
|
||||||
|
elif aqi <= 100:
|
||||||
|
return "very poor"
|
||||||
|
else:
|
||||||
|
return "hazardous"
|
||||||
|
|
||||||
|
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
# News Models
|
# News Models
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
@@ -163,6 +247,11 @@ class WeatherProvider(ABC):
|
|||||||
"""Get weather forecast for a location."""
|
"""Get weather forecast for a location."""
|
||||||
pass
|
pass
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
async def get_sun_times(self, location: GeoLocation) -> SunTimes:
|
||||||
|
"""Get sunrise/sunset times for today."""
|
||||||
|
pass
|
||||||
|
|
||||||
async def get_weather_for_city(self, city: str) -> CurrentWeather:
|
async def get_weather_for_city(self, city: str) -> CurrentWeather:
|
||||||
"""Convenience method: geocode and get current weather."""
|
"""Convenience method: geocode and get current weather."""
|
||||||
location = await self.geocode(city)
|
location = await self.geocode(city)
|
||||||
@@ -171,6 +260,15 @@ class WeatherProvider(ABC):
|
|||||||
return await self.get_current(location)
|
return await self.get_current(location)
|
||||||
|
|
||||||
|
|
||||||
|
class AirQualityProvider(ABC):
|
||||||
|
"""Abstract base class for air quality API providers."""
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
async def get_air_quality(self, location: GeoLocation) -> AirQuality:
|
||||||
|
"""Get current air quality for a location."""
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
class NewsProvider(ABC):
|
class NewsProvider(ABC):
|
||||||
"""Abstract base class for news API providers."""
|
"""Abstract base class for news API providers."""
|
||||||
|
|
||||||
|
|||||||
+138
-8
@@ -14,11 +14,14 @@ from typing import Optional
|
|||||||
|
|
||||||
from .base import (
|
from .base import (
|
||||||
WeatherProvider,
|
WeatherProvider,
|
||||||
|
AirQualityProvider,
|
||||||
WeatherCondition,
|
WeatherCondition,
|
||||||
CurrentWeather,
|
CurrentWeather,
|
||||||
DayForecast,
|
DayForecast,
|
||||||
WeatherForecast,
|
WeatherForecast,
|
||||||
GeoLocation,
|
GeoLocation,
|
||||||
|
SunTimes,
|
||||||
|
AirQuality,
|
||||||
)
|
)
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
@@ -89,11 +92,12 @@ WMO_DESCRIPTIONS: dict[int, str] = {
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
class OpenMeteoProvider(WeatherProvider):
|
class OpenMeteoProvider(WeatherProvider, AirQualityProvider):
|
||||||
"""Open-Meteo weather API implementation."""
|
"""Open-Meteo weather and air quality API implementation."""
|
||||||
|
|
||||||
GEOCODING_URL = "https://geocoding-api.open-meteo.com/v1/search"
|
GEOCODING_URL = "https://geocoding-api.open-meteo.com/v1/search"
|
||||||
WEATHER_URL = "https://api.open-meteo.com/v1/forecast"
|
WEATHER_URL = "https://api.open-meteo.com/v1/forecast"
|
||||||
|
AIR_QUALITY_URL = "https://air-quality-api.open-meteo.com/v1/air-quality"
|
||||||
|
|
||||||
def __init__(
|
def __init__(
|
||||||
self,
|
self,
|
||||||
@@ -194,9 +198,11 @@ class OpenMeteoProvider(WeatherProvider):
|
|||||||
"wind_speed_10m",
|
"wind_speed_10m",
|
||||||
"wind_direction_10m"
|
"wind_direction_10m"
|
||||||
],
|
],
|
||||||
|
"daily": ["uv_index_max"],
|
||||||
"timezone": self.timezone,
|
"timezone": self.timezone,
|
||||||
"temperature_unit": "celsius",
|
"temperature_unit": "celsius",
|
||||||
"wind_speed_unit": "kmh"
|
"wind_speed_unit": "kmh",
|
||||||
|
"forecast_days": 1
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
response.raise_for_status()
|
response.raise_for_status()
|
||||||
@@ -205,6 +211,12 @@ class OpenMeteoProvider(WeatherProvider):
|
|||||||
current = data.get("current", {})
|
current = data.get("current", {})
|
||||||
weather_code = current.get("weather_code", 0)
|
weather_code = current.get("weather_code", 0)
|
||||||
|
|
||||||
|
# Get today's UV index from daily data
|
||||||
|
daily = data.get("daily", {})
|
||||||
|
uv_index = None
|
||||||
|
if daily.get("uv_index_max"):
|
||||||
|
uv_index = daily["uv_index_max"][0]
|
||||||
|
|
||||||
return CurrentWeather(
|
return CurrentWeather(
|
||||||
temperature=current.get("temperature_2m", 0.0),
|
temperature=current.get("temperature_2m", 0.0),
|
||||||
feels_like=current.get("apparent_temperature"),
|
feels_like=current.get("apparent_temperature"),
|
||||||
@@ -214,7 +226,8 @@ class OpenMeteoProvider(WeatherProvider):
|
|||||||
condition=WMO_CODE_MAP.get(weather_code, WeatherCondition.UNKNOWN),
|
condition=WMO_CODE_MAP.get(weather_code, WeatherCondition.UNKNOWN),
|
||||||
condition_text=WMO_DESCRIPTIONS.get(weather_code, "Unknown"),
|
condition_text=WMO_DESCRIPTIONS.get(weather_code, "Unknown"),
|
||||||
timestamp=datetime.now(),
|
timestamp=datetime.now(),
|
||||||
location=location.name
|
location=location.name,
|
||||||
|
uv_index=uv_index
|
||||||
)
|
)
|
||||||
except httpx.HTTPError as e:
|
except httpx.HTTPError as e:
|
||||||
logger.error(f"Weather request failed for {location.name}: {e}")
|
logger.error(f"Weather request failed for {location.name}: {e}")
|
||||||
@@ -259,7 +272,8 @@ class OpenMeteoProvider(WeatherProvider):
|
|||||||
"temperature_2m_max",
|
"temperature_2m_max",
|
||||||
"temperature_2m_min",
|
"temperature_2m_min",
|
||||||
"precipitation_sum",
|
"precipitation_sum",
|
||||||
"precipitation_probability_max"
|
"precipitation_probability_max",
|
||||||
|
"uv_index_max"
|
||||||
],
|
],
|
||||||
"timezone": self.timezone,
|
"timezone": self.timezone,
|
||||||
"temperature_unit": "celsius",
|
"temperature_unit": "celsius",
|
||||||
@@ -272,7 +286,14 @@ class OpenMeteoProvider(WeatherProvider):
|
|||||||
|
|
||||||
# Parse current weather
|
# Parse current weather
|
||||||
current_data = data.get("current", {})
|
current_data = data.get("current", {})
|
||||||
|
daily_data = data.get("daily", {})
|
||||||
weather_code = current_data.get("weather_code", 0)
|
weather_code = current_data.get("weather_code", 0)
|
||||||
|
|
||||||
|
# Get today's UV from daily data
|
||||||
|
uv_index = None
|
||||||
|
if daily_data.get("uv_index_max"):
|
||||||
|
uv_index = daily_data["uv_index_max"][0]
|
||||||
|
|
||||||
current = CurrentWeather(
|
current = CurrentWeather(
|
||||||
temperature=current_data.get("temperature_2m", 0.0),
|
temperature=current_data.get("temperature_2m", 0.0),
|
||||||
feels_like=current_data.get("apparent_temperature"),
|
feels_like=current_data.get("apparent_temperature"),
|
||||||
@@ -282,15 +303,16 @@ class OpenMeteoProvider(WeatherProvider):
|
|||||||
condition=WMO_CODE_MAP.get(weather_code, WeatherCondition.UNKNOWN),
|
condition=WMO_CODE_MAP.get(weather_code, WeatherCondition.UNKNOWN),
|
||||||
condition_text=WMO_DESCRIPTIONS.get(weather_code, "Unknown"),
|
condition_text=WMO_DESCRIPTIONS.get(weather_code, "Unknown"),
|
||||||
timestamp=datetime.now(),
|
timestamp=datetime.now(),
|
||||||
location=location.name
|
location=location.name,
|
||||||
|
uv_index=uv_index
|
||||||
)
|
)
|
||||||
|
|
||||||
# Parse daily forecast
|
# Parse daily forecast
|
||||||
daily_data = data.get("daily", {})
|
|
||||||
daily = []
|
daily = []
|
||||||
dates = daily_data.get("time", [])
|
dates = daily_data.get("time", [])
|
||||||
for i, date_str in enumerate(dates):
|
for i, date_str in enumerate(dates):
|
||||||
code = daily_data.get("weather_code", [])[i] if i < len(daily_data.get("weather_code", [])) else 0
|
code = daily_data.get("weather_code", [])[i] if i < len(daily_data.get("weather_code", [])) else 0
|
||||||
|
uv_max = daily_data.get("uv_index_max", [])[i] if i < len(daily_data.get("uv_index_max", [])) else None
|
||||||
daily.append(DayForecast(
|
daily.append(DayForecast(
|
||||||
date=datetime.fromisoformat(date_str),
|
date=datetime.fromisoformat(date_str),
|
||||||
temp_high=daily_data.get("temperature_2m_max", [])[i] if i < len(daily_data.get("temperature_2m_max", [])) else 0.0,
|
temp_high=daily_data.get("temperature_2m_max", [])[i] if i < len(daily_data.get("temperature_2m_max", [])) else 0.0,
|
||||||
@@ -298,7 +320,8 @@ class OpenMeteoProvider(WeatherProvider):
|
|||||||
condition=WMO_CODE_MAP.get(code, WeatherCondition.UNKNOWN),
|
condition=WMO_CODE_MAP.get(code, WeatherCondition.UNKNOWN),
|
||||||
condition_text=WMO_DESCRIPTIONS.get(code, "Unknown"),
|
condition_text=WMO_DESCRIPTIONS.get(code, "Unknown"),
|
||||||
precipitation_chance=daily_data.get("precipitation_probability_max", [])[i] if i < len(daily_data.get("precipitation_probability_max", [])) else None,
|
precipitation_chance=daily_data.get("precipitation_probability_max", [])[i] if i < len(daily_data.get("precipitation_probability_max", [])) else None,
|
||||||
precipitation_mm=daily_data.get("precipitation_sum", [])[i] if i < len(daily_data.get("precipitation_sum", [])) else None
|
precipitation_mm=daily_data.get("precipitation_sum", [])[i] if i < len(daily_data.get("precipitation_sum", [])) else None,
|
||||||
|
uv_index_max=uv_max
|
||||||
))
|
))
|
||||||
|
|
||||||
return WeatherForecast(
|
return WeatherForecast(
|
||||||
@@ -309,3 +332,110 @@ class OpenMeteoProvider(WeatherProvider):
|
|||||||
except httpx.HTTPError as e:
|
except httpx.HTTPError as e:
|
||||||
logger.error(f"Forecast request failed for {location.name}: {e}")
|
logger.error(f"Forecast request failed for {location.name}: {e}")
|
||||||
raise ValueError(f"Failed to get forecast: {e}")
|
raise ValueError(f"Failed to get forecast: {e}")
|
||||||
|
|
||||||
|
async def get_sun_times(self, location: GeoLocation) -> SunTimes:
|
||||||
|
"""
|
||||||
|
Get sunrise/sunset times for today.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
location: GeoLocation with lat/long
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
SunTimes with sunrise, sunset, and daylight duration
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
ValueError: If API request fails
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
response = await self.client.get(
|
||||||
|
self.WEATHER_URL,
|
||||||
|
params={
|
||||||
|
"latitude": location.latitude,
|
||||||
|
"longitude": location.longitude,
|
||||||
|
"daily": [
|
||||||
|
"sunrise",
|
||||||
|
"sunset",
|
||||||
|
"daylight_duration"
|
||||||
|
],
|
||||||
|
"timezone": self.timezone,
|
||||||
|
"forecast_days": 1
|
||||||
|
}
|
||||||
|
)
|
||||||
|
response.raise_for_status()
|
||||||
|
data = response.json()
|
||||||
|
|
||||||
|
daily = data.get("daily", {})
|
||||||
|
date_str = daily.get("time", [""])[0]
|
||||||
|
sunrise_str = daily.get("sunrise", [""])[0]
|
||||||
|
sunset_str = daily.get("sunset", [""])[0]
|
||||||
|
daylight = daily.get("daylight_duration", [0])[0]
|
||||||
|
|
||||||
|
return SunTimes(
|
||||||
|
location=location.name,
|
||||||
|
date=datetime.fromisoformat(date_str) if date_str else datetime.now(),
|
||||||
|
sunrise=datetime.fromisoformat(sunrise_str) if sunrise_str else datetime.now(),
|
||||||
|
sunset=datetime.fromisoformat(sunset_str) if sunset_str else datetime.now(),
|
||||||
|
daylight_duration=int(daylight) if daylight else 0
|
||||||
|
)
|
||||||
|
except httpx.HTTPError as e:
|
||||||
|
logger.error(f"Sun times request failed for {location.name}: {e}")
|
||||||
|
raise ValueError(f"Failed to get sun times: {e}")
|
||||||
|
|
||||||
|
async def get_air_quality(self, location: GeoLocation) -> AirQuality:
|
||||||
|
"""
|
||||||
|
Get current air quality for a location.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
location: GeoLocation with lat/long
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
AirQuality with pollutant measurements and AQI
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
ValueError: If API request fails
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
response = await self.client.get(
|
||||||
|
self.AIR_QUALITY_URL,
|
||||||
|
params={
|
||||||
|
"latitude": location.latitude,
|
||||||
|
"longitude": location.longitude,
|
||||||
|
"current": [
|
||||||
|
"european_aqi",
|
||||||
|
"us_aqi",
|
||||||
|
"pm2_5",
|
||||||
|
"pm10",
|
||||||
|
"ozone",
|
||||||
|
"nitrogen_dioxide",
|
||||||
|
"sulphur_dioxide",
|
||||||
|
"carbon_monoxide",
|
||||||
|
"grass_pollen",
|
||||||
|
"birch_pollen",
|
||||||
|
"alder_pollen"
|
||||||
|
],
|
||||||
|
"timezone": self.timezone
|
||||||
|
}
|
||||||
|
)
|
||||||
|
response.raise_for_status()
|
||||||
|
data = response.json()
|
||||||
|
|
||||||
|
current = data.get("current", {})
|
||||||
|
|
||||||
|
return AirQuality(
|
||||||
|
location=location.name,
|
||||||
|
timestamp=datetime.now(),
|
||||||
|
aqi_european=current.get("european_aqi"),
|
||||||
|
aqi_us=current.get("us_aqi"),
|
||||||
|
pm2_5=current.get("pm2_5"),
|
||||||
|
pm10=current.get("pm10"),
|
||||||
|
ozone=current.get("ozone"),
|
||||||
|
nitrogen_dioxide=current.get("nitrogen_dioxide"),
|
||||||
|
sulphur_dioxide=current.get("sulphur_dioxide"),
|
||||||
|
carbon_monoxide=current.get("carbon_monoxide"),
|
||||||
|
pollen_grass=current.get("grass_pollen"),
|
||||||
|
pollen_birch=current.get("birch_pollen"),
|
||||||
|
pollen_alder=current.get("alder_pollen")
|
||||||
|
)
|
||||||
|
except httpx.HTTPError as e:
|
||||||
|
logger.error(f"Air quality request failed for {location.name}: {e}")
|
||||||
|
raise ValueError(f"Failed to get air quality: {e}")
|
||||||
|
|||||||
+88
-1
@@ -18,7 +18,7 @@ from pathlib import Path
|
|||||||
|
|
||||||
from src.config import Settings, get_settings, __version__
|
from src.config import Settings, get_settings, __version__
|
||||||
from src.core.dependencies import (
|
from src.core.dependencies import (
|
||||||
verify_api_key, QdrantDep, WikiJSDep, OllamaDep, Neo4jDep
|
verify_api_key, QdrantDep, WikiJSDep, OllamaDep, Neo4jDep, PaperlessDep
|
||||||
)
|
)
|
||||||
from src.core.multi_tenancy import DEFAULT_USER
|
from src.core.multi_tenancy import DEFAULT_USER
|
||||||
|
|
||||||
@@ -85,6 +85,14 @@ class HealthResponse(BaseModel):
|
|||||||
services: Dict[str, Any]
|
services: Dict[str, Any]
|
||||||
|
|
||||||
|
|
||||||
|
class StatsResponse(BaseModel):
|
||||||
|
"""System statistics response model."""
|
||||||
|
neo4j: Dict[str, int]
|
||||||
|
qdrant: Dict[str, Any]
|
||||||
|
wiki_pages: int
|
||||||
|
paperless: Dict[str, Any]
|
||||||
|
|
||||||
|
|
||||||
# Routes
|
# Routes
|
||||||
@app.get("/", tags=["Root"])
|
@app.get("/", tags=["Root"])
|
||||||
async def root() -> Dict[str, str]:
|
async def root() -> Dict[str, str]:
|
||||||
@@ -141,6 +149,85 @@ async def health(settings: Settings = Depends(get_settings)) -> HealthResponse:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/stats", response_model=StatsResponse, tags=["System"])
|
||||||
|
async def stats(
|
||||||
|
user: str = Query(default=DEFAULT_USER, description="User identifier"),
|
||||||
|
neo4j: Neo4jDep = None,
|
||||||
|
qdrant: QdrantDep = None,
|
||||||
|
wikijs: WikiJSDep = None,
|
||||||
|
paperless: PaperlessDep = None,
|
||||||
|
api_key: str = Depends(verify_api_key)
|
||||||
|
) -> StatsResponse:
|
||||||
|
"""
|
||||||
|
Get system statistics.
|
||||||
|
|
||||||
|
Returns counts for:
|
||||||
|
- Neo4j: nodes by type (Document, Entity, Collection, Search)
|
||||||
|
- Qdrant: vectors per collection
|
||||||
|
- Wiki.js: total page count
|
||||||
|
- Paperless: documents, tags, correspondents, document types
|
||||||
|
"""
|
||||||
|
# Neo4j node counts by label
|
||||||
|
neo4j_stats = {}
|
||||||
|
try:
|
||||||
|
for label in ["Document", "Entity", "Collection", "Search"]:
|
||||||
|
result = await neo4j.execute_query(
|
||||||
|
f"MATCH (n:{label}) RETURN count(n) as count"
|
||||||
|
)
|
||||||
|
neo4j_stats[label.lower() + "_nodes"] = result[0]["count"] if result else 0
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Failed to get Neo4j stats: {e}")
|
||||||
|
neo4j_stats = {"error": str(e)}
|
||||||
|
|
||||||
|
# Qdrant collection stats
|
||||||
|
qdrant_stats = {}
|
||||||
|
try:
|
||||||
|
collections = await qdrant.list_collections()
|
||||||
|
qdrant_stats["collections"] = len(collections)
|
||||||
|
qdrant_stats["total_vectors"] = sum(c.get("vectors_count", 0) for c in collections)
|
||||||
|
qdrant_stats["by_collection"] = {
|
||||||
|
c["name"]: c["vectors_count"] for c in collections
|
||||||
|
}
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Failed to get Qdrant stats: {e}")
|
||||||
|
qdrant_stats = {"error": str(e)}
|
||||||
|
|
||||||
|
# Wiki.js page count
|
||||||
|
wiki_pages = 0
|
||||||
|
try:
|
||||||
|
pages = await wikijs.list_all_pages(user)
|
||||||
|
wiki_pages = len(pages)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Failed to get Wiki.js stats: {e}")
|
||||||
|
|
||||||
|
# Paperless-ngx document stats
|
||||||
|
paperless_stats = {}
|
||||||
|
try:
|
||||||
|
# Get document count (page_size=1 for efficiency, we just need the count)
|
||||||
|
docs_result = await paperless.list_documents(page_size=1)
|
||||||
|
paperless_stats["documents"] = docs_result.get("count", 0)
|
||||||
|
|
||||||
|
# Get metadata counts
|
||||||
|
tags = await paperless.list_tags()
|
||||||
|
paperless_stats["tags"] = len(tags)
|
||||||
|
|
||||||
|
correspondents = await paperless.list_correspondents()
|
||||||
|
paperless_stats["correspondents"] = len(correspondents)
|
||||||
|
|
||||||
|
doc_types = await paperless.list_document_types()
|
||||||
|
paperless_stats["document_types"] = len(doc_types)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Failed to get Paperless stats: {e}")
|
||||||
|
paperless_stats = {"error": str(e)}
|
||||||
|
|
||||||
|
return StatsResponse(
|
||||||
|
neo4j=neo4j_stats,
|
||||||
|
qdrant=qdrant_stats,
|
||||||
|
wiki_pages=wiki_pages,
|
||||||
|
paperless=paperless_stats
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
@app.post("/ingest/check-updates", tags=["Ingestion"])
|
@app.post("/ingest/check-updates", tags=["Ingestion"])
|
||||||
async def check_updates(
|
async def check_updates(
|
||||||
documents: Dict[str, Any],
|
documents: Dict[str, Any],
|
||||||
|
|||||||
@@ -18,7 +18,9 @@ class VolatileNamespace(str, Enum):
|
|||||||
Each namespace can have different default TTLs and refresh schedules.
|
Each namespace can have different default TTLs and refresh schedules.
|
||||||
"""
|
"""
|
||||||
# Real-time external data
|
# Real-time external data
|
||||||
WEATHER = "weather" # Current conditions, forecasts
|
WEATHER = "weather" # Current conditions (temperature, humidity, wind)
|
||||||
|
FORECAST = "forecast" # Multi-day weather outlook
|
||||||
|
SUN = "sun" # Sunrise, sunset, daylight duration
|
||||||
NEWS = "news" # Headlines, breaking news
|
NEWS = "news" # Headlines, breaking news
|
||||||
FINANCIAL = "financial" # Stock prices, exchange rates, crypto
|
FINANCIAL = "financial" # Stock prices, exchange rates, crypto
|
||||||
TRANSIT = "transit" # Train/bus schedules, delays, disruptions
|
TRANSIT = "transit" # Train/bus schedules, delays, disruptions
|
||||||
@@ -37,7 +39,9 @@ class VolatileNamespace(str, Enum):
|
|||||||
|
|
||||||
# Default TTLs per namespace (in seconds)
|
# Default TTLs per namespace (in seconds)
|
||||||
NAMESPACE_DEFAULT_TTL: Dict[str, int] = {
|
NAMESPACE_DEFAULT_TTL: Dict[str, int] = {
|
||||||
VolatileNamespace.WEATHER: 1800, # 30 min - weather changes slowly
|
VolatileNamespace.WEATHER: 3600, # 1 hour - current conditions
|
||||||
|
VolatileNamespace.FORECAST: 43200, # 12 hours - forecast stable longer
|
||||||
|
VolatileNamespace.SUN: 86400, # 24 hours - sun times change daily
|
||||||
VolatileNamespace.NEWS: 3600, # 1 hour - news cycles
|
VolatileNamespace.NEWS: 3600, # 1 hour - news cycles
|
||||||
VolatileNamespace.FINANCIAL: 300, # 5 min - markets move fast
|
VolatileNamespace.FINANCIAL: 300, # 5 min - markets move fast
|
||||||
VolatileNamespace.TRANSIT: 300, # 5 min - schedules update frequently
|
VolatileNamespace.TRANSIT: 300, # 5 min - schedules update frequently
|
||||||
|
|||||||
+139
-5
@@ -233,16 +233,16 @@ async def store_volatile(
|
|||||||
async def fetch_weather(
|
async def fetch_weather(
|
||||||
city: str,
|
city: str,
|
||||||
user: str = Query(default=DEFAULT_USER, description="User identifier"),
|
user: str = Query(default=DEFAULT_USER, description="User identifier"),
|
||||||
ttl: int = Query(default=86400, ge=60, le=604800, description="TTL in seconds"),
|
ttl: int = Query(default=3600, ge=60, le=86400, description="TTL in seconds (default 1 hour)"),
|
||||||
qdrant: QdrantDep = None,
|
qdrant: QdrantDep = None,
|
||||||
ollama: OllamaDep = None,
|
ollama: OllamaDep = None,
|
||||||
api_key: str = Depends(verify_api_key)
|
api_key: str = Depends(verify_api_key)
|
||||||
):
|
):
|
||||||
"""
|
"""
|
||||||
Fetch current weather for a city and store in volatile cache.
|
Fetch current weather conditions for a city and store in volatile cache.
|
||||||
|
|
||||||
Called by scheduler for prefetch or on-demand. Geocodes city name
|
Stores temperature, humidity, wind, UV index. For forecasts use /fetch/forecast.
|
||||||
and fetches weather from Open-Meteo API.
|
Called by scheduler for hourly prefetch or on-demand.
|
||||||
|
|
||||||
**Example:**
|
**Example:**
|
||||||
```
|
```
|
||||||
@@ -257,7 +257,49 @@ async def fetch_weather(
|
|||||||
weather_provider=weather_provider,
|
weather_provider=weather_provider,
|
||||||
)
|
)
|
||||||
|
|
||||||
result = await fetch_service.fetch_weather(user, city, ttl=ttl)
|
result = await fetch_service.fetch_current_weather(user, city, ttl=ttl)
|
||||||
|
|
||||||
|
if not result.success:
|
||||||
|
raise HTTPException(status_code=500, detail=result.error)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"success": True,
|
||||||
|
"namespace": result.namespace,
|
||||||
|
"key": result.key,
|
||||||
|
"record": result.record,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/fetch/forecast/{city}")
|
||||||
|
async def fetch_forecast(
|
||||||
|
city: str,
|
||||||
|
user: str = Query(default=DEFAULT_USER, description="User identifier"),
|
||||||
|
days: int = Query(default=7, ge=1, le=16, description="Forecast days (1-16)"),
|
||||||
|
ttl: int = Query(default=43200, ge=60, le=604800, description="TTL in seconds (default 12 hours)"),
|
||||||
|
qdrant: QdrantDep = None,
|
||||||
|
ollama: OllamaDep = None,
|
||||||
|
api_key: str = Depends(verify_api_key)
|
||||||
|
):
|
||||||
|
"""
|
||||||
|
Fetch weather forecast for a city and store in volatile cache.
|
||||||
|
|
||||||
|
Stores multi-day outlook with highs/lows, precipitation, UV.
|
||||||
|
For current conditions use /fetch/weather.
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```
|
||||||
|
POST /volatile/fetch/forecast/amsterdam?user=jpmschweitzer&days=7
|
||||||
|
```
|
||||||
|
"""
|
||||||
|
volatile_service = get_volatile_service(qdrant, ollama)
|
||||||
|
weather_provider = get_weather_provider()
|
||||||
|
|
||||||
|
fetch_service = VolatileFetchService(
|
||||||
|
volatile_service=volatile_service,
|
||||||
|
weather_provider=weather_provider,
|
||||||
|
)
|
||||||
|
|
||||||
|
result = await fetch_service.fetch_forecast(user, city, days=days, ttl=ttl)
|
||||||
|
|
||||||
if not result.success:
|
if not result.success:
|
||||||
raise HTTPException(status_code=500, detail=result.error)
|
raise HTTPException(status_code=500, detail=result.error)
|
||||||
@@ -411,6 +453,98 @@ async def fetch_crypto(
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/fetch/sun/{city}")
|
||||||
|
async def fetch_sun_times(
|
||||||
|
city: str,
|
||||||
|
user: str = Query(default=DEFAULT_USER, description="User identifier"),
|
||||||
|
ttl: int = Query(default=86400, ge=60, le=604800, description="TTL in seconds"),
|
||||||
|
qdrant: QdrantDep = None,
|
||||||
|
ollama: OllamaDep = None,
|
||||||
|
api_key: str = Depends(verify_api_key)
|
||||||
|
):
|
||||||
|
"""
|
||||||
|
Fetch sunrise/sunset times for a city and store in volatile cache.
|
||||||
|
|
||||||
|
Fetches from Open-Meteo API. Useful for home automation triggers.
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```
|
||||||
|
POST /volatile/fetch/sun/rotterdam?user=jpmschweitzer
|
||||||
|
```
|
||||||
|
|
||||||
|
**Response data includes:**
|
||||||
|
- sunrise/sunset times (both HH:MM and ISO formats)
|
||||||
|
- daylight_duration_seconds
|
||||||
|
- daylight_hours
|
||||||
|
- Natural language text summary
|
||||||
|
"""
|
||||||
|
volatile_service = get_volatile_service(qdrant, ollama)
|
||||||
|
weather_provider = get_weather_provider()
|
||||||
|
|
||||||
|
fetch_service = VolatileFetchService(
|
||||||
|
volatile_service=volatile_service,
|
||||||
|
weather_provider=weather_provider,
|
||||||
|
)
|
||||||
|
|
||||||
|
result = await fetch_service.fetch_sun_times(user, city, ttl=ttl)
|
||||||
|
|
||||||
|
if not result.success:
|
||||||
|
raise HTTPException(status_code=500, detail=result.error)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"success": True,
|
||||||
|
"namespace": result.namespace,
|
||||||
|
"key": result.key,
|
||||||
|
"record": result.record,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/fetch/air_quality/{city}")
|
||||||
|
async def fetch_air_quality(
|
||||||
|
city: str,
|
||||||
|
user: str = Query(default=DEFAULT_USER, description="User identifier"),
|
||||||
|
ttl: int = Query(default=3600, ge=60, le=86400, description="TTL in seconds"),
|
||||||
|
qdrant: QdrantDep = None,
|
||||||
|
ollama: OllamaDep = None,
|
||||||
|
api_key: str = Depends(verify_api_key)
|
||||||
|
):
|
||||||
|
"""
|
||||||
|
Fetch air quality data for a city and store in volatile cache.
|
||||||
|
|
||||||
|
Fetches from Open-Meteo Air Quality API.
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```
|
||||||
|
POST /volatile/fetch/air_quality/rotterdam?user=jpmschweitzer
|
||||||
|
```
|
||||||
|
|
||||||
|
**Response data includes:**
|
||||||
|
- European and US AQI indices
|
||||||
|
- Pollutants: PM2.5, PM10, ozone, nitrogen dioxide, etc.
|
||||||
|
- Pollen data (European locations, seasonal)
|
||||||
|
- Natural language text summary
|
||||||
|
"""
|
||||||
|
volatile_service = get_volatile_service(qdrant, ollama)
|
||||||
|
weather_provider = get_weather_provider()
|
||||||
|
|
||||||
|
fetch_service = VolatileFetchService(
|
||||||
|
volatile_service=volatile_service,
|
||||||
|
weather_provider=weather_provider,
|
||||||
|
)
|
||||||
|
|
||||||
|
result = await fetch_service.fetch_air_quality(user, city, ttl=ttl)
|
||||||
|
|
||||||
|
if not result.success:
|
||||||
|
raise HTTPException(status_code=500, detail=result.error)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"success": True,
|
||||||
|
"namespace": result.namespace,
|
||||||
|
"key": result.key,
|
||||||
|
"record": result.record,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
@router.get("/{namespace}/{key}", response_model=VolatileRecordResponse)
|
@router.get("/{namespace}/{key}", response_model=VolatileRecordResponse)
|
||||||
async def get_record(
|
async def get_record(
|
||||||
namespace: str,
|
namespace: str,
|
||||||
|
|||||||
@@ -14,6 +14,9 @@ from src.apis import (
|
|||||||
AggregatedNewsProvider,
|
AggregatedNewsProvider,
|
||||||
AlphaVantageProvider,
|
AlphaVantageProvider,
|
||||||
CurrentWeather,
|
CurrentWeather,
|
||||||
|
WeatherForecast,
|
||||||
|
SunTimes,
|
||||||
|
AirQuality,
|
||||||
NewsFeed,
|
NewsFeed,
|
||||||
StockQuote,
|
StockQuote,
|
||||||
)
|
)
|
||||||
@@ -64,25 +67,25 @@ class VolatileFetchService:
|
|||||||
self.news = news_provider
|
self.news = news_provider
|
||||||
self.financial = financial_provider
|
self.financial = financial_provider
|
||||||
|
|
||||||
async def fetch_weather(
|
async def fetch_current_weather(
|
||||||
self,
|
self,
|
||||||
user: str,
|
user: str,
|
||||||
city: str,
|
city: str,
|
||||||
ttl: int = 86400, # 24 hours
|
ttl: int = 3600, # 1 hour
|
||||||
) -> FetchResult:
|
) -> FetchResult:
|
||||||
"""
|
"""
|
||||||
Fetch current weather for a city and store in volatile cache.
|
Fetch current weather conditions for a city and store in volatile cache.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
user: User identifier
|
user: User identifier
|
||||||
city: City name (will be geocoded)
|
city: City name (will be geocoded)
|
||||||
ttl: Time-to-live in seconds
|
ttl: Time-to-live in seconds (default 1 hour)
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
FetchResult with success status and stored record
|
FetchResult with success status and stored record
|
||||||
"""
|
"""
|
||||||
try:
|
try:
|
||||||
# Geocode city and get weather
|
# Geocode city and get current conditions
|
||||||
location = await self.weather.geocode(city)
|
location = await self.weather.geocode(city)
|
||||||
if not location:
|
if not location:
|
||||||
return FetchResult(
|
return FetchResult(
|
||||||
@@ -92,19 +95,23 @@ class VolatileFetchService:
|
|||||||
error=f"Could not geocode city: {city}"
|
error=f"Could not geocode city: {city}"
|
||||||
)
|
)
|
||||||
|
|
||||||
weather = await self.weather.get_current(location)
|
current = await self.weather.get_current(location)
|
||||||
|
|
||||||
|
# Generate natural language summary
|
||||||
|
text = current.to_text()
|
||||||
|
|
||||||
# Convert to storage format
|
# Convert to storage format
|
||||||
data = {
|
data = {
|
||||||
"temperature": weather.temperature,
|
"temperature": current.temperature,
|
||||||
"feels_like": weather.feels_like,
|
"feels_like": current.feels_like,
|
||||||
"humidity": weather.humidity,
|
"humidity": current.humidity,
|
||||||
"wind_speed": weather.wind_speed,
|
"wind_speed": current.wind_speed,
|
||||||
"wind_direction": weather.wind_direction,
|
"wind_direction": current.wind_direction,
|
||||||
"conditions": weather.condition_text,
|
"conditions": current.condition_text,
|
||||||
"condition_code": weather.condition.value,
|
"condition_code": current.condition.value,
|
||||||
"location": weather.location,
|
"uv_index": current.uv_index,
|
||||||
"text": weather.to_text(),
|
"location": current.location,
|
||||||
|
"text": text,
|
||||||
}
|
}
|
||||||
|
|
||||||
# Store in volatile cache
|
# Store in volatile cache
|
||||||
@@ -117,7 +124,7 @@ class VolatileFetchService:
|
|||||||
ttl=ttl,
|
ttl=ttl,
|
||||||
)
|
)
|
||||||
|
|
||||||
logger.info(f"Stored weather for {city} (user={user})")
|
logger.info(f"Stored current weather for {city} (user={user})")
|
||||||
return FetchResult(
|
return FetchResult(
|
||||||
success=True,
|
success=True,
|
||||||
namespace="weather",
|
namespace="weather",
|
||||||
@@ -126,7 +133,7 @@ class VolatileFetchService:
|
|||||||
)
|
)
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Failed to fetch weather for {city}: {e}")
|
logger.error(f"Failed to fetch current weather for {city}: {e}")
|
||||||
return FetchResult(
|
return FetchResult(
|
||||||
success=False,
|
success=False,
|
||||||
namespace="weather",
|
namespace="weather",
|
||||||
@@ -134,6 +141,94 @@ class VolatileFetchService:
|
|||||||
error=str(e)
|
error=str(e)
|
||||||
)
|
)
|
||||||
|
|
||||||
|
async def fetch_forecast(
|
||||||
|
self,
|
||||||
|
user: str,
|
||||||
|
city: str,
|
||||||
|
days: int = 7,
|
||||||
|
ttl: int = 43200, # 12 hours
|
||||||
|
) -> FetchResult:
|
||||||
|
"""
|
||||||
|
Fetch weather forecast for a city and store in volatile cache.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
user: User identifier
|
||||||
|
city: City name (will be geocoded)
|
||||||
|
days: Number of forecast days (1-16)
|
||||||
|
ttl: Time-to-live in seconds (default 12 hours)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
FetchResult with success status and stored record
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
# Geocode city and get forecast
|
||||||
|
location = await self.weather.geocode(city)
|
||||||
|
if not location:
|
||||||
|
return FetchResult(
|
||||||
|
success=False,
|
||||||
|
namespace="forecast",
|
||||||
|
key=city.lower(),
|
||||||
|
error=f"Could not geocode city: {city}"
|
||||||
|
)
|
||||||
|
|
||||||
|
forecast = await self.weather.get_forecast(location, days=days)
|
||||||
|
|
||||||
|
# Build daily forecast array
|
||||||
|
daily_forecasts = []
|
||||||
|
for day in forecast.daily:
|
||||||
|
daily_forecasts.append({
|
||||||
|
"date": day.date.isoformat(),
|
||||||
|
"day_name": day.date.strftime("%A"),
|
||||||
|
"temp_high": day.temp_high,
|
||||||
|
"temp_low": day.temp_low,
|
||||||
|
"conditions": day.condition_text,
|
||||||
|
"condition_code": day.condition.value,
|
||||||
|
"precipitation_chance": day.precipitation_chance,
|
||||||
|
"precipitation_mm": day.precipitation_mm,
|
||||||
|
"uv_index_max": day.uv_index_max,
|
||||||
|
})
|
||||||
|
|
||||||
|
# Generate natural language summary
|
||||||
|
forecast_lines = [f"{city} {days}-day forecast:"]
|
||||||
|
for day in forecast.daily:
|
||||||
|
forecast_lines.append(day.to_text())
|
||||||
|
text = "\n".join(forecast_lines)
|
||||||
|
|
||||||
|
# Convert to storage format
|
||||||
|
data = {
|
||||||
|
"days": days,
|
||||||
|
"daily": daily_forecasts,
|
||||||
|
"location": forecast.current.location,
|
||||||
|
"text": text,
|
||||||
|
}
|
||||||
|
|
||||||
|
# Store in volatile cache
|
||||||
|
record = await self.volatile.store(
|
||||||
|
user=user,
|
||||||
|
namespace=VolatileNamespace.FORECAST,
|
||||||
|
key=city.lower(),
|
||||||
|
data=data,
|
||||||
|
source="openmeteo",
|
||||||
|
ttl=ttl,
|
||||||
|
)
|
||||||
|
|
||||||
|
logger.info(f"Stored {days}-day forecast for {city} (user={user})")
|
||||||
|
return FetchResult(
|
||||||
|
success=True,
|
||||||
|
namespace="forecast",
|
||||||
|
key=city.lower(),
|
||||||
|
record=record
|
||||||
|
)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Failed to fetch forecast for {city}: {e}")
|
||||||
|
return FetchResult(
|
||||||
|
success=False,
|
||||||
|
namespace="forecast",
|
||||||
|
key=city.lower(),
|
||||||
|
error=str(e)
|
||||||
|
)
|
||||||
|
|
||||||
async def fetch_news(
|
async def fetch_news(
|
||||||
self,
|
self,
|
||||||
user: str,
|
user: str,
|
||||||
@@ -357,3 +452,147 @@ class VolatileFetchService:
|
|||||||
key=f"{symbol.lower()}_{market.lower()}",
|
key=f"{symbol.lower()}_{market.lower()}",
|
||||||
error=str(e)
|
error=str(e)
|
||||||
)
|
)
|
||||||
|
|
||||||
|
async def fetch_sun_times(
|
||||||
|
self,
|
||||||
|
user: str,
|
||||||
|
city: str,
|
||||||
|
ttl: int = 86400, # 24 hours
|
||||||
|
) -> FetchResult:
|
||||||
|
"""
|
||||||
|
Fetch sunrise/sunset times for a city and store in volatile cache.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
user: User identifier
|
||||||
|
city: City name (will be geocoded)
|
||||||
|
ttl: Time-to-live in seconds
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
FetchResult with success status and stored record
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
# Geocode city and get sun times
|
||||||
|
location = await self.weather.geocode(city)
|
||||||
|
if not location:
|
||||||
|
return FetchResult(
|
||||||
|
success=False,
|
||||||
|
namespace="sun",
|
||||||
|
key=city.lower(),
|
||||||
|
error=f"Could not geocode city: {city}"
|
||||||
|
)
|
||||||
|
|
||||||
|
sun_times = await self.weather.get_sun_times(location)
|
||||||
|
|
||||||
|
# Convert to storage format
|
||||||
|
data = {
|
||||||
|
"location": sun_times.location,
|
||||||
|
"date": sun_times.date.isoformat(),
|
||||||
|
"sunrise": sun_times.sunrise.strftime("%H:%M"),
|
||||||
|
"sunset": sun_times.sunset.strftime("%H:%M"),
|
||||||
|
"sunrise_iso": sun_times.sunrise.isoformat(),
|
||||||
|
"sunset_iso": sun_times.sunset.isoformat(),
|
||||||
|
"daylight_duration_seconds": sun_times.daylight_duration,
|
||||||
|
"daylight_hours": sun_times.daylight_duration / 3600,
|
||||||
|
"text": sun_times.to_text(),
|
||||||
|
}
|
||||||
|
|
||||||
|
# Store in volatile cache
|
||||||
|
record = await self.volatile.store(
|
||||||
|
user=user,
|
||||||
|
namespace=VolatileNamespace.SUN,
|
||||||
|
key=city.lower(),
|
||||||
|
data=data,
|
||||||
|
source="openmeteo",
|
||||||
|
ttl=ttl,
|
||||||
|
)
|
||||||
|
|
||||||
|
logger.info(f"Stored sun times for {city} (user={user})")
|
||||||
|
return FetchResult(
|
||||||
|
success=True,
|
||||||
|
namespace="sun",
|
||||||
|
key=city.lower(),
|
||||||
|
record=record
|
||||||
|
)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Failed to fetch sun times for {city}: {e}")
|
||||||
|
return FetchResult(
|
||||||
|
success=False,
|
||||||
|
namespace="sun",
|
||||||
|
key=city.lower(),
|
||||||
|
error=str(e)
|
||||||
|
)
|
||||||
|
|
||||||
|
async def fetch_air_quality(
|
||||||
|
self,
|
||||||
|
user: str,
|
||||||
|
city: str,
|
||||||
|
ttl: int = 3600, # 1 hour
|
||||||
|
) -> FetchResult:
|
||||||
|
"""
|
||||||
|
Fetch air quality data for a city and store in volatile cache.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
user: User identifier
|
||||||
|
city: City name (will be geocoded)
|
||||||
|
ttl: Time-to-live in seconds
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
FetchResult with success status and stored record
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
# Geocode city and get air quality
|
||||||
|
location = await self.weather.geocode(city)
|
||||||
|
if not location:
|
||||||
|
return FetchResult(
|
||||||
|
success=False,
|
||||||
|
namespace="air_quality",
|
||||||
|
key=city.lower(),
|
||||||
|
error=f"Could not geocode city: {city}"
|
||||||
|
)
|
||||||
|
|
||||||
|
air_quality = await self.weather.get_air_quality(location)
|
||||||
|
|
||||||
|
# Convert to storage format
|
||||||
|
data = {
|
||||||
|
"location": air_quality.location,
|
||||||
|
"aqi_european": air_quality.aqi_european,
|
||||||
|
"aqi_us": air_quality.aqi_us,
|
||||||
|
"pm2_5": air_quality.pm2_5,
|
||||||
|
"pm10": air_quality.pm10,
|
||||||
|
"ozone": air_quality.ozone,
|
||||||
|
"nitrogen_dioxide": air_quality.nitrogen_dioxide,
|
||||||
|
"sulphur_dioxide": air_quality.sulphur_dioxide,
|
||||||
|
"carbon_monoxide": air_quality.carbon_monoxide,
|
||||||
|
"pollen_grass": air_quality.pollen_grass,
|
||||||
|
"pollen_birch": air_quality.pollen_birch,
|
||||||
|
"pollen_alder": air_quality.pollen_alder,
|
||||||
|
"text": air_quality.to_text(),
|
||||||
|
}
|
||||||
|
|
||||||
|
# Store in volatile cache
|
||||||
|
record = await self.volatile.store(
|
||||||
|
user=user,
|
||||||
|
namespace=VolatileNamespace.AIR_QUALITY,
|
||||||
|
key=city.lower(),
|
||||||
|
data=data,
|
||||||
|
source="openmeteo",
|
||||||
|
ttl=ttl,
|
||||||
|
)
|
||||||
|
|
||||||
|
logger.info(f"Stored air quality for {city} (user={user})")
|
||||||
|
return FetchResult(
|
||||||
|
success=True,
|
||||||
|
namespace="air_quality",
|
||||||
|
key=city.lower(),
|
||||||
|
record=record
|
||||||
|
)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Failed to fetch air quality for {city}: {e}")
|
||||||
|
return FetchResult(
|
||||||
|
success=False,
|
||||||
|
namespace="air_quality",
|
||||||
|
key=city.lower(),
|
||||||
|
error=str(e)
|
||||||
|
)
|
||||||
|
|||||||
@@ -116,24 +116,6 @@ class WikiChangeListener:
|
|||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Failed to handle notification: {e}", exc_info=True)
|
logger.error(f"Failed to handle notification: {e}", exc_info=True)
|
||||||
|
|
||||||
def _is_automated_user(self, email: str) -> bool:
|
|
||||||
"""
|
|
||||||
Check if email belongs to an automated system user.
|
|
||||||
|
|
||||||
These are edits made by library-desk via Wiki.js API (entity linking).
|
|
||||||
We skip processing these to prevent loops.
|
|
||||||
|
|
||||||
Customize this list based on your Wiki.js username for library-desk.
|
|
||||||
"""
|
|
||||||
automated_users = [
|
|
||||||
self.settings.wikijs_username, # Library-desk's Wiki.js API user
|
|
||||||
"library-desk@system",
|
|
||||||
"automation@system",
|
|
||||||
"bot@system"
|
|
||||||
]
|
|
||||||
|
|
||||||
return email.lower() in [u.lower() for u in automated_users]
|
|
||||||
|
|
||||||
def _is_recently_processed(self, page_id: int) -> bool:
|
def _is_recently_processed(self, page_id: int) -> bool:
|
||||||
"""Check if page was processed recently (debouncing)."""
|
"""Check if page was processed recently (debouncing)."""
|
||||||
if page_id not in self._recent_notifications:
|
if page_id not in self._recent_notifications:
|
||||||
|
|||||||
@@ -98,7 +98,7 @@ class TestVolatileNamespaces:
|
|||||||
|
|
||||||
def test_weather_default_ttl(self):
|
def test_weather_default_ttl(self):
|
||||||
"""Test weather namespace default TTL."""
|
"""Test weather namespace default TTL."""
|
||||||
assert NAMESPACE_DEFAULT_TTL[VolatileNamespace.WEATHER] == 1800 # 30 min
|
assert NAMESPACE_DEFAULT_TTL[VolatileNamespace.WEATHER] == 3600 # 1 hour (current conditions)
|
||||||
|
|
||||||
def test_financial_default_ttl(self):
|
def test_financial_default_ttl(self):
|
||||||
"""Test financial namespace default TTL."""
|
"""Test financial namespace default TTL."""
|
||||||
@@ -110,7 +110,7 @@ class TestVolatileNamespaces:
|
|||||||
|
|
||||||
def test_namespace_count(self):
|
def test_namespace_count(self):
|
||||||
"""Test we have the expected number of namespaces."""
|
"""Test we have the expected number of namespaces."""
|
||||||
assert len(VolatileNamespace) == 11
|
assert len(VolatileNamespace) == 13 # Including SUN, FORECAST
|
||||||
|
|
||||||
|
|
||||||
class TestVolatileListResponse:
|
class TestVolatileListResponse:
|
||||||
@@ -274,7 +274,7 @@ class TestVolatileService:
|
|||||||
def test_get_default_ttl_known_namespace(self, volatile_service):
|
def test_get_default_ttl_known_namespace(self, volatile_service):
|
||||||
"""Test default TTL for known namespace."""
|
"""Test default TTL for known namespace."""
|
||||||
ttl = volatile_service._get_default_ttl("weather")
|
ttl = volatile_service._get_default_ttl("weather")
|
||||||
assert ttl == 1800 # Weather namespace default
|
assert ttl == 3600 # Weather namespace default (1 hour)
|
||||||
|
|
||||||
def test_get_default_ttl_unknown_namespace(self, volatile_service):
|
def test_get_default_ttl_unknown_namespace(self, volatile_service):
|
||||||
"""Test default TTL for unknown namespace."""
|
"""Test default TTL for unknown namespace."""
|
||||||
@@ -366,7 +366,7 @@ class TestVolatileService:
|
|||||||
ttl=None # Not specified
|
ttl=None # Not specified
|
||||||
)
|
)
|
||||||
|
|
||||||
assert result.ttl == 1800 # Weather default
|
assert result.ttl == 3600 # Weather default (1 hour)
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_search_empty_collection(self, volatile_service, mock_qdrant, mock_ollama):
|
async def test_search_empty_collection(self, volatile_service, mock_qdrant, mock_ollama):
|
||||||
|
|||||||
@@ -50,22 +50,6 @@ class TestWikiChangeListener:
|
|||||||
assert listener._debounce_seconds == 5
|
assert listener._debounce_seconds == 5
|
||||||
assert len(listener._recent_notifications) == 0
|
assert len(listener._recent_notifications) == 0
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_automated_user_filtering(self, listener):
|
|
||||||
"""Test that automated users are correctly identified."""
|
|
||||||
# Automated users should be filtered
|
|
||||||
assert listener._is_automated_user("librarian@schweitz.net") is True
|
|
||||||
assert listener._is_automated_user("library-desk@system") is True
|
|
||||||
assert listener._is_automated_user("automation@system") is True
|
|
||||||
assert listener._is_automated_user("bot@system") is True
|
|
||||||
|
|
||||||
# Case insensitive
|
|
||||||
assert listener._is_automated_user("LIBRARIAN@SCHWEITZ.NET") is True
|
|
||||||
|
|
||||||
# Regular users should not be filtered
|
|
||||||
assert listener._is_automated_user("user@example.com") is False
|
|
||||||
assert listener._is_automated_user("john@example.com") is False
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_debouncing_prevents_duplicates(self, listener):
|
async def test_debouncing_prevents_duplicates(self, listener):
|
||||||
"""Test that debouncing prevents duplicate processing."""
|
"""Test that debouncing prevents duplicate processing."""
|
||||||
@@ -163,19 +147,29 @@ class TestWikiChangeListener:
|
|||||||
assert mock_process.call_args[1]['event'] == 'page.delete'
|
assert mock_process.call_args[1]['event'] == 'page.delete'
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_automated_user_notification_filtered(self, listener):
|
async def test_any_user_notification_processed(self, listener):
|
||||||
"""Test that notifications from automated users are filtered out."""
|
"""Test that notifications are processed regardless of user email.
|
||||||
|
|
||||||
|
Note: The user_email in PostgreSQL notifications is the page CREATOR,
|
||||||
|
not the editor. We cannot filter by user email because:
|
||||||
|
- A page created by 'librarian' but edited by a human should be processed
|
||||||
|
- Filtering by creator would break legitimate page ingestion
|
||||||
|
Loop prevention is handled by debouncing instead.
|
||||||
|
"""
|
||||||
mock_connection = AsyncMock()
|
mock_connection = AsyncMock()
|
||||||
|
|
||||||
with patch.object(listener, '_process_page_change', new_callable=AsyncMock) as mock_process:
|
with patch.object(listener, '_process_page_change', new_callable=AsyncMock) as mock_process:
|
||||||
# Notification from automated user should be skipped
|
# Even system user notifications should be processed
|
||||||
|
# (debouncing handles loop prevention, not user filtering)
|
||||||
await listener._handle_notification(
|
await listener._handle_notification(
|
||||||
mock_connection, 1234, 'wiki_page_changes',
|
mock_connection, 1234, 'wiki_page_changes',
|
||||||
'UPDATE:123:librarian@schweitz.net'
|
'UPDATE:123:librarian@schweitz.net'
|
||||||
)
|
)
|
||||||
|
|
||||||
# Process should NOT be called
|
# Process SHOULD be called (user filtering is not used)
|
||||||
mock_process.assert_not_called()
|
mock_process.assert_called_once()
|
||||||
|
assert mock_process.call_args[1]['page_id'] == 123
|
||||||
|
assert mock_process.call_args[1]['event'] == 'page.update'
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_duplicate_notification_filtered(self, listener):
|
async def test_duplicate_notification_filtered(self, listener):
|
||||||
|
|||||||
Reference in New Issue
Block a user