Compare commits

...
11 Commits
Author SHA1 Message Date
jpmschweitzerandClaude Opus 4.5 31486018a5 fix: update endpoint TTL defaults to match namespace TTLs
Build and Push / release (push) Successful in 3s
Build and Push / build (push) Successful in 1m16s
All fetch endpoints now use 2x refresh interval as default TTL.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 17:35:28 +01:00
jpmschweitzerandClaude Opus 4.5 4cfad2e8bc fix: double volatile TTLs to survive missed scheduler runs
Build and Push / release (push) Successful in 3s
Build and Push / build (push) Successful in 2m13s
TTL = 2x refresh interval ensures data remains valid even if a
scheduled refresh is delayed or fails.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 16:54:18 +01:00
jpmschweitzerandClaude Opus 4.5 1e31e74ad6 fix: update CI workflow to trigger on tag push
Build and Push / release (push) Successful in 3s
Build and Push / build (push) Successful in 1m14s
Matches core-api workflow pattern for auto release/build on version tags.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-07 12:05:31 +01:00
jpmschweitzerandClaude Opus 4.5 72f515bf61 feat: add combined environment endpoint for concurrent weather + air quality fetch
Build and Push / release (release) Failing after 3s
Build and Push / build (release) Successful in 1m19s
- POST /volatile/fetch/environment/{city} fetches both in parallel
- Single geocode lookup shared between API calls
- Uses asyncio.gather() for concurrent external requests
- Fix scheduler executor name (rest_api → rest_api_executor)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-07 11:53:46 +01:00
jpmschweitzer 0c085d603e auto release/build on version tag 2026-01-03 20:39:00 +01:00
jpmschweitzerandClaude Opus 4.5 152b2f28c4 release: v1.6.2 - Stats endpoint, weather/forecast separation
Build and Push / build (release) Successful in 30s
- GET /stats endpoint with Neo4j, Qdrant, Wiki.js, Paperless stats
- Split weather into current (1hr TTL) and forecast (12hr TTL)
- New FORECAST namespace for multi-day outlook

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-30 12:50:44 +01:00
jpmschweitzerandClaude Opus 4.5 46b9bcd7a0 feat: separate current weather from forecast into distinct namespaces
- Add FORECAST namespace for multi-day outlook (12hr TTL)
- WEATHER namespace now stores only current conditions (1hr TTL)
- Split fetch_weather into fetch_current_weather + fetch_forecast
- Add POST /volatile/fetch/forecast/{city} endpoint
- Different update frequencies for efficient caching

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-30 12:44:16 +01:00
jpmschweitzerandClaude Opus 4.5 68eb1add3d feat: add GET /stats endpoint with system statistics
Returns counts for:
- Neo4j: nodes by type (Document, Entity, Collection, Search)
- Qdrant: vectors per collection
- Wiki.js: total page count
- Paperless: documents, tags, correspondents, document types

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-30 12:07:02 +01:00
jpmschweitzerandClaude Opus 4.5 d6b30570a0 release: v1.6.1 - Weather forecasts, sun times, air quality
Build and Push / build (release) Successful in 52s
- Weather fetch now returns 7-day forecasts with UV index
- New /volatile/fetch/sun/{city} endpoint for sunrise/sunset
- New /volatile/fetch/air_quality/{city} endpoint for AQI and pollutants
- OpenMeteoProvider now implements AirQualityProvider interface

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-30 10:13:06 +01:00
jpmschweitzerandClaude Opus 4.5 6b0530ed79 fix: remove dead automated user filtering code
The _is_automated_user method was never called - loop prevention is
handled by debouncing instead. User email filtering was intentionally
removed because the notification email is the page CREATOR, not editor.

- Remove unused _is_automated_user method
- Update test to verify notifications are processed regardless of user
- Remove obsolete test_automated_user_filtering test

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-29 21:47:55 +01:00
jpmschweitzerandClaude Opus 4.5 0a8c2639a0 docs: update MEMORY_REMEMBER_PLAN with implementation status
Mark all phases as complete (v1.5.0-v1.6.0):
- Settings DB, Phase A, B, C all implemented
- Updated files summary with actual implementations
- Added remaining work section for file upload placeholder

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-29 21:33:10 +01:00
17 changed files with 1221 additions and 138 deletions
+15 -2
View File
@@ -1,12 +1,25 @@
name: Build and Push
on:
release:
types: [published]
push:
tags:
- 'v*'
jobs:
release:
runs-on: ubuntu-latest
steps:
- name: Create Gitea Release
run: |
curl -sf -X POST \
-H "Authorization: token ${{ secrets.GITHUB_TOKEN }}" \
-H "Content-Type: application/json" \
-d '{"tag_name": "${{ github.ref_name }}", "name": "Release ${{ github.ref_name }}", "body": "Automated release for ${{ github.ref_name }}"}' \
"${{ github.server_url }}/api/v1/repos/${{ github.repository }}/releases"
build:
runs-on: ubuntu-latest
needs: release
steps:
- uses: actions/checkout@v4
+82
View File
@@ -5,6 +5,88 @@ All notable changes to Library Desk will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [1.7.3] - 2026-01-07
### Fixed
- **Endpoint TTL defaults** - Updated all fetch endpoint defaults to match namespace TTLs (2x refresh interval)
## [1.7.2] - 2026-01-07
### Fixed
- **Volatile TTL doubled** - TTL now 2x refresh interval to survive missed/delayed scheduler runs
## [1.7.1] - 2026-01-07
### Fixed
- **CI workflow** - Updated Gitea Actions to trigger on tag push (matching core-api)
## [1.7.0] - 2026-01-07
### Added
- **Combined Environment Endpoint** - `POST /volatile/fetch/environment/{city}`
- Fetches weather and air quality concurrently with `asyncio.gather()`
- Single geocode lookup shared between both API calls
- More efficient than calling weather and air_quality separately
- Reduces wall-clock time and eliminates redundant geocoding
### Fixed
- **Scheduler executor name** - Fixed `rest_api``rest_api_executor` in SchedulerTask model and register_volatile_fetch() to prevent "Executor module not found" errors
## [1.6.2] - 2025-12-30
### Added
- **System Statistics Endpoint** - `GET /stats`
- Neo4j: node counts by type (Document, Entity, Collection, Search)
- Qdrant: collection counts, total vectors, per-collection breakdown
- Wiki.js: total page count
- Paperless: documents, tags, correspondents, document types
- **Weather/Forecast Separation** - Split weather into two distinct namespaces
- `POST /volatile/fetch/weather/{city}` - Current conditions only (1hr TTL)
- `POST /volatile/fetch/forecast/{city}` - 7-day outlook (12hr TTL)
- Different update frequencies for efficient caching
- `FORECAST` namespace added to volatile namespaces
### Changed
- Weather namespace TTL changed from 30 minutes to 1 hour (current conditions)
- Forecast data now stored separately with 12 hour TTL
## [1.6.1] - 2025-12-30
### Added
- **Weather Forecast Support** - Enhanced weather fetch with 7-day daily forecasts
- Current conditions now include UV index
- Daily forecasts with high/low temps, conditions, precipitation chance, UV max
- Natural language text summary with multi-day outlook
- **Sun Times Endpoint** - `POST /volatile/fetch/sun/{city}`
- Sunrise and sunset times (HH:MM and ISO formats)
- Daylight duration in seconds and hours
- Separate volatile namespace with 24hr TTL
- Useful for home automation light triggers
- **Air Quality Endpoint** - `POST /volatile/fetch/air_quality/{city}`
- European and US AQI indices
- Pollutants: PM2.5, PM10, ozone, nitrogen dioxide, sulphur dioxide, carbon monoxide
- Pollen data (grass, birch, alder) for European locations (seasonal)
- Hourly refresh (1hr TTL)
- **New Base Models**
- `SunTimes` dataclass for sunrise/sunset data
- `AirQuality` dataclass with AQI and pollutants
- `AirQualityProvider` abstract interface
- **New Volatile Namespace** - `SUN` for sunrise/sunset times (86400s default TTL)
### Changed
- Weather fetch now uses `get_forecast()` instead of `get_current()` for richer data
- `OpenMeteoProvider` now implements both `WeatherProvider` and `AirQualityProvider`
## [1.6.0] - 2025-12-29
### Added
-9
View File
@@ -41,12 +41,3 @@ Check for duplicate or highly similar documents using vector similarity and grap
3. Check graph relationships
4. Return candidates with similarity scores
## System Statistics
#### `GET /stats`
Get system statistics (wiki pages, neo4j nodes, qdrant vectors).
**Implementation needed:**
- Query Neo4j for node count
- Query Qdrant for vector count
- Query Wiki.js for page count
+102
View File
@@ -0,0 +1,102 @@
#!/usr/bin/env python3
"""Quick script to check environmental data in Qdrant volatile cache."""
import asyncio
import os
import time
from dotenv import load_dotenv
load_dotenv()
async def main():
from qdrant_client import QdrantClient
from qdrant_client.models import Filter, FieldCondition, MatchValue
from src.config import get_settings
settings = get_settings()
qdrant = QdrantClient(url=settings.qdrant_url)
user = "jpmschweitzer"
collection = f"volatile_{user}"
print(f"\n{'='*60}")
print(f"Volatile Data in Qdrant ({collection})")
print(f"{'='*60}\n")
namespaces = ["weather", "air_quality", "forecast", "sun", "news"]
for ns in namespaces:
try:
results = qdrant.scroll(
collection_name=collection,
scroll_filter=Filter(
must=[FieldCondition(key="namespace", match=MatchValue(value=ns))]
),
limit=10,
with_payload=True,
with_vectors=False,
)
points = results[0]
print(f"=== {ns.upper()} ({len(points)} records) ===")
if not points:
print(" (no data)")
print()
continue
for point in points:
payload = point.payload
raw_data = payload.get("raw_data", {})
if ns == "weather":
print(f" Temperature: {raw_data.get('temperature')}°C (feels like {raw_data.get('feels_like')}°C)")
print(f" Conditions: {raw_data.get('conditions')}")
print(f" Humidity: {raw_data.get('humidity')}%")
print(f" Wind: {raw_data.get('wind_speed')} km/h")
print(f" UV Index: {raw_data.get('uv_index')}")
elif ns == "air_quality":
print(f" European AQI: {raw_data.get('aqi_european')}")
print(f" US AQI: {raw_data.get('aqi_us')}")
print(f" PM2.5: {raw_data.get('pm2_5')} µg/m³")
print(f" PM10: {raw_data.get('pm10')} µg/m³")
print(f" Ozone: {raw_data.get('ozone')} µg/m³")
print(f" NO₂: {raw_data.get('nitrogen_dioxide')} µg/m³")
elif ns == "forecast":
daily = raw_data.get("daily", [])
for day in daily[:5]:
print(f" {day.get('day_name', 'N/A')[:3]}: {day.get('temp_low'):.0f}-{day.get('temp_high'):.0f}°C, {day.get('conditions')}")
elif ns == "sun":
print(f" Sunrise: {raw_data.get('sunrise')}")
print(f" Sunset: {raw_data.get('sunset')}")
print(f" Daylight: {raw_data.get('daylight_hours', 0):.1f} hours")
elif ns == "news":
headlines = raw_data.get("headlines", [])
print(f" Category: {raw_data.get('category', 'general')}")
print(f" Headlines ({len(headlines)}):")
for item in headlines[:5]:
title = item.get("title", "")[:60]
source = item.get("source", "")
print(f" - [{source}] {title}...")
# Show TTL info
ttl_expiry = payload.get("ttl_expiry")
if ttl_expiry:
remaining = (ttl_expiry / 1000) - time.time()
if remaining > 0:
print(f" TTL remaining: {int(remaining)}s ({int(remaining/60)} min)")
else:
print(f" TTL: EXPIRED")
print()
except Exception as e:
print(f" Error fetching {ns}: {e}")
print()
if __name__ == "__main__":
asyncio.run(main())
+53 -31
View File
@@ -16,8 +16,16 @@ This document outlines the implementation of "remember" triggers for the memory
| Memory Tier | Remember Trigger | Recall | Status |
|-------------|------------------|--------|--------|
| Wiki | Wiki.js webhook, Consolidation | HybridRAG vector+graph | ✅ Complete |
| Documents | Paperless webhook | Manual `/documents/search` | ⚠️ Partial (no HybridRAG recall) |
| Volatile | Manual `/volatile/store` only | HybridRAG volatile search | ⚠️ Partial (no auto-triggers) |
| Documents | Paperless webhook | HybridRAG document search | ✅ Complete (v1.6.0) |
| Volatile | Scheduler prefetch, HybridRAG post-processor | HybridRAG volatile search | ✅ Complete (v1.6.0) |
### Implementation Summary (v1.6.0)
- **Settings DB**: Central `system_settings` PostgreSQL database with `SettingsClient`
- **Phase A**: Volatile fetch endpoints (`/volatile/fetch/{namespace}/{key}`) with weather, news, financial providers
- **Phase B**: Unified memory routing in consolidation service (wiki/volatile/file/prefetch/skip classification)
- **Phase C**: Document recall in HybridRAG (4-source parallel retrieval)
- **Scheduler Integration**: `SchedulerClient` for external scheduler task registration
---
@@ -662,46 +670,60 @@ document_threshold: float = Field(default=0.6)
## Implementation Order
| Phase | Priority | Effort | Description |
|-------|----------|--------|-------------|
| **Settings DB** | High | Low | PostgreSQL schema + settings client |
| **B.4** | High | Low | Scheduler client |
| **B.1-B.3** | High | Medium | HybridRAG post-processor |
| **B.5** | High | Low | Config options |
| **C.1-C.2** | High | Low | Document recall in HybridRAG |
| **A.1** | Medium | Low | `/volatile/fetch` endpoint |
| **A.2** | Medium | Medium | Weather + News API clients |
| **A.3** | Medium | Low | Fetch service |
| Phase | Priority | Effort | Description | Status |
|-------|----------|--------|-------------|--------|
| **Settings DB** | High | Low | PostgreSQL schema + settings client | ✅ v1.5.0 |
| **B.4** | High | Low | Scheduler client | ✅ v1.6.0 |
| **B.1-B.3** | High | Medium | HybridRAG post-processor | ✅ v1.6.0 |
| **B.5** | High | Low | Config options | ✅ v1.6.0 |
| **C.1-C.2** | High | Low | Document recall in HybridRAG | ✅ v1.6.0 |
| **A.1** | Medium | Low | `/volatile/fetch` endpoint | ✅ v1.6.0 |
| **A.2** | Medium | Medium | Weather + News API clients | ✅ v1.5.0 |
| **A.3** | Medium | Low | Fetch service | ✅ v1.6.0 |
### Remaining Work
| Item | Description | Status |
|------|-------------|--------|
| File upload | Download PDFs and upload to Paperless | ⚠️ Placeholder (logs only) |
| Prefetch patterns | More sophisticated pattern detection | Optional enhancement |
---
## Files Summary
### New Files
### New Files (Implemented)
| Path | Purpose |
|------|---------|
| `src/clients/settings_client.py` | Read-only central settings access |
| `src/clients/scheduler_client.py` | Register/manage scheduler tasks |
| `src/clients/weather_client.py` | Open-Meteo API (with geocoding) |
| `src/clients/news_client.py` | NOS.nl RSS feeds |
| `src/services/volatile_fetch_service.py` | Orchestrates fetch + store |
| Path | Purpose | Version |
|------|---------|---------|
| `src/clients/settings_client.py` | Central settings database access | v1.5.0 |
| `src/clients/scheduler_client.py` | External scheduler task management | v1.6.0 |
| `src/apis/__init__.py` | External API providers package | v1.5.0 |
| `src/apis/base.py` | Abstract base classes for providers | v1.5.0 |
| `src/apis/weather.py` | OpenMeteoProvider (geocoding + forecast) | v1.5.0 |
| `src/apis/news.py` | AggregatedNewsProvider | v1.5.0 |
| `src/apis/nos.py` | NOSProvider (Dutch news RSS) | v1.5.0 |
| `src/apis/bbc.py` | BBCProvider (English news RSS) | v1.5.0 |
| `src/apis/financial.py` | AlphaVantageProvider (stocks/crypto) | v1.5.0 |
| `src/services/volatile_fetch_service.py` | Orchestrates fetch + store | v1.6.0 |
### Modified Files
| Path | Changes |
|------|---------|
| `src/services/hybrid_rag_service.py` | Post-processor, prefetch detection, document search |
| `src/models/hybrid_rag.py` | Memory config options |
| `src/routers/volatile.py` | `/volatile/fetch/{namespace}/{key}` |
| `src/core/dependencies.py` | Settings client, scheduler client DI |
| `src/config.py` | `SYSTEM_SETTINGS_*` connection vars |
| Path | Changes | Version |
|------|---------|---------|
| `src/services/hybrid_rag_service.py` | Document search (4-source parallel retrieval) | v1.6.0 |
| `src/services/consolidation_service.py` | Unified memory routing, scheduler integration | v1.6.0 |
| `src/models/hybrid_rag.py` | Document config options (`enable_documents`, `document_limit`) | v1.6.0 |
| `src/models/consolidation.py` | Memory routing models | v1.6.0 |
| `src/routers/volatile.py` | `/volatile/fetch/{namespace}/{key}` endpoints | v1.6.0 |
| `src/core/dependencies.py` | Settings, scheduler, provider DI | v1.5.0-v1.6.0 |
| `src/config.py` | `SYSTEM_SETTINGS_*`, `SCHEDULER_URL` vars | v1.5.0-v1.6.0 |
### Database
| Item | Details |
|------|---------|
| Database | `system_settings` (PostgreSQL) |
| Table | `settings (key, value JSONB, category, ...)` |
| Library-desk user | `library_desk_ro` (read-only) |
| Management | Direct psql commands |
| Database | `system_settings` (PostgreSQL on postgres-shared) |
| Table | `settings (key, user_scope, value JSONB, schema JSONB, ...)` |
| Library-desk access | Read-only via `SettingsClient` |
| Management | Direct psql commands (future: CRUD manager UI) |
+1 -1
View File
@@ -1,6 +1,6 @@
[project]
name = "library-desk"
version = "1.6.0"
version = "1.7.3"
description = "Coordination service for The Library system - HybridRAG queries, document ingestion, entity extraction, and knowledge consolidation"
readme = "README.md"
requires-python = ">=3.12"
+8
View File
@@ -27,6 +27,9 @@ from .base import (
DayForecast,
WeatherForecast,
GeoLocation,
SunTimes,
# Air quality models
AirQuality,
# News models
NewsItem,
NewsFeed,
@@ -34,6 +37,7 @@ from .base import (
StockQuote,
# Abstract providers
WeatherProvider,
AirQualityProvider,
NewsProvider,
FinancialProvider,
)
@@ -53,8 +57,12 @@ __all__ = [
"DayForecast",
"WeatherForecast",
"GeoLocation",
"SunTimes",
"WeatherProvider",
"OpenMeteoProvider",
# Air quality
"AirQuality",
"AirQualityProvider",
# News
"NewsItem",
"NewsFeed",
+102 -4
View File
@@ -44,14 +44,18 @@ class CurrentWeather:
condition_text: str # Human-readable description
timestamp: datetime
location: str # City/location name
uv_index: Optional[float] = None # UV index 0-11+
def to_text(self) -> str:
"""Generate natural language description."""
return (
f"Currently {self.temperature:.1f}°C "
f"({self.condition_text}) in {self.location}. "
parts = [
f"Currently {self.temperature:.1f}°C",
f"({self.condition_text}) in {self.location}.",
f"Humidity {self.humidity}%, wind {self.wind_speed:.0f} km/h."
)
]
if self.uv_index is not None:
parts.append(f"UV index: {self.uv_index:.0f}.")
return " ".join(parts)
@dataclass
@@ -64,6 +68,14 @@ class DayForecast:
condition_text: str
precipitation_chance: Optional[int] # Percentage 0-100
precipitation_mm: Optional[float]
uv_index_max: Optional[float] = None # Max UV index for the day
def to_text(self) -> str:
"""Generate natural language description."""
date_str = self.date.strftime("%A") # Day name
precip = f", {self.precipitation_chance}% rain" if self.precipitation_chance else ""
uv = f", UV {self.uv_index_max:.0f}" if self.uv_index_max else ""
return f"{date_str}: {self.temp_high:.0f}°/{self.temp_low:.0f}°C, {self.condition_text}{precip}{uv}"
@dataclass
@@ -84,6 +96,78 @@ class GeoLocation:
admin_area: Optional[str] = None # State/province
@dataclass
class SunTimes:
"""Sunrise/sunset times for a location."""
location: str
date: datetime
sunrise: datetime
sunset: datetime
daylight_duration: int # seconds
solar_noon: Optional[datetime] = None
def to_text(self) -> str:
"""Generate natural language description."""
sunrise_str = self.sunrise.strftime("%H:%M")
sunset_str = self.sunset.strftime("%H:%M")
hours = self.daylight_duration // 3600
minutes = (self.daylight_duration % 3600) // 60
return (
f"Sun times for {self.location} on {self.date.strftime('%A %d %B')}: "
f"Sunrise at {sunrise_str}, sunset at {sunset_str}. "
f"Daylight duration: {hours}h {minutes}m."
)
@dataclass
class AirQuality:
"""Air quality measurements for a location."""
location: str
timestamp: datetime
aqi_european: Optional[int] # European AQI 0-500+
aqi_us: Optional[int] # US AQI 0-500+
pm2_5: Optional[float] # µg/m³
pm10: Optional[float] # µg/m³
ozone: Optional[float] # µg/m³
nitrogen_dioxide: Optional[float] # µg/m³
sulphur_dioxide: Optional[float] # µg/m³
carbon_monoxide: Optional[float] # µg/m³
# Pollen (European data only, seasonal)
pollen_grass: Optional[float] = None
pollen_birch: Optional[float] = None
pollen_alder: Optional[float] = None
def to_text(self) -> str:
"""Generate natural language description."""
parts = [f"Air quality in {self.location}:"]
if self.aqi_european is not None:
level = self._aqi_level(self.aqi_european)
parts.append(f"European AQI {self.aqi_european} ({level}).")
if self.pm2_5 is not None:
parts.append(f"PM2.5: {self.pm2_5:.1f} µg/m³.")
if self.pm10 is not None:
parts.append(f"PM10: {self.pm10:.1f} µg/m³.")
if self.ozone is not None:
parts.append(f"Ozone: {self.ozone:.1f} µg/m³.")
return " ".join(parts)
@staticmethod
def _aqi_level(aqi: int) -> str:
"""Convert AQI to human-readable level."""
if aqi <= 20:
return "good"
elif aqi <= 40:
return "fair"
elif aqi <= 60:
return "moderate"
elif aqi <= 80:
return "poor"
elif aqi <= 100:
return "very poor"
else:
return "hazardous"
# =============================================================================
# News Models
# =============================================================================
@@ -163,6 +247,11 @@ class WeatherProvider(ABC):
"""Get weather forecast for a location."""
pass
@abstractmethod
async def get_sun_times(self, location: GeoLocation) -> SunTimes:
"""Get sunrise/sunset times for today."""
pass
async def get_weather_for_city(self, city: str) -> CurrentWeather:
"""Convenience method: geocode and get current weather."""
location = await self.geocode(city)
@@ -171,6 +260,15 @@ class WeatherProvider(ABC):
return await self.get_current(location)
class AirQualityProvider(ABC):
"""Abstract base class for air quality API providers."""
@abstractmethod
async def get_air_quality(self, location: GeoLocation) -> AirQuality:
"""Get current air quality for a location."""
pass
class NewsProvider(ABC):
"""Abstract base class for news API providers."""
+138 -8
View File
@@ -14,11 +14,14 @@ from typing import Optional
from .base import (
WeatherProvider,
AirQualityProvider,
WeatherCondition,
CurrentWeather,
DayForecast,
WeatherForecast,
GeoLocation,
SunTimes,
AirQuality,
)
logger = logging.getLogger(__name__)
@@ -89,11 +92,12 @@ WMO_DESCRIPTIONS: dict[int, str] = {
}
class OpenMeteoProvider(WeatherProvider):
"""Open-Meteo weather API implementation."""
class OpenMeteoProvider(WeatherProvider, AirQualityProvider):
"""Open-Meteo weather and air quality API implementation."""
GEOCODING_URL = "https://geocoding-api.open-meteo.com/v1/search"
WEATHER_URL = "https://api.open-meteo.com/v1/forecast"
AIR_QUALITY_URL = "https://air-quality-api.open-meteo.com/v1/air-quality"
def __init__(
self,
@@ -194,9 +198,11 @@ class OpenMeteoProvider(WeatherProvider):
"wind_speed_10m",
"wind_direction_10m"
],
"daily": ["uv_index_max"],
"timezone": self.timezone,
"temperature_unit": "celsius",
"wind_speed_unit": "kmh"
"wind_speed_unit": "kmh",
"forecast_days": 1
}
)
response.raise_for_status()
@@ -205,6 +211,12 @@ class OpenMeteoProvider(WeatherProvider):
current = data.get("current", {})
weather_code = current.get("weather_code", 0)
# Get today's UV index from daily data
daily = data.get("daily", {})
uv_index = None
if daily.get("uv_index_max"):
uv_index = daily["uv_index_max"][0]
return CurrentWeather(
temperature=current.get("temperature_2m", 0.0),
feels_like=current.get("apparent_temperature"),
@@ -214,7 +226,8 @@ class OpenMeteoProvider(WeatherProvider):
condition=WMO_CODE_MAP.get(weather_code, WeatherCondition.UNKNOWN),
condition_text=WMO_DESCRIPTIONS.get(weather_code, "Unknown"),
timestamp=datetime.now(),
location=location.name
location=location.name,
uv_index=uv_index
)
except httpx.HTTPError as e:
logger.error(f"Weather request failed for {location.name}: {e}")
@@ -259,7 +272,8 @@ class OpenMeteoProvider(WeatherProvider):
"temperature_2m_max",
"temperature_2m_min",
"precipitation_sum",
"precipitation_probability_max"
"precipitation_probability_max",
"uv_index_max"
],
"timezone": self.timezone,
"temperature_unit": "celsius",
@@ -272,7 +286,14 @@ class OpenMeteoProvider(WeatherProvider):
# Parse current weather
current_data = data.get("current", {})
daily_data = data.get("daily", {})
weather_code = current_data.get("weather_code", 0)
# Get today's UV from daily data
uv_index = None
if daily_data.get("uv_index_max"):
uv_index = daily_data["uv_index_max"][0]
current = CurrentWeather(
temperature=current_data.get("temperature_2m", 0.0),
feels_like=current_data.get("apparent_temperature"),
@@ -282,15 +303,16 @@ class OpenMeteoProvider(WeatherProvider):
condition=WMO_CODE_MAP.get(weather_code, WeatherCondition.UNKNOWN),
condition_text=WMO_DESCRIPTIONS.get(weather_code, "Unknown"),
timestamp=datetime.now(),
location=location.name
location=location.name,
uv_index=uv_index
)
# Parse daily forecast
daily_data = data.get("daily", {})
daily = []
dates = daily_data.get("time", [])
for i, date_str in enumerate(dates):
code = daily_data.get("weather_code", [])[i] if i < len(daily_data.get("weather_code", [])) else 0
uv_max = daily_data.get("uv_index_max", [])[i] if i < len(daily_data.get("uv_index_max", [])) else None
daily.append(DayForecast(
date=datetime.fromisoformat(date_str),
temp_high=daily_data.get("temperature_2m_max", [])[i] if i < len(daily_data.get("temperature_2m_max", [])) else 0.0,
@@ -298,7 +320,8 @@ class OpenMeteoProvider(WeatherProvider):
condition=WMO_CODE_MAP.get(code, WeatherCondition.UNKNOWN),
condition_text=WMO_DESCRIPTIONS.get(code, "Unknown"),
precipitation_chance=daily_data.get("precipitation_probability_max", [])[i] if i < len(daily_data.get("precipitation_probability_max", [])) else None,
precipitation_mm=daily_data.get("precipitation_sum", [])[i] if i < len(daily_data.get("precipitation_sum", [])) else None
precipitation_mm=daily_data.get("precipitation_sum", [])[i] if i < len(daily_data.get("precipitation_sum", [])) else None,
uv_index_max=uv_max
))
return WeatherForecast(
@@ -309,3 +332,110 @@ class OpenMeteoProvider(WeatherProvider):
except httpx.HTTPError as e:
logger.error(f"Forecast request failed for {location.name}: {e}")
raise ValueError(f"Failed to get forecast: {e}")
async def get_sun_times(self, location: GeoLocation) -> SunTimes:
"""
Get sunrise/sunset times for today.
Args:
location: GeoLocation with lat/long
Returns:
SunTimes with sunrise, sunset, and daylight duration
Raises:
ValueError: If API request fails
"""
try:
response = await self.client.get(
self.WEATHER_URL,
params={
"latitude": location.latitude,
"longitude": location.longitude,
"daily": [
"sunrise",
"sunset",
"daylight_duration"
],
"timezone": self.timezone,
"forecast_days": 1
}
)
response.raise_for_status()
data = response.json()
daily = data.get("daily", {})
date_str = daily.get("time", [""])[0]
sunrise_str = daily.get("sunrise", [""])[0]
sunset_str = daily.get("sunset", [""])[0]
daylight = daily.get("daylight_duration", [0])[0]
return SunTimes(
location=location.name,
date=datetime.fromisoformat(date_str) if date_str else datetime.now(),
sunrise=datetime.fromisoformat(sunrise_str) if sunrise_str else datetime.now(),
sunset=datetime.fromisoformat(sunset_str) if sunset_str else datetime.now(),
daylight_duration=int(daylight) if daylight else 0
)
except httpx.HTTPError as e:
logger.error(f"Sun times request failed for {location.name}: {e}")
raise ValueError(f"Failed to get sun times: {e}")
async def get_air_quality(self, location: GeoLocation) -> AirQuality:
"""
Get current air quality for a location.
Args:
location: GeoLocation with lat/long
Returns:
AirQuality with pollutant measurements and AQI
Raises:
ValueError: If API request fails
"""
try:
response = await self.client.get(
self.AIR_QUALITY_URL,
params={
"latitude": location.latitude,
"longitude": location.longitude,
"current": [
"european_aqi",
"us_aqi",
"pm2_5",
"pm10",
"ozone",
"nitrogen_dioxide",
"sulphur_dioxide",
"carbon_monoxide",
"grass_pollen",
"birch_pollen",
"alder_pollen"
],
"timezone": self.timezone
}
)
response.raise_for_status()
data = response.json()
current = data.get("current", {})
return AirQuality(
location=location.name,
timestamp=datetime.now(),
aqi_european=current.get("european_aqi"),
aqi_us=current.get("us_aqi"),
pm2_5=current.get("pm2_5"),
pm10=current.get("pm10"),
ozone=current.get("ozone"),
nitrogen_dioxide=current.get("nitrogen_dioxide"),
sulphur_dioxide=current.get("sulphur_dioxide"),
carbon_monoxide=current.get("carbon_monoxide"),
pollen_grass=current.get("grass_pollen"),
pollen_birch=current.get("birch_pollen"),
pollen_alder=current.get("alder_pollen")
)
except httpx.HTTPError as e:
logger.error(f"Air quality request failed for {location.name}: {e}")
raise ValueError(f"Failed to get air quality: {e}")
+2 -2
View File
@@ -18,7 +18,7 @@ class SchedulerTask(BaseModel):
task_name: str = Field(..., description="Unique task identifier")
service: str = Field(default="library-desk", description="Service that owns this task")
executor: str = Field(default="rest_api", description="Executor type")
executor: str = Field(default="rest_api_executor", description="Executor type")
priority: int = Field(default=50, ge=1, le=100, description="Priority (lower = higher)")
description: Optional[str] = Field(None, description="Human-readable description")
enabled: bool = Field(default=True, description="Whether task is enabled")
@@ -291,7 +291,7 @@ class SchedulerClient:
task = SchedulerTask(
task_name=task_name,
service="library-desk",
executor="rest_api",
executor="rest_api_executor",
priority=60, # Background maintenance priority
description=description or f"Prefetch {namespace}/{key} for {user}",
minute=schedule.get("minute", -1),
+88 -1
View File
@@ -18,7 +18,7 @@ from pathlib import Path
from src.config import Settings, get_settings, __version__
from src.core.dependencies import (
verify_api_key, QdrantDep, WikiJSDep, OllamaDep, Neo4jDep
verify_api_key, QdrantDep, WikiJSDep, OllamaDep, Neo4jDep, PaperlessDep
)
from src.core.multi_tenancy import DEFAULT_USER
@@ -85,6 +85,14 @@ class HealthResponse(BaseModel):
services: Dict[str, Any]
class StatsResponse(BaseModel):
"""System statistics response model."""
neo4j: Dict[str, int]
qdrant: Dict[str, Any]
wiki_pages: int
paperless: Dict[str, Any]
# Routes
@app.get("/", tags=["Root"])
async def root() -> Dict[str, str]:
@@ -141,6 +149,85 @@ async def health(settings: Settings = Depends(get_settings)) -> HealthResponse:
)
@app.get("/stats", response_model=StatsResponse, tags=["System"])
async def stats(
user: str = Query(default=DEFAULT_USER, description="User identifier"),
neo4j: Neo4jDep = None,
qdrant: QdrantDep = None,
wikijs: WikiJSDep = None,
paperless: PaperlessDep = None,
api_key: str = Depends(verify_api_key)
) -> StatsResponse:
"""
Get system statistics.
Returns counts for:
- Neo4j: nodes by type (Document, Entity, Collection, Search)
- Qdrant: vectors per collection
- Wiki.js: total page count
- Paperless: documents, tags, correspondents, document types
"""
# Neo4j node counts by label
neo4j_stats = {}
try:
for label in ["Document", "Entity", "Collection", "Search"]:
result = await neo4j.execute_query(
f"MATCH (n:{label}) RETURN count(n) as count"
)
neo4j_stats[label.lower() + "_nodes"] = result[0]["count"] if result else 0
except Exception as e:
logger.error(f"Failed to get Neo4j stats: {e}")
neo4j_stats = {"error": str(e)}
# Qdrant collection stats
qdrant_stats = {}
try:
collections = await qdrant.list_collections()
qdrant_stats["collections"] = len(collections)
qdrant_stats["total_vectors"] = sum(c.get("vectors_count", 0) for c in collections)
qdrant_stats["by_collection"] = {
c["name"]: c["vectors_count"] for c in collections
}
except Exception as e:
logger.error(f"Failed to get Qdrant stats: {e}")
qdrant_stats = {"error": str(e)}
# Wiki.js page count
wiki_pages = 0
try:
pages = await wikijs.list_all_pages(user)
wiki_pages = len(pages)
except Exception as e:
logger.warning(f"Failed to get Wiki.js stats: {e}")
# Paperless-ngx document stats
paperless_stats = {}
try:
# Get document count (page_size=1 for efficiency, we just need the count)
docs_result = await paperless.list_documents(page_size=1)
paperless_stats["documents"] = docs_result.get("count", 0)
# Get metadata counts
tags = await paperless.list_tags()
paperless_stats["tags"] = len(tags)
correspondents = await paperless.list_correspondents()
paperless_stats["correspondents"] = len(correspondents)
doc_types = await paperless.list_document_types()
paperless_stats["document_types"] = len(doc_types)
except Exception as e:
logger.warning(f"Failed to get Paperless stats: {e}")
paperless_stats = {"error": str(e)}
return StatsResponse(
neo4j=neo4j_stats,
qdrant=qdrant_stats,
wiki_pages=wiki_pages,
paperless=paperless_stats
)
@app.post("/ingest/check-updates", tags=["Ingestion"])
async def check_updates(
documents: Dict[str, Any],
+17 -12
View File
@@ -18,7 +18,9 @@ class VolatileNamespace(str, Enum):
Each namespace can have different default TTLs and refresh schedules.
"""
# Real-time external data
WEATHER = "weather" # Current conditions, forecasts
WEATHER = "weather" # Current conditions (temperature, humidity, wind)
FORECAST = "forecast" # Multi-day weather outlook
SUN = "sun" # Sunrise, sunset, daylight duration
NEWS = "news" # Headlines, breaking news
FINANCIAL = "financial" # Stock prices, exchange rates, crypto
TRANSIT = "transit" # Train/bus schedules, delays, disruptions
@@ -36,18 +38,21 @@ class VolatileNamespace(str, Enum):
# Default TTLs per namespace (in seconds)
# TTL = 2x refresh interval to ensure data survives missed/delayed refreshes
NAMESPACE_DEFAULT_TTL: Dict[str, int] = {
VolatileNamespace.WEATHER: 1800, # 30 min - weather changes slowly
VolatileNamespace.NEWS: 3600, # 1 hour - news cycles
VolatileNamespace.FINANCIAL: 300, # 5 min - markets move fast
VolatileNamespace.TRANSIT: 300, # 5 min - schedules update frequently
VolatileNamespace.TRAFFIC: 600, # 10 min - traffic patterns
VolatileNamespace.AIR_QUALITY: 3600, # 1 hour - air quality stable
VolatileNamespace.SPORTS: 60, # 1 min - live scores
VolatileNamespace.SOCIAL: 600, # 10 min - social notifications
VolatileNamespace.SYSTEM: 60, # 1 min - system health
VolatileNamespace.CONTEXT: 3600, # 1 hour - session context
VolatileNamespace.CUSTOM: 3600, # 1 hour - default for custom
VolatileNamespace.WEATHER: 7200, # 2 hours (hourly refresh)
VolatileNamespace.FORECAST: 86400, # 24 hours (12hr refresh)
VolatileNamespace.SUN: 172800, # 48 hours (daily refresh)
VolatileNamespace.NEWS: 7200, # 2 hours (hourly refresh)
VolatileNamespace.FINANCIAL: 600, # 10 min (5 min refresh)
VolatileNamespace.TRANSIT: 600, # 10 min (5 min refresh)
VolatileNamespace.TRAFFIC: 1200, # 20 min (10 min refresh)
VolatileNamespace.AIR_QUALITY: 7200, # 2 hours (hourly refresh)
VolatileNamespace.SPORTS: 120, # 2 min (1 min refresh)
VolatileNamespace.SOCIAL: 1200, # 20 min (10 min refresh)
VolatileNamespace.SYSTEM: 120, # 2 min (1 min refresh)
VolatileNamespace.CONTEXT: 7200, # 2 hours - session context
VolatileNamespace.CUSTOM: 7200, # 2 hours - default for custom
}
+199 -7
View File
@@ -233,16 +233,16 @@ async def store_volatile(
async def fetch_weather(
city: str,
user: str = Query(default=DEFAULT_USER, description="User identifier"),
ttl: int = Query(default=86400, ge=60, le=604800, description="TTL in seconds"),
ttl: int = Query(default=7200, ge=60, le=86400, description="TTL in seconds (default 2 hours)"),
qdrant: QdrantDep = None,
ollama: OllamaDep = None,
api_key: str = Depends(verify_api_key)
):
"""
Fetch current weather for a city and store in volatile cache.
Fetch current weather conditions for a city and store in volatile cache.
Called by scheduler for prefetch or on-demand. Geocodes city name
and fetches weather from Open-Meteo API.
Stores temperature, humidity, wind, UV index. For forecasts use /fetch/forecast.
Called by scheduler for hourly prefetch or on-demand.
**Example:**
```
@@ -257,7 +257,49 @@ async def fetch_weather(
weather_provider=weather_provider,
)
result = await fetch_service.fetch_weather(user, city, ttl=ttl)
result = await fetch_service.fetch_current_weather(user, city, ttl=ttl)
if not result.success:
raise HTTPException(status_code=500, detail=result.error)
return {
"success": True,
"namespace": result.namespace,
"key": result.key,
"record": result.record,
}
@router.post("/fetch/forecast/{city}")
async def fetch_forecast(
city: str,
user: str = Query(default=DEFAULT_USER, description="User identifier"),
days: int = Query(default=7, ge=1, le=16, description="Forecast days (1-16)"),
ttl: int = Query(default=86400, ge=60, le=604800, description="TTL in seconds (default 24 hours)"),
qdrant: QdrantDep = None,
ollama: OllamaDep = None,
api_key: str = Depends(verify_api_key)
):
"""
Fetch weather forecast for a city and store in volatile cache.
Stores multi-day outlook with highs/lows, precipitation, UV.
For current conditions use /fetch/weather.
**Example:**
```
POST /volatile/fetch/forecast/amsterdam?user=jpmschweitzer&days=7
```
"""
volatile_service = get_volatile_service(qdrant, ollama)
weather_provider = get_weather_provider()
fetch_service = VolatileFetchService(
volatile_service=volatile_service,
weather_provider=weather_provider,
)
result = await fetch_service.fetch_forecast(user, city, days=days, ttl=ttl)
if not result.success:
raise HTTPException(status_code=500, detail=result.error)
@@ -318,7 +360,7 @@ async def fetch_news(
async def fetch_stock(
symbol: str,
user: str = Query(default=DEFAULT_USER, description="User identifier"),
ttl: int = Query(default=300, ge=60, le=3600, description="TTL in seconds"),
ttl: int = Query(default=600, ge=60, le=3600, description="TTL in seconds (default 10 min)"),
qdrant: QdrantDep = None,
ollama: OllamaDep = None,
api_key: str = Depends(verify_api_key)
@@ -367,7 +409,7 @@ async def fetch_crypto(
symbol: str,
market: str = Query(default="USD", description="Market currency"),
user: str = Query(default=DEFAULT_USER, description="User identifier"),
ttl: int = Query(default=300, ge=60, le=3600, description="TTL in seconds"),
ttl: int = Query(default=600, ge=60, le=3600, description="TTL in seconds (default 10 min)"),
qdrant: QdrantDep = None,
ollama: OllamaDep = None,
api_key: str = Depends(verify_api_key)
@@ -411,6 +453,156 @@ async def fetch_crypto(
}
@router.post("/fetch/sun/{city}")
async def fetch_sun_times(
city: str,
user: str = Query(default=DEFAULT_USER, description="User identifier"),
ttl: int = Query(default=172800, ge=60, le=604800, description="TTL in seconds (default 48 hours)"),
qdrant: QdrantDep = None,
ollama: OllamaDep = None,
api_key: str = Depends(verify_api_key)
):
"""
Fetch sunrise/sunset times for a city and store in volatile cache.
Fetches from Open-Meteo API. Useful for home automation triggers.
**Example:**
```
POST /volatile/fetch/sun/rotterdam?user=jpmschweitzer
```
**Response data includes:**
- sunrise/sunset times (both HH:MM and ISO formats)
- daylight_duration_seconds
- daylight_hours
- Natural language text summary
"""
volatile_service = get_volatile_service(qdrant, ollama)
weather_provider = get_weather_provider()
fetch_service = VolatileFetchService(
volatile_service=volatile_service,
weather_provider=weather_provider,
)
result = await fetch_service.fetch_sun_times(user, city, ttl=ttl)
if not result.success:
raise HTTPException(status_code=500, detail=result.error)
return {
"success": True,
"namespace": result.namespace,
"key": result.key,
"record": result.record,
}
@router.post("/fetch/air_quality/{city}")
async def fetch_air_quality(
city: str,
user: str = Query(default=DEFAULT_USER, description="User identifier"),
ttl: int = Query(default=7200, ge=60, le=86400, description="TTL in seconds (default 2 hours)"),
qdrant: QdrantDep = None,
ollama: OllamaDep = None,
api_key: str = Depends(verify_api_key)
):
"""
Fetch air quality data for a city and store in volatile cache.
Fetches from Open-Meteo Air Quality API.
**Example:**
```
POST /volatile/fetch/air_quality/rotterdam?user=jpmschweitzer
```
**Response data includes:**
- European and US AQI indices
- Pollutants: PM2.5, PM10, ozone, nitrogen dioxide, etc.
- Pollen data (European locations, seasonal)
- Natural language text summary
"""
volatile_service = get_volatile_service(qdrant, ollama)
weather_provider = get_weather_provider()
fetch_service = VolatileFetchService(
volatile_service=volatile_service,
weather_provider=weather_provider,
)
result = await fetch_service.fetch_air_quality(user, city, ttl=ttl)
if not result.success:
raise HTTPException(status_code=500, detail=result.error)
return {
"success": True,
"namespace": result.namespace,
"key": result.key,
"record": result.record,
}
@router.post("/fetch/environment/{city}")
async def fetch_environment(
city: str,
user: str = Query(default=DEFAULT_USER, description="User identifier"),
weather_ttl: int = Query(default=7200, ge=60, le=86400, description="Weather TTL in seconds (default 2 hours)"),
air_quality_ttl: int = Query(default=7200, ge=60, le=86400, description="Air quality TTL in seconds (default 2 hours)"),
qdrant: QdrantDep = None,
ollama: OllamaDep = None,
api_key: str = Depends(verify_api_key)
):
"""
Fetch weather and air quality concurrently for a city.
Performs a single geocode lookup and fetches both weather and air quality
data in parallel, storing both in volatile cache. More efficient than
calling /fetch/weather and /fetch/air_quality separately.
**Example:**
```
POST /volatile/fetch/environment/rotterdam?user=jpmschweitzer
```
**Response includes:**
- weather: Current conditions (temperature, humidity, wind, UV)
- air_quality: AQI indices, pollutants, pollen data
"""
volatile_service = get_volatile_service(qdrant, ollama)
weather_provider = get_weather_provider()
fetch_service = VolatileFetchService(
volatile_service=volatile_service,
weather_provider=weather_provider,
)
result = await fetch_service.fetch_environment(
user, city, weather_ttl=weather_ttl, air_quality_ttl=air_quality_ttl
)
if not result.success:
raise HTTPException(status_code=500, detail="; ".join(result.errors))
return {
"success": True,
"key": result.key,
"weather": {
"success": result.weather.success if result.weather else False,
"record": result.weather.record if result.weather else None,
"error": result.weather.error if result.weather else None,
},
"air_quality": {
"success": result.air_quality.success if result.air_quality else False,
"record": result.air_quality.record if result.air_quality else None,
"error": result.air_quality.error if result.air_quality else None,
},
"errors": result.errors,
}
@router.get("/{namespace}/{key}", response_model=VolatileRecordResponse)
async def get_record(
namespace: str,
+395 -18
View File
@@ -5,15 +5,19 @@ Orchestrates fetching data from external APIs and storing in volatile cache.
Called by scheduler for prefetch or by HybridRAG for reactive caching.
"""
import asyncio
import logging
from typing import Optional
from dataclasses import dataclass
from dataclasses import dataclass, field
from src.apis import (
OpenMeteoProvider,
AggregatedNewsProvider,
AlphaVantageProvider,
CurrentWeather,
WeatherForecast,
SunTimes,
AirQuality,
NewsFeed,
StockQuote,
)
@@ -33,6 +37,16 @@ class FetchResult:
error: Optional[str] = None
@dataclass
class EnvironmentFetchResult:
"""Result of combined environment fetch (weather + air quality)."""
success: bool
key: str
weather: Optional[FetchResult] = None
air_quality: Optional[FetchResult] = None
errors: list[str] = field(default_factory=list)
class VolatileFetchService:
"""
Service to fetch external data and store in volatile cache.
@@ -64,25 +78,25 @@ class VolatileFetchService:
self.news = news_provider
self.financial = financial_provider
async def fetch_weather(
async def fetch_current_weather(
self,
user: str,
city: str,
ttl: int = 86400, # 24 hours
ttl: int = 3600, # 1 hour
) -> FetchResult:
"""
Fetch current weather for a city and store in volatile cache.
Fetch current weather conditions for a city and store in volatile cache.
Args:
user: User identifier
city: City name (will be geocoded)
ttl: Time-to-live in seconds
ttl: Time-to-live in seconds (default 1 hour)
Returns:
FetchResult with success status and stored record
"""
try:
# Geocode city and get weather
# Geocode city and get current conditions
location = await self.weather.geocode(city)
if not location:
return FetchResult(
@@ -92,19 +106,23 @@ class VolatileFetchService:
error=f"Could not geocode city: {city}"
)
weather = await self.weather.get_current(location)
current = await self.weather.get_current(location)
# Generate natural language summary
text = current.to_text()
# Convert to storage format
data = {
"temperature": weather.temperature,
"feels_like": weather.feels_like,
"humidity": weather.humidity,
"wind_speed": weather.wind_speed,
"wind_direction": weather.wind_direction,
"conditions": weather.condition_text,
"condition_code": weather.condition.value,
"location": weather.location,
"text": weather.to_text(),
"temperature": current.temperature,
"feels_like": current.feels_like,
"humidity": current.humidity,
"wind_speed": current.wind_speed,
"wind_direction": current.wind_direction,
"conditions": current.condition_text,
"condition_code": current.condition.value,
"uv_index": current.uv_index,
"location": current.location,
"text": text,
}
# Store in volatile cache
@@ -117,7 +135,7 @@ class VolatileFetchService:
ttl=ttl,
)
logger.info(f"Stored weather for {city} (user={user})")
logger.info(f"Stored current weather for {city} (user={user})")
return FetchResult(
success=True,
namespace="weather",
@@ -126,7 +144,7 @@ class VolatileFetchService:
)
except Exception as e:
logger.error(f"Failed to fetch weather for {city}: {e}")
logger.error(f"Failed to fetch current weather for {city}: {e}")
return FetchResult(
success=False,
namespace="weather",
@@ -134,6 +152,94 @@ class VolatileFetchService:
error=str(e)
)
async def fetch_forecast(
self,
user: str,
city: str,
days: int = 7,
ttl: int = 43200, # 12 hours
) -> FetchResult:
"""
Fetch weather forecast for a city and store in volatile cache.
Args:
user: User identifier
city: City name (will be geocoded)
days: Number of forecast days (1-16)
ttl: Time-to-live in seconds (default 12 hours)
Returns:
FetchResult with success status and stored record
"""
try:
# Geocode city and get forecast
location = await self.weather.geocode(city)
if not location:
return FetchResult(
success=False,
namespace="forecast",
key=city.lower(),
error=f"Could not geocode city: {city}"
)
forecast = await self.weather.get_forecast(location, days=days)
# Build daily forecast array
daily_forecasts = []
for day in forecast.daily:
daily_forecasts.append({
"date": day.date.isoformat(),
"day_name": day.date.strftime("%A"),
"temp_high": day.temp_high,
"temp_low": day.temp_low,
"conditions": day.condition_text,
"condition_code": day.condition.value,
"precipitation_chance": day.precipitation_chance,
"precipitation_mm": day.precipitation_mm,
"uv_index_max": day.uv_index_max,
})
# Generate natural language summary
forecast_lines = [f"{city} {days}-day forecast:"]
for day in forecast.daily:
forecast_lines.append(day.to_text())
text = "\n".join(forecast_lines)
# Convert to storage format
data = {
"days": days,
"daily": daily_forecasts,
"location": forecast.current.location,
"text": text,
}
# Store in volatile cache
record = await self.volatile.store(
user=user,
namespace=VolatileNamespace.FORECAST,
key=city.lower(),
data=data,
source="openmeteo",
ttl=ttl,
)
logger.info(f"Stored {days}-day forecast for {city} (user={user})")
return FetchResult(
success=True,
namespace="forecast",
key=city.lower(),
record=record
)
except Exception as e:
logger.error(f"Failed to fetch forecast for {city}: {e}")
return FetchResult(
success=False,
namespace="forecast",
key=city.lower(),
error=str(e)
)
async def fetch_news(
self,
user: str,
@@ -357,3 +463,274 @@ class VolatileFetchService:
key=f"{symbol.lower()}_{market.lower()}",
error=str(e)
)
async def fetch_sun_times(
self,
user: str,
city: str,
ttl: int = 86400, # 24 hours
) -> FetchResult:
"""
Fetch sunrise/sunset times for a city and store in volatile cache.
Args:
user: User identifier
city: City name (will be geocoded)
ttl: Time-to-live in seconds
Returns:
FetchResult with success status and stored record
"""
try:
# Geocode city and get sun times
location = await self.weather.geocode(city)
if not location:
return FetchResult(
success=False,
namespace="sun",
key=city.lower(),
error=f"Could not geocode city: {city}"
)
sun_times = await self.weather.get_sun_times(location)
# Convert to storage format
data = {
"location": sun_times.location,
"date": sun_times.date.isoformat(),
"sunrise": sun_times.sunrise.strftime("%H:%M"),
"sunset": sun_times.sunset.strftime("%H:%M"),
"sunrise_iso": sun_times.sunrise.isoformat(),
"sunset_iso": sun_times.sunset.isoformat(),
"daylight_duration_seconds": sun_times.daylight_duration,
"daylight_hours": sun_times.daylight_duration / 3600,
"text": sun_times.to_text(),
}
# Store in volatile cache
record = await self.volatile.store(
user=user,
namespace=VolatileNamespace.SUN,
key=city.lower(),
data=data,
source="openmeteo",
ttl=ttl,
)
logger.info(f"Stored sun times for {city} (user={user})")
return FetchResult(
success=True,
namespace="sun",
key=city.lower(),
record=record
)
except Exception as e:
logger.error(f"Failed to fetch sun times for {city}: {e}")
return FetchResult(
success=False,
namespace="sun",
key=city.lower(),
error=str(e)
)
async def fetch_air_quality(
self,
user: str,
city: str,
ttl: int = 3600, # 1 hour
) -> FetchResult:
"""
Fetch air quality data for a city and store in volatile cache.
Args:
user: User identifier
city: City name (will be geocoded)
ttl: Time-to-live in seconds
Returns:
FetchResult with success status and stored record
"""
try:
# Geocode city and get air quality
location = await self.weather.geocode(city)
if not location:
return FetchResult(
success=False,
namespace="air_quality",
key=city.lower(),
error=f"Could not geocode city: {city}"
)
air_quality = await self.weather.get_air_quality(location)
# Convert to storage format
data = {
"location": air_quality.location,
"aqi_european": air_quality.aqi_european,
"aqi_us": air_quality.aqi_us,
"pm2_5": air_quality.pm2_5,
"pm10": air_quality.pm10,
"ozone": air_quality.ozone,
"nitrogen_dioxide": air_quality.nitrogen_dioxide,
"sulphur_dioxide": air_quality.sulphur_dioxide,
"carbon_monoxide": air_quality.carbon_monoxide,
"pollen_grass": air_quality.pollen_grass,
"pollen_birch": air_quality.pollen_birch,
"pollen_alder": air_quality.pollen_alder,
"text": air_quality.to_text(),
}
# Store in volatile cache
record = await self.volatile.store(
user=user,
namespace=VolatileNamespace.AIR_QUALITY,
key=city.lower(),
data=data,
source="openmeteo",
ttl=ttl,
)
logger.info(f"Stored air quality for {city} (user={user})")
return FetchResult(
success=True,
namespace="air_quality",
key=city.lower(),
record=record
)
except Exception as e:
logger.error(f"Failed to fetch air quality for {city}: {e}")
return FetchResult(
success=False,
namespace="air_quality",
key=city.lower(),
error=str(e)
)
async def fetch_environment(
self,
user: str,
city: str,
weather_ttl: int = 3600,
air_quality_ttl: int = 3600,
) -> EnvironmentFetchResult:
"""
Fetch weather and air quality concurrently for a city.
Performs a single geocode lookup and fetches both weather and air quality
data in parallel, storing both in volatile cache.
Args:
user: User identifier
city: City name (will be geocoded once)
weather_ttl: TTL for weather data (default 1 hour)
air_quality_ttl: TTL for air quality data (default 1 hour)
Returns:
EnvironmentFetchResult with both weather and air quality results
"""
errors: list[str] = []
key = city.lower()
# Single geocode lookup (shared by both fetches)
try:
location = await self.weather.geocode(city)
if not location:
return EnvironmentFetchResult(
success=False,
key=key,
errors=[f"Could not geocode city: {city}"]
)
except Exception as e:
return EnvironmentFetchResult(
success=False,
key=key,
errors=[f"Geocoding failed: {e}"]
)
# Fetch weather and air quality concurrently
async def fetch_weather_data() -> FetchResult:
try:
current = await self.weather.get_current(location)
text = current.to_text()
data = {
"temperature": current.temperature,
"feels_like": current.feels_like,
"humidity": current.humidity,
"wind_speed": current.wind_speed,
"wind_direction": current.wind_direction,
"conditions": current.condition_text,
"condition_code": current.condition.value,
"uv_index": current.uv_index,
"location": current.location,
"text": text,
}
record = await self.volatile.store(
user=user,
namespace=VolatileNamespace.WEATHER,
key=key,
data=data,
source="openmeteo",
ttl=weather_ttl,
)
return FetchResult(success=True, namespace="weather", key=key, record=record)
except Exception as e:
return FetchResult(success=False, namespace="weather", key=key, error=str(e))
async def fetch_air_quality_data() -> FetchResult:
try:
air_quality = await self.weather.get_air_quality(location)
data = {
"location": air_quality.location,
"aqi_european": air_quality.aqi_european,
"aqi_us": air_quality.aqi_us,
"pm2_5": air_quality.pm2_5,
"pm10": air_quality.pm10,
"ozone": air_quality.ozone,
"nitrogen_dioxide": air_quality.nitrogen_dioxide,
"sulphur_dioxide": air_quality.sulphur_dioxide,
"carbon_monoxide": air_quality.carbon_monoxide,
"pollen_grass": air_quality.pollen_grass,
"pollen_birch": air_quality.pollen_birch,
"pollen_alder": air_quality.pollen_alder,
"text": air_quality.to_text(),
}
record = await self.volatile.store(
user=user,
namespace=VolatileNamespace.AIR_QUALITY,
key=key,
data=data,
source="openmeteo",
ttl=air_quality_ttl,
)
return FetchResult(success=True, namespace="air_quality", key=key, record=record)
except Exception as e:
return FetchResult(success=False, namespace="air_quality", key=key, error=str(e))
# Run both fetches concurrently
weather_result, air_quality_result = await asyncio.gather(
fetch_weather_data(),
fetch_air_quality_data(),
)
# Collect any errors
if not weather_result.success:
errors.append(f"Weather: {weather_result.error}")
if not air_quality_result.success:
errors.append(f"Air quality: {air_quality_result.error}")
success = weather_result.success or air_quality_result.success
logger.info(
f"Environment fetch for {city} (user={user}): "
f"weather={'ok' if weather_result.success else 'failed'}, "
f"air_quality={'ok' if air_quality_result.success else 'failed'}"
)
return EnvironmentFetchResult(
success=success,
key=key,
weather=weather_result,
air_quality=air_quality_result,
errors=errors,
)
-18
View File
@@ -116,24 +116,6 @@ class WikiChangeListener:
except Exception as e:
logger.error(f"Failed to handle notification: {e}", exc_info=True)
def _is_automated_user(self, email: str) -> bool:
"""
Check if email belongs to an automated system user.
These are edits made by library-desk via Wiki.js API (entity linking).
We skip processing these to prevent loops.
Customize this list based on your Wiki.js username for library-desk.
"""
automated_users = [
self.settings.wikijs_username, # Library-desk's Wiki.js API user
"library-desk@system",
"automation@system",
"bot@system"
]
return email.lower() in [u.lower() for u in automated_users]
def _is_recently_processed(self, page_id: int) -> bool:
"""Check if page was processed recently (debouncing)."""
if page_id not in self._recent_notifications:
+4 -4
View File
@@ -98,7 +98,7 @@ class TestVolatileNamespaces:
def test_weather_default_ttl(self):
"""Test weather namespace default TTL."""
assert NAMESPACE_DEFAULT_TTL[VolatileNamespace.WEATHER] == 1800 # 30 min
assert NAMESPACE_DEFAULT_TTL[VolatileNamespace.WEATHER] == 3600 # 1 hour (current conditions)
def test_financial_default_ttl(self):
"""Test financial namespace default TTL."""
@@ -110,7 +110,7 @@ class TestVolatileNamespaces:
def test_namespace_count(self):
"""Test we have the expected number of namespaces."""
assert len(VolatileNamespace) == 11
assert len(VolatileNamespace) == 13 # Including SUN, FORECAST
class TestVolatileListResponse:
@@ -274,7 +274,7 @@ class TestVolatileService:
def test_get_default_ttl_known_namespace(self, volatile_service):
"""Test default TTL for known namespace."""
ttl = volatile_service._get_default_ttl("weather")
assert ttl == 1800 # Weather namespace default
assert ttl == 3600 # Weather namespace default (1 hour)
def test_get_default_ttl_unknown_namespace(self, volatile_service):
"""Test default TTL for unknown namespace."""
@@ -366,7 +366,7 @@ class TestVolatileService:
ttl=None # Not specified
)
assert result.ttl == 1800 # Weather default
assert result.ttl == 3600 # Weather default (1 hour)
@pytest.mark.asyncio
async def test_search_empty_collection(self, volatile_service, mock_qdrant, mock_ollama):
+15 -21
View File
@@ -50,22 +50,6 @@ class TestWikiChangeListener:
assert listener._debounce_seconds == 5
assert len(listener._recent_notifications) == 0
@pytest.mark.asyncio
async def test_automated_user_filtering(self, listener):
"""Test that automated users are correctly identified."""
# Automated users should be filtered
assert listener._is_automated_user("librarian@schweitz.net") is True
assert listener._is_automated_user("library-desk@system") is True
assert listener._is_automated_user("automation@system") is True
assert listener._is_automated_user("bot@system") is True
# Case insensitive
assert listener._is_automated_user("LIBRARIAN@SCHWEITZ.NET") is True
# Regular users should not be filtered
assert listener._is_automated_user("user@example.com") is False
assert listener._is_automated_user("john@example.com") is False
@pytest.mark.asyncio
async def test_debouncing_prevents_duplicates(self, listener):
"""Test that debouncing prevents duplicate processing."""
@@ -163,19 +147,29 @@ class TestWikiChangeListener:
assert mock_process.call_args[1]['event'] == 'page.delete'
@pytest.mark.asyncio
async def test_automated_user_notification_filtered(self, listener):
"""Test that notifications from automated users are filtered out."""
async def test_any_user_notification_processed(self, listener):
"""Test that notifications are processed regardless of user email.
Note: The user_email in PostgreSQL notifications is the page CREATOR,
not the editor. We cannot filter by user email because:
- A page created by 'librarian' but edited by a human should be processed
- Filtering by creator would break legitimate page ingestion
Loop prevention is handled by debouncing instead.
"""
mock_connection = AsyncMock()
with patch.object(listener, '_process_page_change', new_callable=AsyncMock) as mock_process:
# Notification from automated user should be skipped
# Even system user notifications should be processed
# (debouncing handles loop prevention, not user filtering)
await listener._handle_notification(
mock_connection, 1234, 'wiki_page_changes',
'UPDATE:123:librarian@schweitz.net'
)
# Process should NOT be called
mock_process.assert_not_called()
# Process SHOULD be called (user filtering is not used)
mock_process.assert_called_once()
assert mock_process.call_args[1]['page_id'] == 123
assert mock_process.call_args[1]['event'] == 'page.update'
@pytest.mark.asyncio
async def test_duplicate_notification_filtered(self, listener):