core-api/CHANGELOG.md

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

Changed

• Alembic refuses to run without a named migration target (core-api T-3) — alembic/env.py exits non-zero before building any connection unless -x target=test or -x target=production is given; nothing defaults, so the most natural command in the repo, a bare alembic current, can no longer reach production by accident. make migrate-test, make migrate-production and make dev-test-server are the three documented ways to reach either database; none of them print a password or a credential-bearing URL.

Fixed

• Migration round-trip tests leave core_api_test at head (core-api T-14) — every round-trip test in tests/test_book_migration.py, tests/test_scribe_migration.py and tests/test_tenants_migration.py now ends on upgrade head rather than its own revision, so a full suite run leaves the test database at head instead of one or two migrations behind it.
• A failing database test no longer prints the test database password (core-api T-12) — the test_database_url fixture now returns a MaskedURL: identical to a plain string for every functional use, but its repr() — which pytest prints for a failing test's fixture arguments, without needing -v — masks the password.

[1.15.0] - 2026-10-19

Added

• A feedback message to tower-control may have no sender (scribe_messages.from_identity nullable, migration 010, core-api T-15 / workspace D-70) — the failure channel must not depend on the thing most likely to break. POST /scribe/messages accepts a missing X-Office-Identity for exactly one shape: an office caller, to is ["tower-control"] alone, and tags contains feedback; stored with from_identity null and from_consumer the caller's consumer. Every other send without an identity is still 400 identity_required, and a user-kind caller gains nothing from the exemption. MessageResponse.from_identity is now optional, so mailbox, history, find and the message_id lookup all answer such a message with a null sender.

[1.14.0] - 2026-10-19

Added

• Archive and trash a message in your own mailbox (scribe_mailbox_entries.archived_at and trashed_at, migration 009, core-api T-13) — deletion is per mailbox entry, never per message, since a message is retained and shared by every recipient (D-66/D-70). PATCH /scribe/mailbox/{entry_id} takes archived and trashed booleans beside read (true sets the timestamp, false clears it); the response carries both states. GET /scribe/mailbox takes view: inbox (default, neither archived nor trashed), archive (archived, not trashed), trash (trashed, regardless of archived_at), all. find now excludes trashed entries.
• Look up a mailbox entry by message number (core-api T-9) — GET /scribe/mailbox?message_id=N answers the caller's own entry for that message, zero or one, in any view, bypassing unread/tag/before_id so no client pages the unread-first/ id-descending listing to find one.

[1.13.0] - 2026-10-19

Added

• A default address on each consumer grant (tenant_consumers.default_address, migration 008, core-api T-11) — the Office's fallback when a call names no address and holds no session binding, since claude.ai's connector keeps no MCP session between tool calls. A composite foreign key (tenant_id, default_address) -> scribe_addresses(owner_tenant, name) ensures a default always names a real address in the same tenant. jpmschweitzer/office-claude defaults to claude-ai where that address already exists; every other grant stays null. GET /tenants/me answers it for the caller's own consumer grant, or null for a user-kind caller or a grant without one; GET /tenants is unchanged. No write route this round — provisioning is later.

[1.12.0] - 2026-10-19

Added

• The tenant registry (src/domains/tenants/, core-api T-8 / workspace D-73) - tenants and tenant_consumers (migration 007), and TenantsService.require_tenant, the one enforcement point Book's and Scribe's own _require_tenant now delegate to instead of checking only presence: a tenant that is not in the table is 403 tenant_unknown, one marked suspended is 403 tenant_suspended, and an Office caller whose consumer holds no grant in that tenant is 403 consumer_not_granted - closing the gap where a typo'd tatlock_user claim opened a fresh, empty namespace rather than being refused. Every existing owner_tenant column (book_records, book_tenant_counters, scribe_addresses, scribe_messages, scribe_mailbox_entries) now carries a foreign key to tenants.id (ON DELETE RESTRICT). Migration 007 seeds the two tenants in use today - jpmschweitzer (household) and llm_tester (test) - both granted to office-claude, the only Office consumer configured in Authentik. Read-only this round: GET /tenants/me answers the caller's own tenant and its granted consumers (the cheap "am I allowed here" probe); GET /tenants answers every tenant for a tatlock/scheduler caller or a list of one for a tenant-bearing caller, never revealing another tenant's existence. No write routes - provisioning a tenant is later (workspace T-195).
• The Scribe (src/domains/scribe/) - the household's message bus (dossier core-api T-5 / workspace D-66, D-68, D-73): register/list addresses, send, mailbox, history, find, mark, tag/untag, all under /scribe/*, behind require_caller("office", "user"). An address is a topic owned by a tenant and a consumer; send's from is always the caller's own identity, never the request body. Every query filters on the caller's tenant (D-73) - an address or message in another tenant answers exactly like one that does not exist, so no id is ever confirmed across that boundary. Messaging only this round: groups, subscriptions, the notification matrix and delivery are later work (workspace T-176). Migration 006 creates scribe_addresses, scribe_messages (retained, full-text indexed), scribe_mailbox_entries.
• Caller carries consumer and tenant from the Office (src/domains/auth/ callers.py, core-api T-4 / workspace D-73) - Caller gains consumer and tenant. For an office caller both X-Office-Consumer and X-Office-Tenant are required on every call, reads included: missing is 400 consumer_required/tenant_required, malformed is 400 consumer_invalid/tenant_invalid. Every other kind ignores both headers outright - tatlock and scheduler carry no tenant, and a user caller's tenant comes only from their own JWT's tatlock_user claim, never from a header.
• Caller auth (src/domains/auth/callers.py) - the first thing in this repo that reads a service key. require_caller() resolves a static office/tatlock/ scheduler key (constant-time compare, checked before any JWT parsing) or falls through to the existing OIDC path. New settings: office_service_key, tatlock_service_key, scheduler_service_key, book_timezone.
• The Book (src/domains/book/) - Notes, Tasks, Decisions, Questions and Rejected options (workspace D-68/D-69/D-70). Routes: add, show, list, find, change, close, link/unlink, tag/untag, history, all under /book/*, behind require_caller("office", "user"). Migration 005 creates the whole schema: book_records, book_tags, book_record_links, book_attachments, book_events (append-only, DB trigger-enforced), book_tenant_counters.
• The Book is behind the tenant boundary (core-api T-7 / workspace D-73) - every route now requires a tenant on the caller (400 tenant_required for a user-kind caller with no tatlock_user claim; an office caller already has one from every call per T-4) and filters on it: a record in another tenant answers exactly like one that does not exist (404 not_found), in both directions, and link/add's parent convenience refuse a target in another tenant the same way. add sets owner_tenant from the caller and allocates number - a short outward id numbered per tenant (each tenant's Book starts at 1, unique on (owner_tenant, number), never reused), allocated via a per-tenant counter row (book_tenant_counters) locked FOR UPDATE in the same transaction as the insert. Responses include number; list accepts number= to find one. Migration 005 (unreleased - production was still at 004) is amended in place to add book_records.owner_tenant/number and book_tenant_counters, rather than patching them in with a later migration.

Changed

• X-Office-Client is retired in favour of X-Office-Consumer (core-api T-4) - the informational header is no longer read; book_events.client (no migration this round) now records the consumer instead.
• Housekeeping reads are locked - GET /housekeeping/* (health, devices, areas, scenes, scripts, automations, history) now require a caller (office, tatlock, or user) instead of being unauthenticated. Writes are unchanged (get_admin_user). api.schweitz.net carries no forward-auth, so this was an open read of the house's device list once Home Assistant's token is restored (T-1 / D-68).

[1.11.0] - 2026-01-08

Added

• News Headlines API - New endpoint for news ticker integration
  • GET /tools/news - Fetch news headlines from user's volatile collection
  • Returns headlines with title, description, source, and URL
  • Data sourced from volatile_{user} Qdrant collection (news namespace)
  • Uses preferred_username from OIDC, falls back to default
• News subdomain under tools (src/domains/tools/news/)
• NewsHeadline and NewsResponse Pydantic schemas
• NewsService for parsing news data from Qdrant

[1.10.12] - 2026-01-08

Fixed

• Forecast data retrieval - Pass raw_data to service instead of extracting wrong field
  • Qdrant client was extracting days (integer 7) instead of daily (list)
  • Now passes full raw_data for service to parse correctly

[1.10.11] - 2026-01-08

Fixed

• Weather wind direction - Convert integer degrees to cardinal direction string
  • Scheduler stores wind_direction as degrees (e.g., 135)
  • Schema expects string (e.g., "SE")
  • Added _degrees_to_cardinal() conversion

[1.10.10] - 2026-01-08

Fixed

• Environment data parsing - Fix parsing of scheduler-generated Qdrant data
  • Forecast: Check daily key first (scheduler stores day count in days, list in daily)
  • Sun times: Use sunrise_iso/sunset_iso fields, handle time-only format fallback
  • Sun times: Calculate daylight from daylight_duration_seconds or daylight_hours
  • Air quality: Support aqi_us/aqi_european and nitrogen_dioxide field names

[1.10.9] - 2026-01-08

Fixed

• Environment user ID cleanup - Strip email domain from user identifier
  • If preferred_username is an email, extract just the username part
  • Ensures Qdrant collection name matches (e.g., volatile_jpmschweitzer not volatile_jpmschweitzer@gmail.com)

[1.10.8] - 2026-01-08

Fixed

• OIDC audience validation - python-jose requires string audience, not list
  • Extract and validate audience from unverified claims first
  • Use token's actual audience for JWT decode (after validating it's allowed)
  • Fixes "audience must be a string or None" error

[1.10.7] - 2026-01-08

Added

• Multi-issuer OIDC support - Accept tokens from multiple OAuth providers
  • Changed oidc_issuer (string) to oidc_issuers (list)
  • Each issuer has its own JWKS endpoint, now cached per-issuer
  • Validates token issuer against allowed list before fetching JWKS
  • Supports tokens from: core-api, tatlock-ui, tatlock OAuth applications
  • Completes fix for environment endpoint user resolution

Removed

• Deprecated src/config.py - consolidated to src/shared/config.py
• Deprecated src/security.py - consolidated to src/shared/security.py

[1.10.6] - 2026-01-08

Fixed

• OIDC audience mismatch - Accept tokens from multiple clients
  • Changed oidc_audience (string) to oidc_audiences (list)
  • Now accepts tokens with audience: core-api, tatlock-ui, or tatlock
  • Fixes environment endpoint returning "default" user instead of authenticated username
  • Fixed main.py to use oidc_audiences[0] for Swagger UI OAuth client

Changed

• Documentation cleanup in README

[1.10.4] - 2026-01-07

Removed

• Ollama integration removed - AI inference is no longer handled by this API
  • Removed src/models/ollama_client.py and all Ollama-related configuration
  • Removed src/models/embeddings.py and src/models/embeddings_ollama.py
  • Removed model aliases and AI configuration from settings
  • Health endpoints no longer check Ollama status
  • Tests updated to reflect database-only health checks

Changed

• Health check /health/full now only checks database connectivity
• Diagnostics endpoint simplified (removed Ollama component info)

[1.10.3] - 2026-01-07

Fixed

• OIDC config not applied to domains module - Both src.auth.oidc and src.domains.auth.oidc configs are now initialized
  • Previously only src.auth.oidc was configured, leaving domains tools using hardcoded "local" user
  • Environment endpoint now correctly uses authenticated user from OIDC token

[1.10.2] - 2026-01-07

Changed

• Enhanced environment endpoint logging - Added detailed user claim logging for debugging
  • Logs both preferred_username and sub claims when resolving user
  • Distinguishes between authenticated and unauthenticated requests

[1.10.1] - 2026-01-06

Fixed

• Fix OIDC import path in tools controller (src.oidc.dependencies → src.auth.oidc)
  • Environment endpoint was returning user: "local" instead of authenticated username
  • Caused queries to wrong Qdrant collection (volatile_local vs volatile_{username})

[1.10.0] - 2026-01-06

Added

• Environment Data API - Qdrant-backed endpoint for weather, forecast, and sun position data
  • GET /tools/environment - Fetch environment data from user's volatile collection
  • Weather: current temperature, conditions, humidity, wind speed
  • Forecast: multi-day outlook with high/low temperatures
  • Sun times: sunrise, sunset, daylight duration
  • Air quality: AQI and quality level (when available)
  • Data sourced from volatile_{user} Qdrant collection
  • Uses preferred_username from OIDC, falls back to default
• qdrant-client dependency for vector database access
• QdrantReadClient wrapper for read-only collection queries
• Comprehensive test suite for environment service parsing

[1.9.4] - 2026-01-04

Fixed

• Fix SQLAlchemy async lazy loading error for new users in /auth/sync
  • Initialize user.roles = [] and user.preferences to avoid greenlet error
  • Was causing "MissingGreenlet: greenlet_spawn has not been called" on new user creation

[1.9.3] - 2026-01-04

Fixed

• Handle non-UUID sub claim in /auth/sync - Authentik JWT may return non-UUID subject identifiers
  • Now derives deterministic UUID from sub string if direct parsing fails

[1.9.2] - 2026-01-04

Fixed

• /auth/users/me endpoint now supports both NPM forward auth headers AND JWT Bearer tokens
  • Added get_current_user_or_forward_auth() combined auth dependency
  • Fixes web authentication where NPM passes X-authentik-* headers instead of JWT
  • Mobile/native clients continue to use JWT Bearer tokens as before

[1.9.1] - 2026-01-03

Fixed

• CORS configuration now uses explicit origins instead of "*"
  • When allow_credentials=True, wildcard origins are rejected by browsers
  • Added home.schweitz.net, tatlock.schweitz.net, and localhost origins

[1.9.0] - 2026-01-03

Added

• NPM Forward Auth Support - Web authentication via Nginx Proxy Manager forward auth
  • GET /auth/me - Get current user from NPM forward auth headers (X-authentik-uid, X-authentik-email, etc.)
  • Auto-creates user on first web login if not in database
  • Syncs roles from NPM forward auth groups header
• get_user_by_email and get_user_by_authentik_id methods in AuthService
• Comprehensive tests for /auth/me endpoint

[1.8.0] - 2026-01-03

Added

• Group-Role Mapping & Permissions (Phase 3)
  • Decoupled group-role architecture (groups from Authentik, roles admin-managed)
  • Permission format: domain.category:action with action hierarchy
  • require_permission and require_any_permission dependency factories
  • Global admin override (admin.general:admin)
• User Profile & API Keys (Phase 4)
  • GET /auth/users/me - Full user profile with roles and preferences
  • GET/PATCH /auth/users/me/preferences - User preferences management
  • GET/POST/DELETE /auth/users/me/api-keys - API key lifecycle
  • API keys with tak_ prefix, SHA-256 hashing, shown only once on creation

[1.7.0] - 2026-01-03

Added

• System Stats API - Host system resource monitoring for dashboard widgets
  • GET /tools/system/stats - Real-time host system statistics
  • CPU: usage percentage, core count, load averages
  • Memory: usage percentage, total/used/available bytes
  • Disks: all mounted filesystems with usage stats (auto-discovers mounts)
  • Network: total bytes sent/received
  • GPU/VRAM: NVIDIA GPU memory usage (via nvidia-smi if available)
• psutil dependency for cross-platform system metrics

[1.6.1] - 2026-01-03

Added

• link_type field to quick links for iframe vs new tab behavior

[1.6.0] - 2026-01-03

Added

• Dashboard API - Quick links and widgets management for Organizr-style dashboard
  • GET /dashboard/quick-links - List quick links with category/visibility filtering
  • GET /dashboard/quick-links/{id} - Get single quick link
  • POST /dashboard/quick-links - Create quick link
  • PUT /dashboard/quick-links/{id} - Update quick link
  • DELETE /dashboard/quick-links/{id} - Delete quick link
  • POST /dashboard/quick-links/reorder - Reorder quick links by position
  • GET /dashboard/widgets - List dashboard widgets
  • GET /dashboard/widgets/{id} - Get single widget
  • POST /dashboard/widgets - Create widget
  • PUT /dashboard/widgets/{id} - Update widget
  • DELETE /dashboard/widgets/{id} - Delete widget
• Database migrations for quick_links and dashboard_widgets tables
• Static file controller for serving Organizr widgets (/static/widgets)
• Default local user authentication when OIDC is disabled

Changed

• Domain-based architecture - Refactored codebase to domain-driven structure
  • src/domains/ - Domain modules (auth, dashboard, health, housekeeping, infrastructure, tools)
  • src/shared/ - Shared utilities (base, config, database, logging, security, clients)
• Test suite updated for new domain structure (285 tests passing)

[1.5.0] - 2026-01-01

Added

• Groups Management - Authentik group synchronization
  • GET /auth/groups - List all groups with search and pagination
  • POST /auth/groups/sync-from-authentik - Bulk sync groups from Authentik admin API
• Database model and migration for groups table

[1.4.6] - 2026-01-01

Changed

• Code cleanup: move inline re import to top of auth/service.py

[1.4.5] - 2026-01-01

Fixed

• Separate httpx and SQLAlchemy async contexts in bulk sync (fixes greenlet error)

[1.4.4] - 2026-01-01

Fixed

• Use uuid field instead of pk for Authentik user sync (pk is integer, uuid is proper UUID)
• Skip internal_service_account type users during bulk sync

[1.4.3] - 2026-01-01

Fixed

• Manually extract and send session cookies for Authentik flow auth (fixes cross-domain cookie handling)

[1.4.2] - 2026-01-01

Fixed

• Use Authentik domain URL (auth.schweitz.net) instead of IP to fix cookie domain matching

[1.4.1] - 2026-01-01

Fixed

• Authentik config now uses AUTHENTIK_USERNAME/PASSWORD to match production env vars

[1.4.0] - 2026-01-01

Added

• Authentication & User Management - Authentik integration for user synchronization
  • GET /auth/me - Get current authenticated user info
  • GET /auth/users - List all users with search and pagination
  • POST /auth/users/sync-from-authentik - Bulk sync users from Authentik admin API
• PostgreSQL database integration with async SQLAlchemy
• Alembic database migrations for schema management
• Database models: User, Role, UserPreferences, ApiKey
• Token validation via Authentik userinfo endpoint
• Role synchronization from Authentik groups
• Health check now includes database connectivity status

Changed

• Authentik configuration now uses username/password for admin API access
• Health endpoint includes database status in diagnostics

[1.3.1] - 2025-12-31

Changed

• Simplified configuration: all settings now read from environment variables/.env only
• Removed Docker socket fallback for container operations (Portainer API is now required)
• Updated default search provider to SearXNG

Removed

• src/credentials.py - credentials now managed via environment variables
• Docker socket fallback methods from Portainer client
• Unused search API settings (Brave, Google)

[1.3.0] - 2025-12-31

Added

• Stack Management Endpoints for Tatlock Control Room integration
  • GET /infrastructure/stacks/{stackId}/compose - Get stack Docker Compose YAML
  • PUT /infrastructure/stacks/{stackId}/compose - Update stack Docker Compose YAML
  • GET /infrastructure/stacks/{stackId}/env - Get stack environment variables
  • PUT /infrastructure/stacks/{stackId}/env - Update stack environment variables
  • POST /infrastructure/stacks/{stackId}/deploy - Redeploy stack
  • POST /infrastructure/stacks/{stackId}/rebuild - Pull images and recreate containers
• DELETE /infrastructure/containers/{id} - Delete container with optional force flag
• Portainer client methods: get_stack_file, redeploy_stack, update_stack_env, delete_container, restart_container

Removed

• REQUESTED_SERVICES.md - endpoint specifications now implemented

[1.2.1] - 2025-12-17

Fixed

• Device control endpoint now returns correct new state after action (added 300ms delay for HA state propagation)

Added

• AGENTS.md with project coding guidelines and release flow documentation

Changed

• Removed obsolete web scraper settings from .env.example

[1.2.0] - 2025-12-17

Added

• Housekeeping API - Home Assistant integration for smart home control
  • GET /housekeeping/health - HA connection status
  • GET /housekeeping/devices - List controllable devices with optional domain/area filtering
  • GET /housekeeping/devices/{entity_id} - Get device details
  • POST /housekeeping/devices/{entity_id}/control - Control devices (turn_on, turn_off, toggle, set_brightness)
  • GET /housekeeping/scenes - List available scenes
  • POST /housekeeping/scenes/{scene_id}/activate - Activate a scene
  • GET /housekeeping/scripts - List available scripts
  • POST /housekeeping/scripts/{script_id}/run - Run a script
  • GET /housekeeping/automations - List automations
  • POST /housekeeping/automations/{automation_id}/toggle - Enable/disable automation
  • GET /housekeeping/history - Query state history
  • GET /housekeeping/areas - List rooms/areas
• Home Assistant REST API client (src/clients/homeassistant_client.py)
• Home Assistant configuration in credentials and settings
• Comprehensive test suite with 65% code coverage (285 tests)
• Tests for NPM client, Ollama client, AI client, OIDC authentication
• Tests for infrastructure, health, tools, and housekeeping endpoints

Removed

• Web Scraper - Entire web scraping module removed
  • src/web_scraper/ directory deleted
  • /web-scraper/scrape endpoint removed
  • Trafilatura and BeautifulSoup dependencies removed from scraping use

Changed

• Updated README.md with current architecture and all endpoints
• Tools controller now only contains DNS lookup functionality
• Health controller endpoints list updated to reflect current features

[1.1.2] - 2024-12-14

Added

• Version field to /health endpoint response

[1.1.1] - 2024-12-14

Added

• Watchtower trigger step in CI workflow for automatic container updates

[1.1.0] - 2024-12-14

Removed

• Uptime Kuma integration (kuma_client.py deleted)
• All /infrastructure/monitors endpoints
• Kuma monitor pause/resume from service start/stop operations
• Uptime percentage display from service control widget
• Kuma-related configuration and credentials
• Stale memory tests (memory functionality moved to core-ai service)

Changed

• Service control widget now uses Portainer container status exclusively
• Simplified widget-data endpoint response (removed monitors field)
• Updated service start/stop to only manage containers via Portainer

[1.0.0] - 2024-12-14

Added

• Initial release of Core Code API service extracted from portainer-core
• Infrastructure management endpoints for Portainer, NPM, and Uptime Kuma
• Web scraping service with content extraction
• DNS management endpoints
• Health check endpoints with diagnostics
• OIDC authentication support via Authentik
• Ollama integration for embeddings
• OpenAPI documentation (Swagger UI and ReDoc)
• Docker containerization with CI/CD workflow