# Changelog All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] ## [1.16.0] - 2026-10-20 ### Added - **`GET /scribe/mailbox` takes `since_message_id`, and answers with `newest_message_id`** (core-api T-16) — the Office's `glance` call needs "has anything changed since I last looked?" with counts. `since_message_id=N` keeps only entries whose message number is above `N`, combined with `view`/`unread`/`tag` as today (`message_id`, which already bypasses the other filters, keeps bypassing `since_message_id` too). `total` already counted every match independent of `limit`; that is unchanged, `since_message_id` is just one more filter it counts against. `newest_message_id` is the caller's newest message number across its whole mailbox — every view, regardless of any filter applied — null when the mailbox is empty; it exists because the unread-first sort can return an older unread entry on a `limit=1` page while a newer message is already read. ### Changed - **Alembic refuses to run without a named migration target** (core-api T-3) — `alembic/env.py` exits non-zero before building any connection unless `-x target=test` or `-x target=production` is given; nothing defaults, so the most natural command in the repo, a bare `alembic current`, can no longer reach production by accident. `make migrate-test`, `make migrate-production` and `make dev-test-server` are the three documented ways to reach either database; none of them print a password or a credential-bearing URL. ### Fixed - **Migration round-trip tests leave `core_api_test` at head** (core-api T-14) — every round-trip test in `tests/test_book_migration.py`, `tests/test_scribe_migration.py` and `tests/test_tenants_migration.py` now ends on `upgrade head` rather than its own revision, so a full suite run leaves the test database at head instead of one or two migrations behind it. - **A failing database test no longer prints the test database password** (core-api T-12) — the `test_database_url` fixture now returns a `MaskedURL`: identical to a plain string for every functional use, but its `repr()` — which pytest prints for a failing test's fixture arguments, without needing `-v` — masks the password. ## [1.15.0] - 2026-10-19 ### Added - **A `feedback` message to `tower-control` may have no sender** (`scribe_messages.from_identity` nullable, migration 010, core-api T-15 / workspace D-70) — the failure channel must not depend on the thing most likely to break. `POST /scribe/messages` accepts a missing `X-Office-Identity` for exactly one shape: an office caller, `to` is `["tower-control"]` alone, and `tags` contains `feedback`; stored with `from_identity` null and `from_consumer` the caller's consumer. Every other send without an identity is still 400 `identity_required`, and a `user`-kind caller gains nothing from the exemption. `MessageResponse.from_identity` is now optional, so `mailbox`, `history`, `find` and the `message_id` lookup all answer such a message with a null sender. ## [1.14.0] - 2026-10-19 ### Added - **Archive and trash a message in your own mailbox** (`scribe_mailbox_entries.archived_at` and `trashed_at`, migration 009, core-api T-13) — deletion is per mailbox entry, never per message, since a message is retained and shared by every recipient (D-66/D-70). `PATCH /scribe/mailbox/{entry_id}` takes `archived` and `trashed` booleans beside `read` (true sets the timestamp, false clears it); the response carries both states. `GET /scribe/mailbox` takes `view`: `inbox` (default, neither archived nor trashed), `archive` (archived, not trashed), `trash` (trashed, regardless of `archived_at`), `all`. `find` now excludes trashed entries. - **Look up a mailbox entry by message number** (core-api T-9) — `GET /scribe/mailbox?message_id=N` answers the caller's own entry for that message, zero or one, in any view, bypassing `unread`/`tag`/`before_id` so no client pages the unread-first/ id-descending listing to find one. ## [1.13.0] - 2026-10-19 ### Added - **A default address on each consumer grant** (`tenant_consumers.default_address`, migration 008, core-api T-11) — the Office's fallback when a call names no address and holds no session binding, since claude.ai's connector keeps no MCP session between tool calls. A composite foreign key `(tenant_id, default_address)` -> `scribe_addresses(owner_tenant, name)` ensures a default always names a real address in the same tenant. `jpmschweitzer`/`office-claude` defaults to `claude-ai` where that address already exists; every other grant stays null. `GET /tenants/me` answers it for the caller's own consumer grant, or null for a user-kind caller or a grant without one; `GET /tenants` is unchanged. No write route this round — provisioning is later. ## [1.12.0] - 2026-10-19 ### Added - **The tenant registry** (`src/domains/tenants/`, core-api T-8 / workspace D-73) - `tenants` and `tenant_consumers` (migration 007), and `TenantsService.require_tenant`, the one enforcement point Book's and Scribe's own `_require_tenant` now delegate to instead of checking only presence: a tenant that is not in the table is 403 `tenant_unknown`, one marked `suspended` is 403 `tenant_suspended`, and an Office caller whose consumer holds no grant in that tenant is 403 `consumer_not_granted` - closing the gap where a typo'd `tatlock_user` claim opened a fresh, empty namespace rather than being refused. Every existing `owner_tenant` column (`book_records`, `book_tenant_counters`, `scribe_addresses`, `scribe_messages`, `scribe_mailbox_entries`) now carries a foreign key to `tenants.id` (`ON DELETE RESTRICT`). Migration 007 seeds the two tenants in use today - `jpmschweitzer` (household) and `llm_tester` (test) - both granted to `office-claude`, the only Office consumer configured in Authentik. Read-only this round: `GET /tenants/me` answers the caller's own tenant and its granted consumers (the cheap "am I allowed here" probe); `GET /tenants` answers every tenant for a `tatlock`/`scheduler` caller or a list of one for a tenant-bearing caller, never revealing another tenant's existence. No write routes - provisioning a tenant is later (workspace T-195). - **The Scribe** (`src/domains/scribe/`) - the household's message bus (dossier core-api T-5 / workspace D-66, D-68, D-73): `register`/`list` addresses, `send`, `mailbox`, `history`, `find`, `mark`, `tag`/`untag`, all under `/scribe/*`, behind `require_caller("office", "user")`. An address is a topic owned by a tenant and a consumer; `send`'s `from` is always the caller's own identity, never the request body. Every query filters on the caller's tenant (D-73) - an address or message in another tenant answers exactly like one that does not exist, so no id is ever confirmed across that boundary. Messaging only this round: groups, subscriptions, the notification matrix and delivery are later work (workspace T-176). Migration 006 creates `scribe_addresses`, `scribe_messages` (retained, full-text indexed), `scribe_mailbox_entries`. - **Caller carries consumer and tenant from the Office** (`src/domains/auth/ callers.py`, core-api T-4 / workspace D-73) - `Caller` gains `consumer` and `tenant`. For an `office` caller both `X-Office-Consumer` and `X-Office-Tenant` are required on every call, reads included: missing is 400 `consumer_required`/`tenant_required`, malformed is 400 `consumer_invalid`/`tenant_invalid`. Every other kind ignores both headers outright - `tatlock` and `scheduler` carry no tenant, and a `user` caller's tenant comes only from their own JWT's `tatlock_user` claim, never from a header. - **Caller auth** (`src/domains/auth/callers.py`) - the first thing in this repo that reads a service key. `require_caller()` resolves a static office/tatlock/ scheduler key (constant-time compare, checked before any JWT parsing) or falls through to the existing OIDC path. New settings: `office_service_key`, `tatlock_service_key`, `scheduler_service_key`, `book_timezone`. - **The Book** (`src/domains/book/`) - Notes, Tasks, Decisions, Questions and Rejected options (workspace D-68/D-69/D-70). Routes: `add`, `show`, `list`, `find`, `change`, `close`, `link`/`unlink`, `tag`/`untag`, `history`, all under `/book/*`, behind `require_caller("office", "user")`. Migration 005 creates the whole schema: `book_records`, `book_tags`, `book_record_links`, `book_attachments`, `book_events` (append-only, DB trigger-enforced), `book_tenant_counters`. - **The Book is behind the tenant boundary** (core-api T-7 / workspace D-73) - every route now requires a tenant on the caller (400 `tenant_required` for a `user`-kind caller with no `tatlock_user` claim; an `office` caller already has one from every call per T-4) and filters on it: a record in another tenant answers exactly like one that does not exist (404 `not_found`), in both directions, and `link`/`add`'s `parent` convenience refuse a target in another tenant the same way. `add` sets `owner_tenant` from the caller and allocates `number` - a short outward id numbered per tenant (each tenant's Book starts at 1, unique on `(owner_tenant, number)`, never reused), allocated via a per-tenant counter row (`book_tenant_counters`) locked `FOR UPDATE` in the same transaction as the insert. Responses include `number`; `list` accepts `number=` to find one. Migration 005 (unreleased - production was still at 004) is amended in place to add `book_records.owner_tenant`/`number` and `book_tenant_counters`, rather than patching them in with a later migration. ### Changed - **`X-Office-Client` is retired in favour of `X-Office-Consumer`** (core-api T-4) - the informational header is no longer read; `book_events.client` (no migration this round) now records the consumer instead. - **Housekeeping reads are locked** - `GET /housekeeping/*` (health, devices, areas, scenes, scripts, automations, history) now require a caller (`office`, `tatlock`, or `user`) instead of being unauthenticated. Writes are unchanged (`get_admin_user`). `api.schweitz.net` carries no forward-auth, so this was an open read of the house's device list once Home Assistant's token is restored (T-1 / D-68). ## [1.11.0] - 2026-01-08 ### Added - **News Headlines API** - New endpoint for news ticker integration - `GET /tools/news` - Fetch news headlines from user's volatile collection - Returns headlines with title, description, source, and URL - Data sourced from `volatile_{user}` Qdrant collection (news namespace) - Uses `preferred_username` from OIDC, falls back to `default` - News subdomain under tools (`src/domains/tools/news/`) - `NewsHeadline` and `NewsResponse` Pydantic schemas - `NewsService` for parsing news data from Qdrant ## [1.10.12] - 2026-01-08 ### Fixed - **Forecast data retrieval** - Pass raw_data to service instead of extracting wrong field - Qdrant client was extracting `days` (integer 7) instead of `daily` (list) - Now passes full raw_data for service to parse correctly ## [1.10.11] - 2026-01-08 ### Fixed - **Weather wind direction** - Convert integer degrees to cardinal direction string - Scheduler stores wind_direction as degrees (e.g., 135) - Schema expects string (e.g., "SE") - Added `_degrees_to_cardinal()` conversion ## [1.10.10] - 2026-01-08 ### Fixed - **Environment data parsing** - Fix parsing of scheduler-generated Qdrant data - Forecast: Check `daily` key first (scheduler stores day count in `days`, list in `daily`) - Sun times: Use `sunrise_iso`/`sunset_iso` fields, handle time-only format fallback - Sun times: Calculate daylight from `daylight_duration_seconds` or `daylight_hours` - Air quality: Support `aqi_us`/`aqi_european` and `nitrogen_dioxide` field names ## [1.10.9] - 2026-01-08 ### Fixed - **Environment user ID cleanup** - Strip email domain from user identifier - If `preferred_username` is an email, extract just the username part - Ensures Qdrant collection name matches (e.g., `volatile_jpmschweitzer` not `volatile_jpmschweitzer@gmail.com`) ## [1.10.8] - 2026-01-08 ### Fixed - **OIDC audience validation** - python-jose requires string audience, not list - Extract and validate audience from unverified claims first - Use token's actual audience for JWT decode (after validating it's allowed) - Fixes "audience must be a string or None" error ## [1.10.7] - 2026-01-08 ### Added - **Multi-issuer OIDC support** - Accept tokens from multiple OAuth providers - Changed `oidc_issuer` (string) to `oidc_issuers` (list) - Each issuer has its own JWKS endpoint, now cached per-issuer - Validates token issuer against allowed list before fetching JWKS - Supports tokens from: `core-api`, `tatlock-ui`, `tatlock` OAuth applications - Completes fix for environment endpoint user resolution ### Removed - Deprecated `src/config.py` - consolidated to `src/shared/config.py` - Deprecated `src/security.py` - consolidated to `src/shared/security.py` ## [1.10.6] - 2026-01-08 ### Fixed - **OIDC audience mismatch** - Accept tokens from multiple clients - Changed `oidc_audience` (string) to `oidc_audiences` (list) - Now accepts tokens with audience: `core-api`, `tatlock-ui`, or `tatlock` - Fixes environment endpoint returning "default" user instead of authenticated username - Fixed main.py to use `oidc_audiences[0]` for Swagger UI OAuth client ### Changed - Documentation cleanup in README ## [1.10.4] - 2026-01-07 ### Removed - **Ollama integration removed** - AI inference is no longer handled by this API - Removed `src/models/ollama_client.py` and all Ollama-related configuration - Removed `src/models/embeddings.py` and `src/models/embeddings_ollama.py` - Removed model aliases and AI configuration from settings - Health endpoints no longer check Ollama status - Tests updated to reflect database-only health checks ### Changed - Health check `/health/full` now only checks database connectivity - Diagnostics endpoint simplified (removed Ollama component info) ## [1.10.3] - 2026-01-07 ### Fixed - **OIDC config not applied to domains module** - Both `src.auth.oidc` and `src.domains.auth.oidc` configs are now initialized - Previously only `src.auth.oidc` was configured, leaving domains tools using hardcoded "local" user - Environment endpoint now correctly uses authenticated user from OIDC token ## [1.10.2] - 2026-01-07 ### Changed - **Enhanced environment endpoint logging** - Added detailed user claim logging for debugging - Logs both `preferred_username` and `sub` claims when resolving user - Distinguishes between authenticated and unauthenticated requests ## [1.10.1] - 2026-01-06 ### Fixed - Fix OIDC import path in tools controller (`src.oidc.dependencies` → `src.auth.oidc`) - Environment endpoint was returning `user: "local"` instead of authenticated username - Caused queries to wrong Qdrant collection (`volatile_local` vs `volatile_{username}`) ## [1.10.0] - 2026-01-06 ### Added - **Environment Data API** - Qdrant-backed endpoint for weather, forecast, and sun position data - `GET /tools/environment` - Fetch environment data from user's volatile collection - Weather: current temperature, conditions, humidity, wind speed - Forecast: multi-day outlook with high/low temperatures - Sun times: sunrise, sunset, daylight duration - Air quality: AQI and quality level (when available) - Data sourced from `volatile_{user}` Qdrant collection - Uses `preferred_username` from OIDC, falls back to `default` - `qdrant-client` dependency for vector database access - `QdrantReadClient` wrapper for read-only collection queries - Comprehensive test suite for environment service parsing ## [1.9.4] - 2026-01-04 ### Fixed - Fix SQLAlchemy async lazy loading error for new users in `/auth/sync` - Initialize `user.roles = []` and `user.preferences` to avoid greenlet error - Was causing "MissingGreenlet: greenlet_spawn has not been called" on new user creation ## [1.9.3] - 2026-01-04 ### Fixed - Handle non-UUID `sub` claim in `/auth/sync` - Authentik JWT may return non-UUID subject identifiers - Now derives deterministic UUID from sub string if direct parsing fails ## [1.9.2] - 2026-01-04 ### Fixed - `/auth/users/me` endpoint now supports both NPM forward auth headers AND JWT Bearer tokens - Added `get_current_user_or_forward_auth()` combined auth dependency - Fixes web authentication where NPM passes `X-authentik-*` headers instead of JWT - Mobile/native clients continue to use JWT Bearer tokens as before ## [1.9.1] - 2026-01-03 ### Fixed - CORS configuration now uses explicit origins instead of `"*"` - When `allow_credentials=True`, wildcard origins are rejected by browsers - Added `home.schweitz.net`, `tatlock.schweitz.net`, and localhost origins ## [1.9.0] - 2026-01-03 ### Added - **NPM Forward Auth Support** - Web authentication via Nginx Proxy Manager forward auth - `GET /auth/me` - Get current user from NPM forward auth headers (X-authentik-uid, X-authentik-email, etc.) - Auto-creates user on first web login if not in database - Syncs roles from NPM forward auth groups header - `get_user_by_email` and `get_user_by_authentik_id` methods in AuthService - Comprehensive tests for `/auth/me` endpoint ## [1.8.0] - 2026-01-03 ### Added - **Group-Role Mapping & Permissions** (Phase 3) - Decoupled group-role architecture (groups from Authentik, roles admin-managed) - Permission format: `domain.category:action` with action hierarchy - `require_permission` and `require_any_permission` dependency factories - Global admin override (`admin.general:admin`) - **User Profile & API Keys** (Phase 4) - `GET /auth/users/me` - Full user profile with roles and preferences - `GET/PATCH /auth/users/me/preferences` - User preferences management - `GET/POST/DELETE /auth/users/me/api-keys` - API key lifecycle - API keys with `tak_` prefix, SHA-256 hashing, shown only once on creation ## [1.7.0] - 2026-01-03 ### Added - **System Stats API** - Host system resource monitoring for dashboard widgets - `GET /tools/system/stats` - Real-time host system statistics - CPU: usage percentage, core count, load averages - Memory: usage percentage, total/used/available bytes - Disks: all mounted filesystems with usage stats (auto-discovers mounts) - Network: total bytes sent/received - GPU/VRAM: NVIDIA GPU memory usage (via nvidia-smi if available) - `psutil` dependency for cross-platform system metrics ## [1.6.1] - 2026-01-03 ### Added - `link_type` field to quick links for iframe vs new tab behavior ## [1.6.0] - 2026-01-03 ### Added - **Dashboard API** - Quick links and widgets management for Organizr-style dashboard - `GET /dashboard/quick-links` - List quick links with category/visibility filtering - `GET /dashboard/quick-links/{id}` - Get single quick link - `POST /dashboard/quick-links` - Create quick link - `PUT /dashboard/quick-links/{id}` - Update quick link - `DELETE /dashboard/quick-links/{id}` - Delete quick link - `POST /dashboard/quick-links/reorder` - Reorder quick links by position - `GET /dashboard/widgets` - List dashboard widgets - `GET /dashboard/widgets/{id}` - Get single widget - `POST /dashboard/widgets` - Create widget - `PUT /dashboard/widgets/{id}` - Update widget - `DELETE /dashboard/widgets/{id}` - Delete widget - Database migrations for `quick_links` and `dashboard_widgets` tables - Static file controller for serving Organizr widgets (`/static/widgets`) - Default local user authentication when OIDC is disabled ### Changed - **Domain-based architecture** - Refactored codebase to domain-driven structure - `src/domains/` - Domain modules (auth, dashboard, health, housekeeping, infrastructure, tools) - `src/shared/` - Shared utilities (base, config, database, logging, security, clients) - Test suite updated for new domain structure (285 tests passing) ## [1.5.0] - 2026-01-01 ### Added - **Groups Management** - Authentik group synchronization - `GET /auth/groups` - List all groups with search and pagination - `POST /auth/groups/sync-from-authentik` - Bulk sync groups from Authentik admin API - Database model and migration for groups table ## [1.4.6] - 2026-01-01 ### Changed - Code cleanup: move inline `re` import to top of auth/service.py ## [1.4.5] - 2026-01-01 ### Fixed - Separate httpx and SQLAlchemy async contexts in bulk sync (fixes greenlet error) ## [1.4.4] - 2026-01-01 ### Fixed - Use `uuid` field instead of `pk` for Authentik user sync (pk is integer, uuid is proper UUID) - Skip internal_service_account type users during bulk sync ## [1.4.3] - 2026-01-01 ### Fixed - Manually extract and send session cookies for Authentik flow auth (fixes cross-domain cookie handling) ## [1.4.2] - 2026-01-01 ### Fixed - Use Authentik domain URL (auth.schweitz.net) instead of IP to fix cookie domain matching ## [1.4.1] - 2026-01-01 ### Fixed - Authentik config now uses AUTHENTIK_USERNAME/PASSWORD to match production env vars ## [1.4.0] - 2026-01-01 ### Added - **Authentication & User Management** - Authentik integration for user synchronization - `GET /auth/me` - Get current authenticated user info - `GET /auth/users` - List all users with search and pagination - `POST /auth/users/sync-from-authentik` - Bulk sync users from Authentik admin API - PostgreSQL database integration with async SQLAlchemy - Alembic database migrations for schema management - Database models: User, Role, UserPreferences, ApiKey - Token validation via Authentik userinfo endpoint - Role synchronization from Authentik groups - Health check now includes database connectivity status ### Changed - Authentik configuration now uses username/password for admin API access - Health endpoint includes database status in diagnostics ## [1.3.1] - 2025-12-31 ### Changed - Simplified configuration: all settings now read from environment variables/.env only - Removed Docker socket fallback for container operations (Portainer API is now required) - Updated default search provider to SearXNG ### Removed - `src/credentials.py` - credentials now managed via environment variables - Docker socket fallback methods from Portainer client - Unused search API settings (Brave, Google) ## [1.3.0] - 2025-12-31 ### Added - **Stack Management Endpoints** for Tatlock Control Room integration - `GET /infrastructure/stacks/{stackId}/compose` - Get stack Docker Compose YAML - `PUT /infrastructure/stacks/{stackId}/compose` - Update stack Docker Compose YAML - `GET /infrastructure/stacks/{stackId}/env` - Get stack environment variables - `PUT /infrastructure/stacks/{stackId}/env` - Update stack environment variables - `POST /infrastructure/stacks/{stackId}/deploy` - Redeploy stack - `POST /infrastructure/stacks/{stackId}/rebuild` - Pull images and recreate containers - `DELETE /infrastructure/containers/{id}` - Delete container with optional force flag - Portainer client methods: `get_stack_file`, `redeploy_stack`, `update_stack_env`, `delete_container`, `restart_container` ### Removed - REQUESTED_SERVICES.md - endpoint specifications now implemented ## [1.2.1] - 2025-12-17 ### Fixed - Device control endpoint now returns correct new state after action (added 300ms delay for HA state propagation) ### Added - AGENTS.md with project coding guidelines and release flow documentation ### Changed - Removed obsolete web scraper settings from .env.example ## [1.2.0] - 2025-12-17 ### Added - **Housekeeping API** - Home Assistant integration for smart home control - `GET /housekeeping/health` - HA connection status - `GET /housekeeping/devices` - List controllable devices with optional domain/area filtering - `GET /housekeeping/devices/{entity_id}` - Get device details - `POST /housekeeping/devices/{entity_id}/control` - Control devices (turn_on, turn_off, toggle, set_brightness) - `GET /housekeeping/scenes` - List available scenes - `POST /housekeeping/scenes/{scene_id}/activate` - Activate a scene - `GET /housekeeping/scripts` - List available scripts - `POST /housekeeping/scripts/{script_id}/run` - Run a script - `GET /housekeeping/automations` - List automations - `POST /housekeeping/automations/{automation_id}/toggle` - Enable/disable automation - `GET /housekeeping/history` - Query state history - `GET /housekeeping/areas` - List rooms/areas - Home Assistant REST API client (`src/clients/homeassistant_client.py`) - Home Assistant configuration in credentials and settings - Comprehensive test suite with 65% code coverage (285 tests) - Tests for NPM client, Ollama client, AI client, OIDC authentication - Tests for infrastructure, health, tools, and housekeeping endpoints ### Removed - **Web Scraper** - Entire web scraping module removed - `src/web_scraper/` directory deleted - `/web-scraper/scrape` endpoint removed - Trafilatura and BeautifulSoup dependencies removed from scraping use ### Changed - Updated README.md with current architecture and all endpoints - Tools controller now only contains DNS lookup functionality - Health controller endpoints list updated to reflect current features ## [1.1.2] - 2024-12-14 ### Added - Version field to /health endpoint response ## [1.1.1] - 2024-12-14 ### Added - Watchtower trigger step in CI workflow for automatic container updates ## [1.1.0] - 2024-12-14 ### Removed - Uptime Kuma integration (kuma_client.py deleted) - All /infrastructure/monitors endpoints - Kuma monitor pause/resume from service start/stop operations - Uptime percentage display from service control widget - Kuma-related configuration and credentials - Stale memory tests (memory functionality moved to core-ai service) ### Changed - Service control widget now uses Portainer container status exclusively - Simplified widget-data endpoint response (removed monitors field) - Updated service start/stop to only manage containers via Portainer ## [1.0.0] - 2024-12-14 ### Added - Initial release of Core Code API service extracted from portainer-core - Infrastructure management endpoints for Portainer, NPM, and Uptime Kuma - Web scraping service with content extraction - DNS management endpoints - Health check endpoints with diagnostics - OIDC authentication support via Authentik - Ollama integration for embeddings - OpenAPI documentation (Swagger UI and ReDoc) - Docker containerization with CI/CD workflow