portainer-core/CHANGELOG.md

Changelog

All notable changes to the portainer-core project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

In Progress

• System Monitoring: Post-migration stability monitoring and performance optimization

Planned

• AI Orchestrator Phases 5-6: Multi-agent workflows, RAG optimization, production hardening
• Authentik SSO Milestones 4-5: Protect remaining services (deferred)
• Disaster recovery and offsite backup strategy

[0.10.1-phase-completion] - 2025-11-26

Added

• Phase 2/3 Completion Documentation
  • Added phase2-memory-system-complete.md with full implementation details
  • Added phase3-multi-agent-workflows-complete.md documenting research capabilities
  • Added system prompts file (prompts.py) with 7 tested variants for A/B testing
  • Session documentation for model testing and VRAM optimization
• Test Results & Analysis
  • Comprehensive prompt testing results (87/100 score for v1_verbose)
  • Model comparison testing (Mistral, Gemma, tool calling validation)
  • Tool logging implementation documentation
  • Verified test results for production readiness
• Portainer Client Enhancements
  • Added comprehensive Portainer API client (148 lines)
  • Stack management, service monitoring, container operations
  • Full error handling and async support

Changed

• Memory System Improvements
  • Added user_id parameter for multi-tenancy support
  • Skip storing system messages (part of agent state_modifier)
  • Enhanced memory manager with better user isolation
  • Improved conversation turn tracking
• AI Controller Enhancements
  • Better memory integration with user_id support
  • Improved error handling for memory operations
  • Enhanced token tracking for usage monitoring
• Architecture Documentation
  • Updated agent flow diagrams to reflect ADK architecture
  • Enhanced core-api README with current setup
  • Updated Docker compose stack configuration

Technical Details

• Files Added:
  • services/core-api/src/agent/prompts.py - 7 system prompt variants
  • plans/completed/phase2-memory-system-complete.md
  • plans/completed/phase3-multi-agent-workflows-complete.md
  • services/core-api/COMPREHENSIVE_PROMPT_TEST_RESULTS.md
  • services/core-api/TOOL_LOGGING_IMPLEMENTATION.md
  • docs/sessions/2025-11-24-*.md - Model testing documentation
• Memory System:
  • Multi-tenancy support with user_id throughout
  • System message filtering (not stored in history)
  • Improved conversation metadata tracking

[0.10.0-adk-migration] - 2025-11-26

Added

• AI Orchestrator: Google ADK Framework Migration ✅ MAJOR ARCHITECTURAL CHANGE
  • New Framework: Google ADK 1.3.0
    • Migrated from LangChain/LangGraph to Google's Agent Development Kit
    • LiteLLM 1.80.5 integration for Ollama compatibility
    • Improved tool calling reliability with local models
    • Better streaming support with ADK event system
  • Model Upgrade: gemma3:12b
    • Upgraded from mistral:7b for better capability
    • Optimized for tool calling with ADK framework
    • ~8GB VRAM usage (vs ~4GB with mistral:7b)
  • Tool Migration
    • All 9 tools converted to ADK async generator format
    • Infrastructure tools (7): time, services, stacks, NPM
    • Research tools (2): web_search, web_scrape
    • Improved error handling and streaming progress
  • System Prompt Optimization
    • New v7_adk_best_practice prompt variant
    • Optimized for ADK agent behavior
    • Better tool usage patterns
  • Enhanced Health Checks
    • Agent-specific health monitoring
    • Tool availability verification
    • LiteLLM connection validation

Changed

• Agent Architecture
  • Replaced LangGraph create_react_agent with ADK Agent
  • Changed from LangChain tools to ADK async generators
  • Updated streaming pipeline for ADK event format
  • Simplified orchestrator.py (more maintainable)
• Dependencies
  • Removed: langchain, langgraph, langchain-community, langchain-core, langchain-ollama
  • Added: google-adk==1.3.0, google-genai==1.17.0, litellm==1.80.5
  • Updated: pydantic>=2.11.1, uvicorn>=0.34.0, httpx>=0.28.0
• Configuration
  • Default model: mistral:7b → gemma3:12b
  • Agent model: mistral:7b → gemma3:12b
  • System prompt: v1_verbose → v7_adk_best_practice

Fixed

• Tool Calling Reliability
  • Issue: LangGraph agents not calling tools with Ollama models
  • Root cause: LangGraph ReAct agent incompatibility with local models
  • Fix: Migrated to Google ADK with proven Ollama support
  • Result: Consistent tool calling across all query types
• Model Compatibility
  • Issue: Gemma models failing with LangChain (status 400)
  • Fix: ADK supports Gemma family natively via LiteLLM
  • Result: Can now use gemma3:12b, gemma3:4b, and other Gemma variants
• Streaming Consistency
  • Issue: Inconsistent streaming behavior with LangGraph
  • Fix: ADK provides unified event streaming
  • Result: Clean, consistent SSE output for frontend

Technical Details

• Files Modified:
  • services/core-api/requirements.txt - Dependency overhaul
  • services/core-api/src/agent/orchestrator.py - Complete rewrite for ADK
  • services/core-api/src/agent/tools.py - Converted 9 tools to ADK format
  • services/core-api/src/agent/streaming.py - Updated for ADK events
  • services/core-api/src/controllers/ai_controller.py - Enhanced error handling
  • services/core-api/src/controllers/health_controller.py - Added agent health checks
  • services/core-api/src/config.py - Updated model and prompt settings
  • services/core-api/Dockerfile - Updated base dependencies
• Architecture Impact:
  • Migration preserves existing API contracts
  • Memory system (Qdrant) completely unaffected
  • Tool implementations (logic) unchanged, only decorators updated
  • Frontend integration (SSE streaming) maintained
• Performance:
  • Simple queries: ~0.3-1s (similar to LangChain)
  • Tool-using queries: ~2-5s (improved from LangChain)
  • Research queries: ~4-7s (maintained from Phase 3)
  • VRAM: ~8GB with gemma3:12b (~4GB increase)

Migration Notes

• Reason for Migration: LangChain/LangGraph's create_react_agent failed to trigger tool calls with Ollama models despite proper configuration
• Migration Duration: ~6 hours (as estimated in migration plan)
• Testing: Validated with time queries, service queries, and web searches
• Rollback: Previous LangChain implementation preserved in git history
• Documentation: See MIGRATION_PLAN_LANGCHAIN_TO_ADK.md for details

Known Issues

• None currently identified - monitoring in progress

Performance Monitoring

• Tool calling success rate: Being tracked post-migration
• Response latency: Within targets (<5s for research)
• VRAM utilization: ~8GB (acceptable for RTX 2080 Ti 11GB)
• Error rate: Monitoring for ADK-specific issues

[0.9.0-ai-memory-system] - 2025-11-23

Added

• AI Orchestrator Phase 2: Memory System ✅ COMPLETE
  • Tier 1: ConversationBufferMemory
    • In-memory storage for last 10 turns per conversation
    • < 1ms access time, automatic pruning
    • OrderedDict-based storage with conversation metadata
  • Tier 2/3: Unified Qdrant Storage
    • Collection: core_api_conversations
    • Embedding model: nomic-embed-text (768 dimensions via Ollama)
    • Persistent storage + semantic search capabilities
    • Chronological retrieval (Tier 2 mode)
    • Semantic similarity search (Tier 3 mode)
  • Auto-Consolidation Service
    • Triggers at 10 turns (when buffer fills)
    • Moves buffer turns → Qdrant automatically
    • Maintains conversation continuity
  • Dual-Retrieval System
    • Checks both buffer (Tier 1) AND Qdrant (Tier 2/3)
    • Cross-restart persistence working
    • Combines memory tiers via get_full_history()
  • Phase 2.5: Multi-Tenancy
    • Added user_id field to all memory schemas
    • Default user: "llm-testuser" for unauthenticated traffic
    • Single collection with user_id filtering approach
    • Tested with multiple users successfully

Changed

• Memory Architecture
  • Original plan: Tier 2 (SQLite summaries) + Tier 3 (Qdrant semantic)
  • Implemented: Unified Tier 2/3 in Qdrant (simpler, more efficient)
  • Rationale: Qdrant handles both persistent storage and semantic search
• Embedding Dimension
  • Changed from 384d (all-MiniLM-L6-v2) to 768d (nomic-embed-text)
  • Better semantic quality, still efficient for local deployment
• Logging Level
  • Changed memory storage logs from DEBUG to INFO for visibility
  • Helps verify storage execution without verbose output

Fixed

• Memory Storage Integration (Priority 1.1)
  • Issue: Memory storage code reached but not executing
  • Root cause: Log level set to WARNING, logger.debug() invisible
  • Fix: Changed logger.debug() → logger.info() for storage paths
  • Verification: 32 points successfully stored in Qdrant
• Memory Persistence (Priority 2.1)
  • Issue: Agent didn't recall conversations after restart
  • Root cause: Only checking buffer (Tier 1), empty after restart
  • Fix: Added dual-check (buffer_exists OR qdrant_exists)
  • Verification: Agent correctly recalled "purple" after restart

Technical Details

• Files Modified:
  • src/api/v1/schemas.py - Added user_id to ChatCompletionRequest
  • src/memory/schemas.py - Added user_id to ConversationTurn and ConversationMetadata
  • src/controllers/ai_controller.py - Memory storage and retrieval integration
  • src/memory/manager.py - Updated add_turn() for user_id
  • src/memory/qdrant_memory.py - Added user_id to payload and retrieval
• Testing:
  • Manual testing with curl commands
  • Multi-user testing (llm-testuser, alice-testuser)
  • Cross-restart persistence verified
  • Semantic search verified (AI-related queries ranked correctly)

Deferred

• Optional Future Enhancements:
  • Time-based consolidation for short conversations (< 10 turns)
  • User filtering in Qdrant queries
  • User management API (list users, delete user data)
  • Migration to LangGraph checkpointers (Phase 3 roadmap)

[0.8.1-authentik-organizr] - 2025-11-21

Added

• Standalone Authentik Proxy Outpost
  • Container: authentik-proxy (port 9445:9443)
  • Redis configuration: redis-shared:6379/0
  • Memory usage: ~150MB
  • API token authentication with Authentik server
  • WebSocket connection to Authentik for config updates
• Forward Authentication for Organizr
  • NPM configuration for home.schweitz.net
  • auth_request directive pointing to standalone outpost
  • Authentication header forwarding (X-authentik-username, email, groups, name, uid)
  • Signin redirect handler for unauthenticated requests
  • WebSocket support enabled
• Documentation
  • Session summary: docs/sessions/2025-11-21-authentik-troubleshooting.md
  • NPM configuration template: docs/npm-configs/organizr-forward-auth.conf
  • Deployment scripts in /tmp for reference

Fixed

• Embedded Outpost Issue: Authentik 2024.8.4 embedded outpost not initializing auth endpoint (version-specific bug)
• Network Connectivity: NPM on host network cannot resolve docker-dataplane container names - use localhost:9445
• NPM Config Generation: API updates don't generate config files - manually created /data/nginx/proxy_host/2.conf
• Redirect Loop: Initial redirect to /outpost.goauthentik.io/start returned 404 - changed to use application domain
• Post-Login Redirect: Direct flow redirect sent users to /if/user/#/library - use outpost start endpoint instead
• Organizr Auto-Login: Headers set at server level don't forward - moved proxy_set_header to location / block

Changed

• Outpost Architecture: Moved from embedded to standalone for reliability (port 9445:9443)

[0.8.0-authentik-sso] - 2025-11-20

Added

• Authentik Identity Provider (version 2024.8.4)
  • Server container (port 9000) with 512MB memory limit
  • Worker container with 384MB memory limit
  • Total memory usage: 563MB (80-90% reduction vs previous attempt)
  • Embedded outpost on port 9444
• Shared Infrastructure Integration
  • PostgreSQL: authentik database with authentik_user
  • Redis: Database 0 for sessions and cache
  • Docker network: docker-dataplane
• Google OAuth Integration
  • OAuth source configured via API
  • Google login button on authentication flow
  • Automatic user creation for external OAuth users
  • Successful test: jpmschweitzer@gmail.com user created
• NPM Configuration
  • Reverse proxy for https://auth.schweitz.net
  • Let's Encrypt SSL with HSTS
  • No forward auth on auth.schweitz.net (prevents redirect loops)
• API Automation
  • Created proxy provider "Organizr Proxy" via API
  • Created application "Organizr" via API
  • Assigned provider to embedded outpost via API
• Documentation
  • Session summary: docs/sessions/2025-11-20-authentik-deployment.md
  • Updated STATUS.md with SSO progress
  • Updated security implementation plan

Fixed

• Health check failing due to missing wget/curl - switched to Python urllib
• Database user authentik_user not created - manually created with grants
• Port 9443 conflict - mapped to 9444 on host
• NPM proxy host marked as deleted - recreated via UI
• Google OAuth enrollment flow error - cleared browser cookies

Changed

• Container count: 19 → 21 (added authentik-server, authentik-worker)
• Active priority: AI Orchestrator → Security & SSO Implementation
• Deferred AI Orchestrator Phase 2 to focus on security

Known Issues

• Embedded outpost auth endpoint returns 404
  • Endpoint: /outpost.goauthentik.io/auth/nginx not available
  • Ping endpoint works, but auth endpoint not initialized
  • Blocking forward authentication for Organizr
  • Investigating provider mode and initialization sequence

[0.7.1-gitea-deployment] - 2025-11-14

Added

• Gitea Git repository hosting service (port 3002, SSH port 2222)
• PostgreSQL database backend for Gitea
• NPM reverse proxy configuration for https://git.schweitz.net with Let's Encrypt SSL
• Uptime Kuma monitoring integration for Gitea
• Organizr dashboard integration for Gitea
• Complete Gitea documentation in CONTAINERS.md

Changed

• Updated infrastructure status to reflect 19 deployed services

[0.7.0-ai-orchestrator-phase1] - 2025-11-13

Added

• AI Orchestrator Phase 1: OpenAI-Compatible API
  • /v1/chat/completions endpoint with streaming and non-streaming support
  • /v1/models endpoint for model discovery
  • Model aliasing system (gpt-3.5-turbo → gemma:7b, etc.)
  • Ollama client with connection pooling
  • Pydantic request/response schemas
  • Server-Sent Events (SSE) streaming format
• Comprehensive Phase 1 testing suite
  • 10/10 tests passing, zero issues
  • 245ms average response time
  • 100% OpenAI API compatibility verified
• Phase 1 implementation guide and test results documentation

Fixed

• Model ID formatting issue (removed extra quotes in model names)

Security

• Deployed on isolated ai-dataplane network

[0.6.0-applications] - 2025-11-13

Added

• Nextcloud cloud storage and collaboration platform (port 8082)
  • MariaDB database backend
  • Redis caching
  • NPM reverse proxy with https://cloud.schweitz.net
  • Database optimization (indices, bigint conversion)
  • Cron background jobs via maintenance container
• Samba network file sharing (ports 139/445)
  • Media share (R/W)
  • Downloads share (R/W)
  • Backups share (R/O)
• UFW firewall rules for Samba ports
• Uptime Kuma multi-network bridge for monitoring all services

Changed

• Disabled host Samba service to prevent port conflicts
• Relocated Nextcloud cron to maintenance container for centralized scheduling

Fixed

• Uptime Kuma network connectivity issues (added bridges to all service networks)

[0.5.2-core-api] - 2025-11-13

Added

• Core API service for Open WebUI integration (port 8083)
• Web scraper module with Trafilatura and BeautifulSoup
• Infrastructure management API (Portainer/NPM/Kuma integration)
• OpenAPI documentation at /docs endpoint
• Health check endpoint
• Uptime Kuma monitoring integration
• Organizr dashboard integration

Changed

• Upgraded system Python from 3.8 (EOL) to 3.12

Security

• Runs as non-root user (uid 1000)
• CORS configured for same-network access only

[0.5.1-open-webui] - 2025-11-12

Added

• Open WebUI LLM chat interface (port 8081)
• Built-in voice capabilities (local STT/TTS)
• Ollama integration for local model inference
• Uptime Kuma monitoring integration
• Organizr dashboard integration (tab + homepage)
• Complete Open WebUI documentation in CONTAINERS.md

[0.5.0-optimization] - 2025-11-11

Added

• Phase 4: Optimization & Security
  • Watchtower for automatic container updates (daily at 4 AM)
  • Maintenance container for automated backups and scheduled tasks
  • Automated Docker config backups (daily at 3 AM, 30-day retention)
• Docker log rotation configuration (10MB max, 3 files per container)
• UFW firewall rules (SSH, Tailscale, infrastructure services)

Security

• Firewall enabled and configured for all public-facing services
• Automated backup system with 30-day retention (~94MB per backup)

[0.4.0-monitoring] - 2025-11-11

Added

• Phase 3: Monitoring Stack
  • Uptime Kuma service monitoring (port 3001)
  • Netdata real-time system metrics (port 19999)
  • Heimdall unified dashboard (port 8888)
• Complete monitoring documentation

[0.3.0-networking] - 2025-11-11

Added

• Phase 2: Networking & External Access
  • Headscale mesh VPN control server (port 8085)
  • Custom 10.99.0.0/16 network range
  • Homelab user and pre-auth key system
  • Device connection procedures for all platforms
• Headscale setup documentation

[0.2.0-foundation] - 2025-11-11

Added

• Phase 1: Foundation Setup
  • Portainer container management (port 8001, host networking)
  • Nginx Proxy Manager reverse proxy (port 81, host networking)
  • Ollama ML model serving (port 11434, GPU-enabled)
  • NVIDIA Container Toolkit (v1.17.9-1 for driver 470 compatibility)
  • GPU management via docker-compose deploy configuration
  • 4TB media drive mounted at /mnt/media
  • User added to docker group

Fixed

• Docker networking issues (iptables FORWARD chain, host networking solution)
• AMP integration (kept on port 8080, no conflicts)

Security

• GPU passthrough configured securely
• Storage permissions set for dual-disk strategy

[0.1.0-planning] - 2025-11-11

Added

• Initial project structure and documentation
• Comprehensive research document (containers/research.md)
  • Evaluated 8 different home server solutions
  • Identified Portainer + Docker Compose as optimal choice
  • Researched SDN solutions (Headscale vs Tailscale)
• Detailed implementation plan (containers/implementation-plan.md)
  • 4-phase deployment strategy
  • Phase 1: Foundation (Portainer, NPM, Ollama, storage)
  • Phase 2: Networking (Headscale)
  • Phase 3: Monitoring (Uptime Kuma, Netdata, Heimdall)
  • Phase 4: Optimization (Watchtower, backups, security)
  • Application backlog (Jellyfin, Nextcloud, Samba)
• System documentation (SYSTEM.md)
  • Hardware specifications
  • Dual-disk storage configuration
  • Software inventory
• Agent guidelines (AGENTS.md)
  • Project-specific conventions
  • Docker Compose standards
  • GPU service requirements
  • Testing procedures
  • Commit message format
• Project status tracking (STATUS.md)
• Version-controlled infrastructure (stacks/ directory)
• Maintenance automation (scripts/ directory, Makefile)

Documented

• Storage architecture: SSD (489GB) for configs, HDD (3.7TB) for content
• Port allocation strategy
• AMP game server integration approach
• GPU passthrough requirements for Jellyfin and Ollama
• Security considerations (Headscale, UFW, credentials management)

Decisions

• Architecture: Portainer + Docker Compose (chosen over TrueNAS Scale, Unraid, Proxmox)
  • Reason: No OS reinstall required, leverages existing Docker, minimal storage footprint
• Reverse Proxy: Nginx Proxy Manager on port 8000 (unified web interface)
• SDN: Headscale (self-hosted Tailscale control server)
• ML Infrastructure: Ollama with GPU support (RTX 2080 Ti)
• Monitoring: Uptime Kuma + Netdata + Heimdall
• Storage Strategy: Dual-disk approach (SSD for performance, HDD for capacity)

---

Changelog Guidelines

Categories

Use these categories for changes:

• Added - New features, services, or capabilities
• Changed - Changes to existing functionality
• Deprecated - Soon-to-be-removed features
• Removed - Removed features
• Fixed - Bug fixes
• Security - Security improvements

Version Numbering

• Major (X.0.0): Breaking changes, major architecture changes
• Minor (0.X.0): New features, service additions, phase completions
• Patch (0.0.X): Bug fixes, configuration tweaks, documentation updates
• Suffix: -planning, -alpha, -beta for pre-release stages

Example Entry Template

## [X.Y.Z] - YYYY-MM-DD

### Added
- feat(stack): deployed nginx proxy manager for unified web interface
- feat(ollama): configured GPU passthrough for ML model inference

### Changed
- config(amp): moved from port 8080 to 8081 to avoid conflicts

### Fixed
- fix(storage): corrected permissions on media drive mount

### Security
- chore(firewall): configured UFW rules for service isolation

---

This changelog is updated as features are implemented and phases are completed