portainer-core/STATUS.md

Project Status

Last Updated: 2025-11-26 Version: 0.10.0-adk-migration

Current Phase

Active Work: AI Infrastructure Optimization & System Hardening Status: ✅ STABLE - ADK Migration Complete, All Systems Operational

See PLANS.md for complete implementation roadmap and CHANGELOG.md for version history.

In Progress

Priority 1: Security & SSO Implementation (Authentik)

• Milestone 1: Authentik Deployment

  • Deploy Authentik server and worker containers
  • Configure shared PostgreSQL database (authentik_user, authentik database)
  • Configure shared Redis (DB 0)
  • Fix health checks (Python urllib instead of wget/curl)
  • Create NPM proxy host for auth.schweitz.net
  • Generate admin recovery key and set password
  • Memory optimization: 563MB total (80-90% reduction vs previous attempt)

• Milestone 2: Google OAuth Integration

  • Create Google OAuth credentials (Client ID/Secret)
  • Configure Authentik Google source via API
  • Configure identification stage to show social login
  • Test Google OAuth login (successful)
  • Verify user creation (jpmschweitzer@gmail.com - external type)

• Milestone 3: Forward Auth for Organizr ✅ COMPLETE (2025-11-21)

  • Create Authentik Proxy Provider (Organizr Proxy) via API
  • Create Authentik Application (Organizr) via API
  • Assign provider to embedded outpost (embedded outpost failed)
  • Deploy standalone outpost container (authentik-proxy on port 9443)
  • Configure Redis connection for standalone outpost
  • Verify outpost endpoints operational
  • Configure NPM forward auth for home.schweitz.net
  • Test SSO access to Organizr (Google OAuth login working)
  • Verify no redirect loops
  • Fix Organizr auto-login (moved headers to location / block)

Resolution: Embedded outpost has version-specific issues in 2024.8.4. Deployed standalone authentik-proxy container successfully. Forward auth fully operational with Organizr auto-login working.

Standalone Outpost Details:

• Container: authentik-proxy (port 9445:9443)
• Status: ✅ Healthy (websocket connected, ping endpoint responding)
• Memory: ~150MB
• Provider: Organizr Proxy (forward_single mode)
• Token: 9blMGz71CFMJszs7AedQefgydpTnwvybjmMn0AlYilIKBV5LIq7snqnCodwX

NPM Configuration:

• Applied to: home.schweitz.net (Organizr) ONLY
• Forward auth: https://localhost:9445/outpost.goauthentik.io (NPM on host network)
• WebSocket support: Enabled
• Headers: X-authentik-username, X-authentik-email, X-authentik-groups, X-authentik-name, X-authentik-uid
• Status: ✅ Fully operational, tested in incognito

Critical Fix: Authentication headers must be set inside location / block, not at server level, for proper forwarding to backend applications.

Priority 2: Core-API Refactoring & Infrastructure Management ✅ COMPLETE

• Code Cleanup: Restructure Core API into function-specific controller files

  • Create /controllers directory structure
  • Create /clients directory structure
  • Create base.py controller base class
  • Add infrastructure settings to config.py
  • Create credentials management system
  • Update main.py routing to include infrastructure controller
  • Separate AI Orchestrator logic into ai_controller.py
  • Extract webscraper to tools_controller.py
  • Create health_controller.py for monitoring endpoints

• Infrastructure Management Controller: Build automation API for service management

  • Portainer Integration (HTTP client with access token)
  • NPM Integration (HTTP client with JWT bearer token + auto-refresh)
  • Read/List Endpoints (all implemented & tested)
  • Write Endpoints (POST/PUT/DELETE all implemented & tested)
  • Portainer API Token generated programmatically
  • Uptime Kuma Integration (deferred - complex Socket.IO)
  • Replace ad-hoc shell scripts in /stacks with API endpoints
  • Add CLI wrapper for common operations

Priority 3: AI Orchestrator Phase 2 (Memory Systems) ✅ COMPLETE

• Implement Tier 1: ConversationBufferMemory (in-memory, last 10 turns)
• Implement Tier 2/3: Unified Qdrant storage (persistent + semantic search)
• Create Qdrant collection (core_api_conversations with 768d nomic-embed-text)
• Implement auto-consolidation service (triggers at 10 turns)
• Add memory persistence across container restarts
• Implement dual-retrieval (buffer + Qdrant)
• Phase 2.5: Multi-Tenancy (user_id isolation with default "llm-testuser")

Implementation Details:

• Tier 1 (Buffer): In-memory storage for last 10 turns (< 1ms access)
• Tier 2/3 (Qdrant): Unified persistent storage + semantic search (768d embeddings)
• Auto-Consolidation: Automatically moves buffer → Qdrant at 10 turns
• Multi-Tenancy: Single collection with user_id filtering (default: "llm-testuser")
• Embedding Model: nomic-embed-text (768 dimensions, via Ollama)
• Memory Retrieval: Dual-check buffer + Qdrant for cross-restart persistence
• Status: 32 points stored, tested with multiple users, recall working after restarts

Priority 4: AI Orchestrator - Framework Migration ✅ COMPLETE (2025-11-26)

• Phase 3: Research Capabilities (DuckDuckGo, web scraping) - COMPLETE (2025-11-24)
• Framework Migration: LangChain/LangGraph → Google ADK - COMPLETE (2025-11-26)
• Migrate agent orchestrator to Google ADK with LiteLLM
• Convert all 9 tools to ADK async generator format
• Update streaming pipeline for ADK event format
• Switch model to gemma3:12b with ADK-optimized prompts
• Implement comprehensive agent health checks
• Update requirements.txt (remove langchain*, add google-adk)
• Production testing and validation

Current Implementation (as of 2025-11-26):

• Framework: Google ADK 1.3.0 with LiteLLM 1.80.5 (migrated from LangChain)
• Model: gemma3:12b (upgraded from mistral:7b)
• System Prompt: v7_adk_best_practice (optimized for ADK)
• Total Tool Count: 9 tools (7 infrastructure + 2 research)
• Tools Format: ADK async generators with proper streaming support
• Architecture: UnifiedAgent with stateless sessions

Tools Available:

• Infrastructure (7): get_current_time, list_services, get_service_details, list_stacks, get_stack_details, list_npm_hosts, get_npm_host_details
• Research (2): web_search (DuckDuckGo + auto-scrape), web_scrape (targeted extraction)

Migration Benefits Achieved:

• ✅ Improved tool calling reliability with Ollama models
• ✅ Better streaming support with ADK event system
• ✅ Model flexibility (works with Gemma, Mistral, Qwen families)
• ✅ Cleaner architecture with unified agent pattern
• ✅ Production-ready health monitoring

Performance Metrics (Post-Migration):

• Simple queries: ~0.3-1s response time
• Tool-using queries: ~2-5s response time
• Research queries: ~4-7s response time
• Tool calling success rate: Monitoring in progress
• VRAM usage: ~8GB with gemma3:12b

Optional Future Enhancements (deferred):

• Multi-agent routing patterns (Phase 4+)
• Code specialist agent with codestral (Phase 4+)
• Time-based memory consolidation
• User filtering in Qdrant queries
• User management API endpoints

Current Blockers

None - SSO implementation complete for critical services. Remaining service rollout deferred in favor of other priorities.

Key Metrics

Metric	Target	Current	Status
Containers Running	15+	22	🟢 All Services Operational
GPU Accessible	Yes	Yes	🟢 Working (RTX 2080 Ti)
Storage Used	<80%	58% HDD (3.6TB/3.7TB)	🟢 Healthy
Services Accessible	All	21/21	🟢 Complete
Remote Access	Working	Ready	🟢 Headscale + NPM
Firewall Active	Yes	Yes	🟢 UFW Configured
Backups Configured	Yes	Yes	🟢 Daily @ 3 AM
AI Orchestrator	Phase 6	Phase 4 ✅	🟢 ADK Migration Complete
SSO (Authentik)	Phase 5	Core Complete ✅	🟢 Organizr + Core API Protected

Quick Reference

Documentation

• Implementation Plans - Master plan tracker and roadmap
• Changelog - Version history
• Container Reference - All deployed services
• System Specifications - Hardware and software details
• Agent Guidelines - Development conventions

Key Paths

• SSD configs: /home/jpmschweitzer/docker-data/
• HDD content: /mnt/media/
• Stacks: Managed in Portainer web UI
• Scripts: /mnt/media/Projects/portainer-core/scripts/

Active Services & URLs

Infrastructure:

• Portainer: http://192.168.86.149:8080 (container management)
• Nginx Proxy Manager: http://192.168.86.149:8000 (reverse proxy admin)
• Authentik: https://auth.schweitz.net (SSO identity provider - Google OAuth enabled)
• Ollama: http://192.168.86.149:11434 (ML models API)

Networking:

• Headscale: http://192.168.86.149:8085 (mesh VPN control)

Monitoring:

• Uptime Kuma: http://192.168.86.149:3001 (service monitoring)
• Netdata: http://192.168.86.149:19999 (system metrics)
• Organizr: http://192.168.86.149:8084 OR https://home.schweitz.net (unified dashboard)

Applications:

• Open WebUI: http://192.168.86.149:8081 (LLM chat interface)
• Core API: http://192.168.86.149:8083 (infrastructure management & AI orchestration)
• Jellyfin: http://192.168.86.149:8096 OR https://media.schweitz.net (GPU media server)
• Nextcloud: http://192.168.86.149:8082 OR https://cloud.schweitz.net (cloud storage)
• Gitea: http://192.168.86.149:3002 OR https://git.schweitz.net (Git hosting, SSH: 2222)
• Samba: \\192.168.86.149 or \\tower-of-joy (file shares: Media, Downloads, Backups)

Background Services:

• Watchtower: Automatic updates daily @ 4 AM
• Maintenance: Automated backups daily @ 3 AM

---

For detailed implementation history and completed work, see CHANGELOG.md