portainer-core/services/core-api/REFACTORING_PLAN.md

Core-API Refactoring Plan

Date: 2025-11-14 Goal: Restructure Core-API into controller-based architecture and add Infrastructure Management API

Current Structure

src/
├── api/
│   └── v1/
│       ├── chat.py           # AI chat completions
│       ├── models.py          # Model listing
│       ├── conversations.py   # Conversation memory
│       └── schemas.py         # Pydantic schemas
├── web_scraper/
│   ├── router.py             # Webscraper endpoints
│   ├── service.py
│   └── schemas.py
├── models/
│   ├── ollama_client.py      # Ollama HTTP client
│   └── embeddings.py
├── memory/                    # Memory tier system
├── config.py                  # Global settings
└── main.py                    # FastAPI app

Target Structure

src/
├── controllers/               # NEW: Controller-based routing
│   ├── __init__.py
│   ├── base.py               # Base controller class
│   ├── ai_controller.py      # AI Orchestrator (chat, models, conversations)
│   ├── tools_controller.py   # Utility tools (webscraper, etc.)
│   ├── health_controller.py  # Health & monitoring
│   └── infrastructure_controller.py  # Infrastructure automation
├── clients/                   # NEW: External API clients
│   ├── __init__.py
│   ├── portainer_client.py   # Portainer API
│   ├── npm_client.py         # Nginx Proxy Manager API
│   └── kuma_client.py        # Uptime Kuma Socket.IO API
├── api/v1/                    # Keep existing for backward compat
├── web_scraper/               # Keep as-is for now
├── models/                    # Keep as-is
├── memory/                    # Keep as-is
├── config.py                  # Enhanced with infrastructure settings
└── main.py                    # Updated routing

Implementation Phases

Phase 1: Infrastructure Setup ✅ COMPLETE

• Research API authentication methods
• Add infrastructure settings to config.py
• Create credentials.py for sensitive data (gitignored)
• Create credentials.example.py as template
• Update .gitignore to exclude credentials.py
• Update config.py to import from credentials module
• Create /controllers directory structure
• Create /clients directory structure
• Create base controller class

Phase 2: API Clients ✅ COMPLETE (Portainer & NPM)

• Implement Portainer API client (access token auth)
• Implement NPM API client (JWT with refresh)
• Add token storage/refresh mechanisms
• Implement Uptime Kuma Socket.IO client (DEFERRED - WebSocket complexity)

Phase 3: Infrastructure Controller ✅ COMPLETE

• GET /infrastructure/health - Check connectivity ✅ TESTED
• GET /infrastructure/services - List all services ✅ TESTED
• GET /infrastructure/services/{name} - Get service details ✅ TESTED
• GET /infrastructure/ports - List allocated ports (skeleton)
• GET /infrastructure/domains - List configured domains ✅ TESTED
• Integrate with main.py routing ✅ TESTED
• Fix Pydantic validation issues (status field type conversion)
• POST /infrastructure/services - Deploy new service ✅ TESTED
• PUT /infrastructure/services/{name} - Update service ✅ TESTED
• DELETE /infrastructure/services/{name} - Remove service ✅ TESTED
• POST /infrastructure/proxy - Create NPM proxy host with optional SSL ✅ IMPLEMENTED
• POST /infrastructure/monitoring/add - Auto-add Kuma monitor (DEFERRED - Socket.IO complexity)

Phase 4: Refactor Existing Controllers 📋 PENDING

• Move AI endpoints to ai_controller.py
• Move webscraper to tools_controller.py
• Move health check to health_controller.py
• Update main.py imports and routing

Phase 5: Testing & Documentation 📋 PENDING

• Test all refactored endpoints
• Update API documentation
• Create CLI wrapper scripts
• Remove old shell scripts

---

Progress Notes (2025-11-14)

Session 1: Foundation & Read Endpoints

Completed:

• Created controller and client architecture
• Implemented Portainer client with full CRUD operations for stacks
• Implemented NPM client with JWT refresh and proxy/certificate management
• Built infrastructure controller with 5 read/list endpoints
• Added infrastructure settings to config.py

Files Created:

• src/controllers/__init__.py
• src/controllers/base.py
• src/controllers/infrastructure_controller.py
• src/clients/__init__.py
• src/clients/portainer_client.py
• src/clients/npm_client.py
• REFACTORING_PLAN.md (this file)

Session 2: Credentials & Testing (2025-11-14 Evening)

Completed:

• Created credentials management system (credentials.py gitignored, credentials.example.py committed)
• Updated config.py to import from credentials module with fallback
• Generated Portainer API token programmatically via API
• Integrated infrastructure controller into main.py
• Fixed Pydantic validation bug (status field int→str conversion)
• Tested all read endpoints with live Portainer/NPM infrastructure
• Verified 8 stacks detected, domains with SSL status working

Test Results:

• ✅ GET /infrastructure/health - Portainer connected, NPM accessible
• ✅ GET /infrastructure/services - Returns 8 active stacks
• ✅ GET /infrastructure/services/{name} - Service lookup working
• ✅ GET /infrastructure/domains - Returns proxy hosts with SSL status
• ⚠️ NPM health check shows false (returns 302 redirect instead of 200)

Session 3: Write Endpoints (2025-11-14 Evening)

Completed:

• Created request/response models for write operations (DeployServiceRequest, UpdateServiceRequest, CreateProxyRequest, OperationResult)
• Implemented POST /infrastructure/services - Deploy new service from compose YAML
• Implemented PUT /infrastructure/services/{name} - Update existing service configuration
• Implemented DELETE /infrastructure/services/{name} - Remove service and stack
• Implemented POST /infrastructure/proxy - Create NPM proxy host with optional SSL certificate
• Updated main.py API description with write endpoints
• Tested all service management endpoints (POST/PUT/DELETE) with live Portainer instance

Test Results:

• ✅ POST /infrastructure/services - Created test-nginx stack (ID: 30)
• ✅ PUT /infrastructure/services/test-nginx - Updated compose with environment variable
• ✅ DELETE /infrastructure/services/test-nginx - Removed stack successfully
• ✅ POST /infrastructure/proxy - Implemented (not tested to avoid production interference)

Next Steps:

1. Refactor existing AI/tools/health endpoints into separate controllers (Phase 4)
2. Fix NPM health check to handle redirects
3. Implement port allocation detection logic
4. Create CLI wrappers for common operations

API Authentication Strategy

Portainer

• Method: Access Token (X-API-Key header)
• Setup: Manual creation in UI, store in config/env
• Duration: Long-lived
• Storage: Environment variable PORTAINER_API_KEY

Nginx Proxy Manager

• Method: JWT Bearer Token
• Setup: Login via /api/tokens with credentials
• Duration: ~24 hours
• Strategy: Auto-refresh with stored credentials
• Storage: NPM_EMAIL and NPM_PASSWORD in env

Uptime Kuma

• Method: Socket.IO WebSocket
• Setup: Login via Socket.IO login event
• Duration: Session-based
• Strategy: Maintain persistent connection or re-auth per request
• Storage: KUMA_USERNAME and KUMA_PASSWORD in env

Configuration Changes

Credentials Management Strategy

Use credentials.py for sensitive data (added to .gitignore):

• Keeps secrets out of version control
• Easy terminal-based management with editor
• Python format for type safety and autocomplete
• Separate from config for security isolation

Implementation:

1. Create src/credentials.py with credentials (gitignored)
2. Create src/credentials.example.py as template (committed)
3. Update config.py to import from credentials module
4. Add credentials.py to .gitignore

Example src/credentials.py:

"""
Infrastructure credentials (GITIGNORED)
Copy from credentials.example.py and fill in real values
"""

# Portainer
PORTAINER_URL = "http://localhost:8001"
PORTAINER_API_KEY = "ptr_your_actual_token_here"

# Nginx Proxy Manager
NPM_URL = "http://localhost:81"
NPM_EMAIL = "jpmschweitzer@gmail.com"
NPM_PASSWORD = "your_actual_password"

# Uptime Kuma
KUMA_URL = "http://localhost:3001"
KUMA_USERNAME = "admin"
KUMA_PASSWORD = "your_actual_password"

Updated config.py to use credentials:

from src.credentials import (
    PORTAINER_URL, PORTAINER_API_KEY,
    NPM_URL, NPM_EMAIL, NPM_PASSWORD,
    KUMA_URL, KUMA_USERNAME, KUMA_PASSWORD
)

class Settings(BaseSettings):
    # Infrastructure Management (from credentials.py)
    portainer_url: str = PORTAINER_URL
    portainer_api_key: str = PORTAINER_API_KEY

    npm_url: str = NPM_URL
    npm_email: str = NPM_EMAIL
    npm_password: str = NPM_PASSWORD

    kuma_url: str = KUMA_URL
    kuma_username: str = KUMA_USERNAME
    kuma_password: str = KUMA_PASSWORD

Benefits

1. Cleaner Code: Separation of concerns, easier to maintain
2. Automation: Programmatic service deployment and configuration
3. Elimination of Shell Scripts: Replace ad-hoc scripts with proper API
4. Service Discovery: Auto-detect running services and configurations
5. Self-Managing Homelab: Foundation for autonomous infrastructure

Migration Notes

• Existing /v1/ endpoints remain unchanged for backward compatibility
• Web scraper endpoints stay at /web-scraper/ initially
• Old shell scripts in /stacks/ will be replaced with CLI wrappers