portainer-core/services/core-api/REFACTORING_PLAN.md

Core-API Refactoring Plan

Date: 2025-11-14 Goal: Restructure Core-API into controller-based architecture and add Infrastructure Management API

Current Structure

src/
├── api/
│   └── v1/
│       ├── chat.py           # AI chat completions
│       ├── models.py          # Model listing
│       ├── conversations.py   # Conversation memory
│       └── schemas.py         # Pydantic schemas
├── web_scraper/
│   ├── router.py             # Webscraper endpoints
│   ├── service.py
│   └── schemas.py
├── models/
│   ├── ollama_client.py      # Ollama HTTP client
│   └── embeddings.py
├── memory/                    # Memory tier system
├── config.py                  # Global settings
└── main.py                    # FastAPI app

Target Structure

src/
├── controllers/               # NEW: Controller-based routing
│   ├── __init__.py
│   ├── base.py               # Base controller class
│   ├── ai_controller.py      # AI Orchestrator (chat, models, conversations)
│   ├── tools_controller.py   # Utility tools (webscraper, etc.)
│   ├── health_controller.py  # Health & monitoring
│   └── infrastructure_controller.py  # Infrastructure automation
├── clients/                   # NEW: External API clients
│   ├── __init__.py
│   ├── portainer_client.py   # Portainer API
│   ├── npm_client.py         # Nginx Proxy Manager API
│   └── kuma_client.py        # Uptime Kuma Socket.IO API
├── api/v1/                    # Keep existing for backward compat
├── web_scraper/               # Keep as-is for now
├── models/                    # Keep as-is
├── memory/                    # Keep as-is
├── config.py                  # Enhanced with infrastructure settings
└── main.py                    # Updated routing

Implementation Phases

Phase 1: Infrastructure Setup 🔄 IN PROGRESS

• Research API authentication methods
• Add infrastructure settings to config.py
• Create credentials.py for sensitive data (gitignored)
• Create credentials.example.py as template
• Update .gitignore to exclude credentials.py
• Update config.py to import from credentials module
• Create /controllers directory structure
• Create /clients directory structure
• Create base controller class

Phase 2: API Clients ✅ COMPLETE (Portainer & NPM)

• Implement Portainer API client (access token auth)
• Implement NPM API client (JWT with refresh)
• Add token storage/refresh mechanisms
• Implement Uptime Kuma Socket.IO client (DEFERRED - WebSocket complexity)

Phase 3: Infrastructure Controller 🔄 IN PROGRESS

• GET /infrastructure/health - Check connectivity
• GET /infrastructure/services - List all services
• GET /infrastructure/services/{name} - Get service details
• GET /infrastructure/ports - List allocated ports (skeleton)
• GET /infrastructure/domains - List configured domains
• POST /infrastructure/services - Deploy new service
• PUT /infrastructure/services/{name} - Update service
• DELETE /infrastructure/services/{name} - Remove service
• POST /infrastructure/monitoring/add - Auto-add Kuma monitor
• POST /infrastructure/proxy/add - Auto-add NPM proxy host

Phase 4: Refactor Existing Controllers 📋 PENDING

• Move AI endpoints to ai_controller.py
• Move webscraper to tools_controller.py
• Move health check to health_controller.py
• Update main.py imports and routing

Phase 5: Testing & Documentation 📋 PENDING

• Test all refactored endpoints
• Update API documentation
• Create CLI wrapper scripts
• Remove old shell scripts

---

Progress Notes (2025-11-14)

Session 1: Foundation & Read Endpoints

Completed:

• Created controller and client architecture
• Implemented Portainer client with full CRUD operations for stacks
• Implemented NPM client with JWT refresh and proxy/certificate management
• Built infrastructure controller with 5 read/list endpoints
• Added infrastructure settings to config.py

Files Created:

• src/controllers/__init__.py
• src/controllers/base.py
• src/controllers/infrastructure_controller.py
• src/clients/__init__.py
• src/clients/portainer_client.py
• src/clients/npm_client.py
• REFACTORING_PLAN.md (this file)

Next Steps:

1. Create credentials.py for secure credential management
2. Update config.py to import from credentials module
3. Add credentials.py to .gitignore
4. Update main.py to include infrastructure routes
5. Test endpoints with live infrastructure
6. Implement write/deploy operations
7. Refactor existing AI/tools/health endpoints
8. Create CLI wrappers

API Authentication Strategy

Portainer

• Method: Access Token (X-API-Key header)
• Setup: Manual creation in UI, store in config/env
• Duration: Long-lived
• Storage: Environment variable PORTAINER_API_KEY

Nginx Proxy Manager

• Method: JWT Bearer Token
• Setup: Login via /api/tokens with credentials
• Duration: ~24 hours
• Strategy: Auto-refresh with stored credentials
• Storage: NPM_EMAIL and NPM_PASSWORD in env

Uptime Kuma

• Method: Socket.IO WebSocket
• Setup: Login via Socket.IO login event
• Duration: Session-based
• Strategy: Maintain persistent connection or re-auth per request
• Storage: KUMA_USERNAME and KUMA_PASSWORD in env

Configuration Changes

Credentials Management Strategy

Use credentials.py for sensitive data (added to .gitignore):

• Keeps secrets out of version control
• Easy terminal-based management with editor
• Python format for type safety and autocomplete
• Separate from config for security isolation

Implementation:

1. Create src/credentials.py with credentials (gitignored)
2. Create src/credentials.example.py as template (committed)
3. Update config.py to import from credentials module
4. Add credentials.py to .gitignore

Example src/credentials.py:

"""
Infrastructure credentials (GITIGNORED)
Copy from credentials.example.py and fill in real values
"""

# Portainer
PORTAINER_URL = "http://localhost:8001"
PORTAINER_API_KEY = "ptr_your_actual_token_here"

# Nginx Proxy Manager
NPM_URL = "http://localhost:81"
NPM_EMAIL = "jpmschweitzer@gmail.com"
NPM_PASSWORD = "your_actual_password"

# Uptime Kuma
KUMA_URL = "http://localhost:3001"
KUMA_USERNAME = "admin"
KUMA_PASSWORD = "your_actual_password"

Updated config.py to use credentials:

from src.credentials import (
    PORTAINER_URL, PORTAINER_API_KEY,
    NPM_URL, NPM_EMAIL, NPM_PASSWORD,
    KUMA_URL, KUMA_USERNAME, KUMA_PASSWORD
)

class Settings(BaseSettings):
    # Infrastructure Management (from credentials.py)
    portainer_url: str = PORTAINER_URL
    portainer_api_key: str = PORTAINER_API_KEY

    npm_url: str = NPM_URL
    npm_email: str = NPM_EMAIL
    npm_password: str = NPM_PASSWORD

    kuma_url: str = KUMA_URL
    kuma_username: str = KUMA_USERNAME
    kuma_password: str = KUMA_PASSWORD

Benefits

1. Cleaner Code: Separation of concerns, easier to maintain
2. Automation: Programmatic service deployment and configuration
3. Elimination of Shell Scripts: Replace ad-hoc scripts with proper API
4. Service Discovery: Auto-detect running services and configurations
5. Self-Managing Homelab: Foundation for autonomous infrastructure

Migration Notes

• Existing /v1/ endpoints remain unchanged for backward compatibility
• Web scraper endpoints stay at /web-scraper/ initially
• Old shell scripts in /stacks/ will be replaced with CLI wrappers