# Core-API Refactoring Plan

**Date:** 2025-11-14
**Goal:** Restructure Core-API into controller-based architecture and add Infrastructure Management API

## Current Structure

```
src/
├── api/
│   └── v1/
│       ├── chat.py           # AI chat completions
│       ├── models.py          # Model listing
│       ├── conversations.py   # Conversation memory
│       └── schemas.py         # Pydantic schemas
├── web_scraper/
│   ├── router.py             # Webscraper endpoints
│   ├── service.py
│   └── schemas.py
├── models/
│   ├── ollama_client.py      # Ollama HTTP client
│   └── embeddings.py
├── memory/                    # Memory tier system
├── config.py                  # Global settings
└── main.py                    # FastAPI app

```

## Target Structure

```
src/
├── controllers/               # NEW: Controller-based routing
│   ├── __init__.py
│   ├── base.py               # Base controller class
│   ├── ai_controller.py      # AI Orchestrator (chat, models, conversations)
│   ├── tools_controller.py   # Utility tools (webscraper, etc.)
│   ├── health_controller.py  # Health & monitoring
│   └── infrastructure_controller.py  # Infrastructure automation
├── clients/                   # NEW: External API clients
│   ├── __init__.py
│   ├── portainer_client.py   # Portainer API
│   ├── npm_client.py         # Nginx Proxy Manager API
│   └── kuma_client.py        # Uptime Kuma Socket.IO API
├── api/v1/                    # Keep existing for backward compat
├── web_scraper/               # Keep as-is for now
├── models/                    # Keep as-is
├── memory/                    # Keep as-is
├── config.py                  # Enhanced with infrastructure settings
└── main.py                    # Updated routing

```

## Implementation Phases

### Phase 1: Infrastructure Setup ✅ COMPLETE
- [x] Research API authentication methods
- [x] Add infrastructure settings to config.py
- [x] Create credentials.py for sensitive data (gitignored)
- [x] Create credentials.example.py as template
- [x] Update .gitignore to exclude credentials.py
- [x] Update config.py to import from credentials module
- [x] Create /controllers directory structure
- [x] Create /clients directory structure
- [x] Create base controller class

### Phase 2: API Clients ✅ COMPLETE (Portainer & NPM)
- [x] Implement Portainer API client (access token auth)
- [x] Implement NPM API client (JWT with refresh)
- [x] Add token storage/refresh mechanisms
- [ ] Implement Uptime Kuma Socket.IO client (DEFERRED - WebSocket complexity)

### Phase 3: Infrastructure Controller ✅ COMPLETE
- [x] GET /infrastructure/health - Check connectivity ✅ TESTED
- [x] GET /infrastructure/services - List all services ✅ TESTED
- [x] GET /infrastructure/services/{name} - Get service details ✅ TESTED
- [x] GET /infrastructure/ports - List allocated ports ✅ IMPLEMENTED & TESTED
- [x] GET /infrastructure/domains - List configured domains ✅ TESTED
- [x] Integrate with main.py routing ✅ TESTED
- [x] Fix Pydantic validation issues (status field type conversion)
- [x] POST /infrastructure/services - Deploy new service ✅ TESTED
- [x] PUT /infrastructure/services/{name} - Update service ✅ TESTED
- [x] DELETE /infrastructure/services/{name} - Remove service ✅ TESTED
- [x] POST /infrastructure/proxy - Create NPM proxy host with optional SSL ✅ IMPLEMENTED
- [ ] POST /infrastructure/monitoring/add - Auto-add Kuma monitor (DEFERRED - Socket.IO complexity)

### Phase 4: Refactor Existing Controllers ✅ COMPLETE
- [x] Move AI endpoints to ai_controller.py ✅ COMPLETE
- [x] Move webscraper to tools_controller.py ✅ COMPLETE
- [x] Move health check to health_controller.py ✅ COMPLETE
- [x] Update main.py imports and routing ✅ COMPLETE
- [x] Test all refactored endpoints ✅ ALL WORKING

### Phase 5: Testing & Documentation ✅ COMPLETE
- [x] Test all refactored endpoints ✅ ALL WORKING
- [x] Update API documentation (OpenAPI spec auto-generated and validated)
- [~] Create CLI wrapper scripts (SKIPPED - LLMs consume OpenAPI spec directly)
- [~] Remove old shell scripts (DEFERRED - not blocking)

### Phase 6: Infrastructure Improvements 📋 FUTURE
- [ ] Consolidate Docker network topology into single `docker-dataplane` network
  - Currently each stack has its own network (172.22.0.x, 172.25.0.x, 172.20.0.x, etc.)
  - Error-prone and unnecessarily complex
  - Single shared network simplifies inter-service communication
  - Reduces subnet conflicts and improves service discovery
  - Update all compose files to use: `networks: [docker-dataplane]`
  - Create network once: `docker network create docker-dataplane`

---

## Progress Notes (2025-11-14)

### Session 1: Foundation & Read Endpoints
**Completed:**
- Created controller and client architecture
- Implemented Portainer client with full CRUD operations for stacks
- Implemented NPM client with JWT refresh and proxy/certificate management
- Built infrastructure controller with 5 read/list endpoints
- Added infrastructure settings to config.py

**Files Created:**
- `src/controllers/__init__.py`
- `src/controllers/base.py`
- `src/controllers/infrastructure_controller.py`
- `src/clients/__init__.py`
- `src/clients/portainer_client.py`
- `src/clients/npm_client.py`
- `REFACTORING_PLAN.md` (this file)

### Session 2: Credentials & Testing (2025-11-14 Evening)
**Completed:**
- Created credentials management system (credentials.py gitignored, credentials.example.py committed)
- Updated config.py to import from credentials module with fallback
- Generated Portainer API token programmatically via API
- Integrated infrastructure controller into main.py
- Fixed Pydantic validation bug (status field int→str conversion)
- Tested all read endpoints with live Portainer/NPM infrastructure
- Verified 8 stacks detected, domains with SSL status working

**Test Results:**
- ✅ GET /infrastructure/health - Portainer connected, NPM accessible
- ✅ GET /infrastructure/services - Returns 8 active stacks
- ✅ GET /infrastructure/services/{name} - Service lookup working
- ✅ GET /infrastructure/domains - Returns proxy hosts with SSL status
- ✅ NPM health check fixed (now accepts 2xx/3xx status codes and follows redirects)

### Session 3: Write Endpoints (2025-11-14 Evening)
**Completed:**
- Created request/response models for write operations (DeployServiceRequest, UpdateServiceRequest, CreateProxyRequest, OperationResult)
- Implemented POST /infrastructure/services - Deploy new service from compose YAML
- Implemented PUT /infrastructure/services/{name} - Update existing service configuration
- Implemented DELETE /infrastructure/services/{name} - Remove service and stack
- Implemented POST /infrastructure/proxy - Create NPM proxy host with optional SSL certificate
- Updated main.py API description with write endpoints
- Tested all service management endpoints (POST/PUT/DELETE) with live Portainer instance

**Test Results:**
- ✅ POST /infrastructure/services - Created test-nginx stack (ID: 30)
- ✅ PUT /infrastructure/services/test-nginx - Updated compose with environment variable
- ✅ DELETE /infrastructure/services/test-nginx - Removed stack successfully
- ✅ POST /infrastructure/proxy - Implemented (not tested to avoid production interference)

**Next Steps:**
1. ~~Refactor existing AI/tools/health endpoints into separate controllers (Phase 4)~~ ✅ DONE (2025-11-14)
2. ~~Fix NPM health check to handle redirects~~ ✅ DONE (2025-11-14)
3. ~~Implement port allocation detection logic~~ ✅ DONE (2025-11-14)
4. ~~Create CLI wrappers for common operations~~ ⊘ SKIPPED (LLMs use OpenAPI)
5. (OPTIONAL) Consolidate Docker networks into `docker-dataplane` (Phase 6)

### Session 4: NPM Health Check & Port Detection (2025-11-14 Afternoon)
**Completed:**
- Fixed NPM health check to handle redirects properly
- Updated `npm_client.py` to accept 2xx/3xx status codes as healthy
- Enabled explicit redirect following in httpx client
- Verified fix with live NPM instance (now shows 9 proxy hosts)
- Implemented comprehensive port detection in `GET /infrastructure/ports` endpoint
- Added `get_containers()` and `get_container()` methods to PortainerClient
- Enhanced PortInfo model with internal/external hostname and IP fields
- Implemented domain mapping from NPM proxy hosts to services
- Added deduplication logic for port entries (Docker returns duplicates per bind address)

**Port Detection Features:**
- Scans all running containers across all Portainer endpoints
- Extracts internal port, host port, and protocol for each container
- Maps container names to service names via Docker Compose labels
- Retrieves internal Docker hostnames and IP addresses per network
- Cross-references NPM proxy hosts to identify external domains
- Returns 22 unique port mappings with complete metadata

**Technical Details:**

*NPM Health Check:*
- Issue: NPM's `/api` endpoint returns 302 redirect, old code only accepted 200
- Solution: Accept `200 <= status_code < 400` as healthy response
- Result: NPM health check now returns `true` and proxy hosts are enumerated correctly

*Port Detection:*
- Queries Portainer Docker API for container list and port mappings
- Extracts NetworkSettings for internal IPs and hostnames
- Builds port→domain map from NPM proxy hosts configuration
- Matches services to external domains using multiple strategies:
  - By container name + port
  - By internal IP + port
  - By host address + host port (localhost, 127.0.0.1, server IP)
- Deduplicates based on (port, container_name, protocol) tuple
- Example output: Nextcloud port 80 → internal IP 172.25.0.3 → external domain cloud.schweitz.net

### Session 5: Controller Architecture Refactoring (2025-11-14 Evening)
**Completed:**
- Created `ai_controller.py` consolidating chat, models, and conversations endpoints
- Created `tools_controller.py` for web scraper functionality
- Created `health_controller.py` for service health and info endpoints
- Updated `main.py` to use new controller-based architecture
- Removed legacy router imports and inline endpoint definitions
- Tested all refactored endpoints - 16 endpoints working correctly

**Architecture Changes:**
- All endpoints now follow consistent controller pattern inheriting from `BaseController`
- Controllers use `create_router()` method for FastAPI router configuration
- Clean separation of concerns:
  - `ai_controller.py` - AI orchestration and conversation memory (7 endpoints)
  - `tools_controller.py` - Utility tools like web scraper (1 endpoint)
  - `health_controller.py` - Service status and info (2 endpoints)
  - `infrastructure_controller.py` - Infrastructure management (6 endpoints)
- Simplified `main.py` from 220 lines to 152 lines
- Backward compatible - all existing endpoints work identically

**Test Results:**
- ✅ GET / - Service information
- ✅ GET /health - Health check with Ollama status
- ✅ GET /v1/models - Model listing
- ✅ POST /v1/chat/completions - Chat completions
- ✅ GET /v1/conversations/{id} - Conversation history
- ✅ GET /infrastructure/health - Infrastructure health
- ✅ POST /web-scraper/scrape - Web scraping
- ✅ OpenAPI spec generation - 16 endpoints documented

## API Authentication Strategy

### Portainer
- **Method:** Access Token (X-API-Key header)
- **Setup:** Manual creation in UI, store in config/env
- **Duration:** Long-lived
- **Storage:** Environment variable `PORTAINER_API_KEY`

### Nginx Proxy Manager
- **Method:** JWT Bearer Token
- **Setup:** Login via `/api/tokens` with credentials
- **Duration:** ~24 hours
- **Strategy:** Auto-refresh with stored credentials
- **Storage:** `NPM_EMAIL` and `NPM_PASSWORD` in env

### Uptime Kuma
- **Method:** Socket.IO WebSocket
- **Setup:** Login via Socket.IO `login` event
- **Duration:** Session-based
- **Strategy:** Maintain persistent connection or re-auth per request
- **Storage:** `KUMA_USERNAME` and `KUMA_PASSWORD` in env

## Configuration Changes

### Credentials Management Strategy

**Use `credentials.py` for sensitive data** (added to `.gitignore`):
- Keeps secrets out of version control
- Easy terminal-based management with editor
- Python format for type safety and autocomplete
- Separate from config for security isolation

**Implementation:**
1. Create `src/credentials.py` with credentials (gitignored)
2. Create `src/credentials.example.py` as template (committed)
3. Update `config.py` to import from credentials module
4. Add `credentials.py` to `.gitignore`

**Example `src/credentials.py`:**
```python
"""
Infrastructure credentials (GITIGNORED)
Copy from credentials.example.py and fill in real values
"""

# Portainer
PORTAINER_URL = "http://localhost:8001"
PORTAINER_API_KEY = "ptr_your_actual_token_here"

# Nginx Proxy Manager
NPM_URL = "http://localhost:81"
NPM_EMAIL = "jpmschweitzer@gmail.com"
NPM_PASSWORD = "your_actual_password"

# Uptime Kuma
KUMA_URL = "http://localhost:3001"
KUMA_USERNAME = "admin"
KUMA_PASSWORD = "your_actual_password"
```

**Updated `config.py` to use credentials:**
```python
from src.credentials import (
    PORTAINER_URL, PORTAINER_API_KEY,
    NPM_URL, NPM_EMAIL, NPM_PASSWORD,
    KUMA_URL, KUMA_USERNAME, KUMA_PASSWORD
)

class Settings(BaseSettings):
    # Infrastructure Management (from credentials.py)
    portainer_url: str = PORTAINER_URL
    portainer_api_key: str = PORTAINER_API_KEY

    npm_url: str = NPM_URL
    npm_email: str = NPM_EMAIL
    npm_password: str = NPM_PASSWORD

    kuma_url: str = KUMA_URL
    kuma_username: str = KUMA_USERNAME
    kuma_password: str = KUMA_PASSWORD
```

## Benefits

1. **Cleaner Code:** Separation of concerns, easier to maintain
2. **Automation:** Programmatic service deployment and configuration
3. **Elimination of Shell Scripts:** Replace ad-hoc scripts with proper API
4. **Service Discovery:** Auto-detect running services and configurations
5. **Self-Managing Homelab:** Foundation for autonomous infrastructure

## Migration Notes

- Existing `/v1/` endpoints remain unchanged for backward compatibility
- Web scraper endpoints stay at `/web-scraper/` initially
- Old shell scripts in `/stacks/` will be replaced with CLI wrappers