# Core-API Refactoring Plan

**Date:** 2025-11-14
**Goal:** Restructure Core-API into controller-based architecture and add Infrastructure Management API

## Current Structure

```
src/
├── api/
│   └── v1/
│       ├── chat.py           # AI chat completions
│       ├── models.py          # Model listing
│       ├── conversations.py   # Conversation memory
│       └── schemas.py         # Pydantic schemas
├── web_scraper/
│   ├── router.py             # Webscraper endpoints
│   ├── service.py
│   └── schemas.py
├── models/
│   ├── ollama_client.py      # Ollama HTTP client
│   └── embeddings.py
├── memory/                    # Memory tier system
├── config.py                  # Global settings
└── main.py                    # FastAPI app

```

## Target Structure

```
src/
├── controllers/               # NEW: Controller-based routing
│   ├── __init__.py
│   ├── base.py               # Base controller class
│   ├── ai_controller.py      # AI Orchestrator (chat, models, conversations)
│   ├── tools_controller.py   # Utility tools (webscraper, etc.)
│   ├── health_controller.py  # Health & monitoring
│   └── infrastructure_controller.py  # Infrastructure automation
├── clients/                   # NEW: External API clients
│   ├── __init__.py
│   ├── portainer_client.py   # Portainer API
│   ├── npm_client.py         # Nginx Proxy Manager API
│   └── kuma_client.py        # Uptime Kuma Socket.IO API
├── api/v1/                    # Keep existing for backward compat
├── web_scraper/               # Keep as-is for now
├── models/                    # Keep as-is
├── memory/                    # Keep as-is
├── config.py                  # Enhanced with infrastructure settings
└── main.py                    # Updated routing

```

## Implementation Phases

### Phase 1: Infrastructure Setup ✅ COMPLETE
- [x] Research API authentication methods
- [x] Add infrastructure settings to config.py
- [x] Create credentials.py for sensitive data (gitignored)
- [x] Create credentials.example.py as template
- [x] Update .gitignore to exclude credentials.py
- [x] Update config.py to import from credentials module
- [x] Create /controllers directory structure
- [x] Create /clients directory structure
- [x] Create base controller class

### Phase 2: API Clients ✅ COMPLETE (Portainer & NPM)
- [x] Implement Portainer API client (access token auth)
- [x] Implement NPM API client (JWT with refresh)
- [x] Add token storage/refresh mechanisms
- [ ] Implement Uptime Kuma Socket.IO client (DEFERRED - WebSocket complexity)

### Phase 3: Infrastructure Controller ✅ COMPLETE
- [x] GET /infrastructure/health - Check connectivity ✅ TESTED
- [x] GET /infrastructure/services - List all services ✅ TESTED
- [x] GET /infrastructure/services/{name} - Get service details ✅ TESTED
- [x] GET /infrastructure/ports - List allocated ports (skeleton)
- [x] GET /infrastructure/domains - List configured domains ✅ TESTED
- [x] Integrate with main.py routing ✅ TESTED
- [x] Fix Pydantic validation issues (status field type conversion)
- [x] POST /infrastructure/services - Deploy new service ✅ TESTED
- [x] PUT /infrastructure/services/{name} - Update service ✅ TESTED
- [x] DELETE /infrastructure/services/{name} - Remove service ✅ TESTED
- [x] POST /infrastructure/proxy - Create NPM proxy host with optional SSL ✅ IMPLEMENTED
- [ ] POST /infrastructure/monitoring/add - Auto-add Kuma monitor (DEFERRED - Socket.IO complexity)

### Phase 4: Refactor Existing Controllers 📋 PENDING
- [ ] Move AI endpoints to ai_controller.py
- [ ] Move webscraper to tools_controller.py
- [ ] Move health check to health_controller.py
- [ ] Update main.py imports and routing

### Phase 5: Testing & Documentation 📋 PENDING
- [ ] Test all refactored endpoints
- [ ] Update API documentation
- [ ] Create CLI wrapper scripts
- [ ] Remove old shell scripts

---

## Progress Notes (2025-11-14)

### Session 1: Foundation & Read Endpoints
**Completed:**
- Created controller and client architecture
- Implemented Portainer client with full CRUD operations for stacks
- Implemented NPM client with JWT refresh and proxy/certificate management
- Built infrastructure controller with 5 read/list endpoints
- Added infrastructure settings to config.py

**Files Created:**
- `src/controllers/__init__.py`
- `src/controllers/base.py`
- `src/controllers/infrastructure_controller.py`
- `src/clients/__init__.py`
- `src/clients/portainer_client.py`
- `src/clients/npm_client.py`
- `REFACTORING_PLAN.md` (this file)

### Session 2: Credentials & Testing (2025-11-14 Evening)
**Completed:**
- Created credentials management system (credentials.py gitignored, credentials.example.py committed)
- Updated config.py to import from credentials module with fallback
- Generated Portainer API token programmatically via API
- Integrated infrastructure controller into main.py
- Fixed Pydantic validation bug (status field int→str conversion)
- Tested all read endpoints with live Portainer/NPM infrastructure
- Verified 8 stacks detected, domains with SSL status working

**Test Results:**
- ✅ GET /infrastructure/health - Portainer connected, NPM accessible
- ✅ GET /infrastructure/services - Returns 8 active stacks
- ✅ GET /infrastructure/services/{name} - Service lookup working
- ✅ GET /infrastructure/domains - Returns proxy hosts with SSL status
- ⚠️ NPM health check shows `false` (returns 302 redirect instead of 200)

### Session 3: Write Endpoints (2025-11-14 Evening)
**Completed:**
- Created request/response models for write operations (DeployServiceRequest, UpdateServiceRequest, CreateProxyRequest, OperationResult)
- Implemented POST /infrastructure/services - Deploy new service from compose YAML
- Implemented PUT /infrastructure/services/{name} - Update existing service configuration
- Implemented DELETE /infrastructure/services/{name} - Remove service and stack
- Implemented POST /infrastructure/proxy - Create NPM proxy host with optional SSL certificate
- Updated main.py API description with write endpoints
- Tested all service management endpoints (POST/PUT/DELETE) with live Portainer instance

**Test Results:**
- ✅ POST /infrastructure/services - Created test-nginx stack (ID: 30)
- ✅ PUT /infrastructure/services/test-nginx - Updated compose with environment variable
- ✅ DELETE /infrastructure/services/test-nginx - Removed stack successfully
- ✅ POST /infrastructure/proxy - Implemented (not tested to avoid production interference)

**Next Steps:**
1. Refactor existing AI/tools/health endpoints into separate controllers (Phase 4)
2. Fix NPM health check to handle redirects
3. Implement port allocation detection logic
4. Create CLI wrappers for common operations

## API Authentication Strategy

### Portainer
- **Method:** Access Token (X-API-Key header)
- **Setup:** Manual creation in UI, store in config/env
- **Duration:** Long-lived
- **Storage:** Environment variable `PORTAINER_API_KEY`

### Nginx Proxy Manager
- **Method:** JWT Bearer Token
- **Setup:** Login via `/api/tokens` with credentials
- **Duration:** ~24 hours
- **Strategy:** Auto-refresh with stored credentials
- **Storage:** `NPM_EMAIL` and `NPM_PASSWORD` in env

### Uptime Kuma
- **Method:** Socket.IO WebSocket
- **Setup:** Login via Socket.IO `login` event
- **Duration:** Session-based
- **Strategy:** Maintain persistent connection or re-auth per request
- **Storage:** `KUMA_USERNAME` and `KUMA_PASSWORD` in env

## Configuration Changes

### Credentials Management Strategy

**Use `credentials.py` for sensitive data** (added to `.gitignore`):
- Keeps secrets out of version control
- Easy terminal-based management with editor
- Python format for type safety and autocomplete
- Separate from config for security isolation

**Implementation:**
1. Create `src/credentials.py` with credentials (gitignored)
2. Create `src/credentials.example.py` as template (committed)
3. Update `config.py` to import from credentials module
4. Add `credentials.py` to `.gitignore`

**Example `src/credentials.py`:**
```python
"""
Infrastructure credentials (GITIGNORED)
Copy from credentials.example.py and fill in real values
"""

# Portainer
PORTAINER_URL = "http://localhost:8001"
PORTAINER_API_KEY = "ptr_your_actual_token_here"

# Nginx Proxy Manager
NPM_URL = "http://localhost:81"
NPM_EMAIL = "jpmschweitzer@gmail.com"
NPM_PASSWORD = "your_actual_password"

# Uptime Kuma
KUMA_URL = "http://localhost:3001"
KUMA_USERNAME = "admin"
KUMA_PASSWORD = "your_actual_password"
```

**Updated `config.py` to use credentials:**
```python
from src.credentials import (
    PORTAINER_URL, PORTAINER_API_KEY,
    NPM_URL, NPM_EMAIL, NPM_PASSWORD,
    KUMA_URL, KUMA_USERNAME, KUMA_PASSWORD
)

class Settings(BaseSettings):
    # Infrastructure Management (from credentials.py)
    portainer_url: str = PORTAINER_URL
    portainer_api_key: str = PORTAINER_API_KEY

    npm_url: str = NPM_URL
    npm_email: str = NPM_EMAIL
    npm_password: str = NPM_PASSWORD

    kuma_url: str = KUMA_URL
    kuma_username: str = KUMA_USERNAME
    kuma_password: str = KUMA_PASSWORD
```

## Benefits

1. **Cleaner Code:** Separation of concerns, easier to maintain
2. **Automation:** Programmatic service deployment and configuration
3. **Elimination of Shell Scripts:** Replace ad-hoc scripts with proper API
4. **Service Discovery:** Auto-detect running services and configurations
5. **Self-Managing Homelab:** Foundation for autonomous infrastructure

## Migration Notes

- Existing `/v1/` endpoints remain unchanged for backward compatibility
- Web scraper endpoints stay at `/web-scraper/` initially
- Old shell scripts in `/stacks/` will be replaced with CLI wrappers