ok... ok... I'll add it to git...
This commit is contained in:
@@ -0,0 +1,222 @@
|
||||
# Core-API Refactoring Plan
|
||||
|
||||
**Date:** 2025-11-14
|
||||
**Goal:** Restructure Core-API into controller-based architecture and add Infrastructure Management API
|
||||
|
||||
## Current Structure
|
||||
|
||||
```
|
||||
src/
|
||||
├── api/
|
||||
│ └── v1/
|
||||
│ ├── chat.py # AI chat completions
|
||||
│ ├── models.py # Model listing
|
||||
│ ├── conversations.py # Conversation memory
|
||||
│ └── schemas.py # Pydantic schemas
|
||||
├── web_scraper/
|
||||
│ ├── router.py # Webscraper endpoints
|
||||
│ ├── service.py
|
||||
│ └── schemas.py
|
||||
├── models/
|
||||
│ ├── ollama_client.py # Ollama HTTP client
|
||||
│ └── embeddings.py
|
||||
├── memory/ # Memory tier system
|
||||
├── config.py # Global settings
|
||||
└── main.py # FastAPI app
|
||||
|
||||
```
|
||||
|
||||
## Target Structure
|
||||
|
||||
```
|
||||
src/
|
||||
├── controllers/ # NEW: Controller-based routing
|
||||
│ ├── __init__.py
|
||||
│ ├── base.py # Base controller class
|
||||
│ ├── ai_controller.py # AI Orchestrator (chat, models, conversations)
|
||||
│ ├── tools_controller.py # Utility tools (webscraper, etc.)
|
||||
│ ├── health_controller.py # Health & monitoring
|
||||
│ └── infrastructure_controller.py # Infrastructure automation
|
||||
├── clients/ # NEW: External API clients
|
||||
│ ├── __init__.py
|
||||
│ ├── portainer_client.py # Portainer API
|
||||
│ ├── npm_client.py # Nginx Proxy Manager API
|
||||
│ └── kuma_client.py # Uptime Kuma Socket.IO API
|
||||
├── api/v1/ # Keep existing for backward compat
|
||||
├── web_scraper/ # Keep as-is for now
|
||||
├── models/ # Keep as-is
|
||||
├── memory/ # Keep as-is
|
||||
├── config.py # Enhanced with infrastructure settings
|
||||
└── main.py # Updated routing
|
||||
|
||||
```
|
||||
|
||||
## Implementation Phases
|
||||
|
||||
### Phase 1: Infrastructure Setup 🔄 IN PROGRESS
|
||||
- [x] Research API authentication methods
|
||||
- [x] Add infrastructure settings to config.py
|
||||
- [ ] Create credentials.py for sensitive data (gitignored)
|
||||
- [ ] Create credentials.example.py as template
|
||||
- [ ] Update .gitignore to exclude credentials.py
|
||||
- [ ] Update config.py to import from credentials module
|
||||
- [x] Create /controllers directory structure
|
||||
- [x] Create /clients directory structure
|
||||
- [x] Create base controller class
|
||||
|
||||
### Phase 2: API Clients ✅ COMPLETE (Portainer & NPM)
|
||||
- [x] Implement Portainer API client (access token auth)
|
||||
- [x] Implement NPM API client (JWT with refresh)
|
||||
- [x] Add token storage/refresh mechanisms
|
||||
- [ ] Implement Uptime Kuma Socket.IO client (DEFERRED - WebSocket complexity)
|
||||
|
||||
### Phase 3: Infrastructure Controller 🔄 IN PROGRESS
|
||||
- [x] GET /infrastructure/health - Check connectivity
|
||||
- [x] GET /infrastructure/services - List all services
|
||||
- [x] GET /infrastructure/services/{name} - Get service details
|
||||
- [x] GET /infrastructure/ports - List allocated ports (skeleton)
|
||||
- [x] GET /infrastructure/domains - List configured domains
|
||||
- [ ] POST /infrastructure/services - Deploy new service
|
||||
- [ ] PUT /infrastructure/services/{name} - Update service
|
||||
- [ ] DELETE /infrastructure/services/{name} - Remove service
|
||||
- [ ] POST /infrastructure/monitoring/add - Auto-add Kuma monitor
|
||||
- [ ] POST /infrastructure/proxy/add - Auto-add NPM proxy host
|
||||
|
||||
### Phase 4: Refactor Existing Controllers 📋 PENDING
|
||||
- [ ] Move AI endpoints to ai_controller.py
|
||||
- [ ] Move webscraper to tools_controller.py
|
||||
- [ ] Move health check to health_controller.py
|
||||
- [ ] Update main.py imports and routing
|
||||
|
||||
### Phase 5: Testing & Documentation 📋 PENDING
|
||||
- [ ] Test all refactored endpoints
|
||||
- [ ] Update API documentation
|
||||
- [ ] Create CLI wrapper scripts
|
||||
- [ ] Remove old shell scripts
|
||||
|
||||
---
|
||||
|
||||
## Progress Notes (2025-11-14)
|
||||
|
||||
### Session 1: Foundation & Read Endpoints
|
||||
**Completed:**
|
||||
- Created controller and client architecture
|
||||
- Implemented Portainer client with full CRUD operations for stacks
|
||||
- Implemented NPM client with JWT refresh and proxy/certificate management
|
||||
- Built infrastructure controller with 5 read/list endpoints
|
||||
- Added infrastructure settings to config.py
|
||||
|
||||
**Files Created:**
|
||||
- `src/controllers/__init__.py`
|
||||
- `src/controllers/base.py`
|
||||
- `src/controllers/infrastructure_controller.py`
|
||||
- `src/clients/__init__.py`
|
||||
- `src/clients/portainer_client.py`
|
||||
- `src/clients/npm_client.py`
|
||||
- `REFACTORING_PLAN.md` (this file)
|
||||
|
||||
**Next Steps:**
|
||||
1. Create credentials.py for secure credential management
|
||||
2. Update config.py to import from credentials module
|
||||
3. Add credentials.py to .gitignore
|
||||
4. Update main.py to include infrastructure routes
|
||||
5. Test endpoints with live infrastructure
|
||||
6. Implement write/deploy operations
|
||||
7. Refactor existing AI/tools/health endpoints
|
||||
8. Create CLI wrappers
|
||||
|
||||
## API Authentication Strategy
|
||||
|
||||
### Portainer
|
||||
- **Method:** Access Token (X-API-Key header)
|
||||
- **Setup:** Manual creation in UI, store in config/env
|
||||
- **Duration:** Long-lived
|
||||
- **Storage:** Environment variable `PORTAINER_API_KEY`
|
||||
|
||||
### Nginx Proxy Manager
|
||||
- **Method:** JWT Bearer Token
|
||||
- **Setup:** Login via `/api/tokens` with credentials
|
||||
- **Duration:** ~24 hours
|
||||
- **Strategy:** Auto-refresh with stored credentials
|
||||
- **Storage:** `NPM_EMAIL` and `NPM_PASSWORD` in env
|
||||
|
||||
### Uptime Kuma
|
||||
- **Method:** Socket.IO WebSocket
|
||||
- **Setup:** Login via Socket.IO `login` event
|
||||
- **Duration:** Session-based
|
||||
- **Strategy:** Maintain persistent connection or re-auth per request
|
||||
- **Storage:** `KUMA_USERNAME` and `KUMA_PASSWORD` in env
|
||||
|
||||
## Configuration Changes
|
||||
|
||||
### Credentials Management Strategy
|
||||
|
||||
**Use `credentials.py` for sensitive data** (added to `.gitignore`):
|
||||
- Keeps secrets out of version control
|
||||
- Easy terminal-based management with editor
|
||||
- Python format for type safety and autocomplete
|
||||
- Separate from config for security isolation
|
||||
|
||||
**Implementation:**
|
||||
1. Create `src/credentials.py` with credentials (gitignored)
|
||||
2. Create `src/credentials.example.py` as template (committed)
|
||||
3. Update `config.py` to import from credentials module
|
||||
4. Add `credentials.py` to `.gitignore`
|
||||
|
||||
**Example `src/credentials.py`:**
|
||||
```python
|
||||
"""
|
||||
Infrastructure credentials (GITIGNORED)
|
||||
Copy from credentials.example.py and fill in real values
|
||||
"""
|
||||
|
||||
# Portainer
|
||||
PORTAINER_URL = "http://localhost:8001"
|
||||
PORTAINER_API_KEY = "ptr_your_actual_token_here"
|
||||
|
||||
# Nginx Proxy Manager
|
||||
NPM_URL = "http://localhost:81"
|
||||
NPM_EMAIL = "jpmschweitzer@gmail.com"
|
||||
NPM_PASSWORD = "your_actual_password"
|
||||
|
||||
# Uptime Kuma
|
||||
KUMA_URL = "http://localhost:3001"
|
||||
KUMA_USERNAME = "admin"
|
||||
KUMA_PASSWORD = "your_actual_password"
|
||||
```
|
||||
|
||||
**Updated `config.py` to use credentials:**
|
||||
```python
|
||||
from src.credentials import (
|
||||
PORTAINER_URL, PORTAINER_API_KEY,
|
||||
NPM_URL, NPM_EMAIL, NPM_PASSWORD,
|
||||
KUMA_URL, KUMA_USERNAME, KUMA_PASSWORD
|
||||
)
|
||||
|
||||
class Settings(BaseSettings):
|
||||
# Infrastructure Management (from credentials.py)
|
||||
portainer_url: str = PORTAINER_URL
|
||||
portainer_api_key: str = PORTAINER_API_KEY
|
||||
|
||||
npm_url: str = NPM_URL
|
||||
npm_email: str = NPM_EMAIL
|
||||
npm_password: str = NPM_PASSWORD
|
||||
|
||||
kuma_url: str = KUMA_URL
|
||||
kuma_username: str = KUMA_USERNAME
|
||||
kuma_password: str = KUMA_PASSWORD
|
||||
```
|
||||
|
||||
## Benefits
|
||||
|
||||
1. **Cleaner Code:** Separation of concerns, easier to maintain
|
||||
2. **Automation:** Programmatic service deployment and configuration
|
||||
3. **Elimination of Shell Scripts:** Replace ad-hoc scripts with proper API
|
||||
4. **Service Discovery:** Auto-detect running services and configurations
|
||||
5. **Self-Managing Homelab:** Foundation for autonomous infrastructure
|
||||
|
||||
## Migration Notes
|
||||
|
||||
- Existing `/v1/` endpoints remain unchanged for backward compatibility
|
||||
- Web scraper endpoints stay at `/web-scraper/` initially
|
||||
- Old shell scripts in `/stacks/` will be replaced with CLI wrappers
|
||||
Reference in New Issue
Block a user