security rework and memory optimilizations.
This commit is contained in:
@@ -74,7 +74,7 @@ src/
|
||||
- [x] GET /infrastructure/health - Check connectivity ✅ TESTED
|
||||
- [x] GET /infrastructure/services - List all services ✅ TESTED
|
||||
- [x] GET /infrastructure/services/{name} - Get service details ✅ TESTED
|
||||
- [x] GET /infrastructure/ports - List allocated ports (skeleton)
|
||||
- [x] GET /infrastructure/ports - List allocated ports ✅ IMPLEMENTED & TESTED
|
||||
- [x] GET /infrastructure/domains - List configured domains ✅ TESTED
|
||||
- [x] Integrate with main.py routing ✅ TESTED
|
||||
- [x] Fix Pydantic validation issues (status field type conversion)
|
||||
@@ -84,17 +84,27 @@ src/
|
||||
- [x] POST /infrastructure/proxy - Create NPM proxy host with optional SSL ✅ IMPLEMENTED
|
||||
- [ ] POST /infrastructure/monitoring/add - Auto-add Kuma monitor (DEFERRED - Socket.IO complexity)
|
||||
|
||||
### Phase 4: Refactor Existing Controllers 📋 PENDING
|
||||
- [ ] Move AI endpoints to ai_controller.py
|
||||
- [ ] Move webscraper to tools_controller.py
|
||||
- [ ] Move health check to health_controller.py
|
||||
- [ ] Update main.py imports and routing
|
||||
### Phase 4: Refactor Existing Controllers ✅ COMPLETE
|
||||
- [x] Move AI endpoints to ai_controller.py ✅ COMPLETE
|
||||
- [x] Move webscraper to tools_controller.py ✅ COMPLETE
|
||||
- [x] Move health check to health_controller.py ✅ COMPLETE
|
||||
- [x] Update main.py imports and routing ✅ COMPLETE
|
||||
- [x] Test all refactored endpoints ✅ ALL WORKING
|
||||
|
||||
### Phase 5: Testing & Documentation 📋 PENDING
|
||||
- [ ] Test all refactored endpoints
|
||||
- [ ] Update API documentation
|
||||
- [ ] Create CLI wrapper scripts
|
||||
- [ ] Remove old shell scripts
|
||||
### Phase 5: Testing & Documentation ✅ COMPLETE
|
||||
- [x] Test all refactored endpoints ✅ ALL WORKING
|
||||
- [x] Update API documentation (OpenAPI spec auto-generated and validated)
|
||||
- [~] Create CLI wrapper scripts (SKIPPED - LLMs consume OpenAPI spec directly)
|
||||
- [~] Remove old shell scripts (DEFERRED - not blocking)
|
||||
|
||||
### Phase 6: Infrastructure Improvements 📋 FUTURE
|
||||
- [ ] Consolidate Docker network topology into single `docker-dataplane` network
|
||||
- Currently each stack has its own network (172.22.0.x, 172.25.0.x, 172.20.0.x, etc.)
|
||||
- Error-prone and unnecessarily complex
|
||||
- Single shared network simplifies inter-service communication
|
||||
- Reduces subnet conflicts and improves service discovery
|
||||
- Update all compose files to use: `networks: [docker-dataplane]`
|
||||
- Create network once: `docker network create docker-dataplane`
|
||||
|
||||
---
|
||||
|
||||
@@ -132,7 +142,7 @@ src/
|
||||
- ✅ GET /infrastructure/services - Returns 8 active stacks
|
||||
- ✅ GET /infrastructure/services/{name} - Service lookup working
|
||||
- ✅ GET /infrastructure/domains - Returns proxy hosts with SSL status
|
||||
- ⚠️ NPM health check shows `false` (returns 302 redirect instead of 200)
|
||||
- ✅ NPM health check fixed (now accepts 2xx/3xx status codes and follows redirects)
|
||||
|
||||
### Session 3: Write Endpoints (2025-11-14 Evening)
|
||||
**Completed:**
|
||||
@@ -151,10 +161,79 @@ src/
|
||||
- ✅ POST /infrastructure/proxy - Implemented (not tested to avoid production interference)
|
||||
|
||||
**Next Steps:**
|
||||
1. Refactor existing AI/tools/health endpoints into separate controllers (Phase 4)
|
||||
2. Fix NPM health check to handle redirects
|
||||
3. Implement port allocation detection logic
|
||||
4. Create CLI wrappers for common operations
|
||||
1. ~~Refactor existing AI/tools/health endpoints into separate controllers (Phase 4)~~ ✅ DONE (2025-11-14)
|
||||
2. ~~Fix NPM health check to handle redirects~~ ✅ DONE (2025-11-14)
|
||||
3. ~~Implement port allocation detection logic~~ ✅ DONE (2025-11-14)
|
||||
4. ~~Create CLI wrappers for common operations~~ ⊘ SKIPPED (LLMs use OpenAPI)
|
||||
5. (OPTIONAL) Consolidate Docker networks into `docker-dataplane` (Phase 6)
|
||||
|
||||
### Session 4: NPM Health Check & Port Detection (2025-11-14 Afternoon)
|
||||
**Completed:**
|
||||
- Fixed NPM health check to handle redirects properly
|
||||
- Updated `npm_client.py` to accept 2xx/3xx status codes as healthy
|
||||
- Enabled explicit redirect following in httpx client
|
||||
- Verified fix with live NPM instance (now shows 9 proxy hosts)
|
||||
- Implemented comprehensive port detection in `GET /infrastructure/ports` endpoint
|
||||
- Added `get_containers()` and `get_container()` methods to PortainerClient
|
||||
- Enhanced PortInfo model with internal/external hostname and IP fields
|
||||
- Implemented domain mapping from NPM proxy hosts to services
|
||||
- Added deduplication logic for port entries (Docker returns duplicates per bind address)
|
||||
|
||||
**Port Detection Features:**
|
||||
- Scans all running containers across all Portainer endpoints
|
||||
- Extracts internal port, host port, and protocol for each container
|
||||
- Maps container names to service names via Docker Compose labels
|
||||
- Retrieves internal Docker hostnames and IP addresses per network
|
||||
- Cross-references NPM proxy hosts to identify external domains
|
||||
- Returns 22 unique port mappings with complete metadata
|
||||
|
||||
**Technical Details:**
|
||||
|
||||
*NPM Health Check:*
|
||||
- Issue: NPM's `/api` endpoint returns 302 redirect, old code only accepted 200
|
||||
- Solution: Accept `200 <= status_code < 400` as healthy response
|
||||
- Result: NPM health check now returns `true` and proxy hosts are enumerated correctly
|
||||
|
||||
*Port Detection:*
|
||||
- Queries Portainer Docker API for container list and port mappings
|
||||
- Extracts NetworkSettings for internal IPs and hostnames
|
||||
- Builds port→domain map from NPM proxy hosts configuration
|
||||
- Matches services to external domains using multiple strategies:
|
||||
- By container name + port
|
||||
- By internal IP + port
|
||||
- By host address + host port (localhost, 127.0.0.1, server IP)
|
||||
- Deduplicates based on (port, container_name, protocol) tuple
|
||||
- Example output: Nextcloud port 80 → internal IP 172.25.0.3 → external domain cloud.schweitz.net
|
||||
|
||||
### Session 5: Controller Architecture Refactoring (2025-11-14 Evening)
|
||||
**Completed:**
|
||||
- Created `ai_controller.py` consolidating chat, models, and conversations endpoints
|
||||
- Created `tools_controller.py` for web scraper functionality
|
||||
- Created `health_controller.py` for service health and info endpoints
|
||||
- Updated `main.py` to use new controller-based architecture
|
||||
- Removed legacy router imports and inline endpoint definitions
|
||||
- Tested all refactored endpoints - 16 endpoints working correctly
|
||||
|
||||
**Architecture Changes:**
|
||||
- All endpoints now follow consistent controller pattern inheriting from `BaseController`
|
||||
- Controllers use `create_router()` method for FastAPI router configuration
|
||||
- Clean separation of concerns:
|
||||
- `ai_controller.py` - AI orchestration and conversation memory (7 endpoints)
|
||||
- `tools_controller.py` - Utility tools like web scraper (1 endpoint)
|
||||
- `health_controller.py` - Service status and info (2 endpoints)
|
||||
- `infrastructure_controller.py` - Infrastructure management (6 endpoints)
|
||||
- Simplified `main.py` from 220 lines to 152 lines
|
||||
- Backward compatible - all existing endpoints work identically
|
||||
|
||||
**Test Results:**
|
||||
- ✅ GET / - Service information
|
||||
- ✅ GET /health - Health check with Ollama status
|
||||
- ✅ GET /v1/models - Model listing
|
||||
- ✅ POST /v1/chat/completions - Chat completions
|
||||
- ✅ GET /v1/conversations/{id} - Conversation history
|
||||
- ✅ GET /infrastructure/health - Infrastructure health
|
||||
- ✅ POST /web-scraper/scrape - Web scraping
|
||||
- ✅ OpenAPI spec generation - 16 endpoints documented
|
||||
|
||||
## API Authentication Strategy
|
||||
|
||||
|
||||
Reference in New Issue
Block a user