ai-flow improvement / add langchain
This commit is contained in:
@@ -46,6 +46,13 @@ This document is a **complete revision** of the Authentik SSO implementation pla
|
||||
5. Create backup snapshots at every milestone
|
||||
6. Update this document with progress and issues as we go
|
||||
|
||||
**SSO Inclusion Policy:**
|
||||
- ✅ **Include:** Web-based admin interfaces, dashboards, APIs requiring browser access
|
||||
- ❌ **Exclude:** Services with native mobile/desktop apps that work better with username/password
|
||||
- ❌ **Exclude:** Media streaming services (Jellyfin) - app integration priority
|
||||
- ❌ **Exclude:** Development tools (code-server) - IDE integration priority
|
||||
- ⏸️ **Defer:** Disabled/inactive services (Nextcloud) - implement when re-enabled
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
@@ -870,23 +877,52 @@ curl -I https://auth.schweitz.net # Should return error (Authentik not running)
|
||||
|
||||
**Dependencies:** M3 completed successfully
|
||||
|
||||
**Configuration:** See original plan Section 5.1.3 for detailed implementation.
|
||||
**Status:** ✅ **COMPLETE** - Using forward auth (shares Organizr Proxy provider)
|
||||
|
||||
**Expected Duration:** 90-120 minutes
|
||||
**Implementation Decision (2025-11-23):**
|
||||
- Core API already protected with forward auth via NPM
|
||||
- Shares "Organizr Proxy" provider with home.schweitz.net
|
||||
- Authentication working correctly with Google OAuth
|
||||
- Headers forwarded: X-authentik-username, X-authentik-email, X-authentik-groups, X-authentik-name, X-authentik-uid
|
||||
- **Decision:** Keep current setup, defer separate admin provider to avoid complexity
|
||||
- **Rationale:** Current implementation is secure and functional for homelab use case
|
||||
|
||||
**Status:** ⏳ Not Started
|
||||
**Configuration:**
|
||||
- Provider: Organizr Proxy (shared)
|
||||
- External host: https://api.schweitz.net
|
||||
- Outpost: Standalone proxy (port 9445)
|
||||
- Mode: forward_single
|
||||
|
||||
**Expected Duration:** ~~90-120 minutes~~ SKIPPED (already functional)
|
||||
|
||||
---
|
||||
|
||||
### Milestone 5: Remaining Services (Gradual Rollout)
|
||||
|
||||
**Objective:** Enable forward auth on remaining 9 services, one at a time, testing each before proceeding.
|
||||
**Objective:** Enable forward auth on remaining services, one at a time, testing each before proceeding.
|
||||
|
||||
**Dependencies:** M3 and M4 completed successfully
|
||||
|
||||
**Services:** Nextcloud, Gitea, Jellyfin, Open WebUI, code-server, Netdata, Uptime Kuma, AMP, Tatlock
|
||||
**Services to Protect:**
|
||||
- Gitea (git.schweitz.net)
|
||||
- Open WebUI (no external domain yet)
|
||||
- Netdata (no external domain yet)
|
||||
- Uptime Kuma (no external domain yet)
|
||||
- AMP (amp.schweitz.net)
|
||||
- Tatlock (tatlock.schweitz.net)
|
||||
|
||||
**Expected Duration:** 4-8 hours (30-60 min per service)
|
||||
**Services EXCLUDED from SSO (Keep Native Auth):**
|
||||
- ❌ **Jellyfin (media.schweitz.net)** - Better mobile app integration with native auth
|
||||
- ❌ **code-server (code.schweitz.net)** - Better VS Code integration with native auth
|
||||
- ❌ **Nextcloud (cloud.schweitz.net)** - Service disabled, SSO deferred until re-enabled
|
||||
|
||||
**Rationale for Exclusions:**
|
||||
- Jellyfin and code-server have excellent native authentication
|
||||
- Mobile apps and desktop clients work better with username/password
|
||||
- SSO adds complexity without significant security benefit for these services
|
||||
- Nextcloud is not currently in active use
|
||||
|
||||
**Expected Duration:** 3-6 hours (30-60 min per service)
|
||||
|
||||
**Status:** ⏳ Not Started
|
||||
|
||||
|
||||
@@ -0,0 +1,427 @@
|
||||
# Unified Agent Architecture Plan
|
||||
|
||||
**Date:** 2025-11-23
|
||||
**Objective:** Build a single intelligent agent that handles all tool routing, multi-modal processing, and agentic reasoning internally, exposing one simple chat endpoint to any UI
|
||||
|
||||
## Vision
|
||||
|
||||
Instead of configuring functions in Open WebUI (or any other UI), the Core API becomes an intelligent orchestrator that:
|
||||
|
||||
1. **Accepts simple chat messages** - Just like talking to ChatGPT
|
||||
2. **Internally routes to specialized tools/models** - Infrastructure management, web search, code execution, etc.
|
||||
3. **Streams reasoning/thinking** - Shows what it's doing ("Searching the web...", "Querying database...", "Using expert model...")
|
||||
4. **Returns unified responses** - Combines results from multiple sources transparently
|
||||
|
||||
### Benefits
|
||||
|
||||
✅ **UI-agnostic** - Works with Open WebUI, CLI, mobile apps, any client
|
||||
✅ **No configuration needed** - Users just chat naturally
|
||||
✅ **Transparent reasoning** - See what's happening under the hood
|
||||
✅ **Tool discovery** - Agent decides when to use tools, not manual triggers
|
||||
✅ **Multi-modal support** - Handle text, images, code, infrastructure queries
|
||||
✅ **Expert model routing** - Use small models for simple tasks, large for complex
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ User Interface │
|
||||
│ (Open WebUI, CLI, Mobile App, etc.) │
|
||||
└──────────────────────┬──────────────────────────────────────┘
|
||||
│ Simple chat: "Deploy nginx proxy"
|
||||
↓
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Core API - Unified Agent │
|
||||
│ /v1/chat/completions (OpenAI-compatible endpoint) │
|
||||
└──────────────────────┬──────────────────────────────────────┘
|
||||
│
|
||||
↓
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Agent Orchestrator (LangGraph) │
|
||||
│ ┌──────────────────────────────────────────────┐ │
|
||||
│ │ Reasoning Loop: │ │
|
||||
│ │ 1. Analyze user intent │ │
|
||||
│ │ 2. Select appropriate tool(s) │ │
|
||||
│ │ 3. Execute tool calls │ │
|
||||
│ │ 4. Synthesize results │ │
|
||||
│ │ 5. Stream thinking/reasoning │ │
|
||||
│ └──────────────────────────────────────────────┘ │
|
||||
└──────────────────────┬──────────────────────────────────────┘
|
||||
│
|
||||
┌──────────────┼──────────────┬──────────────┐
|
||||
│ │ │ │
|
||||
↓ ↓ ↓ ↓
|
||||
┌──────────────┐ ┌──────────┐ ┌──────────┐ ┌──────────────┐
|
||||
│ Tool Catalog │ │ Models │ │ Memory │ │ Knowledge │
|
||||
│ │ │ │ │ │ │ │
|
||||
│ • Infra Mgmt │ │ • Gemma │ │ • Qdrant │ │ • Web Search │
|
||||
│ • Web Scrape │ │ • Codestral│ │ • Buffer│ │ • Docs │
|
||||
│ • File Ops │ │ • Mistral│ │ │ │ │
|
||||
│ • Code Exec │ │ │ │ │ │ │
|
||||
└──────────────┘ └──────────┘ └──────────┘ └──────────────┘
|
||||
```
|
||||
|
||||
## Implementation Options
|
||||
|
||||
### Option 1: LangGraph (Recommended)
|
||||
|
||||
**Pros:**
|
||||
- Built-in agent loops and tool calling
|
||||
- State management for multi-step reasoning
|
||||
- Streaming support for intermediate steps
|
||||
- Well-documented patterns
|
||||
- Active development
|
||||
|
||||
**Cons:**
|
||||
- Additional dependency (~50MB)
|
||||
- Learning curve for LangGraph concepts
|
||||
- Some overhead vs custom implementation
|
||||
|
||||
**Example flow:**
|
||||
```python
|
||||
from langgraph.prebuilt import create_react_agent
|
||||
from langchain_core.tools import tool
|
||||
|
||||
@tool
|
||||
def deploy_service(service_name: str, compose_yaml: str) -> str:
|
||||
"""Deploy a containerized service via Portainer"""
|
||||
# Use existing infrastructure controller
|
||||
return portainer_client.deploy_stack(...)
|
||||
|
||||
@tool
|
||||
def web_search(query: str) -> str:
|
||||
"""Search the web and extract content"""
|
||||
# Use existing web scraper
|
||||
return scraper.scrape(...)
|
||||
|
||||
agent = create_react_agent(
|
||||
model=ChatOllama(model="gemma:7b"),
|
||||
tools=[deploy_service, web_search, ...],
|
||||
state_modifier="You are a homelab infrastructure assistant..."
|
||||
)
|
||||
|
||||
# Streaming with reasoning
|
||||
for chunk in agent.stream({"messages": [user_message]}):
|
||||
if "thinking" in chunk:
|
||||
yield f"data: {json.dumps({'reasoning': chunk['thinking']})}\n\n"
|
||||
if "tool_calls" in chunk:
|
||||
yield f"data: {json.dumps({'tool': chunk['tool_calls'][0]['name']})}\n\n"
|
||||
if "response" in chunk:
|
||||
yield f"data: {json.dumps({'content': chunk['response']})}\n\n"
|
||||
```
|
||||
|
||||
### Option 2: Custom Agent Loop
|
||||
|
||||
**Pros:**
|
||||
- Full control over behavior
|
||||
- Minimal dependencies
|
||||
- Optimized for specific use case
|
||||
- Easier to debug
|
||||
|
||||
**Cons:**
|
||||
- More code to maintain
|
||||
- Need to implement tool calling protocol
|
||||
- Reinventing some wheels
|
||||
|
||||
**Example flow:**
|
||||
```python
|
||||
class UnifiedAgent:
|
||||
def __init__(self):
|
||||
self.tools = ToolCatalog()
|
||||
self.model = OllamaClient()
|
||||
|
||||
async def process(self, user_message: str):
|
||||
# 1. Intent analysis
|
||||
yield {"type": "thinking", "content": "Analyzing your request..."}
|
||||
intent = await self.analyze_intent(user_message)
|
||||
|
||||
# 2. Tool selection
|
||||
if intent.requires_tool:
|
||||
yield {"type": "thinking", "content": f"Using {intent.tool_name}..."}
|
||||
tool_result = await self.tools.execute(intent.tool_name, intent.params)
|
||||
|
||||
# 3. Response generation
|
||||
yield {"type": "thinking", "content": "Generating response..."}
|
||||
response = await self.model.generate(context=tool_result)
|
||||
|
||||
yield {"type": "content", "content": response}
|
||||
```
|
||||
|
||||
### Option 3: Hybrid (LangChain Tools + Custom Orchestration)
|
||||
|
||||
Use LangChain's tool framework but custom agent loop:
|
||||
- Leverage `@tool` decorator for easy tool definitions
|
||||
- Custom routing logic for model selection
|
||||
- Manual streaming control
|
||||
|
||||
## Recommended Approach: LangGraph with Custom Extensions
|
||||
|
||||
**Phase 1: Core Agent (Week 1)**
|
||||
- Set up LangGraph agent with basic tools
|
||||
- Implement streaming with reasoning output
|
||||
- Wire up existing infrastructure tools
|
||||
- Test with simple queries
|
||||
|
||||
**Phase 2: Advanced Routing (Week 2)**
|
||||
- Multi-model routing (small for simple, large for complex)
|
||||
- Parallel tool execution
|
||||
- Error handling and retries
|
||||
- Context management
|
||||
|
||||
**Phase 3: Multi-Modal (Week 3)**
|
||||
- Image analysis (if needed)
|
||||
- Code execution sandbox
|
||||
- File operations
|
||||
- Database queries
|
||||
|
||||
## Tool Catalog Design
|
||||
|
||||
### Tier 1: Infrastructure Tools (Existing)
|
||||
|
||||
```python
|
||||
@tool
|
||||
async def list_services() -> List[Dict]:
|
||||
"""List all running Docker services"""
|
||||
return await portainer_client.list_containers()
|
||||
|
||||
@tool
|
||||
async def deploy_service(name: str, compose: str) -> str:
|
||||
"""Deploy a new service from Docker Compose YAML"""
|
||||
return await portainer_client.deploy_stack(name, compose)
|
||||
|
||||
@tool
|
||||
async def create_proxy(domain: str, target: str) -> str:
|
||||
"""Create Nginx reverse proxy for a service"""
|
||||
return await npm_client.create_proxy_host(domain, target)
|
||||
|
||||
@tool
|
||||
async def check_service_health(service: str) -> Dict:
|
||||
"""Check if a service is healthy"""
|
||||
return await kuma_client.get_monitor_status(service)
|
||||
```
|
||||
|
||||
### Tier 2: Knowledge Tools
|
||||
|
||||
```python
|
||||
@tool
|
||||
async def web_search(query: str) -> str:
|
||||
"""Search the web and extract main content"""
|
||||
return await scraper.scrape_url(query)
|
||||
|
||||
@tool
|
||||
async def query_memory(question: str) -> List[str]:
|
||||
"""Search conversation history for relevant context"""
|
||||
return await memory.semantic_search(question)
|
||||
|
||||
@tool
|
||||
async def read_documentation(topic: str) -> str:
|
||||
"""Read project documentation"""
|
||||
docs_path = f"/docs/{topic}.md"
|
||||
return read_file(docs_path)
|
||||
```
|
||||
|
||||
### Tier 3: Execution Tools (Future)
|
||||
|
||||
```python
|
||||
@tool
|
||||
async def execute_python(code: str) -> str:
|
||||
"""Execute Python code in sandbox"""
|
||||
# Future: Integrate code interpreter
|
||||
pass
|
||||
|
||||
@tool
|
||||
async def query_database(sql: str) -> List[Dict]:
|
||||
"""Query PostgreSQL database"""
|
||||
# Future: Safe SQL execution
|
||||
pass
|
||||
```
|
||||
|
||||
## Streaming Reasoning Output
|
||||
|
||||
### SSE Format for Transparency
|
||||
|
||||
```python
|
||||
# Stream format
|
||||
{
|
||||
"type": "thinking", # or "tool_call", "content", "error"
|
||||
"content": "Searching the web for nginx configuration...",
|
||||
"tool": "web_search", # optional, if type is tool_call
|
||||
"model": "gemma:7b" # optional, which model is being used
|
||||
}
|
||||
|
||||
# Example stream
|
||||
data: {"type": "thinking", "content": "Analyzing your request..."}
|
||||
|
||||
data: {"type": "thinking", "content": "Detected infrastructure task"}
|
||||
|
||||
data: {"type": "tool_call", "tool": "list_services", "content": "Checking current services..."}
|
||||
|
||||
data: {"type": "thinking", "content": "Found 22 running services"}
|
||||
|
||||
data: {"type": "thinking", "content": "Using expert model for response..."}
|
||||
|
||||
data: {"type": "model_switch", "from": "gemma:2b", "to": "mistral:7b"}
|
||||
|
||||
data: {"type": "content", "content": "Here are your running services:\n\n..."}
|
||||
|
||||
data: [DONE]
|
||||
```
|
||||
|
||||
### Open WebUI Integration
|
||||
|
||||
Open WebUI already supports streaming, we just need to format it correctly:
|
||||
|
||||
```javascript
|
||||
// Open WebUI will render thinking/reasoning in a collapsible section
|
||||
// Standard content renders as usual
|
||||
```
|
||||
|
||||
## Model Routing Strategy
|
||||
|
||||
### Intent-Based Routing
|
||||
|
||||
```python
|
||||
class ModelRouter:
|
||||
MODELS = {
|
||||
"simple": "gemma:2b", # Fast, <100 tokens
|
||||
"general": "gemma:7b", # Balanced
|
||||
"expert": "mistral:7b", # Complex reasoning
|
||||
"code": "codestral:latest" # Code tasks
|
||||
}
|
||||
|
||||
async def select_model(self, message: str, context: str) -> str:
|
||||
# Use lightweight model for routing decision
|
||||
prompt = f"""Analyze this request and categorize:
|
||||
|
||||
User: {message}
|
||||
Context: {context}
|
||||
|
||||
Categories:
|
||||
- simple: Greetings, basic facts, short answers
|
||||
- general: Normal conversation, explanations
|
||||
- expert: Complex reasoning, multi-step problems
|
||||
- code: Programming tasks, debugging
|
||||
|
||||
Return ONLY the category.
|
||||
"""
|
||||
|
||||
category = await ollama.generate(model="gemma:2b", prompt=prompt)
|
||||
return self.MODELS[category.strip()]
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Prototype LangGraph agent** (2-3 hours)
|
||||
- Basic agent with 2-3 tools
|
||||
- Streaming with thinking output
|
||||
- Test with Open WebUI
|
||||
|
||||
2. **Integrate existing tools** (3-4 hours)
|
||||
- Wrap infrastructure controller as tools
|
||||
- Wrap web scraper as tool
|
||||
- Test tool calling
|
||||
|
||||
3. **Model routing** (2 hours)
|
||||
- Implement intent analysis
|
||||
- Add model selection logic
|
||||
- Test performance
|
||||
|
||||
4. **Production deployment** (2 hours)
|
||||
- Error handling
|
||||
- Rate limiting
|
||||
- Logging and monitoring
|
||||
- Update API documentation
|
||||
|
||||
**Total effort:** ~12-15 hours (1-2 weeks of focused work)
|
||||
|
||||
## Success Criteria
|
||||
|
||||
✅ User can chat naturally without configuring functions
|
||||
✅ Agent automatically uses tools when appropriate
|
||||
✅ Streaming shows what the agent is doing
|
||||
✅ Works with Open WebUI without changes
|
||||
✅ Can be used from CLI/API directly
|
||||
✅ Performance is acceptable (<5s for tool-using responses)
|
||||
✅ Errors are handled gracefully
|
||||
|
||||
## Example User Flows
|
||||
|
||||
### Flow 1: Infrastructure Query
|
||||
```
|
||||
User: "What services are currently running?"
|
||||
|
||||
[Thinking: Analyzing request...]
|
||||
[Thinking: Detected infrastructure query]
|
||||
[Tool Call: list_services - Fetching service list...]
|
||||
[Thinking: Processing results...]
|
||||
[Content: You have 22 services running:
|
||||
- ollama (healthy)
|
||||
- core-api (healthy)
|
||||
- ...]
|
||||
```
|
||||
|
||||
### Flow 2: Complex Task
|
||||
```
|
||||
User: "Deploy an nginx proxy for my new blog at blog.schweitz.net"
|
||||
|
||||
[Thinking: Breaking down the task...]
|
||||
[Thinking: Need to deploy nginx and configure NPM]
|
||||
[Tool Call: deploy_service - Deploying nginx container...]
|
||||
[Tool Call: create_proxy - Creating proxy host...]
|
||||
[Thinking: Configuring SSL certificate...]
|
||||
[Content: Done! Your blog is now accessible at https://blog.schweitz.net
|
||||
- Nginx container: running
|
||||
- SSL certificate: active
|
||||
- Health check: passing]
|
||||
```
|
||||
|
||||
### Flow 3: Knowledge Query
|
||||
```
|
||||
User: "How do I configure Headscale?"
|
||||
|
||||
[Thinking: Checking documentation...]
|
||||
[Tool Call: read_documentation(headscale)]
|
||||
[Thinking: Extracting relevant steps...]
|
||||
[Content: To configure Headscale on tower-of-joy:
|
||||
|
||||
1. Create a user: `headscale users create homelab`
|
||||
2. Generate auth key: `headscale preauthkeys create...`
|
||||
...]
|
||||
```
|
||||
|
||||
## Technology Stack
|
||||
|
||||
- **Agent Framework:** LangGraph 0.2.x
|
||||
- **LLM Integration:** LangChain-Ollama
|
||||
- **Tool Framework:** LangChain Tools
|
||||
- **Streaming:** SSE (Server-Sent Events)
|
||||
- **State Management:** LangGraph StateGraph
|
||||
- **Memory:** Existing Qdrant integration
|
||||
|
||||
## Risk Mitigation
|
||||
|
||||
**Risk:** LangGraph adds complexity
|
||||
- **Mitigation:** Start simple, add features incrementally
|
||||
|
||||
**Risk:** Tool calling may be slow
|
||||
- **Mitigation:** Parallel execution, caching, optimized tools
|
||||
|
||||
**Risk:** Reasoning output may be verbose
|
||||
- **Mitigation:** Configurable verbosity, collapsible UI elements
|
||||
|
||||
**Risk:** May not work with all UIs
|
||||
- **Mitigation:** Stick to OpenAI-compatible streaming format
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. Should we support function calling format for backwards compatibility?
|
||||
2. How verbose should reasoning output be?
|
||||
3. Should we cache tool results?
|
||||
4. Do we need user confirmation for destructive operations?
|
||||
5. Should tools have permission levels based on user?
|
||||
|
||||
---
|
||||
|
||||
**Ready to implement:** Yes ✓
|
||||
**Estimated timeline:** 1-2 weeks
|
||||
**Priority:** High (enables true agentic behavior)
|
||||
Reference in New Issue
Block a user