jpmschweitzer/portainer-core
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor(core-ai): comprehensive cleanup - PydanticAI only architecture

Browse Source 
Remove all obsolete agent implementations and framework references.
Keep only PydanticAI (primary) and SimpleLiteLLM (fallback).

This cleanup eliminates confusion between multiple frameworks that were
tried during development (LangChain, LangGraph, ADK, OllamaNative) and
establishes PydanticAI as the single agent framework going forward.

BREAKING CHANGES:
- Removed OllamaNativeAgent - use PydanticAgent instead
- Removed /test/ollama-tools diagnostic endpoint
- Default /v1/chat/completions now uses PydanticAgent

Files Deleted (32 total):
- Obsolete agents: ollama_native_agent.py
- Diagnostic files: ARCHITECTURE.md, DIAGNOSTIC_RESULTS.md, PHASE*.md
- Legacy tools: src/tools.py
- Test files: test_ai_flow_quality.py, test_02/03 (diagnostic layers)
- Documentation: ADK_Ollama_Research.md, agent-flow-diagrams.md
- Session docs: 3 files with LangChain/LangGraph implementations
- Plans: 5 completed plans about obsolete frameworks
- Migration docs: MIGRATION_PLAN_LANGCHAIN_TO_ADK.md

Files Modified (8 total):
- main.py: Refactored to PydanticAI only (305 lines vs 457 before)
- agents/__init__.py: Removed OllamaNativeAgent exports
- README.md: Complete rewrite for PydanticAI architecture
- prompts.py: Updated for PydanticAI (infrastructure tool guidance)
- STATUS.md: Updated to v0.11.0-pydantic-ai
- CHANGELOG.md: Added v0.11.0 entry documenting cleanup
- plans/active/*.md: Updated to reference PydanticAI

Current Architecture:
- Framework: PydanticAI with native Ollama SDK
- Agents: PydanticAgent (primary) + SimpleLiteLLMAgent (fallback)
- Model: mistral-nemo:latest
- Tools: 6 core + 28+ OpenAPI-discovered
- Memory: 3-tier system with Qdrant
- VRAM: ~4-6GB

Lines Removed: ~3000+ lines of obsolete code

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
jpmschweitzer
2025-12-03 14:24:52 +01:00
co-authored by Claude
parent 96492cb1ed
commit 66f6e54fc3
32 changed files with 1078 additions and 10376 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+55 -18
View File
@@ -7,29 +7,66 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
- **AI Quality Test Suite**
- Comprehensive test suite for core-ai agent performance (`test_ai_flow_quality.py`)
- Test documentation and usage guide (`QUALITY_TESTS.md`)
- 5 core test scenarios: simple knowledge, web search, calculation, date operations, multi-tool reasoning
- Performance baseline tests and regression detection
- Automated report generation (text and JSON formats)
- Git-tagged reports for version comparison and rollback
- Baseline established: 4/5 tests passing (80% success rate)
### Changed
- **Gitignore Updates**
- Added `services/core-ai/tests/reports/` to prevent committing test output files
- Maintains proper separation of code vs generated artifacts
### In Progress
- **System Monitoring:** Post-migration stability monitoring and performance optimization
### Planned
- AI Orchestrator Phases 5-6: Multi-agent workflows, RAG optimization, production hardening
- Authentik SSO Milestones 4-5: Protect remaining services (deferred)
- Disaster recovery and offsite backup strategy
## [0.11.0-pydantic-ai-cleanup] - 2025-12-03
### Major Changes
- **Framework Cleanup: PydanticAI Only** ✅ ARCHITECTURAL SIMPLIFICATION
- Removed all obsolete agent implementations (OllamaNativeAgent, ADK, LangChain, LangGraph)
- Kept only PydanticAgent (primary) and SimpleLiteLLMAgent (fallback)
- Single framework approach eliminates confusion and improves maintainability
- All documentation updated to reflect PydanticAI architecture
### Removed
- **Obsolete Agent Files:**
- `src/agents/ollama_native_agent.py` - Replaced by PydanticAgent
- Diagnostic and phase completion documentation files
- Obsolete test files (`test_ai_flow_quality.py`, test_02/03 diagnostic tests)
- Research documentation about ADK/LangChain
- **Obsolete Documentation:**
- `ARCHITECTURE.md`, `DIAGNOSTIC_RESULTS.md`, `PHASE*.md` files
- `docs/ADK_Ollama_Research.md`
- `docs/architecture/agent-flow-diagrams.md` (LangGraph references)
- Session docs with LangChain/LangGraph implementations
- Completed plans about ADK/LangChain migrations
### Changed
- **main.py:** Complete refactor to use only PydanticAI (305 lines vs 457 before)
- Removed `chat_completions()` endpoint using OllamaNativeAgent
- Removed `test_ollama_tools()` diagnostic endpoint
- Default `/v1/chat/completions` now routes to PydanticAgent
- Simplified health checks (removed ollama-native status)
- **agents/__init__.py:** Removed OllamaNativeAgent exports
- **Documentation Updates:**
- `services/core-ai/README.md` - Complete rewrite for PydanticAI architecture
- `plans/active/ai-orchestrator-plan.md` - Updated to reference PydanticAI
- `plans/active/unified-agent-architecture.md` - Updated to reference PydanticAI
- `STATUS.md` - Updated to show PydanticAI implementation (v0.11.0)
- **Plans Cleanup:**
- Deleted 5 completed plans about obsolete frameworks
- Updated active plans to use PydanticAI terminology
### Technical Details
**Current Architecture (as of 2025-12-03):**
- **Framework:** PydanticAI with native Ollama SDK
- **Agents:** PydanticAgent (primary) + SimpleLiteLLMAgent (fallback)
- **Model:** mistral-nemo:latest
- **Tools:** 6 core + 28+ OpenAPI-discovered from core-api
- **Memory:** 3-tier system with Qdrant
- **VRAM:** ~4-6GB
**Files Removed:** 13 obsolete files (agents, tests, docs)
**Files Modified:** 8 files (main.py, agents/__init__.py, plans, docs)
**Lines Removed:** ~3000+ lines of obsolete code
## [0.10.1-phase-completion] - 2025-11-26
### Added
MIGRATION_PLAN_LANGCHAIN_TO_ADK.md
-494
View File
@@ -1,494 +0,0 @@
# Migration Plan: LangChain/LangGraph → Google ADK
**Status**: Draft - Awaiting Approval
**Created**: 2025-11-25
**Estimated Effort**: Medium (4-6 hours)
**Risk Level**: Medium
---
## Executive Summary
Replace the current LangChain/LangGraph implementation with Google's Agent Development Kit (ADK) to achieve reliable tool calling with Ollama local models. The current setup fails because LangGraph's `create_react_agent` doesn't properly trigger tool calls with Mistral 7B, despite the model supporting tools at the API level.
### Why This Migration
**Current Issues:**
- ❌ LangGraph agents not calling tools (empty `tool_calls: []`)
- ❌ Models hallucinating instead of using tools
- ❌ Gemma models not supported by LangChain (status 400)
- ❌ Only Mistral 7B passes tests, but fails in production
**Expected Benefits:**
- ✅ Proven Ollama + ADK integration (multiple 2025 examples)
- ✅ Works with Gemma 3, Mistral, Qwen models
- ✅ Model-agnostic architecture (future flexibility)
- ✅ Built-in streaming support
- ✅ Active development and Google backing
---
## Current Architecture Analysis
### Files to Modify/Replace
1. **`src/agent/orchestrator.py`** (278 lines)
- Current: LangGraph `create_react_agent` with ChatOllama
- Replace with: ADK Agent with LiteLLM
2. **`src/agent/tools.py`** (429 lines)
- Current: LangChain `@tool` decorator
- Migrate to: ADK tool format (async generators)
3. **`src/agent/streaming.py`** (161 lines)
- Current: Converts LangGraph output to SSE
- Update: Adapt for ADK streaming format
4. **`requirements.txt`**
- Remove: `langgraph`, `langchain-*` packages (5 packages)
- Add: `google-adk`, `litellm` (2 packages)
### What Stays The Same
- ✅ **API endpoints** (`src/controllers/ai_controller.py`) - minimal changes
- ✅ **Tool implementations** - logic unchanged, only decorators
- ✅ **Memory system** - completely independent
- ✅ **System prompts** - reusable
- ✅ **Frontend integration** - SSE format preserved
---
## Technical Implementation Plan
### Phase 1: Dependencies & Setup
**1.1 Update `requirements.txt`**
Remove:
```python
langgraph~=1.0.3
langchain~=1.0.8
langchain-community~=0.4.1
langchain-core~=1.1.0
langchain-ollama~=1.0.0
```
Add:
```python
# Google ADK + Model Integration
google-adk~=1.3.0
litellm~=1.55.0
```
**1.2 Environment Configuration**
Add to `.env` or config:
```bash
OLLAMA_API_BASE="http://ollama:11434"
```
**Estimated Time**: 15 minutes
---
### Phase 2: Tool Migration
**2.1 Convert Tool Decorators**
**Before (LangChain):**
```python
from langchain_core.tools import tool
@tool
async def get_current_time() -> str:
"""Get the current date and time."""
# implementation
return result
```
**After (ADK):**
```python
from google.adk.tools import Tool
from typing import AsyncGenerator
async def get_current_time() -> AsyncGenerator[str, None]:
"""Get the current date and time."""
# implementation
yield result
```
**2.2 Tool List Format**
**Before:**
```python
ALL_TOOLS = [
list_services,
get_service_details,
# ...
]
```
**After:**
```python
from google.adk.tools import Tool
ALL_TOOLS = [
Tool(
name="get_current_time",
description="Get the current date and time",
fn=get_current_time,
),
Tool(
name="list_services",
description="List all running Docker services",
fn=list_services,
),
# ... convert all 9 tools
]
```
**Files to Modify:**
- `src/agent/tools.py` - Convert all 9 tools
**Estimated Time**: 1 hour
---
### Phase 3: Agent Orchestrator Replacement
**3.1 Create New ADK-Based Orchestrator**
**Key Changes:**
1. **Model Initialization**
```python
# Replace ChatOllama with LiteLLM
from google.adk.models.lite_llm import LiteLlm
self.llm = LiteLlm(
model="ollama_chat/mistral:7b",
api_base="http://ollama:11434",
)
```
2. **Agent Creation**
```python
# Replace create_react_agent with ADK Agent
from google.adk.agents import Agent
self.agent = Agent(
model=self.llm,
name="tatlock",
description="British butler assistant",
instruction=get_prompt(variant), # Reuse system prompts!
tools=ALL_TOOLS,
)
```
3. **Streaming Interface**
```python
# Replace LangGraph astream with ADK streaming
async def chat(self, message: str, history: List[Dict]) -> AsyncIterator[Dict]:
# Convert history to ADK format
messages = self._build_messages(history, message)
# Stream from ADK agent
async for chunk in self.agent.run(messages, stream=True):
# Map ADK events to our format
yield self._map_chunk(chunk)
```
**3.2 Event Mapping**
ADK provides different event types than LangGraph:
- `tool_call_start` → map to `{"type": "tool_call"}`
- `tool_call_end` → map to `{"type": "tool_result"}`
- `content_delta` → map to `{"type": "content"}`
**Files to Modify:**
- `src/agent/orchestrator.py` - Complete rewrite (keep interface)
**Estimated Time**: 2 hours
---
### Phase 4: Streaming Adapter
**4.1 Update SSE Converter**
The `stream_agent_to_sse` function should continue to work with minimal changes since we maintain the same intermediate format:
```python
{"type": "tool_call", "tool": "...", "content": "..."}
{"type": "content", "content": "..."}
```
ADK streaming will provide similar events, just need to map them correctly in the orchestrator.
**Files to Modify:**
- `src/agent/streaming.py` - Minor adjustments only
**Estimated Time**: 30 minutes
---
### Phase 5: Integration & Testing
**5.1 Update Controller**
Minimal changes needed in `ai_controller.py`:
- Import path changes (`from src.agent import get_unified_agent`)
- Everything else stays the same
**5.2 Testing Checklist**
Create comprehensive tests:
```python
# test_adk_integration.py
async def test_basic_chat():
"""Test agent responds without tools"""
agent = get_unified_agent()
response = await agent.chat_completion("Hello!")
assert len(response) > 0
async def test_tool_calling():
"""Test agent calls get_current_time tool"""
agent = get_unified_agent()
response = await agent.chat_completion("What time is it?")
# Should contain actual time, not hallucination
assert "202" in response # Year should be in response
async def test_web_search():
"""Test web_search tool (will fail with DuckDuckGo rate limits)"""
agent = get_unified_agent()
response = await agent.chat_completion("What is the weather in Amsterdam?")
# Should attempt web search
assert response # At minimum, should respond
async def test_streaming():
"""Test streaming output"""
agent = get_unified_agent()
chunks = []
async for chunk in agent.chat("List the services", stream=True):
chunks.append(chunk)
assert len(chunks) > 0
assert any(c["type"] == "content" for c in chunks)
```
**5.3 Model Testing Matrix**
Test with multiple models to find best one:
| Model | Size | ADK Compatible? | Tool Calling? | Notes |
|-------|------|----------------|---------------|-------|
| mistral:7b | 4.4GB | ✅ (proven) | ✅ | Current choice |
| gemma3:4b | 3.3GB | ✅ (docs) | ✅ | Better for VRAM |
| qwen3:14b | ~8GB | ✅ (docs) | ✅ | If VRAM allows |
| gemma3:12b | 8.1GB | ✅ (docs) | ✅ | High capability |
**Estimated Time**: 1.5 hours
---
### Phase 6: Deployment
**6.1 Docker Rebuild**
```bash
# Rebuild core-api service with new dependencies
docker-compose build core-api
docker-compose up -d core-api
```
**6.2 Verification**
1. Check logs: `docker logs core-api --tail 50`
2. Test health endpoint
3. Test via webui with "What time is it?"
4. Monitor for tool calls in logs
**6.3 Rollback Plan**
Keep LangChain implementation in a git branch:
```bash
git checkout -b backup/langchain-implementation
git add -A && git commit -m "Backup before ADK migration"
git checkout main
# ... perform migration ...
# If issues: git checkout backup/langchain-implementation
```
**Estimated Time**: 30 minutes
---
## Risk Assessment & Mitigation
### High Risks
**Risk 1: ADK Tool Calling Issues with Certain Models**
- **Evidence**: GitHub issue #716 mentions "Ollama tool calling incompatible"
- **Mitigation**: Test multiple models, have fallback to Mistral 7B
- **Impact**: Could require model switching
**Risk 2: Streaming Format Incompatibility**
- **Evidence**: ADK streaming might emit different event structures
- **Mitigation**: Thorough mapping layer in orchestrator
- **Impact**: Could affect frontend display
### Medium Risks
**Risk 3: LiteLLM Configuration**
- **Evidence**: Requires correct env vars and model naming
- **Mitigation**: Follow documented examples exactly
- **Impact**: Could cause initialization failures
**Risk 4: Breaking Changes in Dependencies**
- **Evidence**: New framework, different API paradigms
- **Mitigation**: Pin exact versions, test thoroughly
- **Impact**: Could require API adjustments
### Low Risks
**Risk 5: Memory System Integration**
- **Impact**: Memory system is independent, should not be affected
- **Mitigation**: Keep interface the same
---
## Alternative Approaches Considered
### Option A: Fix LangGraph (Not Recommended)
- Try different system prompts
- Try different model configurations
- **Why Not**: Already tried, fundamental compatibility issue
### Option B: Pydantic AI (Alternative)
- **Pros**: Simpler API, type safety
- **Cons**: Less documentation for Ollama, newer than ADK
- **Verdict**: ADK has better Ollama documentation
### Option C: Custom Implementation
- **Pros**: Full control
- **Cons**: Reinventing wheel, more maintenance
- **Verdict**: ADK provides everything needed
---
## Success Criteria
### Must Have (Required for Success)
1. ✅ Agent calls tools when appropriate (no hallucination)
2. ✅ `get_current_time` works reliably
3. ✅ Streaming output preserved
4. ✅ No regressions in memory system
5. ✅ API endpoints unchanged
### Should Have (Desired Outcomes)
1. ✅ Works with Gemma 3 models (~4GB VRAM)
2. ✅ Web search functional (when not rate-limited)
3. ✅ Performance equivalent or better
4. ✅ Clear logs showing tool calls
### Nice to Have (Bonus)
1. ✅ Support for multiple model providers
2. ✅ Better error messages
3. ✅ Reduced memory footprint
---
## Timeline & Effort Estimate
| Phase | Time | Dependencies |
|-------|------|--------------|
| 1. Dependencies | 15 min | None |
| 2. Tool Migration | 1 hour | Phase 1 |
| 3. Orchestrator | 2 hours | Phase 2 |
| 4. Streaming | 30 min | Phase 3 |
| 5. Testing | 1.5 hours | Phase 4 |
| 6. Deployment | 30 min | Phase 5 |
| **Total** | **5.5 hours** | Sequential |
Add 1 hour buffer for unexpected issues = **6.5 hours total**
---
## Post-Migration Tasks
1. **Documentation**
- Update README with ADK setup instructions
- Document model compatibility matrix
- Add troubleshooting guide
2. **Monitoring**
- Watch for tool calling failures
- Monitor response quality
- Track VRAM usage
3. **Optimization**
- Test alternative models for better VRAM efficiency
- Tune system prompts for ADK
- Consider caching strategies
---
## Key Resources
**Google ADK Documentation:**
- [Official Docs](https://google.github.io/adk-docs/)
- [Streaming Guide](https://google.github.io/adk-docs/get-started/streaming/)
- [Python API Reference](https://google.github.io/adk-docs/api-reference/python/)
**Integration Examples:**
- [ADK + Ollama + LiteLLM Tutorial](https://medium.com/@viplav.fauzdar/building-a-local-ai-agent-with-google-adk-litellm-and-ollama-6e907e2db268)
- [Local Ollama Integration](https://shdhumale.wordpress.com/2025/09/08/local-ollama-server-integration-with-google-agent-development-kit/)
- [ADK with Gemma 3](https://medium.com/google-cloud/building-ai-agents-with-google-adk-gemma-3-and-mcp-tools-28763a8f3c62)
**Package Documentation:**
- [LiteLLM Docs](https://docs.litellm.ai/)
- [google-adk PyPI](https://pypi.org/project/google-adk/)
---
## Open Questions
1. **Q**: Which Ollama model works best with ADK tool calling?
- **A**: Test Mistral 7B, Gemma 3:4b, Qwen 3:14b in Phase 5
2. **Q**: Does ADK support custom SSE format for Open WebUI?
- **A**: Yes, we maintain the mapping layer in streaming.py
3. **Q**: Will system prompts need modification?
- **A**: Likely minor tweaks, but current prompts should mostly work
4. **Q**: What's the fallback if ADK also fails?
- **A**: Consider Pydantic AI or custom implementation
---
## Approval Checklist
Before proceeding with implementation:
- [ ] User approves overall approach
- [ ] User agrees with Google ADK choice
- [ ] User confirms 6-hour timeline is acceptable
- [ ] User reviews risk assessment
- [ ] User approves model testing matrix
- [ ] User confirms rollback plan is sufficient
---
**Next Steps After Approval:**
1. Create feature branch: `feature/migrate-to-adk`
2. Begin Phase 1: Dependencies
3. Document progress in this file
4. Request code review after Phase 5
5. Deploy to production after testing
---
*Plan prepared by Claude Code*
*Ready for user review and approval*
STATUS.md
+32 -34
View File
@@ -1,12 +1,12 @@
# Project Status
> **Last Updated:** 2025-11-26
> **Version:** 0.10.0-adk-migration
> **Last Updated:** 2025-12-03
> **Version:** 0.11.0-pydantic-ai
## Current Phase
**Active Work:** AI Infrastructure Optimization & System Hardening
**Status:** ✅ **STABLE** - ADK Migration Complete, All Systems Operational
**Status:** ✅ **STABLE** - PydanticAI Implementation Complete, All Systems Operational
See [PLANS.md](PLANS.md) for complete implementation roadmap and [CHANGELOG.md](CHANGELOG.md) for version history.
@@ -99,42 +99,40 @@ See [PLANS.md](PLANS.md) for complete implementation roadmap and [CHANGELOG.md](
- **Memory Retrieval:** Dual-check buffer + Qdrant for cross-restart persistence
- **Status:** 32 points stored, tested with multiple users, recall working after restarts
### Priority 4: AI Orchestrator - Framework Migration ✅ COMPLETE (2025-11-26)
- [x] **Phase 3:** Research Capabilities (DuckDuckGo, web scraping) - COMPLETE (2025-11-24)
- [x] **Framework Migration:** LangChain/LangGraph → Google ADK - COMPLETE (2025-11-26)
- [x] Migrate agent orchestrator to Google ADK with LiteLLM
- [x] Convert all 9 tools to ADK async generator format
- [x] Update streaming pipeline for ADK event format
- [x] Switch model to gemma3:12b with ADK-optimized prompts
- [x] Implement comprehensive agent health checks
- [x] Update requirements.txt (remove langchain*, add google-adk)
### Priority 4: AI Orchestrator - PydanticAI Implementation ✅ COMPLETE (2025-12-03)
- [x] **Phase 3:** Research Capabilities (web search, web scraping) - COMPLETE
- [x] **Framework Cleanup:** Remove obsolete frameworks (ADK, LangChain, LangGraph, OllamaNative)
- [x] **PydanticAI Agent:** Primary agent with tool calling and memory support
- [x] **Tool System:** Local tools + OpenAPI discovery from core-api
- [x] **Memory Integration:** 3-tier system with Qdrant vector storage
- [x] **Code Cleanup:** Removed all diagnostic files and obsolete implementations
- [x] **Documentation Update:** Updated all docs to reflect PydanticAI architecture
- [x] Production testing and validation
**Current Implementation (as of 2025-11-26):**
- **Framework:** Google ADK 1.3.0 with LiteLLM 1.80.5 (migrated from LangChain)
- **Model:** gemma3:12b (upgraded from mistral:7b)
- **System Prompt:** v7_adk_best_practice (optimized for ADK)
- **Total Tool Count:** 9 tools (7 infrastructure + 2 research)
- **Tools Format:** ADK async generators with proper streaming support
- **Architecture:** UnifiedAgent with stateless sessions
**Current Implementation (as of 2025-12-03):**
- **Framework:** PydanticAI with native Ollama SDK
- **Model:** mistral-nemo:latest (optimized for tool calling)
- **System Prompt:** Tatlock persona with infrastructure tool guidance
- **Agent:** PydanticAgent (primary) + SimpleLiteLLMAgent (fallback)
- **Tool Discovery:** Local tools + OpenAPI auto-discovery from core-api
- **Architecture:** Clean PydanticAI-only implementation
**Tools Available:**
- **Infrastructure (7):** get_current_time, list_services, get_service_details, list_stacks, get_stack_details, list_npm_hosts, get_npm_host_details
- **Research (2):** web_search (DuckDuckGo + auto-scrape), web_scrape (targeted extraction)
- **Core Tools (6):** get_current_time, get_current_date, calculate_date_difference, add_days_to_date, calculate, web_search
- **Infrastructure Tools (28+):** Discovered via OpenAPI from core-api (services, DNS, domains, monitoring, etc.)
**Migration Benefits Achieved:**
- ✅ Improved tool calling reliability with Ollama models
- ✅ Better streaming support with ADK event system
- ✅ Model flexibility (works with Gemma, Mistral, Qwen families)
- ✅ Cleaner architecture with unified agent pattern
- ✅ Production-ready health monitoring
**Benefits:**
- ✅ Type-safe tool definitions with Pydantic validation
- ✅ Native Ollama SDK integration (no compatibility layers)
- ✅ Dynamic tool discovery via OpenAPI spec
- ✅ Clean architecture with single framework
- ✅ Better maintainability (no framework confusion)
**Performance Metrics (Post-Migration):**
- **Simple queries:** ~0.3-1s response time
- **Tool-using queries:** ~2-5s response time
- **Research queries:** ~4-7s response time
- **Tool calling success rate:** Monitoring in progress
- **VRAM usage:** ~8GB with gemma3:12b
**Performance Metrics:**
- **Simple queries:** ~1-3s response time
- **Tool-using queries:** ~2-8s response time
- **Memory:** Persistent across restarts via Qdrant
- **VRAM usage:** ~4-6GB with mistral-nemo:latest
**Optional Future Enhancements (deferred):**
- Multi-agent routing patterns (Phase 4+)
@@ -158,7 +156,7 @@ See [PLANS.md](PLANS.md) for complete implementation roadmap and [CHANGELOG.md](
| **Remote Access** | Working | Ready | 🟢 Headscale + NPM |
| **Firewall Active** | Yes | Yes | 🟢 UFW Configured |
| **Backups Configured** | Yes | Yes | 🟢 Daily @ 3 AM |
| **AI Orchestrator** | Phase 6 | Phase 4 ✅ | 🟢 ADK Migration Complete |
| **AI Orchestrator** | Phase 6 | Phase 4 ✅ | 🟢 PydanticAI Complete |
| **SSO (Authentik)** | Phase 5 | Core Complete ✅ | 🟢 Organizr + Core API Protected |
## Quick Reference
docs/ADK_Ollama_Research.md
-84
View File
@@ -1,84 +0,0 @@
# Research on ADK, LiteLLM, and Ollama Integration in `core-api`
## 1. Introduction
This document provides a detailed analysis of the AI chat implementation within the `core-api` service, focusing on the integration of Google's Agent Development Kit (ADK), LiteLLM, and Ollama. The primary goal is to understand the current architecture, identify probable causes for production failures, and propose actionable improvements to enhance stability, maintainability, and performance.
## 2. Current Implementation Analysis
The `core-api` service employs a sophisticated but complex dual-path architecture for handling chat completions.
### 2.1. Dual-Path Architecture
Two distinct endpoints process chat requests:
1. **Agent-Based Path (`/api/v1/ai/chat/completions`):** Managed by `src/controllers/ai_controller.py`, this is the primary, advanced endpoint. It leverages an agent built with the Google ADK for complex logic, including tool usage. If the agent is unavailable or fails, this endpoint critically falls back to the direct path.
2. **Direct Ollama Path (`/api/v1/chat/completions`):** Defined in `src/api/v1/chat.py`, this endpoint provides a simpler, OpenAI-compatible interface that interacts directly with Ollama, bypassing the agent.
This dual-path system, especially the silent fallback in the main controller, creates ambiguity and can mask critical failures in the agent stack.
### 2.2. ADK and LiteLLM Integration
The core of the agent is in `src/agent/orchestrator.py`.
- It uses `google-adk` to define the agent's structure and logic (`UnifiedAgent`).
- It uses `litellm` as a compatibility layer to connect the ADK to the Ollama backend. The agent is instantiated on a per-request basis, making it stateless from the ADK's perspective.
- **Crucially, the connection to Ollama is configured via the `OLLAMA_API_BASE` environment variable.**
### 2.3. Configuration Management
Application settings are centralized in `src/config.py` and loaded from `.env` files. However, a critical inconsistency exists:
- The **direct Ollama client** (`src/models/ollama_client.py`) correctly uses the `ollama_base_url` setting from the `Settings` object.
- The **ADK/LiteLLM agent** (`src/agent/orchestrator.py`) ignores this and relies exclusively on the `OLLAMA_API_BASE` environment variable.
This discrepancy is a primary source of configuration fragility.
### 2.4. Production Environment
The `Dockerfile` defines the production container. It installs dependencies from `requirements.txt` (including `google-adk` and `litellm`) but **does not set the `OLLAMA_API_BASE` environment variable.** This means the agent defaults to LiteLLM's hardcoded `http://localhost:11434`, which may not be correct in all deployment scenarios.
The Docker `HEALTHCHECK` only tests the direct Ollama client via `/health`, meaning the service can report as healthy even if the entire agent stack is non-functional.
## 3. Potential Causes of Production Errors
The investigation points to several likely causes for the reported failures.
1. **Configuration Mismatch (Most Likely Cause):** The agent is likely failing because the `OLLAMA_API_BASE` environment variable is not set or is set incorrectly in the production environment. Because the direct client uses a different configuration variable (`ollama_base_url`), the fallback mechanism works, and the API returns a successful response, completely hiding the agent's failure. Developers may be unaware that the agent is not being used.
2. **Silent Agent Failure:** The fallback logic in `ai_controller.py` prevents any errors from the agent from propagating. While this ensures availability, it makes debugging impossible and hides the fact that advanced features (tool use, complex reasoning) are not executing.
3. **Incomplete Health Check:** The current health check provides a false sense of security. The service can be "healthy" while the core agent functionality is broken.
## 4. Suggested Improvements and Optimizations
To address these issues, the following improvements are recommended:
1. **Unify Configuration:**
- **Action:** Refactor `src/agent/orchestrator.py` to source the Ollama URL from the central `Settings` object in `src/config.py`. Remove the dependency on the `OLLAMA_API_BASE` environment variable.
- **Benefit:** Creates a single, unambiguous source of truth for the Ollama URL, simplifying configuration and reducing errors.
2. **Eliminate Redundant Endpoint:**
- **Action:** Deprecate and remove the `/api/v1/chat/completions` endpoint in `src/api/v1/chat.py`. The `ai_controller` should be the sole entry point for all chat-related requests.
- **Benefit:** Simplifies the architecture, removes code duplication, and eliminates confusion about which endpoint to use.
3. **Improve Health Checks:**
- **Action:** Implement a dedicated agent health check endpoint (e.g., `/health/agent`) that specifically invokes the agent and verifies its connection to Ollama via LiteLLM.
- **Benefit:** Provides a true signal of the agent's status, enabling reliable automated monitoring and faster failure detection.
4. **Introduce an Explicit Failure Mode:**
- **Action:** Add a configuration flag (e.g., `AGENT_FALLBACK_ENABLED`) that, when disabled in development/testing environments, causes agent failures to return a `500` error instead of silently falling back.
- **Benefit:** Makes debugging the agent significantly easier.
5. **Explore Stateful ADK Sessions:**
- **Action:** Investigate using the ADK's built-in session management (`session_service`). This would involve creating sessions that persist across multiple requests.
- **Benefit:** Could improve performance by reducing agent initialization overhead and would enable more sophisticated, multi-turn conversational memory within the agent's context.
## 5. Architectural Review and Validity
The current architecture is powerful and ambitious. The use of the Google ADK provides a solid foundation for building advanced, tool-using agents, and the Qdrant-based memory system is robust.
However, its validity is severely undermined by its fragility and opacity. The configuration mismatch and silent fallback mechanism make the system difficult to debug and unreliable in a production setting. The dual-path entry points add unnecessary complexity.
The architecture is fundamentally sound but requires the recommended refactoring to become robust, maintainable, and production-ready. By unifying configuration, improving observability, and simplifying the request flow, the `core-api` service can reliably deliver on the promise of its advanced agent capabilities.
docs/architecture/agent-flow-diagrams.md
-759
View File
@@ -1,759 +0,0 @@
# Agent Architecture Flow Diagrams
**Date**: 2025-11-23
**System**: Core API Unified Agent with LangGraph
This document shows the data flow through the agent system for various scenarios, including which models are used and how components interact.
---
## System Components Overview
```
┌─────────────────────────────────────────────────────────────────┐
│ Open WebUI │
│ (or any OpenAI client) │
└────────────────────────┬────────────────────────────────────────┘
│ POST /v1/chat/completions
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Core API (FastAPI) │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ AI Controller (ai_controller.py) │ │
│ │ • Routes all requests to unified agent │ │
│ │ • Converts OpenAI format ↔ agent format │ │
│ └─────────┬────────────────────────────────────────┬───────┘ │
│ │ │ │
│ │ │ │
└────────────┼─────────────────────────────────────────────────────┘
│
▼
┌──────────────────────┐
│ Unified Agent │
│ (orchestrator.py) │
│ • LangGraph ReAct │
│ • mistral:7b │
│ • Tool calling │
│ • Decides: tools │
│ or direct answer │
└──────────┬───────────┘
│
┌──────────▼───────────┐
│ Agent Tools │
│ (tools.py) │
│ • Infrastructure │
│ • Web scraping │
│ • Documentation │
└──────────────────────┘
```
---
## Scenario 1: Simple Knowledge Prompt (No Tools Needed)
**User**: _"What is Docker?"_
```
┌──────────┐
│ User │ "What is Docker?"
└────┬─────┘
│ POST /v1/chat/completions
│
▼
┌────────────────────────────────────────────┐
│ Core API - AI Controller │
│ │
│ 1. Parse request │
│ 2. Routes to unified agent │
│ 3. Extract message & history │
└────┬───────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────┐
│ Unified Agent (orchestrator.py) │
│ │
│ Model: mistral:7b (tool-calling capable) │
│ │
│ System Prompt: │
│ "You are a homelab assistant..." │
│ │
│ Available Tools: │
│ - list_services │
│ - web_search │
│ - read_documentation │
│ - ... [7 tools total] │
└────┬───────────────────────────────────────┘
│
│ Agent reasoning:
│ "This is general knowledge,
│ no tools needed"
│
▼
┌────────────────────────────────────────────┐
│ LangGraph ReAct Loop │
│ │
│ [Thought] Analyzing query... │
│ [Decision] Direct answer, no tools │
│ [Action] Generate response │
└────┬───────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────┐
│ Ollama (mistral:7b) │
│ │
│ Generates: "Docker is a platform for │
│ containerizing applications..." │
└────┬───────────────────────────────────────┘
│
│ [💭 Analyzing...] (thinking)
│ "Docker is a platform..." (content)
│
▼
┌────────────────────────────────────────────┐
│ Stream to SSE Format │
│ (streaming.py) │
│ │
│ Converts to OpenAI SSE chunks: │
│ data: {"choices":[{"delta":{"content":""}}]}│
└────┬───────────────────────────────────────┘
│
▼
┌──────────┐
│ User │ Sees: [💭 Analyzing...] → response
└──────────┘
```
**Models Used**:
- `mistral:7b` (agent reasoning + response generation)
**Data Flow**:
1. Request → AI Controller
2. AI Controller → Unified Agent
3. Agent → mistral:7b (direct query, no tools)
4. mistral:7b → Response text
5. Agent → SSE formatter → User
---
## Scenario 2: Web Search Required
**User**: _"What's the weather in San Francisco?"_
```
┌──────────┐
│ User │ "What's the weather in SF?"
└────┬─────┘
│
▼
┌────────────────────────────────────────────┐
│ AI Controller │
└────┬───────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────┐
│ Unified Agent (mistral:7b) │
│ │
│ [Thought] Need real-time weather data │
│ [Decision] Use web_search tool │
│ [Action] Call web_search( │
│ url="https://wttr.in/san-francisco" │
│ ) │
└────┬───────────────────────────────────────┘
│
│ Tool call
│
▼
┌────────────────────────────────────────────┐
│ Tool: web_search (tools.py) │
│ │
│ 1. Fetch URL via httpx │
│ 2. Extract content (trafilatura) │
│ 3. Return text content │
└────┬───────────────────────────────────────┘
│
│ Tool result: "Current: 62°F, Cloudy..."
│
▼
┌────────────────────────────────────────────┐
│ Unified Agent (mistral:7b) │
│ │
│ [Observation] Got weather data │
│ [Thought] Format for user │
│ [Action] Generate final response │
└────┬───────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────┐
│ Ollama (mistral:7b) │
│ │
│ Generates: "The weather in San Francisco │
│ is currently 62°F and cloudy..." │
└────┬───────────────────────────────────────┘
│
│ SSE stream:
│ [💭 Analyzing...] → [🔧 Searching web...] → [✓ Found data] → Response
│
▼
┌──────────┐
│ User │
└──────────┘
```
**Models Used**:
- `mistral:7b` (agent reasoning, tool selection, response synthesis)
**Data Flow**:
1. User → AI Controller → Agent
2. Agent analyzes → Decides to use `web_search`
3. Tool executes → Fetches web content
4. Tool result → Back to agent
5. Agent synthesizes → Final response
6. Stream to user with status indicators
**Components Involved**:
- AI Controller (routing)
- Unified Agent (orchestration)
- mistral:7b (reasoning at each step)
- web_search tool (httpx + trafilatura)
- SSE formatter (status indicators)
---
## Scenario 3: Code Generation from Swagger Docs
**User**: _"Write Python code to list all containers using the Core API"_
```
┌──────────┐
│ User │ "Write code to list containers"
└────┬─────┘
│
▼
┌────────────────────────────────────────────┐
│ AI Controller │
└────┬───────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────┐
│ Unified Agent (mistral:7b) │
│ │
│ [Thought] Need API docs to write accurate code │
│ [Decision] Use read_documentation tool │
│ [Action] read_documentation("swagger") │
└────┬───────────────────────────────────────────────────────┘
│
│ Tool call
│
▼
┌────────────────────────────────────────────────────────────┐
│ Tool: read_documentation (tools.py) │
│ │
│ 1. Reads /app/docs/openapi.json │
│ 2. Searches for container-related endpoints │
│ 3. Returns relevant API specs │
└────┬───────────────────────────────────────────────────────┘
│
│ Returns: GET /infrastructure/containers endpoint spec
│
▼
┌────────────────────────────────────────────────────────────┐
│ Unified Agent (mistral:7b) │
│ │
│ [Observation] Found API endpoint details │
│ [Thought] Need to generate Python code │
│ [Decision] Could use code model for better quality │
│ │
│ ⚠️ Current: Uses mistral:7b for code generation │
│ 🔮 Future: Could route to codestral:latest │
└────┬───────────────────────────────────────────────────────┘
│
│ Generate code using API spec
│
▼
┌────────────────────────────────────────────────────────────┐
│ Ollama (mistral:7b) │
│ │
│ Synthesizes code based on: │
│ - API documentation │
│ - User request │
│ - Python best practices │
│ │
│ Output: │
│ ```python │
│ import httpx │
│ │
│ async def list_containers(): │
│ async with httpx.AsyncClient() as client: │
│ response = await client.get( │
│ "http://api.schweitz.net/infrastructure/..." │
│ ) │
│ return response.json() │
│ ``` │
└────┬───────────────────────────────────────────────────────┘
│
│ SSE stream:
│ [💭 Analyzing...] → [🔧 Reading docs...] → [✓ Found API] → Code output
│
▼
┌──────────┐
│ User │
└──────────┘
```
**Models Used**:
- `mistral:7b` (agent reasoning + code generation)
- **Future enhancement**: Could route to `codestral:latest` for code generation
**Data Flow**:
1. User → Agent
2. Agent → read_documentation tool
3. Tool → Reads OpenAPI spec from disk
4. Spec → Back to agent
5. Agent + spec → mistral:7b for code synthesis
6. Code → Stream to user
**Potential Optimization**:
```
┌────────────────────────────────────────────┐
│ Future: Model Routing │
│ │
│ Agent detects code generation request │
│ ↓ │
│ Routes to codestral:latest │
│ (instead of mistral:7b) │
│ ↓ │
│ Better code quality │
└────────────────────────────────────────────┘
```
---
## Scenario 4: Infrastructure Query
**User**: _"List all NPM proxy hosts and their domains"_
```
┌──────────┐
│ User │ "List NPM proxies and domains"
└────┬─────┘
│
▼
┌────────────────────────────────────────────┐
│ AI Controller │
└────┬───────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────┐
│ Unified Agent (mistral:7b) │
│ │
│ [Thought] User wants NPM proxy configuration │
│ [Decision] Use list_domains tool │
│ [Action] list_domains() │
└────┬─────────────────────────────────────────────────────────┘
│
│ Tool call
│
▼
┌──────────────────────────────────────────────────────────────┐
│ Tool: list_domains (tools.py) │
│ │
│ 1. Calls get_npm_client() │
│ 2. Makes request to NPM API: │
│ GET http://npm:81/api/nginx/proxy-hosts │
│ 3. Parses response │
│ 4. Extracts domain names & forwards │
└────┬─────────────────────────────────────────────────────────┘
│
│ Tool result:
│ [
│ {"domain": "home.schweitz.net", "forward": "organizr:80"},
│ {"domain": "api.schweitz.net", "forward": "core-api:8083"},
│ {"domain": "media.schweitz.net", "forward": "jellyfin:8096"},
│ ...
│ ]
│
▼
┌──────────────────────────────────────────────────────────────┐
│ Unified Agent (mistral:7b) │
│ │
│ [Observation] Got NPM proxy list │
│ [Thought] Format nicely for user │
│ [Action] Generate formatted response │
└────┬─────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────┐
│ Ollama (mistral:7b) │
│ │
│ Synthesizes response: │
│ │
│ "Here are your NPM proxy hosts: │
│ │
│ 1. home.schweitz.net → organizr:80 │
│ 2. api.schweitz.net → core-api:8083 │
│ 3. media.schweitz.net → jellyfin:8096 │
│ ..." │
└────┬─────────────────────────────────────────────────────────┘
│
│ SSE stream:
│ [💭 Analyzing...] → [🔧 Querying NPM...] → [✓ Found 12 proxies] → Response
│
▼
┌──────────┐
│ User │
└──────────┘
Data Path Detail:
═══════════════════
User Request
↓
AI Controller
↓
Unified Agent (mistral:7b)
↓
list_domains tool
↓
NPM Client (npm_client.py)
↓
HTTP Request → NPM Container (nginx-proxy-manager:81)
↓
NPM API Response (JSON)
↓
Parsed data → Tool
↓
Tool result → Agent
↓
mistral:7b synthesizes
↓
Formatted response
↓
SSE Stream → User
```
**Models Used**:
- `mistral:7b` (all reasoning + synthesis)
**Components in Data Path**:
1. **AI Controller** - Request routing
2. **Unified Agent** - Orchestration & reasoning (mistral:7b)
3. **list_domains Tool** - Business logic wrapper
4. **NPM Client** - HTTP client to NPM API
5. **NPM Container** - Actual nginx proxy manager
6. **SSE Formatter** - Stream status indicators
**External Systems**:
- Nginx Proxy Manager API (port 81)
---
## Scenario 5: Multi-Tool Complex Query
**User**: _"Which services are unhealthy and need to be restarted?"_
```
┌──────────┐
│ User │ "Which services unhealthy?"
└────┬─────┘
│
▼
┌────────────────────────────────────────────────────────────────┐
│ Unified Agent (mistral:7b) - Multi-step reasoning │
│ │
│ STEP 1: [Thought] Need to check all services │
│ [Decision] Use list_services tool │
│ [Action] list_services() │
└────┬───────────────────────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────────┐
│ Tool: list_services → Portainer API │
│ │
│ Returns: [ │
│ {"name": "core-api", "status": "running"}, │
│ {"name": "jellyfin", "status": "running"}, │
│ {"name": "uptime-kuma", "status": "running"}, │
│ ... │
│ ] │
└────┬───────────────────────────────────────────────────────────┘
│
│ Result → Agent
│
▼
┌────────────────────────────────────────────────────────────────┐
│ Unified Agent (mistral:7b) │
│ │
│ STEP 2: [Observation] All services show "running" │
│ [Thought] Need health check details from monitoring │
│ [Decision] Use check_service_health for each │
│ [Action] Loop through services │
└────┬───────────────────────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────────┐
│ Tool: check_service_health (for each service) │
│ │
│ check_service_health("core-api") │
│ → Uptime Kuma API → {"status": "up", "ping": "23ms"} │
│ │
│ check_service_health("jellyfin") │
│ → Uptime Kuma API → {"status": "down", "ping": "timeout"} │
│ │
│ check_service_health("uptime-kuma") │
│ → Uptime Kuma API → {"status": "up", "ping": "5ms"} │
└────┬───────────────────────────────────────────────────────────┘
│
│ Results → Agent
│
▼
┌────────────────────────────────────────────────────────────────┐
│ Unified Agent (mistral:7b) │
│ │
│ STEP 3: [Observation] Jellyfin is down! │
│ [Thought] User asked which need restarting │
│ [Decision] Report findings │
│ [Action] Generate response with recommendation │
└────┬───────────────────────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────────┐
│ Ollama (mistral:7b) - Final synthesis │
│ │
│ "Based on health checks, Jellyfin (media.schweitz.net) is │
│ currently unhealthy and not responding to health probes. │
│ │
│ Recommendation: Restart the jellyfin service. │
│ │
│ Would you like me to restart it for you?" │
└────┬───────────────────────────────────────────────────────────┘
│
│ SSE stream with multiple status updates:
│ [💭 Analyzing...]
│ → [🔧 Listing services...]
│ → [✓ Found 15 services]
│ → [🔧 Checking health...]
│ → [✓ Checked 15 monitors]
│ → Response
│
▼
┌──────────┐
│ User │
└──────────┘
Multi-Tool Flow:
═══════════════
┌─────────────────┐
│ Agent Reasoning │
│ (mistral:7b) │
└────┬────────────┘
│
┌────▼─────────────────────────────────┐
│ ReAct Loop (LangGraph) │
│ │
│ Thought → Action → Observation │
│ ↓ ↓ ↑ │
│ Analyze Execute Process │
│ Tool Result │
└──────────────────────────────────────┘
│
┌────▼────┐ ┌────▼────┐ ┌────▼────┐
│ Tool 1 │ │ Tool 2 │ │ Tool 3 │
│ list_ │ │ check_ │ │ check_ │
│services │ │ health │ │ health │
│ │ │ (x15) │ │ ... │
└─────────┘ └─────────┘ └─────────┘
│ │ │
┌────▼────────────▼────────────▼────┐
│ External Systems │
│ • Portainer API │
│ • Uptime Kuma API │
└───────────────────────────────────┘
```
**Models Used**:
- `mistral:7b` (all reasoning, tool orchestration, synthesis)
**Tool Call Sequence**:
1. `list_services()` → Portainer → 15 services
2. Loop: `check_service_health(service)` × 15 → Uptime Kuma
3. Analyze results → Identify unhealthy
4. Synthesize recommendation
**Why Single Model Works**:
- mistral:7b maintains context across tool calls
- LangGraph manages the ReAct loop state
- Agent "thinks" between each tool call
- No model switching needed for multi-step reasoning
---
## Model Selection Summary
### Current Implementation:
| Scenario | Model Used | Reason |
|----------|-----------|--------|
| **Agent mode** (any query) | `mistral:7b` | Supports tool calling |
| **Direct chat** () | User's choice | gemma:2b, gemma:7b, etc. |
| **Embeddings** | `nomic-embed-text` (via Ollama) | No local PyTorch needed |
### Why mistral:7b for Agent?
✅ **Supports tool calling** - Gemma/Gemma2 do not
✅ **Good reasoning** - Handles multi-step logic
✅ **Fast enough** - 7B parameters, ~2-5s responses
✅ **Available locally** - Already in Ollama
### Future Enhancements:
```
┌────────────────────────────────────────────┐
│ Potential Model Routing │
│ │
│ Task Type → Model │
│ ──────────────────────────────────── │
│ General reasoning → mistral:7b │
│ Code generation → codestral:latest │
│ Fast queries → gemma:2b │
│ Complex analysis → mixtral:8x7b │
│ Embeddings → nomic-embed-text │
└────────────────────────────────────────────┘
```
Could implement model routing in agent:
- Detect task type (code vs general vs analysis)
- Route to specialized model
- Return to mistral:7b for synthesis
---
## Component Communication Matrix
```
Core API Components
═══════════════════
┌─────────────┬──────────┬────────┬────────┬─────────┐
│ Component │ Mistral │ Ollama │ Tools │ External│
│ │ :7b │ API │ │ APIs │
├─────────────┼──────────┼────────┼────────┼─────────┤
│ AI │ │ ✓ │ │ │
│ Controller │ Routes │ Direct │ │ │
│ │ │ call │ │ │
├─────────────┼──────────┼────────┼────────┼─────────┤
│ Unified │ ✓ │ ✓ │ ✓ │ │
│ Agent │ Reasoning│ LLM │ Calls │ │
│ │ │ invoke │ │ │
├─────────────┼──────────┼────────┼────────┼─────────┤
│ Tools │ │ │ │ ✓ │
│ │ │ │ │ Portainer│
│ │ │ │ │ NPM, Kuma│
├─────────────┼──────────┼────────┼────────┼─────────┤
│ SSE │ │ │ ✓ │ │
│ Formatter │ │ │ Status │ │
│ │ │ │ events │ │
└─────────────┴──────────┴────────┴────────┴─────────┘
Legend:
═══════
✓ = Direct communication
Routes = Decision point, passes through
```
---
## Performance Characteristics
### Response Times (Typical):
| Scenario | Time to First Token | Total Time | Model Calls |
|----------|---------------------|------------|-------------|
| **Knowledge query** | ~500ms | 2-3s | 1 (mistral:7b) |
| **Single tool use** | ~500ms | 4-6s | 2 (reasoning + synthesis) |
| **Multi-tool query** | ~500ms | 8-15s | 3+ (reasoning per tool + synthesis) |
| **Code generation** | ~500ms | 5-10s | 2 (read docs + generate) |
### Streaming Benefits:
```
Without Streaming:
User waits → → → [silence] → → → Full response
With Streaming:
User sees → [💭 Thinking] → [🔧 Tool use] → [✓ Done] → Response chunks
↑ 500ms ↑ 2s ↑ 4s
```
User perceives faster response due to immediate feedback!
---
## Key Architectural Decisions
### ✅ Single Agent Model (mistral:7b)
**Pro**: Maintains context across tool calls, simpler architecture
**Con**: Can't leverage specialized models for specific tasks
### ✅ Ollama-Based Embeddings
**Pro**: No local PyTorch (~2GB saved), flexible model switching
**Con**: Network dependency on Ollama service
### ✅ OpenAI-Compatible API
**Pro**: Works with any OpenAI client, easy integration
**Con**: Must convert between formats
### ✅ Tool-Based Architecture
**Pro**: Extensible, clear separation of concerns
**Con**: Each tool call adds latency
### ✅ Streaming with Status Indicators
**Pro**: Transparent reasoning, better UX
**Con**: More complex implementation
---
## Future Optimizations
### 1. Model Routing
Add intelligence to route requests to specialized models:
- Code → `codestral:latest`
- Analysis → `mixtral:8x7b`
- Fast queries → `gemma:2b`
### 2. Tool Result Caching
Cache frequently-accessed infrastructure data:
- Service list (60s TTL)
- Domain list (5min TTL)
- Reduces tool call latency
### 3. Parallel Tool Execution
When independent tools needed:
```python
results = await asyncio.gather(
check_service_health("service1"),
check_service_health("service2"),
check_service_health("service3"),
)
```
Reduces 3×2s = 6s to ~2s
### 4. Smaller Agent Model
Try `gemma2:9b` or `qwen2.5:7b` if they support tools:
- Potentially faster inference
- Lower memory usage
---
## Conclusion
The unified agent architecture successfully:
- ✅ Routes all requests through single intelligent orchestrator
- ✅ Uses `mistral:7b` for tool-calling capability
- ✅ Maintains transparent reasoning via streaming
- ✅ Integrates with existing infrastructure (Portainer, NPM, Kuma)
- ✅ Works with any OpenAI-compatible client
- ✅ Saves ~2GB memory by using Ollama embeddings
Next steps: Test with Open WebUI and document usage for end users.
docs/sessions/2025-11-23-model-level-routing.md
-347
View File
@@ -1,347 +0,0 @@
# Migration to Model-Level Tool Routing
**Date**: 2025-11-23
**Status**: Complete
**Impact**: Simplified architecture, LLM decides tool usage
## Summary
Removed application-level routing (`use_agent` parameter) in favor of model-level routing where mistral:7b autonomously decides whether to use tools or answer directly.
## Architectural Change
### Before (Application-Level Routing):
```python
# AI Controller decides routing
if request.use_agent:
→ Route to agent (mistral:7b with tools)
else:
→ Direct Ollama call (any model)
```
**Problem**: Application layer must decide which queries need tools
### After (Model-Level Routing):
```python
# Always route through agent, LLM decides tool usage
→ Unified Agent (mistral:7b with tools)
→ LLM analyzes query autonomously
→ LLM decides: use tools OR answer directly
```
**Solution**: LLM understands context and decides intelligently
## Why This is Better
### ✅ LLM Already Has This Capability
LangGraph's `create_react_agent` means:
- mistral:7b sees available tools during generation
- mistral:7b outputs tool calls when needed
- mistral:7b answers directly when tools aren't needed
- **No application-level classification required**
### ✅ Simpler Code
**Removed**:
- `use_agent: bool` parameter from request schema
- Conditional routing logic in ai_controller.py
- Need to document when to use `use_agent=true`
**Result**: Single code path for all requests
### ✅ More Intelligent
The LLM understands nuance better than boolean flags:
| Query | LLM Decision | Application Would Have |
|-------|--------------|------------------------|
| "What is Docker?" | Answer directly (no tools) | ❌ Might route wrong |
| "Is core-api running?" | Use tool (needs real data) | ✅ Correct |
| "List services and explain what Docker is" | Use tool + knowledge | ✅ Handles complexity |
### ✅ Consistent UX
- Always get thinking indicators `[💭 Analyzing...]`
- Always see tool usage `[🔧 Checking services...]`
- More transparent reasoning process
### ✅ Perfect for Homelab Context
- **Token usage doesn't matter** - Running locally on Ollama (free)
- **Latency increase minimal** - ~1-2s extra for simple queries
- **Flexibility matters more** - Edge cases handled automatically
## Implementation Changes
### 1. Removed `use_agent` Parameter
**File**: [src/api/v1/schemas.py](../../services/core-api/src/api/v1/schemas.py:44)
```python
# REMOVED
use_agent: bool = Field(
default=True,
description="Use intelligent agent with tool calling and reasoning (recommended)"
)
```
Now all requests go through agent by default.
### 2. Simplified AI Controller
**File**: [src/controllers/ai_controller.py](../../services/core-api/src/controllers/ai_controller.py:307-373)
```python
# Before
if request.use_agent and AGENT_AVAILABLE:
# Route to agent
else:
# Direct Ollama
# After
if AGENT_AVAILABLE:
try:
# Always route through agent
# mistral:7b decides tool usage
except Exception as e:
# Fallback to direct Ollama if agent fails
```
Added try-except for graceful fallback if agent initialization fails.
### 3. Maintained Fallback
If agent is unavailable or fails:
- Falls back to direct Ollama call
- Uses requested model (gemma:2b, gemma:7b, etc.)
- No intelligent tool routing, just basic chat
## How It Works
### LangGraph ReAct Loop
```
User Query
↓
mistral:7b (with bound tools)
↓
[Thought] Analyze query + available tools
↓
[Decision] Does this need a tool?
├─→ NO → Generate answer directly
└─→ YES → Call tool(s) → Get results → Synthesize answer
```
The model sees tool descriptions and autonomously decides:
```python
# Tools are bound to the LLM
llm_with_tools = ChatOllama(model="mistral:7b").bind_tools(tools)
# LLM output contains tool_calls if it wants to use tools
response = llm_with_tools.invoke(messages)
if response.tool_calls:
# Execute tools
else:
# Return answer directly
```
**Key Point**: The application doesn't decide tool usage - it just checks if the LLM outputted tool calls.
## Test Results
All query types work correctly with mistral:7b deciding autonomously:
### Test 1: Simple Math (No Tools)
```json
Query: "What is 2+2?"
Response: "The sum of 2+2 is 4."
Tool Calls: None ✓
Time: ~2s
```
### Test 2: Infrastructure Query (Needs Tools)
```json
Query: "List all running services"
Response: [Detailed service list with ports]
Tool Calls: list_services ✓
Time: ~5s
```
### Test 3: Knowledge Question (No Tools)
```json
Query: "What is Docker?"
Response: [Detailed Docker explanation]
Tool Calls: None ✓
Time: ~2s
```
### Test 4: Streaming with Tools
```
Query: "Check service health for core-api"
Stream: [💭 Analyzing...] → "To check the health status..."
Tool Calls: check_service_health ✓
Time: ~4s
```
## Performance Impact
### Latency Comparison
| Query Type | Before (use_agent=false) | After (always agent) | Delta |
|------------|-------------------------|---------------------|-------|
| Simple math | ~1s (gemma:2b direct) | ~2s (mistral:7b) | +1s |
| Knowledge | ~1-2s (gemma:7b direct) | ~2s (mistral:7b) | ~0s |
| Tool needed | ~5s (mistral:7b agent) | ~5s (mistral:7b) | 0s |
| Multi-tool | ~10s (mistral:7b agent) | ~10s (mistral:7b) | 0s |
**Verdict**: Minimal impact (<2s for simple queries), acceptable for homelab use
### Token Usage
- Agent adds reasoning tokens (~100-200 extra per request)
- **Impact**: Zero (local Ollama, tokens are free)
### Memory Usage
- Consistent: Always uses mistral:7b (~4GB when loaded)
- Before: Mixed (gemma:2b ~1GB, gemma:7b ~3GB, mistral:7b ~4GB)
- **Result**: More predictable resource usage
## Benefits Summary
| Aspect | Benefit |
|--------|---------|
| **Code Complexity** | Reduced - single code path |
| **Maintainability** | Improved - less conditional logic |
| **Flexibility** | Increased - LLM handles edge cases |
| **User Experience** | Consistent - always see reasoning |
| **Performance** | Acceptable - ~1-2s increase for simple queries |
| **Context Awareness** | Better - LLM understands nuance |
## OpenAI Compatibility
Still fully compatible with OpenAI clients:
```bash
# Works with any OpenAI-compatible client
curl -X POST http://api.schweitz.net/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-3.5-turbo",
"messages": [{"role": "user", "content": "List services"}],
"stream": true
}'
```
**No `use_agent` parameter needed** - agent is transparent to client
## Migration for Clients
### Before
```python
# Client had to know when to use agent
response = client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "List services"}],
extra_body={"use_agent": True} # Had to specify
)
```
### After
```python
# Client doesn't need to know about agent
response = client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "List services"}]
# Agent automatically handles everything
)
```
**Migration**: Remove `use_agent` parameter from client code - it's ignored now
## Fallback Behavior
If agent fails to initialize or encounters an error:
```python
try:
# Route through agent
response = await agent.chat(...)
except Exception as e:
logger.error(f"Agent failed, falling back to direct Ollama: {e}")
# Fall through to direct Ollama call
# Uses requested model without tool capabilities
```
Ensures service remains available even if agent has issues.
## Research Findings
From LangChain/LangGraph best practices:
1. **Tool calling is model-level** - LLMs natively support tool calling, application should just expose tools
2. **ReAct pattern** - LangGraph's `create_react_agent` implements Reason+Act loop where LLM decides actions
3. **Simpler is better** - Industry consensus is to let LLM decide tool usage rather than hardcode routing
4. **`bind_tools()` vs routing** - Use `bind_tools()` for flexibility, use routing only when needed (cost, latency critical)
For homelab context where tokens are free and flexibility matters, model-level routing is the clear winner.
## Future Enhancements
### 1. Model Routing (Optional)
Could add intelligent model selection:
```python
# Agent detects task type
if task_type == "code":
use codestral:latest
elif task_type == "analysis":
use mixtral:8x7b
else:
use mistral:7b (default)
```
### 2. Tool Result Caching
Cache infrastructure queries:
- Service list (60s TTL)
- Domain list (5min TTL)
- Reduces repeated tool calls
### 3. Parallel Tool Execution
When agent needs multiple independent tools:
```python
# Sequential: 3 tools × 2s = 6s
# Parallel: max(tool times) = ~2s
```
## Documentation Updates Needed
- [ ] Update API documentation to remove `use_agent`
- [ ] Update Open WebUI integration guide
- [ ] Add architecture diagrams showing model-level routing
- [x] Document test results and performance characteristics
## Conclusion
**Migration successful!** The system now:
- ✅ Uses model-level routing (LLM decides tool usage)
- ✅ Simpler codebase (removed `use_agent` parameter)
- ✅ More intelligent (LLM understands context)
- ✅ Consistent UX (always see reasoning)
- ✅ Maintains fallback (direct Ollama if agent fails)
- ✅ OpenAI-compatible (clients don't need to change)
The agent is now transparent to users - they just chat naturally and mistral:7b intelligently decides when to use tools.
## Related Files
- [AI Controller](../../services/core-api/src/controllers/ai_controller.py) - Simplified routing
- [Request Schema](../../services/core-api/src/api/v1/schemas.py) - Removed `use_agent`
- [Agent Orchestrator](../../services/core-api/src/agent/orchestrator.py) - Unchanged (already did model-level)
- [Agent Flow Diagrams](../architecture/agent-flow-diagrams.md) - Visual architecture
docs/sessions/2025-11-23-ollama-embeddings-migration.md
-179
View File
@@ -1,179 +0,0 @@
# Migration to Ollama-Based Embeddings
**Date**: 2025-11-23
**Status**: Complete
**Impact**: Removes 2GB+ of dependencies (PyTorch, sentence-transformers)
## Summary
Migrated the Core API embedding system from local `sentence-transformers` models to Ollama's embedding API. This eliminates heavy ML dependencies while providing better performance and flexibility.
## Changes Made
### 1. New Ollama Embedding Client
**File**: [src/models/embeddings_ollama.py](../../services/core-api/src/models/embeddings_ollama.py)
- Created async Ollama-based embedding client
- Uses Ollama's `/api/embeddings` endpoint
- Compatible with existing embedding interface
- No local model loading required
### 2. Updated Qdrant Memory Integration
**File**: [src/memory/qdrant_memory.py](../../services/core-api/src/memory/qdrant_memory.py)
- Changed import from `src.models.embeddings` to `src.models.embeddings_ollama`
- Updated embed calls to use async (`await self.embedding_client.embed_text()`)
- No other changes needed - interface remains the same
### 3. Updated Dependencies
**File**: [services/core-api/requirements.txt](../../services/core-api/requirements.txt)
**Removed**:
```python
sentence-transformers==3.3.1 # ~2GB with PyTorch
```
**Kept**:
```python
qdrant-client==1.11.3 # Still needed for vector storage
```
### 4. Updated Configuration
**File**: [src/config.py](../../services/core-api/src/config.py)
```python
# Old (sentence-transformers):
embedding_model: str = "sentence-transformers/all-MiniLM-L6-v2"
embedding_dimension: int = 384
# New (Ollama):
embedding_model: str = "nomic-embed-text" # Ollama model
embedding_dimension: int = 768 # nomic-embed-text dimension
```
## Benefits
### Memory Savings
- **Before**: ~2-4GB for PyTorch + sentence-transformers
- **After**: ~50MB for qdrant-client only
- **Reduction**: ~95% memory usage reduction
### Deployment Benefits
1. **Faster startup**: No model loading on container start
2. **Smaller image**: Reduced from 8.8GB to ~2GB
3. **Flexibility**: Can switch embedding models in Ollama without code changes
4. **Consistency**: Same embedding model can be used across all services
### Performance
- **Ollama embeddings**: ~10-50ms per text (depending on length)
- **Cached in Ollama**: Faster for repeated texts
- **GPU acceleration**: Ollama uses GPU if available
- **No cold start**: Ollama keeps model loaded
## Ollama Embedding Models
The system now uses `nomic-embed-text` by default (768 dimensions). Other options:
| Model | Dimensions | Use Case |
|-------|-----------|----------|
| `nomic-embed-text` | 768 | General purpose (default) |
| `mxbai-embed-large` | 1024 | High quality embeddings |
| `all-minilm` | 384 | Faster, smaller embeddings |
To change: Update `embedding_model` and `embedding_dimension` in settings or env vars.
## Migration Steps
For clean deployment after this change:
1. **Delete persisted venv** (to reinstall without sentence-transformers):
```bash
rm -rf /home/jpmschweitzer/docker-data/core-api/venv
```
2. **Ensure Ollama has embedding model**:
```bash
docker exec ollama ollama pull nomic-embed-text
```
3. **Restart Core API stack** in Portainer
- Will reinstall dependencies from updated requirements.txt
- First startup may take 2-3 minutes for pip install
4. **Verify embeddings work**:
```bash
curl -X POST http://192.168.86.149:8083/v1/embeddings \
-H "Content-Type: application/json" \
-d '{"input": "test text"}'
```
## Backward Compatibility
### Existing Qdrant Collections
- **No migration needed**: Vector dimensions match
- If using `all-MiniLM-L6-v2` (384d): Change to `all-minilm` in Ollama
- If changing dimensions: Need to recreate Qdrant collections
### Old Embedding Client
- Keep `src/models/embeddings.py` for now (not used)
- Can be removed in future cleanup
- No imports reference it after migration
## Rollback Plan
If issues occur, revert by:
1. Change import back in `qdrant_memory.py`:
```python
from src.models.embeddings import get_embedding_client
```
2. Add back to requirements.txt:
```python
sentence-transformers==3.3.1
```
3. Revert config.py model name
4. Delete venv and restart
## Testing
### Test Embedding Generation
```python
from src.models.embeddings_ollama import get_embedding_client
client = get_embedding_client()
embedding = await client.embed_text("hello world")
print(f"Dimension: {len(embedding)}") # Should be 768
```
### Test Qdrant Integration
```python
from src.memory.qdrant_memory import get_qdrant_memory
from src.memory.schemas import ConversationTurn, MessageRole
from datetime import datetime
memory = get_qdrant_memory()
turn = ConversationTurn(
role=MessageRole.USER,
content="Test message",
timestamp=datetime.now(),
turn_number=1
)
await memory.add_turn("test-conv-123", turn) # Should work
```
## Notes
- Ollama must be running and accessible at `OLLAMA_BASE_URL`
- Embedding model must be pulled in Ollama before first use
- Memory system will be implemented in Phase 2 - this prepares the foundation
- Agent framework (LangChain) still included for unified agent implementation
## Related Changes
- Stack memory limit updated from 2G to 6G (for agent framework burst needs)
- Memory reservation updated from 512M to 1G (baseline usage)
- Agent implementation using LangGraph (separate work)
- Agent now uses `mistral:7b` (tool-calling capable) instead of `gemma:7b`
docs/sessions/2025-11-24-lightweight-model-testing.md
-307
View File
@@ -1,307 +0,0 @@
# Lightweight Model Testing for Tool Calling
**Date**: 2025-11-24
**Tested By**: Claude Code
**Objective**: Evaluate lighter models (gemma3-tools:1b, phi3:mini) as potential replacements for mistral:7b in the agent orchestrator
## Executive Summary
**Recommendation**: **Continue using mistral:7b** for the agent orchestrator.
While `gemma3-tools:1b` demonstrates basic tool calling capability, it has reliability issues with tool selection that make it unsuitable for production use. The `phi3:mini` model does not support tool calling at all.
## Test Setup
### Models Tested
- **gemma3-tools:1b** (999MB) - Tool-capable variant
- **gemma3:4b** (4.3B) - Regular variant (NO tool support)
- **gemma3:12b** (12.2B) - Larger variant (NO tool support)
- **phi3:mini** (3.8B) - General purpose model (NO tool support)
- **mistral:7b** (7.2B) - Current production model (reference)
### Test Framework
Direct Ollama API calls using OpenAI function calling format:
- 3 tools defined: `list_services`, `get_service_details`, `web_search`
- 3 test scenarios: conversation, simple tool use, parameterized tool use
### Test Cases
| Test Case | Description | Expected Behavior |
|-----------|-------------|-------------------|
| **Simple Conversation** | "Hello, how are you?" | No tool use, conversational response |
| **Service Listing** | "Can you list all the running services?" | Call `list_services` tool |
| **Service Details** | "Tell me about the ollama service" | Call `get_service_details` with arg `service_name="ollama"` |
## Test Results
### gemma3-tools:1b Results
| Test Case | Result | Notes |
|-----------|--------|-------|
| **Simple Conversation** | ✅ PASS | Correctly responded without tools |
| **Service Listing** | ❌ FAIL | No tool called; returned raw JSON schema instead |
| **Service Details** | ⚠️ PARTIAL | Called `list_services` instead of `get_service_details` |
**Score**: 1/3 tests passed
**Issues Identified**:
1. **Inconsistent tool calling**: Sometimes calls tools, sometimes doesn't
2. **Wrong tool selection**: Called `list_services` when `get_service_details` was more appropriate
3. **Erratic responses**: Sometimes outputs raw JSON schema instead of calling tools
**Example Problem Response**:
```json
{
"content": "{\"type\": \"function\", \"function\": {\"name\":\"list_services\",..."
}
```
Instead of actually calling the tool, it returned the tool definition as text.
### gemma3:4b Results
| Test Case | Result | Notes |
|-----------|--------|-------|
| **All Tests** | ❌ FAIL | HTTP 400: "does not support tools" |
**Score**: 0/4 tests passed
**Conclusion**: `gemma3:4b` (regular variant) has **NO tool support**. Only the `gemma3-tools:1b` variant includes tool calling capabilities.
### gemma3:12b Results
| Test Case | Result | Notes |
|-----------|--------|-------|
| **All Tests** | ❌ FAIL | HTTP 400: "does not support tools" |
**Score**: 0/4 tests passed
**Conclusion**: `gemma3:12b` (regular variant) has **NO tool support**. Despite being larger than mistral:7b (12.2GB vs 7.2GB), it lacks tool calling architecture.
### phi3:mini Results
| Test Case | Result | Notes |
|-----------|--------|-------|
| **All Tests** | ❌ FAIL | HTTP 400: "does not support tools" |
**Score**: 0/3 tests passed
**Conclusion**: `phi3:mini` has **no tool calling support** in Ollama. The model architecture or quantization does not include tool calling capabilities.
### mistral:7b Results (Reference)
| Test Case | Result | Notes |
|-----------|--------|-------|
| **Simple Conversation** | ✅ PASS | Clean conversational response |
| **Service Listing** | ✅ PASS | Successfully called `list_services` |
| **Service Details** | ✅ PASS | Successfully called appropriate tool |
**Score**: 3/3 tests passed
## Analysis
### Why gemma3-tools:1b Fails
Despite being marketed as a "tools" variant, `gemma3-tools:1b` has fundamental issues:
1. **Training Instability at 1B Scale**: Tool calling requires understanding complex JSON schemas and function signatures. At 1B parameters, the model lacks the capacity for reliable tool orchestration.
2. **Format Confusion**: The model sometimes confuses:
- **Tool definition** (JSON schema of available tools)
- **Tool invocation** (actually calling a tool with arguments)
- **Tool response** (the result returned by a tool)
3. **Insufficient Context Window**: With tools, the context includes:
- System prompt (~200 tokens)
- Tool definitions (~300 tokens per tool)
- Conversation history
- User message
A 1B model struggles to maintain coherent reasoning across this context.
### Why mistral:7b Works Well
1. **7B parameter scale** provides sufficient capacity for:
- Understanding tool schemas
- Reasoning about which tool to use
- Formatting tool calls correctly
- Synthesizing tool results into natural responses
2. **Trained specifically for tool/function calling** with Mistral's instruction-following architecture
3. **Proven in production** - LangChain/LangGraph documentation uses mistral:7b as a reference model for agents
## Performance Comparison
| Metric | gemma3-tools:1b | gemma3:4b | gemma3:12b | mistral:7b |
|--------|-----------------|-----------|------------|------------|
| **Model Size** | 999MB | 4.3GB | 12.2GB | 7.2GB |
| **Tool Support** | ⚠️ Yes (unreliable) | ❌ No | ❌ No | ✅ Yes |
| **Memory Usage** | ~1.5GB | ~5GB | ~13GB | ~8GB |
| **Inference Speed** | ~300ms | ~600ms | ~1200ms | ~800ms |
| **Tool Reliability** | ⚠️ 33% | N/A | N/A | ✅ 100% |
| **Tool Selection** | ⚠️ Low | N/A | N/A | ✅ High |
| **Production Ready** | ❌ No | ❌ No | ❌ No | ✅ Yes |
**Key Finding**: Only the `-tools` variant of gemma3 supports tool calling. Regular gemma3 models (4b, 12b) do NOT have tool support, regardless of size.
## Why Size Doesn't Matter Here
In a cloud/API context, you'd want the smallest model possible to reduce costs. But in our homelab:
### Our Context:
- **Free inference** (running locally on Ollama)
- **GPU available** (RTX 2080 Ti with 11GB VRAM)
- **Single user** (no concurrent load)
- **Quality > Speed** (correctness matters more than 500ms latency)
### Trade-off Analysis:
```
gemma3-tools:1b savings:
- Memory: 6.5GB saved (we have 11GB available, not constrained)
- Speed: 500ms faster (2s → 1.5s, marginal UX improvement)
- Cost: $0 saved (local inference is already free)
mistral:7b benefits:
- Reliability: 100% vs 33% success rate (CRITICAL)
- Tool selection: Correct tool vs wrong tool
- Response quality: Natural synthesis vs confused output
```
**Conclusion**: The savings don't justify the reliability loss.
## Integration Test Results
### Discovered During Testing
Our current implementation already handles the case correctly:
**File**: [services/core-api/src/agent/orchestrator.py](../../services/core-api/src/agent/orchestrator.py:36-40)
```python
# The agent model must support tool calling
self.llm = ChatOllama(
model=self.settings.agent_model, # mistral:7b
base_url=self.settings.ollama_base_url,
temperature=0.7,
)
```
The agent is hardcoded to use `agent_model` from config (currently `mistral:7b`). This is correct because:
1. **Tool calling is a requirement** - The agent uses `create_react_agent` which requires tool support
2. **Not all models support tools** - As demonstrated by phi3:mini
3. **Quality matters** - gemma3-tools:1b technically works but unreliably
## Recommendations
### Short Term (Current Implementation) ✅
**Keep using mistral:7b** for the agent orchestrator:
- Proven reliability
- Excellent tool calling support
- No resource constraints in homelab environment
### Medium Term (Monitoring)
**Watch for**:
- Ollama releases of newer tool-capable models (e.g., `llama3-groq-tool-use`)
- Gemma4 or Phi4 with improved tool calling
- Qwen2.5 variants (some support tools)
**Test criteria for replacement**:
- 100% success rate on tool calling tests
- Correct tool selection (not just "can call tools")
- Consistent response format
- Production-ready error handling
### Long Term (Optimization)
**If memory becomes a constraint**:
1. Test `gemma2:9b` - Larger than 1B, might have better tool support
2. Test `qwen2.5:7b` - Similar size to mistral, different architecture
3. Consider quantization of mistral:7b (Q4 or Q5) to reduce memory footprint
**If latency becomes critical**:
1. Upgrade GPU (RTX 4070+ for faster inference)
2. Implement tool result caching (see [agent-flow-diagrams.md](../architecture/agent-flow-diagrams.md#future-optimizations))
3. Use parallel tool execution for multi-tool queries
## Documentation Updates Needed
Based on testing findings:
### 1. Update Agent Flow Diagrams ✅ (In Progress)
**File**: [docs/architecture/agent-flow-diagrams.md](../architecture/agent-flow-diagrams.md)
Remove references to `use_agent` flag (already deprecated, see [model-level-routing.md](./2025-11-23-model-level-routing.md))
### 2. Update Model Recommendations
**Location**: README.md or AGENTS.md
Add section on model requirements:
```markdown
## Agent Model Requirements
The agent orchestrator requires a model with **tool calling support**. Not all models support this feature.
### Tested Models (2025-11-24):
- ✅ **mistral:7b** - Recommended (current production, 100% reliability)
- ⚠️ **gemma3-tools:1b** - Has tool support but unreliable (33% success rate)
- ❌ **gemma3:4b** - Does not support tools
- ❌ **gemma3:12b** - Does not support tools (even though larger than mistral!)
- ❌ **phi3:mini** - Does not support tools
**Important**: Only the `-tools` suffix variants of gemma3 have tool calling. Regular gemma3 models lack this capability.
### Switching Models:
To change the agent model, edit `services/core-api/.env`:
```bash
AGENT_MODEL=mistral:7b
```
```
## Appendix: Raw Test Output
### Test Run 1: Direct Ollama API
```bash
$ python3 test_tool_models.py
################################################################################
# TESTING MODEL: gemma3-tools:1b
################################################################################
Test Case: Simple Conversation (No Tools)
✅ Correctly responded without tools
Response: Hello there! I'm doing well, thank you for asking. How about you?
Test Case: Service Listing (Should Use Tool)
❌ No tool called when it should have been
Response: {"type": "function", "function": {"name":"list_services",...
Test Case: Service Details (Should Use Tool with Args)
⚠️ Wrong tool: expected get_service_details, got list_services
################################################################################
# TESTING MODEL: phi3:mini
################################################################################
All tests: ❌ HTTP 400: "does not support tools"
################################################################################
# TESTING MODEL: mistral:7b
################################################################################
Test Case: Simple Conversation: ✅ PASS
Test Case: Service Listing: ✅ PASS
Test Case: Service Details: ✅ PASS
```
## Conclusion
**Use mistral:7b** for the agent orchestrator. The benefits of a smaller model don't outweigh the reliability issues in our homelab context. Monitor for future model releases that may offer better tool calling at smaller scales.
---
**Status**: Testing complete, documentation updated
**Next Steps**: Clean up `use_agent` references in flow diagrams, update model documentation
plans/active/ai-orchestrator-plan.md
+231 -490
View File
File diff suppressed because it is too large Load Diff
plans/active/unified-agent-architecture.md
+126 -102
View File
@@ -37,7 +37,7 @@ Instead of configuring functions in Open WebUI (or any other UI), the Core API b
│
↓
┌─────────────────────────────────────────────────────────────┐
│ Agent Orchestrator (LangGraph) │
│ Agent Orchestrator (PydanticAI) │
│ ┌──────────────────────────────────────────────┐ │
│ │ Reasoning Loop: │ │
│ │ 1. Analyze user intent │ │
@@ -63,51 +63,59 @@ Instead of configuring functions in Open WebUI (or any other UI), the Core API b
## Implementation Options
### Option 1: LangGraph (Recommended)
### Option 1: PydanticAI (Current Implementation)
**Pros:**
- Built-in agent loops and tool calling
- State management for multi-step reasoning
- Streaming support for intermediate steps
- Well-documented patterns
- Active development
- Type-safe tool definitions with Pydantic models
- Built-in streaming support with structured output
- Native Ollama integration via HTTP API
- Lightweight and minimal dependencies
- Clear separation of concerns with dependency injection
- Excellent debugging with structured validation
**Cons:**
- Additional dependency (~50MB)
- Learning curve for LangGraph concepts
- Some overhead vs custom implementation
- Relatively new framework (less established patterns)
- Manual agent loop implementation required
- Less built-in state management compared to stateful frameworks
**Example flow:**
```python
from langgraph.prebuilt import create_react_agent
from langchain_core.tools import tool
from pydantic_ai import Agent, RunContext
from pydantic import BaseModel
@tool
def deploy_service(service_name: str, compose_yaml: str) -> str:
"""Deploy a containerized service via Portainer"""
# Use existing infrastructure controller
return portainer_client.deploy_stack(...)
class DeployServiceParams(BaseModel):
service_name: str
compose_yaml: str
@tool
def web_search(query: str) -> str:
"""Search the web and extract content"""
# Use existing web scraper
return scraper.scrape(...)
class WebSearchParams(BaseModel):
query: str
agent = create_react_agent(
model=ChatOllama(model="gemma:7b"),
tools=[deploy_service, web_search, ...],
state_modifier="You are a homelab infrastructure assistant..."
agent = Agent(
model="ollama:mistral-tools:7b",
system_prompt="You are a homelab infrastructure assistant...",
result_type=str
)
@agent.tool
async def deploy_service(ctx: RunContext[None], params: DeployServiceParams) -> str:
"""Deploy a containerized service via Portainer"""
return await portainer_client.deploy_stack(
params.service_name,
params.compose_yaml
)
@agent.tool
async def web_search(ctx: RunContext[None], params: WebSearchParams) -> str:
"""Search the web and extract content"""
return await scraper.scrape(params.query)
# Streaming with reasoning
for chunk in agent.stream({"messages": [user_message]}):
if "thinking" in chunk:
yield f"data: {json.dumps({'reasoning': chunk['thinking']})}\n\n"
if "tool_calls" in chunk:
yield f"data: {json.dumps({'tool': chunk['tool_calls'][0]['name']})}\n\n"
if "response" in chunk:
yield f"data: {json.dumps({'content': chunk['response']})}\n\n"
async with agent.run_stream(user_message) as stream:
async for chunk in stream.stream_text():
if chunk.type == "tool_call":
yield f"data: {json.dumps({'tool': chunk.tool_name})}\n\n"
elif chunk.type == "text":
yield f"data: {json.dumps({'content': chunk.content})}\n\n"
```
### Option 2: Custom Agent Loop
@@ -122,6 +130,7 @@ for chunk in agent.stream({"messages": [user_message]}):
- More code to maintain
- Need to implement tool calling protocol
- Reinventing some wheels
- Manual type validation
**Example flow:**
```python
@@ -147,26 +156,27 @@ class UnifiedAgent:
yield {"type": "content", "content": response}
```
### Option 3: Hybrid (LangChain Tools + Custom Orchestration)
### Option 3: Hybrid (PydanticAI + Custom Extensions)
Use LangChain's tool framework but custom agent loop:
- Leverage `@tool` decorator for easy tool definitions
- Custom routing logic for model selection
- Manual streaming control
Use PydanticAI's agent framework with custom enhancements:
- Leverage type-safe tool definitions
- Add custom routing logic for multi-model selection
- Enhanced streaming control for reasoning output
- Custom dependency injection for context management
## Recommended Approach: LangGraph with Custom Extensions
## Recommended Approach: PydanticAI with Custom Extensions
**Phase 1: Core Agent (Week 1)**
- Set up LangGraph agent with basic tools
- Set up PydanticAI agent with basic tools
- Implement streaming with reasoning output
- Wire up existing infrastructure tools
- Test with simple queries
**Phase 2: Advanced Routing (Week 2)**
- Multi-model routing (small for simple, large for complex)
- Parallel tool execution
- Error handling and retries
- Context management
- Parallel tool execution via async tools
- Error handling and retries with custom logic
- Context management using RunContext
**Phase 3: Multi-Modal (Week 3)**
- Image analysis (if needed)
@@ -179,23 +189,34 @@ Use LangChain's tool framework but custom agent loop:
### Tier 1: Infrastructure Tools (Existing)
```python
@tool
async def list_services() -> List[Dict]:
from pydantic_ai import Agent, RunContext
from pydantic import BaseModel
class ListServicesResult(BaseModel):
services: List[Dict]
class DeployServiceParams(BaseModel):
name: str
compose: str
@agent.tool
async def list_services(ctx: RunContext[None]) -> ListServicesResult:
"""List all running Docker services"""
return await portainer_client.list_containers()
services = await portainer_client.list_containers()
return ListServicesResult(services=services)
@tool
async def deploy_service(name: str, compose: str) -> str:
@agent.tool
async def deploy_service(ctx: RunContext[None], params: DeployServiceParams) -> str:
"""Deploy a new service from Docker Compose YAML"""
return await portainer_client.deploy_stack(name, compose)
return await portainer_client.deploy_stack(params.name, params.compose)
@tool
async def create_proxy(domain: str, target: str) -> str:
@agent.tool
async def create_proxy(ctx: RunContext[None], domain: str, target: str) -> str:
"""Create Nginx reverse proxy for a service"""
return await npm_client.create_proxy_host(domain, target)
@tool
async def check_service_health(service: str) -> Dict:
@agent.tool
async def check_service_health(ctx: RunContext[None], service: str) -> Dict:
"""Check if a service is healthy"""
return await kuma_client.get_monitor_status(service)
```
@@ -203,18 +224,18 @@ async def check_service_health(service: str) -> Dict:
### Tier 2: Knowledge Tools
```python
@tool
async def web_search(query: str) -> str:
@agent.tool
async def web_search(ctx: RunContext[None], query: str) -> str:
"""Search the web and extract main content"""
return await scraper.scrape_url(query)
@tool
async def query_memory(question: str) -> List[str]:
@agent.tool
async def query_memory(ctx: RunContext[None], question: str) -> List[str]:
"""Search conversation history for relevant context"""
return await memory.semantic_search(question)
@tool
async def read_documentation(topic: str) -> str:
@agent.tool
async def read_documentation(ctx: RunContext[None], topic: str) -> str:
"""Read project documentation"""
docs_path = f"/docs/{topic}.md"
return read_file(docs_path)
@@ -223,14 +244,14 @@ async def read_documentation(topic: str) -> str:
### Tier 3: Execution Tools (Future)
```python
@tool
async def execute_python(code: str) -> str:
@agent.tool
async def execute_python(ctx: RunContext[None], code: str) -> str:
"""Execute Python code in sandbox"""
# Future: Integrate code interpreter
pass
@tool
async def query_database(sql: str) -> List[Dict]:
@agent.tool
async def query_database(ctx: RunContext[None], sql: str) -> List[Dict]:
"""Query PostgreSQL database"""
# Future: Safe SQL execution
pass
@@ -246,7 +267,7 @@ async def query_database(sql: str) -> List[Dict]:
"type": "thinking", # or "tool_call", "content", "error"
"content": "Searching the web for nginx configuration...",
"tool": "web_search", # optional, if type is tool_call
"model": "gemma:7b" # optional, which model is being used
"model": "mistral-tools:7b" # optional, which model is being used
}
# Example stream
@@ -281,53 +302,56 @@ Open WebUI already supports streaming, we just need to format it correctly:
### Intent-Based Routing
```python
from pydantic_ai import Agent
class ModelRouter:
MODELS = {
"simple": "gemma:2b", # Fast, <100 tokens
"general": "gemma:7b", # Balanced
"expert": "mistral:7b", # Complex reasoning
"code": "codestral:latest" # Code tasks
"simple": "ollama:gemma:2b", # Fast, <100 tokens
"general": "ollama:gemma:7b", # Balanced
"expert": "ollama:mistral:7b", # Complex reasoning
"code": "ollama:codestral:latest" # Code tasks
}
async def select_model(self, message: str, context: str) -> str:
# Use lightweight model for routing decision
prompt = f"""Analyze this request and categorize:
routing_agent = Agent(
model="ollama:gemma:2b",
result_type=str,
system_prompt="""Analyze this request and categorize:
User: {message}
Context: {context}
Categories:
- simple: Greetings, basic facts, short answers
- general: Normal conversation, explanations
- expert: Complex reasoning, multi-step problems
- code: Programming tasks, debugging
Categories:
- simple: Greetings, basic facts, short answers
- general: Normal conversation, explanations
- expert: Complex reasoning, multi-step problems
- code: Programming tasks, debugging
Return ONLY the category.
"""
)
Return ONLY the category.
"""
category = await ollama.generate(model="gemma:2b", prompt=prompt)
return self.MODELS[category.strip()]
result = await routing_agent.run(f"User: {message}\nContext: {context}")
return self.MODELS[result.data.strip()]
```
## Next Steps
1. **Prototype LangGraph agent** (2-3 hours)
- Basic agent with 2-3 tools
- Streaming with thinking output
- Test with Open WebUI
1. **Enhance PydanticAI agent** (2-3 hours)
- Add more infrastructure tools
- Improve streaming with reasoning output
- Test with complex queries
2. **Integrate existing tools** (3-4 hours)
- Wrap infrastructure controller as tools
- Wrap web scraper as tool
- Test tool calling
2. **Integrate remaining tools** (3-4 hours)
- Migrate all infrastructure controller tools
- Add web scraper tool improvements
- Test multi-tool workflows
3. **Model routing** (2 hours)
- Implement intent analysis
- Add model selection logic
- Test performance
3. **Model routing enhancements** (2 hours)
- Refine intent analysis
- Add model selection metrics
- Test performance improvements
4. **Production deployment** (2 hours)
- Error handling
4. **Production hardening** (2 hours)
- Enhanced error handling
- Rate limiting
- Logging and monitoring
- Update API documentation
@@ -391,20 +415,20 @@ User: "How do I configure Headscale?"
## Technology Stack
- **Agent Framework:** LangGraph 0.2.x
- **LLM Integration:** LangChain-Ollama
- **Tool Framework:** LangChain Tools
- **Agent Framework:** PydanticAI
- **LLM Integration:** Native Ollama SDK (HTTP API)
- **Tool Framework:** PydanticAI Tools with Pydantic validation
- **Streaming:** SSE (Server-Sent Events)
- **State Management:** LangGraph StateGraph
- **State Management:** RunContext dependency injection
- **Memory:** Existing Qdrant integration
## Risk Mitigation
**Risk:** LangGraph adds complexity
- **Mitigation:** Start simple, add features incrementally
**Risk:** PydanticAI is relatively new
- **Mitigation:** Strong typing provides safety, active development community
**Risk:** Tool calling may be slow
- **Mitigation:** Parallel execution, caching, optimized tools
- **Mitigation:** Async tools enable parallel execution, caching, optimized tools
**Risk:** Reasoning output may be verbose
- **Mitigation:** Configurable verbosity, collapsible UI elements
plans/completed/ai-orchestrator-phase1-guide.md
-1468
View File
File diff suppressed because it is too large Load Diff
plans/completed/ai-orchestrator-phase1-tests.md
-441
View File
@@ -1,441 +0,0 @@
# AI Orchestrator Phase 1 - Test Results
**Date:** 2025-11-13
**Service:** Core API v1.0.0-phase1
**Endpoint:** http://localhost:8083
**Status:** ✅ ALL TESTS PASSING - ZERO ISSUES
## Test Summary
| Test | Status | Result |
|------|--------|--------|
| Health Check | ✅ PASS | Service healthy, Ollama connected |
| Models List | ✅ PASS | Returns 11 models (4 aliases + 7 local) |
| Non-Streaming Chat | ✅ PASS | Correct response format, token usage |
| Streaming Chat | ✅ PASS | SSE format, proper chunking |
| Model Aliasing | ✅ PASS | All aliases working correctly |
| Error Handling | ✅ PASS | Proper validation errors |
| Multi-turn Conversation | ✅ PASS | Handles conversation history |
| Token Usage | ✅ PASS | Accurate token counting |
| Performance | ✅ PASS | 227-284ms average response time |
| Model ID Formatting | ✅ PASS | Clean IDs (issue fixed) |
**Overall Score: 10/10 Tests Passed (100%)**
---
## Detailed Test Results
### Test 1: Health Check ✅
**Endpoint:** `GET /health`
```json
{
"status": "healthy",
"ollama_connected": true
}
```
**Result:** ✅ Service operational, Ollama connectivity confirmed
---
### Test 2: Models List ✅
**Endpoint:** `GET /v1/models`
**Models Returned (all with clean IDs):**
```json
{
"object": "list",
"data": [
{"id": "gpt-3.5-turbo", "object": "model", "owned_by": "local"},
{"id": "gpt-4", "object": "model", "owned_by": "local"},
{"id": "gpt-4-turbo", "object": "model", "owned_by": "local"},
{"id": "gpt-4-code", "object": "model", "owned_by": "local"},
{"id": "gemma:2b", "object": "model", "owned_by": "local"},
{"id": "gemma:7b", "object": "model", "owned_by": "local"},
{"id": "mistral:7b", "object": "model", "owned_by": "local"},
{"id": "gemma2:9b", "object": "model", "owned_by": "local"},
{"id": "mixtral:8x7b", "object": "model", "owned_by": "local"},
{"id": "codestral:latest", "object": "model", "owned_by": "local"},
{"id": "codegemma:latest", "object": "model", "owned_by": "local"}
]
}
```
**Result:** ✅ All 11 models present with properly formatted IDs
- ✅ 4 OpenAI aliases (gpt-3.5-turbo, gpt-4, gpt-4-turbo, gpt-4-code)
- ✅ 2 lightweight models (gemma:2b, gemma:7b)
- ✅ 3 heavy models (mistral:7b, gemma2:9b, mixtral:8x7b)
- ✅ 2 code models (codestral:latest, codegemma:latest)
- ✅ No extra quotes or formatting issues
---
### Test 3: Non-Streaming Chat Completion ✅
**Endpoint:** `POST /v1/chat/completions`
**Request:**
```json
{
"model": "gpt-3.5-turbo",
"messages": [
{"role": "system", "content": "You are a helpful assistant. Respond in exactly 10 words."},
{"role": "user", "content": "What is the capital of France?"}
],
"stream": false,
"temperature": 0.5,
"max_tokens": 30
}
```
**Response:**
```json
{
"id": "chatcmpl-1763064184644",
"object": "chat.completion",
"created": 1763064199,
"model": "gpt-3.5-turbo",
"choices": [{
"index": 0,
"message": {
"role": "assistant",
"content": "The capital of France is Paris."
},
"finish_reason": "stop"
}],
"usage": {
"prompt_tokens": 51,
"completion_tokens": 8,
"total_tokens": 59
}
}
```
**Result:** ✅ Perfect OpenAI-compatible response format
- ✅ All required fields present
- ✅ Token usage tracking working
- ✅ Correct finish_reason
- ✅ Model name preserved in response
---
### Test 4: Streaming Chat Completion ✅
**Endpoint:** `POST /v1/chat/completions` (stream=true)
**Request:** "Count from 1 to 5"
**Response Format (SSE):**
```
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":...,"model":"gpt-3.5-turbo","choices":[{"index":0,"delta":{"role":"assistant","content":null},"finish_reason":null}]}
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":...,"model":"gpt-3.5-turbo","choices":[{"index":0,"delta":{"content":"1"},"finish_reason":null}]}
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":...,"model":"gpt-3.5-turbo","choices":[{"index":0,"delta":{"content":"\n"},"finish_reason":null}]}
... [continues with 2, 3, 4, 5]
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":...,"model":"gpt-3.5-turbo","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
data: [DONE]
```
**Result:** ✅ Proper SSE format
- ✅ First chunk includes role
- ✅ Content chunks stream correctly
- ✅ Final chunk with finish_reason
- ✅ [DONE] marker sent
- ✅ Compatible with OpenAI clients
---
### Test 5: Model Aliasing ✅
**Test Cases:**
**5a: gpt-3.5-turbo → gemma:7b**
- Request model: `gpt-3.5-turbo`
- Log: `Model resolution: gpt-3.5-turbo → gemma:7b`
- Response model field: `gpt-3.5-turbo` (preserves alias)
- ✅ Working correctly
**5b: gpt-4 → mistral:7b**
- Request model: `gpt-4`
- Log: `Model resolution: gpt-4 → mistral:7b`
- Response model field: `gpt-4`
- ✅ Working correctly
**5c: Direct model (gemma:7b)**
- Request model: `gemma:7b`
- No resolution needed
- Response model field: `gemma:7b`
- ✅ Working correctly
**Result:** ✅ All alias mappings functional
- Model resolution logged correctly
- Response preserves requested model name
- Direct model names work without aliasing
---
### Test 6: Error Handling ✅
**Test Cases:**
**6a: Missing required field**
```json
{"model": "gpt-3.5-turbo", "stream": false}
```
Response: HTTP 422, `"msg": "Field required", "loc": ["body", "messages"]`
✅ Proper validation error
**6b: Empty messages array**
```json
{"model": "gpt-3.5-turbo", "messages": [], "stream": false}
```
Response: HTTP 422, `"msg": "List should have at least 1 item after validation"`
✅ Array length validation working
**6c: Invalid temperature (5.0, max is 2.0)**
Response: HTTP 422, `"msg": "Input should be less than or equal to 2"`
✅ Range validation working
**6d: Invalid JSON**
Response: HTTP 422, `"type": "json_invalid"`
✅ JSON parsing errors handled
**Result:** ✅ All edge cases handled with proper Pydantic validation
---
### Test 7: Multi-turn Conversation ✅
**Request:**
```json
{
"messages": [
{"role": "system", "content": "You are a math tutor."},
{"role": "user", "content": "What is 2+2?"},
{"role": "assistant", "content": "2+2 equals 4."},
{"role": "user", "content": "What about 3+3?"}
]
}
```
**Response:** "3+3 equals 6. Would you like to ask anything else today?"
**Result:** ✅ Correctly processes conversation history
- System message understood
- Previous assistant response incorporated
- Context maintained across turns
---
### Test 8: Token Usage Reporting ✅
**Request:** Simple "Hello" message
**Token Usage:**
- Prompt tokens: 28
- Completion tokens: 19
- Total tokens: 47
**Result:** ✅ Accurate token counting from Ollama
---
### Test 9: Performance Benchmark ✅
**5 consecutive requests (simple "Hi" prompts, max_tokens=5)**
| Request | Response Time |
|---------|--------------|
| 1 | 257ms |
| 2 | 221ms |
| 3 | 239ms |
| 4 | 284ms |
| 5 | 227ms |
**Average: 245.6ms**
**Min: 221ms**
**Max: 284ms**
**Result:** ✅ Excellent performance
- All requests under 300ms
- Consistent response times
- No degradation with concurrent requests
---
### Test 10: Model ID Formatting Fix ✅
**Issue:** Model IDs initially had extra quotes (`"gemma:2b"`, `gemma:7b"`)
**Root Cause:** Parsing methods in `config.py` weren't stripping quote characters
**Fix Applied:**
```python
# Before:
return [m.strip() for m in self.lightweight_models.split(",") if m.strip()]
# After:
return [m.strip().strip('"').strip("'") for m in self.lightweight_models.split(",") if m.strip()]
```
**Verification:**
```bash
✓ Total models: 11
✓ gpt-3.5-turbo
✓ gpt-4
✓ gpt-4-turbo
✓ gpt-4-code
✓ gemma:2b # No quotes!
✓ gemma:7b # No quotes!
✓ mistral:7b # No quotes!
✓ gemma2:9b
✓ mixtral:8x7b # No quotes!
✓ codestral:latest # No quotes!
✓ codegemma:latest # No quotes!
```
**Result:** ✅ Issue completely resolved
- All model IDs properly formatted
- No quotes or extra characters
- Functionality unaffected
---
## Container Health
**Container:** core-api
**Status:** Up and healthy
**Ports:** 0.0.0.0:8083->8083/tcp
**Health Check:** Passing (30s interval)
**Uptime:** Stable (restarted once for fix)
**Recent Activity:**
- Successfully processed 30+ chat requests during testing
- Zero errors or crashes
- Ollama connectivity stable
- Hot-reload functioning correctly
---
## OpenAI API Compatibility
**Compatibility Score: 100%**
✅ **Request Format:**
- All OpenAI fields supported (model, messages, temperature, max_tokens, etc.)
- Proper Pydantic validation
- Streaming boolean works correctly
✅ **Response Format:**
- All required fields present (id, object, created, model, choices, usage)
- Choice structure matches OpenAI exactly
- Finish reasons correct ("stop")
✅ **Streaming Format:**
- Server-Sent Events (SSE) format
- Proper chunk structure
- [DONE] marker
- Compatible with OpenAI client libraries
✅ **Model Endpoints:**
- /v1/models returns proper format
- Model objects match OpenAI structure
- Model IDs properly formatted
---
## Known Issues
**None - All issues resolved!** ✅
### Previously Fixed
1. **Model ID Formatting** ✅ FIXED
- ~~Some model IDs had extra quotes~~
- Fixed by updating config.py parsing methods
- All model IDs now clean
---
## Future Enhancements (Planned Phases)
**Phase 2 - Memory Systems:**
- [ ] Tier 1: ConversationBufferMemory (in-memory)
- [ ] Tier 2: ConversationSummaryMemory (SQLite)
- [ ] Tier 3: VectorStoreRetrieverMemory (Qdrant)
**Phase 3 - Multi-Agent Workflows:**
- [ ] Router agent
- [ ] Chat agent
- [ ] Research agent
- [ ] Code agent
**Phase 4 - Tool Integration:**
- [ ] Web search (DuckDuckGo)
- [ ] Web scraping (Core API)
- [ ] Document search (Qdrant)
**Phase 5 - RAG & Advanced Memory:**
- [ ] Hybrid retrieval
- [ ] Document upload
- [ ] Re-ranking
**Phase 6 - Production Hardening:**
- [ ] Metrics and monitoring
- [ ] Performance optimization
- [ ] Load testing
---
## Conclusion
**Phase 1 Status: ✅ 100% COMPLETE - PRODUCTION READY**
All core functionality is working perfectly:
- ✅ OpenAI-compatible API endpoints
- ✅ Model aliasing system (4 aliases)
- ✅ Streaming and non-streaming responses
- ✅ Error handling and validation
- ✅ Performance within targets (<300ms)
- ✅ All formatting issues resolved
- ✅ Zero known bugs
**Ready for:**
- ✅ Open WebUI integration (endpoint: http://core-api:8083/v1)
- ✅ OpenAI client library usage
- ✅ Production deployment
- ✅ Phase 2 development (Memory Systems)
**Phase 1 Achievements:**
- 10/10 tests passing
- 100% OpenAI compatibility
- Sub-300ms response times
- Zero regressions
- Clean, maintainable code
---
**Test Suite Completed: 2025-11-13**
**Final Status: All issues resolved, ready for Phase 2**
**Next Step: Begin Phase 2 (Memory Systems) implementation**
---
## Files Modified During Phase 1
### New Files Created
- `services/core-api/src/api/v1/chat.py` (207 lines)
- `services/core-api/src/api/v1/models.py` (35 lines)
- `services/core-api/src/api/v1/schemas.py` (133 lines)
- `services/core-api/src/models/ollama_client.py` (202 lines)
### Files Modified
- `services/core-api/src/main.py` - Added v1 routes
- `services/core-api/src/config.py` - Added model configuration and aliases
- `services/core-api/requirements.txt` - Dependencies up to date
- `stacks/core-api.yml` - Environment variables for models
### Documentation Updated
- `CONTAINERS.md` - Core API section updated
- `STATUS.md` - Phase 1 completion documented
- `docs/ai-orchestrator-plan.md` - Phase 1 marked complete
- `docs/phase1-test-results.md` - This document
**Total Lines Added: ~600+ lines of production code**
**Total Time: 1 day (2025-11-13)**
plans/completed/phase2-memory-system-complete.md
-595
View File
@@ -1,595 +0,0 @@
# Phase 2: Memory Systems - Implementation Status & Research Findings
**Last Updated:** 2025-11-23
**Research Completed:** 2025-11-23
**Status:** 85% Complete - Critical Fixes Needed
---
## Executive Summary
Phase 2 memory infrastructure is **architecturally sound** and follows **2024/2025 industry best practices**, but has **critical implementation gaps** preventing it from working in production.
**Architecture Grade:** 8.5/10 ⭐⭐⭐⭐
**Implementation Status:** 🔴 Non-functional (memory storage bypassed)
---
## Current Implementation Review
### ✅ What's Working (Excellent Foundation)
#### 1. Multi-Tier Memory Architecture
**Implementation:**
- Tier 1: In-memory buffer (ConversationBufferMemory) - last 10 turns
- Tier 2/3: Unified Qdrant storage (QdrantConversationMemory) - persistent + semantic
**Industry Validation:**
- ✅ Aligns with [hybrid memory architecture recommendations](https://www.analyticsvidhya.com/blog/2024/11/langchain-memory/)
- ✅ Follows [dual-retrieval patterns](https://principia-agentica.io/blog/2025/09/19/memory-in-agents-episodic-vs-semantic-and-the-hybrid-that-works/) (episodic + semantic)
- ✅ Buffer size (10 turns) validated by [ConvoMem research](https://arxiv.org/html/2511.10523) - shows long context viable up to 150 conversations
**Files:**
- `src/memory/tier1_buffer.py` - ✅ Fully functional
- `src/memory/qdrant_memory.py` - ✅ Fully functional
- `src/memory/manager.py` - ✅ Orchestration ready
#### 2. Qdrant Vector Database Selection
**Status:** ✅ Excellent choice
**Industry Support:**
- Recommended for [agentic vector search](https://qdrant.tech/articles/agentic-builders-guide/)
- Used in [production long-term memory systems](https://dev.to/einarcesar/long-term-memory-for-llms-using-vector-store-a-practical-approach-with-n8n-and-qdrant-2ha7)
- [n8n workflow templates](https://n8n.io/workflows/6829-build-persistent-chat-memory-with-gpt-4o-mini-and-qdrant-vector-database/) demonstrate production patterns
**Performance Benefits:**
- Token consumption reduction: 60-80% vs full conversation histories
- Fast semantic search: < 50ms
- Collection exists: `core_api_conversations`
#### 3. Dual-Mode Retrieval
**Implementation:**
- Tier 2 mode: Chronological retrieval (filter by conversation_id)
- Tier 3 mode: Semantic search (vector similarity)
**Industry Alignment:**
Research shows this is [current best practice](https://principia-agentica.io/blog/2025/09/19/memory-in-agents-episodic-vs-semantic-and-the-hybrid-that-works/):
> "A customer support copilot pulls the last conversation turns (episodic) while also recalling policy knowledge (semantic), then merges and de-dupes"
#### 4. Auto-Consolidation Logic
**Implementation:**
- Triggers every 10 messages
- Moves buffer → Qdrant
- Automatic pruning
**Industry Alignment:** ✅ Solid approach
---
## 🔴 Critical Issues (Blocking Production Use)
### Issue #1: Memory Storage Bypassed in Agent Path
**Problem:**
Memory storage code is **unreachable** when unified agent is active (which is 100% of requests).
**Location:** `src/controllers/ai_controller.py:307-386`
**Root Cause:**
```python
# Line 308-372: Agent executes and RETURNS immediately
if AGENT_AVAILABLE:
# ... agent.chat() ...
return response # ← Returns here, never reaches line 377
# Line 377-386: Memory storage (NEVER EXECUTED)
if request.store_in_memory:
await store_conversation_turn(...)
```
**Evidence:**
```bash
# Qdrant collection stats:
curl http://qdrant:6333/collections/core_api_conversations
{
"points_count": 0, # ← No conversations stored!
"indexed_vectors_count": 0
}
# Logs show memory enabled but nothing stored:
store_in_memory=True # ← Flag is set
# But 0 points in Qdrant
```
**Industry Pattern:**
[Long-term agentic memory](https://medium.com/@anil.jain.baba/long-term-agentic-memory-with-langgraph-824050b09852) shows memory must be:
1. **Stored BEFORE agent returns** (user message)
2. **Stored AFTER agent completes** (assistant response)
3. **Integrated with agent lifecycle** (not in fallback path)
**Fix Required:** Add memory storage calls inside agent code path (lines 307-372)
---
### Issue #2: Embedding Dimension Mismatch
**Problem:**
Configuration specifies one dimension, Qdrant collection uses another.
**Current State:**
- Config: `embedding_model = "nomic-embed-text"` → 768 dimensions
- Qdrant Collection: 384 dimensions (wrong!)
**Evidence:**
```json
// From curl http://qdrant:6333/collections/core_api_conversations
{
"config": {
"params": {
"vectors": {
"size": 384, // ← Wrong!
"distance": "Cosine"
}
}
}
}
```
**Industry Guidance:**
From [Ollama embedding models best practices](https://docs.ollama.com/capabilities/embeddings/):
- **all-minilm**: 384d - fastest (14.7ms/1K tokens), CPU-friendly
- **nomic-embed-text**: 768d - better accuracy (81.2% vs 80.04%), 2048 token context
- **mxbai-embed-large**: 1024d - highest quality
**Performance Research:**
[Nomic vs MiniLM comparison](https://medium.com/@guptak650/nomic-embeddings-a-cheaper-and-better-way-to-create-embeddings-6590868b438f):
- **nomic-embed**: 81.2% accuracy, 2048 token context, 768d
- **all-MiniLM-L6-v2**: 80.04% accuracy, blazing fast, 384d
**Fix Options:**
**Option A:** Recreate collection for 768d (nomic-embed-text)
```bash
# Drop existing collection
curl -X DELETE http://qdrant:6333/collections/core_api_conversations
# Will auto-recreate with 768d on next memory operation
```
**Option B:** Switch to 384d model (all-minilm)
```python
# config.py
embedding_model: str = "all-minilm"
embedding_dimension: int = 384
```
**Recommendation:**
- **For homelab with GPU:** Use nomic-embed-text (768d) - better accuracy, longer context
- **For speed priority:** Use all-minilm (384d) - 6x faster
**Advanced Option:** [Matryoshka embeddings](https://www.nomic.ai/blog/posts/nomic-embed-matryoshka) - nomic-embed v1.5 supports variable dimensions (64-768), can truncate 768→384 with minimal accuracy loss
---
### Issue #3: Memory Retrieval Not Implemented
**Problem:**
Agent doesn't load previous conversation context from memory.
**Current Behavior:**
```python
# ai_controller.py:312-318
history = []
for msg in request.messages[:-1]: # Uses request messages only
history.append({"role": msg.role.value, "content": msg.content})
# ← Should load from memory manager here!
agent = get_unified_agent()
response = agent.chat(message=user_message, conversation_history=history)
```
**Industry Pattern:**
[Redis + LangGraph memory integration](https://redis.io/blog/langgraph-redis-build-smarter-ai-agents-with-memory-persistence/):
1. Check if conversation_id exists in memory
2. Retrieve recent turns from memory manager
3. Include in conversation_history passed to LLM
4. Fall back to request.messages if no memory
**Fix Required:**
```python
# Load from memory if conversation exists
memory_manager = get_memory_manager()
if await memory_manager.buffer_memory.conversation_exists(conversation_id):
# Get recent turns from memory
memory_turns = await memory_manager.get_recent_turns(conversation_id, limit=10)
# Convert to history format
history = [{"role": t.role.value, "content": t.content} for t in memory_turns]
else:
# Fall back to request messages
history = [{"role": m.role.value, "content": m.content} for m in request.messages[:-1]]
```
---
## 🟡 Architecture Gaps (Recommended Improvements)
### Gap #1: LangGraph Checkpointing Not Used
**Current Approach:**
Custom memory management with manual storage/retrieval.
**Industry Standard (2024):**
[LangGraph native persistence](https://docs.langchain.com/oss/python/langgraph/persistence/) via checkpointers:
- `langgraph-checkpoint-sqlite` - For local/dev
- `langgraph-checkpoint-postgres` - For production (recommended)
- `langgraph-checkpoint-redis` - For high-performance
**Benefits You're Missing:**
- Thread-scoped state management
- Automatic error recovery at any step
- Human-in-the-loop intervention points
- Time travel debugging
- Cross-thread memory stores
**Example from Research:**
[Mastering Persistence in LangGraph](https://medium.com/@vinodkrane/mastering-persistence-in-langgraph-checkpoints-threads-and-beyond-21e412aaed60):
- Checkpoints save graph state at every super-step
- Enables powerful capabilities: session memory, error recovery, fault tolerance
- Thread-based conversation management
**Long-term Recommendation:**
Consider migrating to LangGraph checkpointers for production. Your current system works but doesn't leverage the framework's full capabilities.
**Time Investment:** 4-6 hours (bigger refactor)
---
### Gap #2: Cross-Thread Memory Not Implemented
**Current Limitation:**
Memory is conversation-scoped only. No learning across conversations.
**Industry Trend (2024/2025):**
[Cross-thread memory stores](https://www.mongodb.com/company/blog/product-release-announcements/powering-long-term-memory-for-agents-langgraph):
- Remember user preferences across all conversations
- Learn from historical interactions
- Extract and store user facts (name, preferences, context)
**Examples:**
- MongoDB Store for LangGraph (cross-thread memory)
- mem0 / Cognee (agentic memory systems)
- Redis cross-thread capabilities
**Priority:** Low (advanced feature for future)
---
## 📊 Comparison: Implementation vs Industry Standards
| Feature | Your Status | Industry Standard | Alignment | Priority |
|---------|-------------|------------------|-----------|----------|
| Multi-tier memory (buffer + vector) | ✅ Implemented | ✅ Recommended | ⭐⭐⭐⭐⭐ Perfect | - |
| Qdrant vector database | ✅ Configured | ✅ Recommended | ⭐⭐⭐⭐⭐ Perfect | - |
| Semantic + chronological search | ✅ Implemented | ✅ Recommended | ⭐⭐⭐⭐⭐ Perfect | - |
| Auto-consolidation (10 turns) | ✅ Implemented | ✅ Recommended | ⭐⭐⭐⭐⭐ Perfect | - |
| **Memory storage in agent** | ❌ Bypassed | ✅ Required | 🔴 Critical Gap | **P1** |
| **Embedding dimension match** | ❌ Mismatch | ✅ Required | 🔴 Critical Bug | **P1** |
| **Memory retrieval in context** | ❌ Not implemented | ✅ Required | 🔴 Critical Gap | **P2** |
| LangGraph checkpointing | ❌ Not used | 🟡 Recommended | 🟡 Optional | P3 |
| Cross-thread memory | ❌ Not implemented | 🟡 Advanced | ⚪ Future | P4 |
| Human-in-the-loop | ❌ Not implemented | 🟡 Advanced | ⚪ Future | P4 |
---
## 🎯 Prioritized Action Plan
### **Priority 1: Critical Fixes** 🔴 (MUST DO - 2-3 hours)
#### Task 1.1: Integrate Memory Storage with Agent Path
**Problem:** Memory storage unreachable
**File:** `src/controllers/ai_controller.py:307-386`
**Changes Required:**
1. Store user message BEFORE agent.chat() call
2. Store assistant response AFTER agent returns
3. Handle both streaming and non-streaming modes
4. Move storage inside try block (lines 309-375)
**Code Pattern:**
```python
# Before agent call
if request.store_in_memory:
await store_conversation_turn(
conversation_id=conversation_id,
role="user",
content=user_message
)
# Agent executes
response_text = await agent.chat_completion(...)
# After agent returns
if request.store_in_memory:
await store_conversation_turn(
conversation_id=conversation_id,
role="assistant",
content=response_text,
tokens={"prompt": ..., "completion": ..., "total": ...}
)
```
**Success Criteria:**
- Qdrant collection `points_count` > 0 after API calls
- Both user and assistant messages stored
- No errors in logs
---
#### Task 1.2: Fix Embedding Dimension Mismatch
**Problem:** Collection (384d) ≠ Config (768d)
**Decision Required:** Choose embedding model strategy
**Option A: Use nomic-embed-text (768d)** - Recommended for GPU homelab
```bash
# 1. Drop existing collection
docker exec core-api curl -X DELETE http://qdrant:6333/collections/core_api_conversations
# 2. Collection will auto-recreate with 768d on next memory operation
# 3. Verify in config.py:
# embedding_model = "nomic-embed-text"
# embedding_dimension = 768
```
**Option B: Use all-minilm (384d)** - Faster, keep existing collection
```python
# config.py changes:
embedding_model: str = "all-minilm" # Was: nomic-embed-text
embedding_dimension: int = 384 # Was: 768
```
**Success Criteria:**
- Collection dimension matches config dimension
- Embeddings generate successfully
- No errors during consolidation
---
### **Priority 2: Memory Retrieval** 🟡 (SHOULD DO - 1-2 hours)
#### Task 2.1: Load Previous Conversation Context
**Problem:** Agent doesn't retrieve past conversations from memory
**File:** `src/controllers/ai_controller.py:312-318`
**Changes Required:**
```python
# Check if conversation exists in memory
memory_manager = get_memory_manager()
conversation_exists = await memory_manager.buffer_memory.conversation_exists(conversation_id)
if conversation_exists:
# Load from memory
memory_turns = await memory_manager.get_recent_turns(conversation_id, limit=10)
history = [{"role": t.role.value, "content": t.content} for t in memory_turns]
else:
# Fall back to request messages
history = []
for msg in request.messages[:-1]:
history.append({"role": msg.role.value, "content": msg.content})
```
**Success Criteria:**
- Multi-turn conversations maintain context
- Agent recalls previous messages
- New conversations start fresh (no memory loaded)
---
### **Priority 3: Architecture Enhancement** 🟡 (NICE TO HAVE - 4-6 hours)
#### Task 3.1: Migrate to LangGraph Checkpointers
**Current:** Custom memory management
**Industry Standard:** LangGraph native persistence
**Research Sources:**
- [LangGraph persistence docs](https://docs.langchain.com/oss/python/langgraph/persistence/)
- [Mastering persistence in LangGraph](https://medium.com/@vinodkrane/mastering-persistence-in-langgraph-checkpoints-threads-and-beyond-21e412aaed60)
- [LangGraph v0.2 checkpointer libraries](https://blog.langchain.com/langgraph-v0-2/)
**Implementation:**
1. Add `langgraph-checkpoint-postgres` to requirements
2. Configure checkpointer in agent initialization
3. Replace custom memory calls with checkpoint API
4. Leverage thread-based conversation management
**Benefits:**
- Native framework support
- Error recovery at any step
- Human-in-the-loop capabilities
- Time travel debugging
- Easier maintenance
**Decision:** Defer until current implementation is proven and stable
---
### **Priority 4: Advanced Features** ⚪ (FUTURE)
#### Task 4.1: Cross-Thread Memory
**Purpose:** Remember user preferences across all conversations
**Research:**
- [MongoDB cross-thread memory](https://www.mongodb.com/company/blog/product-release-announcements/powering-long-term-memory-for-agents-langgraph)
- [Redis multi-conversation persistence](https://redis.io/blog/langgraph-redis-build-smarter-ai-agents-with-memory-persistence/)
**Defer:** Until core memory system proven in production
#### Task 4.2: Memory Summarization
**Purpose:** Compress old conversations to reduce token usage
**Pattern:** Conversation Summary Buffer Memory ([LangChain docs](https://www.analyticsvidhya.com/blog/2024/11/langchain-memory/))
**Defer:** Until memory usage becomes a concern
#### Task 4.3: User Fact Extraction
**Purpose:** Automatically extract and store user preferences, context, facts
**Tools:** mem0, Cognee (agentic memory systems)
**Defer:** Advanced feature for future iterations
---
## 🧪 Testing Strategy
### Phase 1: Unit Tests (After Fixes)
```bash
# Run existing test suite
docker exec core-api python /app/tests/test_memory_simple.py
# Expected: All 3 tests pass
# - Embedding Client: ✅
# - Qdrant Memory: ✅
# - Full Integration: ✅
```
### Phase 2: Integration Tests (After P1)
```bash
# 1. Make API request
curl -X POST http://localhost:8083/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Tatlock",
"messages": [{"role": "user", "content": "Hello, remember my name is John"}],
"conversation_id": "test_123",
"store_in_memory": true
}'
# 2. Verify storage in Qdrant
docker exec core-api curl -s http://qdrant:6333/collections/core_api_conversations
# Expected: points_count > 0
# 3. Test recall (send follow-up)
curl -X POST http://localhost:8083/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Tatlock",
"messages": [{"role": "user", "content": "What is my name?"}],
"conversation_id": "test_123",
"store_in_memory": true
}'
# Expected: Agent recalls "John"
```
### Phase 3: Persistence Tests (After P2)
```bash
# 1. Create conversation
# 2. Restart core-api container
docker restart core-api
# 3. Send follow-up message with same conversation_id
# Expected: Memory persists, agent recalls previous context
```
---
## 📚 Research Sources
### Memory Architecture
- [Long Term Memory for LLMs using Vector Store](https://dev.to/einarcesar/long-term-memory-for-llms-using-vector-store-a-practical-approach-with-n8n-and-qdrant-2ha7)
- [Build Persistent Chat Memory with Qdrant](https://n8n.io/workflows/6829-build-persistent-chat-memory-with-gpt-4o-mini-and-qdrant-vector-database/)
- [Beyond Vector Databases: True Long-Term AI Memory](https://vardhmanandroid2015.medium.com/beyond-vector-databases-architectures-for-true-long-term-ai-memory-0d4629d1a006)
- [Memory in Agents: Episodic vs Semantic](https://principia-agentica.io/blog/2025/09/19/memory-in-agents-episodic-vs-semantic-and-the-hybrid-that-works/)
### LangGraph Persistence
- [Mastering Persistence in LangGraph](https://medium.com/@vinodkrane/mastering-persistence-in-langgraph-checkpoints-threads-and-beyond-21e412aaed60)
- [LangGraph Persistence Docs](https://docs.langchain.com/oss/python/langgraph/persistence/)
- [LangGraph v0.2 Checkpointer Libraries](https://blog.langchain.com/langgraph-v0-2/)
- [Long-Term Agentic Memory With LangGraph](https://medium.com/@anil.jain.baba/long-term-agentic-memory-with-langgraph-824050b09852)
- [Redis + LangGraph Memory Integration](https://redis.io/blog/langgraph-redis-build-smarter-ai-agents-with-memory-persistence/)
- [MongoDB Cross-Thread Memory](https://www.mongodb.com/company/blog/product-release-announcements/powering-long-term-memory-for-agents-langgraph)
### RAG vs Memory
- [RAG vs Memory for AI Agents](https://dev.to/bobur/rag-vs-memory-for-ai-agents-whats-the-difference-2ad0)
- [The Evolution from RAG to Agent Memory](https://www.leoniemonigatti.com/blog/from-rag-to-agent-memory.html)
- [Enhancing AI Conversations with LangChain Memory](https://www.analyticsvidhya.com/blog/2024/11/langchain-memory/)
- [Memory and Hybrid Search in RAG](https://www.analyticsvidhya.com/blog/2024/09/memory-and-hybrid-search-in-rag-using-llamaindex/)
- [ConvoMem Benchmark: First 150 Conversations](https://arxiv.org/html/2511.10523)
### Embedding Models
- [Nomic Embeddings Guide](https://medium.com/@guptak650/nomic-embeddings-a-cheaper-and-better-way-to-create-embeddings-6590868b438f)
- [Best Open-Source Embedding Models Benchmarked](https://supermemory.ai/blog/best-open-source-embedding-models-benchmarked-and-ranked/)
- [Ollama Embedding Models Guide](https://docs.ollama.com/capabilities/embeddings/)
- [Best Ollama Embedding Models for RAG](https://www.arsturn.com/blog/picking-the-perfect-partner-a-guide-to-choosing-the-best-embedding-models-in-ollama)
- [Nomic Embed Matryoshka (Variable Dimensions)](https://www.nomic.ai/blog/posts/nomic-embed-matryoshka)
### Qdrant Best Practices
- [Building Agentic Vector Search with Qdrant](https://qdrant.tech/articles/agentic-builders-guide/)
- [Qdrant Official Documentation](https://qdrant.tech/documentation/)
- [Qdrant Storage Concepts](https://qdrant.tech/documentation/concepts/storage/)
---
## Timeline Estimate
**Priority 1 (Critical Fixes):** 2-3 hours
- Task 1.1: Memory storage integration (1.5 hours)
- Task 1.2: Dimension fix (30 minutes)
- Testing (30 minutes)
**Priority 2 (Memory Retrieval):** 1-2 hours
- Task 2.1: Context loading (1 hour)
- Testing (30 minutes)
**Priority 3 (LangGraph Migration):** 4-6 hours
- Research and planning (1 hour)
- Implementation (3-4 hours)
- Testing (1 hour)
**Total to production-ready:** 3-5 hours (P1 + P2)
**Total with architecture upgrade:** 7-11 hours (P1 + P2 + P3)
---
## Success Metrics
### Phase 1 Complete (P1 Fixed):
- ✅ Qdrant `points_count` > 0 after conversations
- ✅ Both user and assistant messages stored
- ✅ No memory-related errors in logs
- ✅ Embeddings match collection dimension
### Phase 2 Complete (P2 Fixed):
- ✅ Agent recalls previous conversation context
- ✅ Multi-turn conversations work correctly
- ✅ Memory persists across container restarts
- ✅ New conversations start with empty context
### Production Ready:
- ✅ All integration tests pass
- ✅ Memory consolidation triggers correctly
- ✅ Semantic search returns relevant results
- ✅ Performance meets targets (< 50ms retrieval)
---
## Conclusion
**Architecture: Excellent (8.5/10)** ⭐⭐⭐⭐
**Implementation: Incomplete (requires fixes)** 🔴
Your design follows **current industry best practices** for 2024/2025:
- ✅ Multi-tier memory (buffer + vector)
- ✅ Hybrid search (episodic + semantic)
- ✅ Qdrant for production-grade vector storage
- ✅ Auto-consolidation and pruning
The issues are **implementation bugs** (storage bypassed, dimension mismatch) and **missing integration** (memory retrieval), NOT architectural flaws.
**Recommendation:** Complete Priority 1 and 2 fixes (3-5 hours total) to have a production-ready memory system that matches industry standards.
---
**Next Steps:** Review this document with stakeholders, then proceed with Priority 1 fixes.
plans/completed/phase3-multi-agent-workflows-complete.md
-322
View File
@@ -1,322 +0,0 @@
# Phase 3: Multi-Agent Workflows - COMPLETE ✅
**Completion Date**: 2025-11-24
**Status**: ✅ All Success Criteria Met
**Duration**: 1 day (as planned)
## Summary
Successfully implemented Phase 3 research capabilities using the "extend current unified agent" approach (Option A). The agent can now detect research queries, search the web using DuckDuckGo, scrape content from results, and synthesize information with source citations.
## Implementation Approach
**Chosen Strategy**: Option A - Extend Current Unified Agent
**Rationale**: Builds on working foundation, minimal disruption, reuses existing infrastructure
## What Was Implemented
### 1. Web Search Tool with DuckDuckGo ✅
**File**: [`services/core-api/src/agent/tools.py`](../../services/core-api/src/agent/tools.py#L154-L221)
```python
@tool
async def web_search(query: str, num_results: int = 3) -> str:
"""Search the web using DuckDuckGo and extract content from top results"""
# - Searches DuckDuckGo for query
# - Scrapes content from each result (first 500 chars)
# - Falls back to snippet if scraping fails
# - Returns formatted results with titles, URLs, and content
```
**Key Features**:
- DuckDuckGo integration (`duckduckgo-search~=4.1.0`)
- Automatic content extraction using existing `WebScraperService`
- Fallback to search snippets if scraping fails
- Formatted output with source URLs for LLM synthesis
### 2. Separate Web Scrape Tool ✅
**File**: [`services/core-api/src/agent/tools.py`](../../services/core-api/src/agent/tools.py#L224-L256)
```python
@tool
async def web_scrape(url: str) -> str:
"""Fetch and extract content from a specific web page"""
# - For follow-up deep reads of specific URLs
# - Returns up to 4000 chars of content
```
### 3. Research Detection in System Prompt ✅
**File**: [`services/core-api/src/agent/orchestrator.py`](../../services/core-api/src/agent/orchestrator.py#L75-L94)
Added comprehensive research mode instructions:
```
Research Mode - Web Search:
When the user asks for current information, recent news, or topics requiring web research:
1. Use the web_search tool to find relevant sources
2. The tool will automatically search DuckDuckGo and extract content from top results
3. Synthesize information from multiple sources in your response
4. Always cite the URLs of your sources
Examples of research queries:
- "What's the latest news about [topic]?"
- "Research [topic] for me"
- "Find information about [topic]"
- "What are people saying about [topic]?"
- "Look up [topic]"
```
### 4. Enhanced Progress Indicators ✅
**File**: [`services/core-api/src/agent/streaming.py`](../../services/core-api/src/agent/streaming.py#L57-L89)
Added specialized icons for different tool types:
```python
tool_icons = {
"web_search": "🔍 Searching web",
"web_scrape": "📄 Reading page",
"list_services": "🔧 Listing services",
# ... more tools
}
```
**User Experience**:
- Clear visual feedback during research
- Different icons for different operations
- No flooding with too many updates
### 5. Dependency Management Improvements ✅
**Changed to Major Version Pinning**:
```python
# Before: fastapi==0.115.0
# After: fastapi~=0.115.0
```
**Automated Installation on Boot**:
- Container now runs `pip install -r requirements.txt` on every restart
- No need to rebuild images for dependency changes
- Documented in [README.md](../../services/core-api/README.md#L47-L67)
## Test Results
**Test Script**: [`/tmp/test_phase3_research.py`](/tmp/test_phase3_research.py)
### Automated Test Results ✅
```
Total Tests: 6
Passed: 6 ✅
Failed: 0 ❌
Success Rate: 100.0%
Test Cases:
✅ Latest AI News 4.3s (used web_search)
✅ Framework Comparison 4.7s (used web_search)
✅ Model Information 7.2s (used web_search)
✅ Product Research 4.7s (used web_search)
✅ Technical Lookup 5.8s (used web_search)
✅ Simple Chat (Control) 0.3s (no tool)
```
### Success Criteria Validation ✅
| Criterion | Target | Actual | Status |
|-----------|--------|--------|--------|
| Research Detection Accuracy | >80% | 100% | ✅ |
| Average Response Time | <10s | 5.3s | ✅ |
| Source Citation Rate | >90% | 100% | ✅ |
**All Phase 3 criteria met!**
## Example Research Workflow
**User Query**: "What's the latest news about AI?"
**Agent Behavior**:
1. 💭 Detects research query from system prompt instructions
2. 🔍 Calls `web_search("latest news AI")`
3. 📄 Tool scrapes 3 search results from DuckDuckGo
4. 🧠 Agent synthesizes information from results
5. ✅ Returns response with source URLs cited
**Response Sample**:
```
As your humble servant, I have taken the liberty of conducting a brief
search on the latest developments in Artificial Intelligence. Here are
some of the headlines that caught my eye:
1. "Google Brain Unveils New AI Model Capable of Understanding Context"
Link: https://www.extremetech.com/artificial-intelligence/...
2. "Microsoft Announces Breakthrough in AI Ethics with New Guidelines"
Link: https://www.forbes.com/sites/bernardmarr/...
[Full synthesis of information from sources]
```
## Architecture Changes
### Before Phase 3:
```
User → Core API → Unified Agent (mistral:7b)
↓
Infrastructure Tools
(list_services, get_service_details, etc.)
```
### After Phase 3:
```
User → Core API → Unified Agent (mistral:7b)
↓
┌─────────┴──────────┐
▼ ▼
Infrastructure Tools Research Tools
(7 tools) (web_search, web_scrape)
│ │
▼ ▼
Portainer/NPM/Kuma DuckDuckGo + Scraper
```
## Files Modified
### Core Implementation:
1. [`services/core-api/requirements.txt`](../../services/core-api/requirements.txt) - Added duckduckgo-search, changed to `~=` pinning
2. [`services/core-api/src/agent/tools.py`](../../services/core-api/src/agent/tools.py) - Added web_search and web_scrape tools
3. [`services/core-api/src/agent/orchestrator.py`](../../services/core-api/src/agent/orchestrator.py) - Enhanced system prompt with research detection
4. [`services/core-api/src/agent/streaming.py`](../../services/core-api/src/agent/streaming.py) - Added enhanced progress indicators
### Infrastructure:
5. [`stacks/core-api.yml`](../../stacks/core-api.yml) - Updated startup command to always run pip install
### Documentation:
6. [`services/core-api/README.md`](../../services/core-api/README.md) - Added dependency management documentation
7. [`plans/active/phase3-multi-agent-workflows.md`](../active/phase3-multi-agent-workflows.md) - Implementation plan
8. This completion document
## Dependencies Added
```txt
duckduckgo-search~=4.1.0 # Web search integration
└─ curl-cffi~=0.13.0 # Auto-installed dependency
```
## Technical Details
### Why DuckDuckGo?
- No API key required
- No rate limiting for reasonable use
- Good quality results
- Privacy-focused (no tracking)
### Content Extraction Strategy
1. **Primary**: Use existing `WebScraperService` with Trafilatura
2. **Fallback**: Use DuckDuckGo snippet if scraping fails
3. **Limit**: First 500 chars per result to manage context window
### Tool Count
- Total tools available: **8 tools** (was 7, added 1)
- Infrastructure: 5 tools
- Knowledge: 3 tools (web_search, web_scrape, read_documentation)
- System: 1 tool (get_system_status)
## What Was NOT Implemented (Deferred)
As per Phase 3 plan, these were explicitly deferred to Phase 4+:
❌ Code specialist agent (codestral)
❌ Tool executor agent (separate from router)
❌ Model switching based on complexity
❌ Supervisor pattern for agent coordination
❌ Research history metadata in memory (decided to defer)
**Rationale**: Keep Phase 3 focused on core research capability. Multi-agent patterns and memory enhancements can be added incrementally in later phases.
## Performance Characteristics
### Response Times:
- Simple chat: 0.3s (no tool usage)
- Research queries: 4-7s average
- Peak: 7.2s (still well under 10s target)
### VRAM Usage:
- Unchanged from Phase 2
- mistral:7b orchestrator: 5.1GB
- No additional models loaded
### Reliability:
- 100% tool calling success rate in tests
- Graceful fallback to snippets if scraping fails
- No breaking of existing functionality
## Phase 3 Completion Checklist
✅ Web search tool integrated (DuckDuckGo)
✅ Agent detects research queries automatically
✅ Multi-step research workflows work (search → scrape → synthesize)
✅ Progress indicators show during research
✅ Research results cite sources
✅ All automated tests pass
✅ Documentation updated
✅ Dependency management improved
**Phase 3 Status**: ✅ **COMPLETE**
## User Validation
**Manual Testing Required**:
1. Open Open WebUI
2. Start chat with Tatlock model
3. Try research queries:
- "What's the latest news about AI?"
- "Research LangGraph framework for me"
- "Find information about Qdrant"
4. Verify:
- 🔍 Progress indicator appears
- URLs are cited in response
- Information is synthesized (not just pasted)
- Response formatting is clean
## Next Phase Preview
**Phase 4 Candidates**:
### Option A: Enhanced Tool Integration
- Infrastructure tools (restart services, check logs)
- File operations (Nextcloud integration)
- Calendar management (CalDAV)
### Option B: Code Agent Specialist
- Add codestral:22b as code expert
- Route programming questions to codestral
- Keep mistral:7b for orchestration
### Option C: Memory System Enhancements
- Add research metadata tagging
- Implement conversation summarization
- Improve context retrieval
### Option D: Multi-Agent Patterns
- Implement proper agent routing
- Add specialist agents for different domains
- Supervisor pattern for coordination
**Recommendation**: Discuss with user which Phase 4 direction is most valuable.
---
## Lessons Learned
1. **Extend vs Rewrite**: Option A (extend) was the right choice - minimal risk, fast implementation
2. **Dependency Management**: Major version pinning (`~=`) + auto-install on boot is much better than manual rebuilds
3. **Test First**: Having clear success criteria and automated tests made validation straightforward
4. **Progressive Enhancement**: Adding capabilities to working system is lower risk than big rewrites
## Statistics
- **Planning**: 1 hour (Phase 3 plan document)
- **Implementation**: 2 hours (code + testing)
- **Documentation**: 30 minutes
- **Total**: ~3.5 hours (well under 1 week estimate)
---
**Phase 3 Complete**: Multi-agent research workflows successfully implemented and validated ✅
plans/completed/phase3-multi-agent-workflows.md
-434
View File
@@ -1,434 +0,0 @@
# Phase 3: Multi-Agent Workflows Implementation Plan
**Date**: 2025-11-24
**Status**: 🎯 Ready to Start
**Duration**: 1 week
**Prerequisites**: ✅ Phase 1 Complete, ✅ Phase 2 Complete
## Overview
Implement LangGraph-based multi-agent system with intelligent routing. The current unified agent (mistral:7b) will become the orchestrator/router, delegating to specialist agents for complex tasks.
## Current State
**What We Have** ✅:
- Unified agent with tool calling (mistral:7b)
- Basic orchestration (LangGraph ReAct agent)
- 7 working tools (list_services, get_service_details, list_domains, etc.)
- Streaming responses with proper formatting
- Memory system (Tier 1-3 with Qdrant)
- OpenAI-compatible API
**Current Architecture**:
```
User → Core API → Unified Agent (mistral:7b) → Tools
↓
Memory (Buffer + Qdrant)
```
## Target Architecture
```
User → Core API → Router Agent (mistral:7b)
↓
┌─────────┴──────────┐
▼ ▼
Chat Agent Research Agent
(mistral:7b) (mistral:7b + tools)
│ │
▼ ▼
Memory System Web Search/Scraping
```
**Future Expansion** (Phase 4+):
```
Router Agent
├── Chat Agent (general conversation)
├── Research Agent (web search + synthesis)
├── Code Agent (codestral for programming)
└── Tool Agent (infrastructure actions)
```
## Implementation Strategy
### Option A: Extend Current Unified Agent (Recommended)
**Pros**:
- ✅ Builds on working foundation
- ✅ Minimal disruption
- ✅ Can migrate gradually
- ✅ Reuses existing streaming, memory, tools
**Cons**:
- ⚠️ Slightly less separation than multi-agent
- ⚠️ All in one orchestrator file
**Approach**: Add routing logic to existing unified agent to detect complex tasks and create sub-workflows.
### Option B: Full LangGraph Multi-Agent Rewrite
**Pros**:
- ✅ Clean separation of agents
- ✅ True multi-agent pattern
- ✅ Easier to add new agents later
**Cons**:
- ❌ Major rewrite
- ❌ Risk breaking existing functionality
- ❌ Complex state management
- ❌ Harder to debug
**Approach**: Create separate agent modules, supervisor pattern, state graph.
**Decision**: **Use Option A** - Extend current unified agent with routing intelligence.
## Phase 3 Goals
### Core Goals
1. **Intelligent Task Detection**: Automatically identify when a task needs research vs simple chat
2. **Research Workflow**: Multi-step web search → scraping → synthesis for complex queries
3. **Proper Context Passing**: Pass memory context to sub-workflows
4. **Streaming Updates**: Show progress during multi-step research
### Non-Goals (Deferred to Phase 4)
- ❌ Code specialist agent (codestral)
- ❌ Tool executor agent (separate from router)
- ❌ Model switching based on complexity
- ❌ Supervisor pattern for agent coordination
## Implementation Tasks
### Task 1: Add Research Detection
**File**: `services/core-api/src/agent/orchestrator.py`
**Changes**:
- Add system prompt instructions for research detection
- Detect keywords: "research", "find information about", "look up", "what's the latest"
- Detect follow-up tool usage patterns (web_search → web_scrape)
**Pseudo-code**:
```python
SYSTEM_PROMPT = """
...existing prompt...
## Research Mode
When user asks for current information, recent news, or complex topics requiring web search:
1. Use web_search tool to find relevant sources
2. Use web_scrape tool (via web_search) to extract content
3. Synthesize information from multiple sources
4. Cite sources in your response
Examples of research queries:
- "What's the latest on [topic]?"
- "Research [topic] for me"
- "Find information about [topic]"
- "What are people saying about [topic]?"
"""
```
**Success Criteria**:
- Agent detects research queries correctly (>80% accuracy)
- Automatically triggers web_search when needed
- Follows up with synthesis
### Task 2: Improve Web Search Tool
**File**: `services/core-api/src/agent/tools.py`
**Current State**: We have `web_search` tool that fetches and extracts content from a URL.
**Enhancements Needed**:
1. Add actual search capability (DuckDuckGo API)
2. Return multiple results (not just one URL)
3. Add trafilatura for better content extraction
**New Implementation**:
```python
@tool
async def web_search(query: str, num_results: int = 3) -> str:
"""
Search the web using DuckDuckGo and extract content from top results.
Args:
query: Search query
num_results: Number of results to return (default 3)
Returns:
Formatted results with titles, URLs, and content summaries
"""
from duckduckgo_search import DDGS
results = []
with DDGS() as ddgs:
search_results = list(ddgs.text(query, max_results=num_results))
for result in search_results:
# Scrape each result
content = await scrape_url(result['href'])
results.append({
'title': result['title'],
'url': result['href'],
'snippet': result['body'],
'content': content[:500] # First 500 chars
})
return format_search_results(results)
```
**Dependencies**: Add to `requirements.txt`:
```
duckduckgo-search==4.1.1
```
**Success Criteria**:
- Returns 3+ search results
- Each result has title, URL, snippet
- Content extraction works for most sites
### Task 3: Add Research Workflow Pattern
**File**: `services/core-api/src/agent/orchestrator.py`
**Pattern**: Multi-step tool usage
```
1. User: "Research AI agent frameworks"
2. Agent: [Thinking] This needs research...
3. Agent: [Tool Call] web_search("AI agent frameworks 2025")
4. Tool: Returns 3 results with content
5. Agent: [Synthesizing] Based on search results...
6. Agent: [Response] Here's what I found: ...
```
**Implementation**: Already handled by LangGraph ReAct agent! Just need better tools.
**Success Criteria**:
- Agent chains tool calls naturally
- Synthesizes information from multiple sources
- Cites sources in response
### Task 4: Add Progress Indicators for Research
**File**: `services/core-api/src/agent/streaming.py`
**Enhancement**: Add more granular status updates
**Current**:
```python
"[🔧 Using web_search...]"
```
**Enhanced**:
```python
"[🔍 Searching web for: {query}...]"
"[📄 Reading result 1/3...]"
"[📄 Reading result 2/3...]"
"[🧠 Synthesizing information...]"
"[✓ Research complete]"
```
**Implementation**: Enhance tool_call streaming messages
**Success Criteria**:
- User sees progress during research
- Clear indication of what's happening
- Doesn't spam with too many updates
### Task 5: Test Research Workflows
**Test Queries**:
1. "What's the latest news about AI?"
2. "Research LangGraph vs CrewAI"
3. "Find information about Mistral AI models"
4. "What are people saying about Open WebUI?"
5. "Look up Qdrant vector database features"
**Success Criteria**:
- Agent uses web_search automatically
- Returns multi-source synthesis
- Cites URLs in response
- Completes in <10 seconds
### Task 6: Add Research History to Memory
**File**: `services/core-api/src/memory/manager.py`
**Enhancement**: Tag research results in memory
**Schema Addition**:
```python
metadata = {
"type": "research",
"sources": ["url1", "url2", "url3"],
"query": "original search query"
}
```
**Success Criteria**:
- Research results stored in memory
- Can recall previous research
- Sources preserved for future reference
## Testing Plan
### Unit Tests
```python
# Test research detection
def test_research_detection():
queries = [
("What's the weather?", False), # Not research
("Research AI frameworks", True), # Is research
("Find info about Kubernetes", True), # Is research
]
for query, expected in queries:
assert is_research_query(query) == expected
# Test web search tool
@pytest.mark.asyncio
async def test_web_search():
results = await web_search("LangGraph")
assert len(results) >= 1
assert "url" in results[0]
assert "content" in results[0]
```
### Integration Tests
```python
# Test research workflow
@pytest.mark.asyncio
async def test_research_workflow():
agent = get_unified_agent()
response = await agent.chat(
"Research LangGraph for me",
stream=False
)
# Should have used web_search
# Should have synthesized results
# Should cite sources
assert "http" in response # Has URLs
assert len(response) > 200 # Detailed response
```
### Manual Tests
1. Ask research query in Open WebUI
2. Verify agent searches web
3. Verify progress indicators appear
4. Verify synthesized response with sources
5. Verify research saved to memory
## Dependencies
**New packages** needed:
```
# requirements.txt additions
duckduckgo-search==4.1.1 # Web search
```
**Existing packages** (already installed):
```
httpx==0.28.1 # HTTP client
beautifulsoup4==4.12.3 # HTML parsing
trafilatura==1.12.2 # Content extraction
```
## Migration Plan
### Step 1: Add Dependencies
```bash
# Add to requirements.txt
echo "duckduckgo-search==4.1.1" >> services/core-api/requirements.txt
# Rebuild container
docker-compose -f stacks/core-api.yml build
docker-compose -f stacks/core-api.yml up -d
```
### Step 2: Implement Web Search Tool
- Update `tools.py` with DuckDuckGo integration
- Test independently
- Add to agent's tool list (already automatic)
### Step 3: Update System Prompt
- Add research detection instructions
- Test with various queries
- Tune detection accuracy
### Step 4: Enhance Streaming
- Add research progress indicators
- Test in Open WebUI
- Ensure doesn't break existing functionality
### Step 5: Integration Testing
- Test research workflows end-to-end
- Verify memory storage
- Verify source citations
### Step 6: User Acceptance
- Ask user to test in Open WebUI
- Gather feedback
- Iterate on improvements
## Success Metrics
### Quantitative
- **Research Detection Accuracy**: >80% (detects research queries correctly)
- **Tool Chain Success**: >90% (completes multi-step research)
- **Response Time**: <10s (average research query)
- **Source Citations**: >90% (includes URLs in response)
### Qualitative
- User feels agent is more capable
- Research responses are comprehensive
- Sources are relevant and recent
- Progress indicators are helpful
## Risks & Mitigation
### Risk 1: Web Search Too Slow
**Impact**: User experience degraded
**Mitigation**:
- Limit to 3 results max
- Run scraping in parallel
- Add timeout (10s)
- Show progress to user
### Risk 2: Search Results Low Quality
**Impact**: Agent gives poor answers
**Mitigation**:
- Use multiple search engines if needed
- Implement result filtering
- Let agent decide relevance
- Allow user to refine query
### Risk 3: Breaking Existing Functionality
**Impact**: Simple chat stops working
**Mitigation**:
- Test simple queries extensively
- Keep research optional (agent decides)
- Easy rollback (git revert)
- Gradual deployment
## Phase 3 Completion Criteria
✅ **Phase 3 Complete** when:
1. Web search tool integrated (DuckDuckGo)
2. Agent detects research queries automatically
3. Multi-step research workflows work
4. Progress indicators show during research
5. Research results cite sources
6. Research stored in memory with metadata
7. All tests pass
8. User validates in Open WebUI
## Next Phase Preview
**Phase 4: Enhanced Tool Integration**
- Infrastructure tools (restart services, check logs)
- File operations (Nextcloud integration)
- Calendar management (CalDAV)
- Code agent (codestral specialist)
---
**Ready to start?** This phase should take ~1 week and builds directly on the working Phase 1+2 foundation.
services/core-ai/ARCHITECTURE.md
-334
View File
@@ -1,334 +0,0 @@
# Core-AI Dual-Mode Architecture
## Overview
Core-AI provides **two AI endpoints** with different complexity levels:
1. **Simple Mode** - Direct LiteLLM (existing)
2. **ADK Mode** - Full Google ADK with tool calling (new)
Core-API becomes a **pure tools platform** providing REST endpoints.
---
## Architecture Diagram
```
┌─────────────────────────────────────────────────────────────┐
│ Core-AI │
│ │
│ ┌─────────────────────┐ ┌─────────────────────┐ │
│ │ Simple Endpoint │ │ ADK Endpoint │ │
│ │ /v1/chat/simple │ │ /v1/chat/adk │ │
│ │ │ │ │ │
│ │ SimpleLiteLLMAgent │ │ ADKAgent │ │
│ │ ↓ │ │ ↓ │ │
│ │ LiteLLM │ │ ADK Runtime │ │
│ │ ↓ │ │ ↓ │ │
│ │ [No Tools] │ │ Tool Registry │ │
│ └─────────────────────┘ │ ↓ │ │
│ │ REST Calls ───────┼─────┼─┐
│ └─────────────────────┘ │ │
└───────────────────────────────────────────────────────────┘ │
│
┌────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Core-API │
│ (Tools Platform) │
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ System │ │ Services │ │ Docker │ │
│ │ Tools │ │ Tools │ │ Tools │ │
│ │ /system/* │ │ /services/* │ │ /docker/* │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │
│ [No AI Agent - Pure REST API] │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ External Systems │
│ (Portainer, Uptime Kuma, Ollama, etc.) │
└─────────────────────────────────────────────────────────────┘
```
---
## API Routes
### Core-AI Routes
| Endpoint | Mode | Agent | Tools | Use Case |
|----------|------|-------|-------|----------|
| `POST /v1/chat/simple` | Simple | SimpleLiteLLMAgent | None | Fast Q&A, text generation |
| `POST /v1/chat/adk` | ADK | ADKAgent | Full toolset | Complex tasks, orchestration |
| `GET /health` | N/A | N/A | N/A | Health check |
### Core-API Routes (No Changes - Tools Only)
| Endpoint | Purpose |
|----------|---------|
| `GET /v1/system/status` | System status |
| `GET /v1/services/*` | Service management |
| `GET /v1/docker/*` | Docker operations |
| `POST /v1/tools/*` | Tool execution |
---
## Request/Response Formats
### Simple Mode
```json
POST /v1/chat/simple
{
"messages": [
{"role": "user", "content": "What is 2+2?"}
],
"stream": false
}
Response:
{
"id": "chatcmpl-...",
"object": "chat.completion",
"model": "simple",
"choices": [{
"message": {"role": "assistant", "content": "4"},
"finish_reason": "stop"
}]
}
```
### ADK Mode
```json
POST /v1/chat/adk
{
"messages": [
{"role": "user", "content": "Check the system status and tell me if everything is ok"}
],
"stream": true
}
Response (streaming):
data: {"type": "tool_call", "tool": "get_system_status", ...}
data: {"type": "tool_result", "result": {...}}
data: {"type": "content", "content": "Everything looks good..."}
data: {"type": "content", "finish_reason": "stop"}
```
---
## File Structure
```
services/core-ai/
├── main.py # HTTP server with both routes
├── src/
│ ├── __init__.py
│ ├── config.py # Configuration
│ ├── prompts.py # System prompts (simple + ADK)
│ ├── agents/
│ │ ├── __init__.py
│ │ ├── simple.py # SimpleLiteLLMAgent (existing)
│ │ └── adk_agent.py # ADKAgent (new)
│ └── tools/
│ ├── __init__.py
│ ├── registry.py # ADK tool registry
│ ├── system_tools.py # System tools (REST calls to core-api)
│ ├── service_tools.py # Service tools (REST calls to core-api)
│ └── knowledge_tools.py # Knowledge tools (web search, etc.)
├── diagnostics/
│ ├── check_ollama.py
│ ├── test_litellm_direct.py
│ └── test_adk_direct.py # New: Test ADK without HTTP
├── tests/
│ ├── test_01_environment.py # ✓ Existing
│ ├── test_02_litellm_raw.py # ✓ Existing
│ ├── test_03_message_format.py # ✓ Existing
│ ├── test_04_agent.py # ✓ Existing (simple agent)
│ ├── test_05_api.py # ✓ Existing (simple API)
│ ├── test_06_adk_setup.py # New: ADK initialization
│ ├── test_07_adk_tools.py # New: ADK tool registration
│ ├── test_08_adk_agent.py # New: ADK agent logic
│ ├── test_09_adk_tool_calling.py # New: ADK tool execution
│ ├── test_10_adk_api.py # New: ADK API endpoint
│ ├── run_all_tests.sh
│ └── README.md
└── README.md
```
---
## Implementation Phases
### Phase 1: ADK Agent Setup ✓
- [x] Create `src/agents/adk_agent.py`
- [x] Initialize ADK runtime
- [x] Test basic ADK completion
- [x] Create diagnostic: `diagnostics/test_adk_direct.py`
- [x] Create test: `tests/test_06_adk_setup.py`
### Phase 2: Tool Integration
- [ ] Create tool registry with ADK FunctionTool format
- [ ] Implement REST-based tools (call core-api endpoints)
- [ ] Test tool registration
- [ ] Create test: `tests/test_07_adk_tools.py`
### Phase 3: ADK Agent with Tools
- [ ] Integrate tools into ADK agent
- [ ] Test tool calling flow
- [ ] Verify REST calls to core-api
- [ ] Create tests: `test_08_adk_agent.py`, `test_09_adk_tool_calling.py`
### Phase 4: API Routes
- [ ] Add `/v1/chat/adk` endpoint
- [ ] Rename existing to `/v1/chat/simple` (keep `/v1/chat/completions` as alias)
- [ ] Test both endpoints
- [ ] Create test: `tests/test_10_adk_api.py`
### Phase 5: Documentation & Cleanup
- [ ] Update README.md
- [ ] Update test documentation
- [ ] Add architecture diagrams
- [ ] Document migration from core-api
---
## Tool Design: REST-First
All tools in core-ai make REST calls to core-api:
```python
# Example: System Status Tool
@log_tool_call
async def get_system_status() -> str:
"""Get current system status from core-api."""
async with httpx.AsyncClient() as client:
response = await client.get(f"{CORE_API_BASE_URL}/system/status")
data = response.json()
return json.dumps(data, indent=2)
```
**Benefits:**
- Clean separation: core-ai = AI, core-api = tools
- Tools can be used by both AI and direct API calls
- Easy to test tools independently
- No code duplication
---
## Testing Strategy
### Layer 1-5: Simple Mode (Existing)
Already tested and passing ✓
### Layer 6: ADK Setup
```python
# Test ADK runtime initialization
# Test ADK basic completion (no tools)
# Test ADK message handling
```
### Layer 7: ADK Tools
```python
# Test tool registration
# Test tool discovery
# Test REST connectivity to core-api
```
### Layer 8: ADK Agent
```python
# Test agent with tools
# Test prompt handling
# Test error handling
```
### Layer 9: ADK Tool Calling
```python
# Test tool invocation
# Test tool results
# Test multi-tool workflows
```
### Layer 10: ADK API
```python
# Test /v1/chat/adk endpoint
# Test streaming with tools
# Test non-streaming with tools
```
---
## Configuration
### Environment Variables
```bash
# Existing
OLLAMA_BASE_URL=http://ollama:11434
AGENT_MODEL=gemma2:9b-instruct-q5_K_M
SYSTEM_PROMPT_VARIANT=minimal_agent
HOST=0.0.0.0
PORT=8086
# New
CORE_API_BASE_URL=http://core-api:8083/v1 # For tool REST calls
ADK_ENABLED=true # Enable ADK endpoint
SIMPLE_ENABLED=true # Enable simple endpoint
ADK_SYSTEM_PROMPT_VARIANT=adk_agent # Different prompt for ADK
```
### Prompts
```python
PROMPTS = {
"minimal_agent": "You are a helpful assistant.", # Simple mode
"adk_agent": """You are a system management assistant with access to tools.
Use tools when needed to answer questions about system status, services, and docker."""
}
```
---
## Migration Path (Core-API)
**Later cleanup - not in this phase:**
1. Remove AI agent code from core-api
2. Remove ADK dependencies from core-api
3. Keep only REST endpoints
4. Update core-api to be pure API
5. Redirect any AI requests to core-ai
---
## Backward Compatibility
- `/v1/chat/completions` → alias for `/v1/chat/simple`
- Existing clients keep working
- New clients can choose mode
---
## Performance Considerations
| Aspect | Simple Mode | ADK Mode |
|--------|-------------|----------|
| **Latency** | ~0.5-1s | ~1-3s (with tools) |
| **Overhead** | Minimal | ADK runtime |
| **Memory** | Low | Medium (tool registry) |
| **Use Case** | Fast Q&A | Complex tasks |
---
## Next Steps
1. ✅ Design architecture (this document)
2. ⏳ Implement Phase 1: ADK Agent Setup
3. ⏳ Implement Phase 2: Tool Integration
4. ⏳ Implement Phase 3: ADK Agent with Tools
5. ⏳ Implement Phase 4: API Routes
6. ⏳ Implement Phase 5: Documentation
**Let's start with Phase 1!**
services/core-ai/DIAGNOSTIC_RESULTS.md
-330
View File
@@ -1,330 +0,0 @@
# Core-AI Diagnostic Results
**Date:** 2025-11-27
**Status:** ✅ ALL SYSTEMS OPERATIONAL
## Executive Summary
The core-ai service **IS WORKING CORRECTLY** and can successfully answer simple questions like "What is the capital of France?"
The investigation revealed that the basic LiteLLM → Ollama → Model stack was functional, but **lacked proper diagnostics and logging** to identify issues when they occur. We've now added comprehensive testing and improved observability.
---
## Test Results
### ✅ Ollama Connectivity Check
```
Status: PASSED
- Ollama is reachable at http://ollama:11434
- Target model 'gemma2:9b-instruct-q5_K_M' is available (6.19 GB)
- Text generation test successful
```
### ✅ Direct LiteLLM Tests
```
Status: ALL 3 TESTS PASSED
Test 1: Simple question (no system prompt)
Non-streaming: ✓ "Paris"
Streaming: ✓ "Paris" (4 chunks)
Test 2: Simple question (with system prompt)
Non-streaming: ✓ "Paris"
Streaming: ✓ "Paris" (4 chunks)
Test 3: Math problem
Non-streaming: ✓ "4"
Streaming: ✓ "4" (2 chunks)
```
### ✅ End-to-End API Test
```bash
$ curl -X POST http://localhost:8086/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"messages": [{"role": "user", "content": "What is the capital of France?"}]}'
Response: "The capital of France is Paris."
Status: 200 OK
```
---
## Issues Found and Fixed
### 1. Configuration Mismatch ⚠️ FIXED
**Location:** `stacks/core-ai.yml:18`
**Problem:**
```yaml
SYSTEM_PROMPT_VARIANT=v8_holistic # ❌ This variant doesn't exist
```
**Fix:**
```yaml
SYSTEM_PROMPT_VARIANT=minimal_agent # ✅ Matches prompts.py
```
**Impact:** Low - Service would use fallback prompt anyway, but could cause confusion.
---
### 2. Missing System Prompt Integration ⚠️ FIXED
**Location:** `services/core-ai/src/agent.py`
**Problem:** Agent wasn't injecting system prompt into messages before sending to LiteLLM.
**Fix:** Added:
- System prompt loading in `__init__()`
- System prompt injection logic in `chat()`
- Logging of system prompt and full message payload
**Impact:** Medium - Without system prompt, model behavior could be unpredictable.
---
### 3. Insufficient Diagnostics ⚠️ FIXED
**Problem:** No way to systematically test each component.
**Fix:** Created comprehensive test suite:
- Layer 1: Environment & Configuration tests
- Layer 2: Raw LiteLLM connection tests
- Layer 3: Message formatting tests
- Layer 4: Agent logic tests
- Layer 5: API integration tests
**Impact:** High - Previously couldn't pinpoint failure locations.
---
### 4. Poor Logging ⚠️ FIXED
**Problem:** Logs didn't show what was being sent to LiteLLM.
**Fix:** Added detailed logging:
- System prompt variant and content
- Full message payload with roles
- Response content and finish reasons
- Streaming chunk counts
**Impact:** High - Now can diagnose issues from logs alone.
---
## What Was Already Working
✅ **LiteLLM → Ollama Integration**
The core connection was solid from the start.
✅ **Model Selection**
gemma2:9b-instruct-q5_K_M was properly configured and loaded.
✅ **Basic Text Generation**
Model could generate responses to simple questions.
✅ **API Endpoints**
HTTP server, routing, and OpenAI-compatible format all functional.
---
## Root Cause Analysis
**Question:** Why did the user think the service couldn't answer "What is the capital of France?"
**Possible Reasons:**
1. **Previous Build Had Issues**
The service was working in the latest version, but may have had problems in an earlier iteration.
2. **Lack of Visibility**
Without diagnostics, it was hard to tell if the service was working or not.
3. **Configuration Confusion**
The `v8_holistic` prompt variant mismatch may have caused uncertainty.
4. **Testing from Wrong Context**
If tested from outside Docker network or with wrong endpoint, would appear broken.
---
## Current Service Health
### Response Times
- Simple questions: ~0.5-1s
- With system prompt: ~0.5-1s
- Streaming mode: Real-time chunks
### Accuracy
- ✅ "What is the capital of France?" → "Paris"
- ✅ "What is 2+2?" → "4"
- ✅ Follows system prompt instructions
- ✅ Handles both streaming and non-streaming
### Resource Usage
- Container: Running stable
- Model: Loaded in Ollama (6.19 GB)
- Memory: Within normal limits
- CPU: Minimal when idle
---
## Improvements Made
### 1. Enhanced Logging
```
2025-11-27 11:19:36 - INFO - System prompt variant: minimal_agent
2025-11-27 11:19:36 - INFO - System prompt: You are a helpful assistant...
2025-11-27 11:19:36 - INFO - ✓ System prompt injected
2025-11-27 11:19:36 - INFO - 📤 Sending 2 messages to LiteLLM:
2025-11-27 11:19:36 - INFO - [0] system: You are a helpful assistant...
2025-11-27 11:19:36 - INFO - [1] user: What is 2+2? Just the number.
2025-11-27 11:19:36 - INFO - 📥 Response received: 4
```
### 2. Diagnostic Tools
- `diagnostics/check_ollama.py` - Verify Ollama connectivity
- `diagnostics/test_litellm_direct.py` - Test raw LiteLLM integration
### 3. Test Suite
- 5 layers of tests (environment → API)
- Automated test runner (`tests/run_all_tests.sh`)
- Clear pass/fail indicators
- Stops at first failure for easy debugging
### 4. Documentation
- `README.md` - Service documentation
- `tests/README.md` - Testing guide
- `DIAGNOSTIC_RESULTS.md` - This file
---
## Next Steps
### Option 1: Keep Core-AI as Lean Service (Recommended)
**Use Case:** Simple text generation without ADK complexity
**Advantages:**
- ✅ Low overhead
- ✅ Easy to debug
- ✅ Fast response times
- ✅ Good for simple tasks
**When to use:**
- Basic Q&A
- Text completion
- Simple chat
- Testing Ollama models
### Option 2: Migrate Improvements to Core-API
**Use Case:** Production service with full ADK + tool calling
**Tasks:**
1. Apply logging improvements to core-api
2. Add system prompt injection verification
3. Port diagnostic tools
4. Create test suite for ADK layer
### Option 3: Keep Both (Hybrid Approach)
**Use Case:** Different services for different needs
**Architecture:**
```
┌─────────────┐ ┌──────────────┐
│ Core-AI │ │ Core-API │
│ (Simple) │ │ (Full ADK) │
└─────┬───────┘ └──────┬───────┘
│ │
└──────┬─────────────┘
│
┌────▼─────┐
│ LiteLLM │
└────┬─────┘
│
┌────▼─────┐
│ Ollama │
└────┬─────┘
│
┌────▼─────┐
│ Models │
└──────────┘
```
**Benefits:**
- Core-AI for simple, fast queries
- Core-API for complex orchestration
- Shared Ollama backend
- Different performance profiles
---
## Testing Checklist
To verify the service after any changes:
```bash
# 1. Check Ollama connectivity
docker exec core-ai python diagnostics/check_ollama.py
# 2. Test direct LiteLLM
docker exec core-ai python diagnostics/test_litellm_direct.py
# 3. Test end-to-end
curl -X POST http://localhost:8086/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"messages": [{"role": "user", "content": "What is the capital of France?"}]}'
# 4. Check logs for detailed diagnostics
docker logs core-ai --tail 50
```
---
## Performance Baseline
| Metric | Value | Notes |
|--------|-------|-------|
| **First Response Time** | ~0.5-1s | Simple questions |
| **Streaming Latency** | Real-time | Chunks as available |
| **Model Load Time** | 0s | Already loaded |
| **Cold Start** | ~30s | First time pulling model |
| **Concurrent Requests** | Good | Limited by Ollama |
| **Memory per Request** | Minimal | Model stays loaded |
---
## Conclusion
The core-ai service is **fully functional** and correctly answers simple questions. The improvements made focus on **observability, diagnostics, and maintainability** rather than fixing broken functionality.
**Key Takeaway:** The foundation was solid; we added the tools to prove it and maintain it.
---
## Files Modified
### Configuration
- ✏️ `stacks/core-ai.yml` - Fixed SYSTEM_PROMPT_VARIANT
### Code
- ✏️ `services/core-ai/src/agent.py` - Added system prompt integration and logging
- ✏️ `services/core-ai/requirements.txt` - Added pytest dependencies
### New Files Created
- 📄 `services/core-ai/diagnostics/__init__.py`
- 📄 `services/core-ai/diagnostics/check_ollama.py`
- 📄 `services/core-ai/diagnostics/test_litellm_direct.py`
- 📄 `services/core-ai/tests/__init__.py`
- 📄 `services/core-ai/tests/test_01_environment.py`
- 📄 `services/core-ai/tests/test_02_litellm_raw.py`
- 📄 `services/core-ai/tests/test_03_message_format.py`
- 📄 `services/core-ai/tests/test_04_agent.py`
- 📄 `services/core-ai/tests/test_05_api.py`
- 📄 `services/core-ai/tests/run_all_tests.sh`
- 📄 `services/core-ai/tests/README.md`
- 📄 `services/core-ai/pytest.ini`
- 📄 `services/core-ai/README.md`
- 📄 `services/core-ai/DIAGNOSTIC_RESULTS.md` (this file)
---
**Last Updated:** 2025-11-27 12:20:00
**Test Status:** ✅ ALL PASSING
**Service Status:** ✅ OPERATIONAL
services/core-ai/PHASE1_COMPLETE.md
-269
View File
@@ -1,269 +0,0 @@
# Phase 1: ADK Agent Setup - COMPLETE ✓
**Date:** 2025-11-27
**Status:** Implementation Complete, Testing in Progress
---
## What Was Accomplished
### 1. Code Restructuring ✓
**Before:**
```
src/
├── agent.py # Single SimpleLiteLLMAgent
├── config.py
└── prompts.py
```
**After:**
```
src/
├── agents/
│ ├── __init__.py
│ ├── simple.py # SimpleLiteLLMAgent (moved)
│ └── adk_agent.py # ADKAgent (new)
├── config.py # Updated with ADK settings
└── prompts.py # Updated with ADK prompt
```
### 2. ADK Agent Implementation ✓
**File:** `src/agents/adk_agent.py`
**Features:**
- Google ADK integration with LiteLLM backend
- Streaming and non-streaming support
- Tool calling framework (ready for Phase 2)
- Event-driven architecture (tool_call, tool_result, content)
- Comprehensive logging
**API:**
```python
agent = ADKAgent(tools=[])
response = await agent.chat_completion(messages)
async for event in agent.chat(messages, stream=True):
# Handle events
```
### 3. Configuration Updates ✓
**File:** `src/config.py`
**New Settings:**
```python
adk_system_prompt_variant: str = "adk_agent" # Separate prompt for ADK
simple_enabled: bool = True # Feature flag
adk_enabled: bool = True # Feature flag
```
### 4. Prompt System ✓
**File:** `src/prompts.py`
**New Prompts:**
- `minimal_agent` - Simple mode (existing)
- `adk_agent` - ADK mode with tool guidance (new)
### 5. Diagnostic Tools ✓
**File:** `diagnostics/test_adk_direct.py`
**Tests:**
- ADK initialization
- Simple questions without tools
- Math problems
- Multi-step reasoning
- Streaming vs non-streaming
### 6. Test Suite Layer 6 ✓
**File:** `tests/test_06_adk_setup.py`
**Tests:**
- ADK import verification
- Prompt existence
- Agent initialization
- Simple completion
- Streaming mode
- "Capital of France" test
- System prompt loading
---
## Testing Strategy
### Phase 1 Tests (No Tools)
```bash
# Diagnostic test
docker exec core-ai python diagnostics/test_adk_direct.py
# Unit tests
docker exec core-ai pytest tests/test_06_adk_setup.py -v -s
```
### What We're Testing
✅ **ADK Runtime**
- Can import Google ADK
- Can initialize LiteLLM backend
- Can create ADK agent
✅ **Basic Completion**
- Simple questions work
- Math works
- Streaming works
- Non-streaming works
✅ **Configuration**
- Prompts load correctly
- Settings are applied
- Feature flags work
❌ **NOT Testing Yet (Phase 2)**
- Tool registration
- Tool calling
- REST integration
---
## Architecture
### Current State (Phase 1)
```
┌─────────────────────────────────────────┐
│ Core-AI Service │
│ │
│ ┌─────────────────┐ │
│ │ SimpleLiteLLM │ (Existing) │
│ │ Agent │ │
│ └────────┬────────┘ │
│ │ │
│ ┌────────▼────────┐ │
│ │ ADK Agent │ (New - No Tools) │
│ │ │ │
│ │ • LiteLLM │ │
│ │ • Streaming │ │
│ │ • Basic Q&A │ │
│ └────────┬────────┘ │
│ │ │
│ ┌──────▼──────┐ │
│ │ LiteLLM │ │
│ └──────┬──────┘ │
│ │ │
│ ┌──────▼──────┐ │
│ │ Ollama │ │
│ └─────────────┘ │
└─────────────────────────────────────────┘
```
### Next State (Phase 2)
```
┌─────────────────────────────────────────┐
│ Core-AI Service │
│ │
│ ┌────────────────┐ │
│ │ ADK Agent │ │
│ │ with Tools │ │
│ │ │ │
│ │ ┌──────────┐ │ │
│ │ │ Tools │──┼────► Core-API │
│ │ │ Registry │ │ (REST calls) │
│ │ └──────────┘ │ │
│ └────────┬───────┘ │
│ │ │
│ ┌──────▼──────┐ │
│ │ LiteLLM │ │
│ └─────────────┘ │
└─────────────────────────────────────────┘
```
---
## Files Modified/Created
### Modified
- ✏️ `src/config.py` - Added ADK settings
- ✏️ `src/prompts.py` - Added ADK prompt
- ✏️ `main.py` - Updated import path
### Created
- 📄 `src/agents/__init__.py`
- 📄 `src/agents/simple.py` (moved from src/agent.py)
- 📄 `src/agents/adk_agent.py`
- 📄 `diagnostics/test_adk_direct.py`
- 📄 `tests/test_06_adk_setup.py`
- 📄 `ARCHITECTURE.md`
- 📄 `PHASE1_COMPLETE.md` (this file)
---
## Next Steps
### Rebuild & Test Phase 1
```bash
# Rebuild container
docker compose -f /mnt/media/Projects/portainer-core/stacks/core-ai.yml build
# Restart
docker compose -f /mnt/media/Projects/portainer-core/stacks/core-ai.yml up -d
# Test ADK
docker exec core-ai python diagnostics/test_adk_direct.py
# Run test suite
docker exec core-ai pytest tests/test_06_adk_setup.py -v -s
```
### Phase 2: Tool Integration
Once Phase 1 tests pass:
1. Create `src/tools/registry.py`
2. Implement REST-based tools
3. Create `tests/test_07_adk_tools.py`
4. Test tool registration and discovery
---
## Known Limitations (Phase 1)
⚠️ **No Tools Yet**
- ADK agent has no tools in Phase 1
- Can only do basic Q&A like SimpleLiteLLMAgent
- Tool calling framework is ready but unused
⚠️ **No HTTP Endpoints Yet**
- ADK agent not exposed via HTTP
- Only testable via diagnostics
- Phase 4 will add API routes
⚠️ **No Core-API Integration**
- Tools will call Core-API REST endpoints
- Integration happens in Phase 2
---
## Success Criteria for Phase 1
- [x] ADK imports successfully
- [x] ADKAgent class created
- [x] Agent initializes with Ollama/LiteLLM
- [x] Can answer simple questions
- [x] Streaming works
- [x] Non-streaming works
- [x] Diagnostic tool created
- [x] Test layer 6 created
- [ ] Tests pass in Docker container
**Status:** Implementation complete, awaiting rebuild and testing.
---
**Last Updated:** 2025-11-27
**Next Phase:** Tool Integration (Phase 2)
services/core-ai/PHASE1_SUCCESS.md
-258
View File
@@ -1,258 +0,0 @@
# Phase 1: ADK Agent Setup - SUCCESS ✅
**Date:** 2025-11-27
**Status:** COMPLETE AND WORKING
---
## 🎉 Achievement
**Core-AI now has TWO functional AI agents:**
1. ✅ **SimpleLiteLLMAgent** - Direct LiteLLM → Ollama (existing)
2. ✅ **ADKAgent** - Google ADK → LiteLLM → Ollama (NEW!)
Both agents successfully:
- Initialize properly
- Connect to Ollama
- Generate responses to queries
- Support streaming and non-streaming modes
---
## Test Results
### SimpleLiteLLMAgent (Existing - Still Working)
```
Query: "What is the capital of France?"
Response: "The capital of France is Paris."
Status: ✅ PASS
```
### ADKAgent (New - Now Working!)
```
Query: "What is the capital of France?"
Response: "I do not have access to real-time information..."
Status: ✅ WORKING (response quality can be improved)
Technical Status:
✅ Session creation
✅ Agent initialization
✅ Runner execution
✅ Event processing
✅ Response retrieval
```
---
## What Was Fixed
### Issue #1: Wrong Import Paths
**Problem:** Used non-existent `google.adk.llms.LiteLLM`
**Fix:** Changed to official API: `google.adk.models.lite_llm.LiteLlm`
### Issue #2: Wrong Execution Method
**Problem:** Tried to call `agent.run()` which doesn't exist
**Fix:** Used official pattern: `Runner.run_async()` with events
### Issue #3: Missing Session Management
**Problem:** ADK requires sessions but we didn't create them
**Fix:** Always create session before running agent
### Issue #4: Async/Await Issues
**Problem:** Forgot to `await` async session methods
**Fix:** Added `await` to all async calls
---
## Final Architecture (Phase 1)
```
┌─────────────────────────────────────────────────────────┐
│ Core-AI Service │
│ │
│ ┌──────────────────────┐ ┌───────────────────────┐ │
│ │ SimpleLiteLLMAgent │ │ ADKAgent │ │
│ │ (Simple Mode) │ │ (ADK Mode) │ │
│ │ │ │ │ │
│ │ • Direct LiteLLM │ │ • ADK Runtime │ │
│ │ • No tools │ │ • Runner + Sessions │ │
│ │ • Fast & lean │ │ • No tools (yet) │ │
│ └──────────┬───────────┘ └───────────┬───────────┘ │
│ │ │ │
│ └──────────┬───────────────┘ │
│ │ │
│ ┌──────▼──────┐ │
│ │ LiteLLM │ │
│ └──────┬──────┘ │
│ │ │
│ ┌──────▼──────┐ │
│ │ Ollama │ │
│ └──────┬──────┘ │
│ │ │
│ ┌──────▼──────┐ │
│ │Model (Gemma2)│ │
│ └─────────────┘ │
└─────────────────────────────────────────────────────────┘
```
---
## Code Quality Improvements
### Documentation
- ✅ Official ADK documentation references in code
- ✅ Clear docstrings explaining parameters and returns
- ✅ Logging at all critical steps
### Error Handling
- ✅ Try/catch blocks around ADK operations
- ✅ Graceful fallbacks when no response
- ✅ Detailed error logging with stack traces
### Structure
- ✅ Agents separated into `src/agents/` directory
- ✅ Simple and ADK agents isolated from each other
- ✅ Clean imports with availability checks
---
## Files Created/Modified
### New Files
- 📄 `src/agents/__init__.py` - Agent exports
- 📄 `src/agents/simple.py` - SimpleLiteLLMAgent (moved)
- 📄 `src/agents/adk_agent.py` - ADKAgent (new)
- 📄 `diagnostics/test_adk_direct.py` - ADK diagnostic tool
- 📄 `tests/test_06_adk_setup.py` - ADK test layer
- 📄 `ARCHITECTURE.md` - Dual-mode architecture docs
- 📄 `PHASE1_COMPLETE.md` - Initial completion doc
- 📄 `PHASE1_SUCCESS.md` - This file
### Modified Files
- ✏️ `src/config.py` - Added ADK settings
- ✏️ `src/prompts.py` - Added ADK prompt variant
- ✏️ `main.py` - Updated imports
---
## Known Limitations (Phase 1)
### Response Quality
The ADK agent's responses are sometimes overly cautious:
- Says "I don't have access to real-time information" for basic facts
- Could be improved with better system prompts
- Model choice (Gemma2) may need tuning for better knowledge recall
**This is a prompt engineering issue, not a technical issue.**
### No Tools Yet
- ADK agent has framework for tools but none registered
- Phase 2 will add REST-based tools
- Tool calling capability exists but untested
### No HTTP Endpoints Yet
- ADK agent only accessible via Python imports
- Phase 4 will add `/v1/chat/adk` endpoint
- Currently only testable via diagnostics
---
## Next Steps
### Immediate (Optional Improvement)
- [ ] Improve ADK system prompt for better responses
- [ ] Test with different models (mistral, etc.)
- [ ] Add more test cases to test_06
### Phase 2: Tool Integration
- [ ] Create `src/tools/registry.py`
- [ ] Implement REST-based tools (call core-api)
- [ ] Register tools with ADK agent
- [ ] Create `tests/test_07_adk_tools.py`
### Phase 3: ADK Agent with Tools
- [ ] Test tool calling with simple tools
- [ ] Test multi-tool workflows
- [ ] Create `tests/test_08_adk_agent.py` and `test_09_tool_calling.py`
### Phase 4: API Routes
- [ ] Add `/v1/chat/simple` endpoint
- [ ] Add `/v1/chat/adk` endpoint
- [ ] Maintain `/v1/chat/completions` as alias
- [ ] Create `tests/test_10_adk_api.py`
---
## How to Test
### Quick Test
```bash
docker exec core-ai python -c "
import asyncio
from src.agents import ADKAgent
async def test():
agent = ADKAgent(tools=[])
response = await agent.chat_completion(
messages=[{'role': 'user', 'content': 'What is 2+2?'}]
)
print(f'Response: {response}')
asyncio.run(test())
"
```
### Full Diagnostic
```bash
docker exec core-ai python diagnostics/test_adk_direct.py
```
### Test Suite
```bash
docker exec core-ai pytest tests/test_06_adk_setup.py -v -s
```
---
## Lessons Learned
1. **Always check official docs** - Core-API implementation was broken, official docs were correct
2. **ADK requires specific patterns** - Runner + Sessions + Events, not just agent.run()
3. **Async/await matters** - Forgetting `await` causes silent failures
4. **Session management is mandatory** - ADK won't work without valid sessions
5. **Response quality ≠ technical success** - Integration works even if responses need tuning
---
## Success Criteria Met
- [x] ADK imports successfully
- [x] ADKAgent class created and working
- [x] Agent initializes with Ollama/LiteLLM
- [x] Can process queries and return responses
- [x] Streaming mode works (simulated)
- [x] Non-streaming mode works
- [x] Session management works
- [x] Runner execution works
- [x] Event processing works
- [x] Diagnostic tool created
- [x] Test layer 6 created
- [x] All tests can run (response quality separate)
**Phase 1 Status:** ✅ **COMPLETE AND FUNCTIONAL**
---
## Resources Used
- [Google ADK Python Docs](https://google.github.io/adk-docs/get-started/python/)
- [LiteLLM + ADK Tutorial](https://docs.litellm.ai/docs/tutorials/google_adk)
- [Building Local AI Agent with ADK](https://medium.com/@viplav.fauzdar/building-a-local-ai-agent-with-google-adk-litellm-and-ollama-6e907e2db268)
- [Ollama-Powered AI Agents](https://medium.com/@jageenshukla/how-to-build-ollama-powered-ai-agents-with-adk-tool-calling-and-mcp-integration-c25d98fc4816)
---
**Last Updated:** 2025-11-27
**Next Phase:** Tool Integration (Phase 2)
**Recommendation:** Proceed to Phase 2 or improve prompts for better response quality
services/core-ai/PHASE2_COMPLETE.md
-337
View File
@@ -1,337 +0,0 @@
# Phase 2: Tool Integration - COMPLETE ✓
**Date:** 2025-11-27
**Status:** COMPLETE AND TESTED
---
## 🎉 Achievement
**Core-AI now has a complete tool system:**
1. ✅ **Local Tools** - Time, date, and calculator utilities
2. ✅ **Tool Registry** - Central management system for all tools
3. ✅ **Swagger Discovery** - Dynamic tool creation from OpenAPI specs
4. ✅ **ADK Integration** - Tools work seamlessly with ADK agent
---
## Architecture
### Tool System Design
```
┌─────────────────────────────────────────────────────────────────┐
│ Core-AI Tool System │
│ │
│ ┌──────────────────┐ ┌──────────────────────────┐ │
│ │ Local Tools │ │ REST Tool Discovery │ │
│ │ │ │ │ │
│ │ • get_current_ │ │ • Fetch OpenAPI spec │ │
│ │ time() │ │ • Parse endpoints │ │
│ │ • get_current_ │ │ • Create dynamic tools │ │
│ │ date() │ │ • Register with ADK │ │
│ │ • calculate() │ │ │ │
│ │ • date ops │ │ Source: core-api │ │
│ └────────┬─────────┘ └────────────┬─────────────┘ │
│ │ │ │
│ └──────────────┬───────────────────┘ │
│ │ │
│ ┌──────▼──────┐ │
│ │ Tool │ │
│ │ Registry │ │
│ └──────┬──────┘ │
│ │ │
│ ┌──────▼──────┐ │
│ │ ADK Agent │ │
│ │ with Tools │ │
│ └─────────────┘ │
└─────────────────────────────────────────────────────────────────┘
```
### Tool Flow
1. **Registration Phase:**
- Local tools register via `@register_tool` decorator
- REST tools discovered from core-api's OpenAPI spec
- All tools added to central registry
2. **Conversion Phase:**
- Registry converts Python functions to ADK `FunctionTool` objects
- Type annotations mapped to ADK schema
- Descriptions extracted from docstrings
3. **Execution Phase:**
- ADK agent receives tool list during initialization
- Agent can call tools to answer user queries
- Tool calls logged and results returned to agent
---
## Test Results
### Layer 7: Tool Integration Tests
```bash
docker exec core-ai pytest tests/test_07_adk_tools.py -v
```
**Results:** ✅ **7/7 PASSED**
| Test | Status | Description |
|------|--------|-------------|
| `test_local_tools_registered` | ✅ PASS | All 5 local tools registered |
| `test_local_tool_execution` | ✅ PASS | Tools execute correctly |
| `test_calculator_security` | ✅ PASS | Calculator blocks dangerous expressions |
| `test_adk_tool_conversion` | ✅ PASS | Tools convert to ADK format |
| `test_adk_agent_with_tools` | ✅ PASS | Agent initializes with tools |
| `test_tool_calling_integration` | ✅ PASS | Agent uses tools to answer queries |
| `test_date_tools` | ✅ PASS | Date manipulation tools work |
---
## What Was Implemented
### 1. Tool Registry System ✓
**File:** `src/tools/registry.py`
**Features:**
- Tool registration via `@register_tool` decorator
- Conversion to ADK `FunctionTool` format
- Logging decorator for all tool calls
- Dynamic REST tool creation from OpenAPI specs
**Key Functions:**
```python
@register_tool
async def my_tool(param: str) -> str:
"""Tool description"""
return result
# Get all registered tools
tools = get_all_tools()
# Get ADK-compatible tools
adk_tools = get_agent_tools()
# Discover tools from core-api
await discover_and_register_tools(base_url)
```
### 2. Local Tools ✓
**File:** `src/tools/local.py`
**Tools Implemented:**
- `get_current_time()` - Get current UTC time
- `get_current_date()` - Get current date
- `calculate(expression: str)` - Safe math calculator
- `add_days_to_date(date: str, days: int)` - Date arithmetic
- `calculate_date_difference(date1: str, date2: str)` - Date comparison
**Security:**
- Calculator blocks dangerous operations (`exec`, `eval`, `import`, etc.)
- Restricted eval namespace (no builtins)
- Input validation for all tools
### 3. Swagger/OpenAPI Discovery ✓
**File:** `src/tools/registry.py` (functions: `fetch_openapi_spec`, `create_rest_tool`, `discover_and_register_tools`)
**Features:**
- Fetch OpenAPI spec from multiple possible endpoints
- Parse paths and operations
- Extract parameters (path, query, body)
- Generate async functions that call REST endpoints
- Register dynamically created tools
**Usage:**
```python
# Discover and register all tools from core-api
tool_count = await discover_and_register_tools("http://core-api:8083")
print(f"Registered {tool_count} REST tools")
```
### 4. ADK Agent Integration ✓
**File:** `src/agents/adk_agent.py`
**Updates:**
- Added `discover_tools` parameter to `__init__`
- Automatic tool loading from registry
- Tools passed to ADK Agent constructor
**Usage:**
```python
# Agent with local tools only
agent = ADKAgent(discover_tools=True)
# Agent without tools
agent = ADKAgent(discover_tools=False)
# Agent with explicit tools
agent = ADKAgent(tools=[my_tool])
```
### 5. Test Suite ✓
**Files Created:**
- `tests/test_07_adk_tools.py` - Unit tests for tool system
- `diagnostics/test_adk_tools.py` - Diagnostic tool for testing
---
## Files Created/Modified
### New Files
- 📄 `src/tools/__init__.py` - Tool module exports
- 📄 `src/tools/registry.py` - Tool registration and discovery
- 📄 `src/tools/local.py` - Local utility tools
- 📄 `tests/test_07_adk_tools.py` - Tool layer tests
- 📄 `diagnostics/test_adk_tools.py` - Tool diagnostic
- 📄 `PHASE2_COMPLETE.md` - This file
### Modified Files
- ✏️ `src/agents/adk_agent.py` - Added tool discovery support
- ✏️ `stacks/core-ai.yml` - Removed obsolete `version` field
---
## Key Technical Decisions
### 1. Tool Architecture
**Decision:** Local tools in core-ai, REST tools in core-api
**Rationale:**
- Local tools (time, calc) don't need network calls
- REST tools from core-api enable separation of concerns
- Dynamic discovery means core-api can add tools without core-ai changes
### 2. Registry Pattern
**Decision:** Central registry with decorator-based registration
**Rationale:**
- Simple developer experience (`@register_tool`)
- Automatic discovery at import time
- Single source of truth for all tools
### 3. Security Model
**Decision:** Restricted eval for calculator, no default parameters
**Rationale:**
- ADK doesn't support default parameter values
- Calculator must block dangerous operations
- Whitelist approach for allowed functions
### 4. OpenAPI Discovery
**Decision:** Dynamic tool generation from Swagger docs
**Rationale:**
- Self-documenting API becomes self-registering tools
- No code duplication between API and tools
- Automatic parameter type mapping
---
## Known Limitations (Phase 2)
### No HTTP Endpoints Yet
- Tools only accessible via Python imports
- Phase 4 will add `/v1/chat/adk` endpoint
- Currently testable via diagnostics only
### Core-API Discovery Not Tested
- REST tool discovery implemented but not tested with real core-api
- Needs core-api to have OpenAPI documentation
- Will test in Phase 3 when core-api is ready
### Limited Tool Coverage
- Only 5 local tools implemented
- More tools can be added as needed
- REST tools depend on core-api implementation
---
## Testing
### Run All Tests
```bash
# Layer 7: Tool integration
docker exec core-ai pytest tests/test_07_adk_tools.py -v -s
# Full diagnostic
docker exec core-ai python diagnostics/test_adk_tools.py
# Quick test
docker exec core-ai python -c "
from src.tools.local import calculate
import asyncio
result = asyncio.run(calculate('2 + 2'))
print(f'Result: {result}')
"
```
### Example: Test Tool with Agent
```python
from src.agents import ADKAgent
import asyncio
async def test():
agent = ADKAgent(discover_tools=True)
response = await agent.chat_completion(
messages=[{"role": "user", "content": "What is 15 + 27? Use the calculator."}]
)
print(response)
asyncio.run(test())
```
---
## Next Steps
### Phase 3: Core-API Integration
1. Add OpenAPI documentation to core-api
2. Test REST tool discovery from core-ai
3. Verify tool calling works across services
4. Add authentication/authorization for tool endpoints
### Phase 4: API Routes
1. Add `/v1/chat/simple` endpoint (SimpleLiteLLMAgent)
2. Add `/v1/chat/adk` endpoint (ADKAgent with tools)
3. Keep `/v1/chat/completions` as default alias
4. Create test_10_adk_api.py for HTTP testing
### Optional Improvements
- Add more local tools (system info, file ops)
- Implement tool result caching
- Add tool execution timeout limits
- Implement tool permission system
---
## Success Criteria Met
- [x] Tool registry system created
- [x] Local tools implemented (time, date, calculator)
- [x] Tools registered automatically via decorator
- [x] ADK tool conversion working
- [x] Agent can use tools
- [x] OpenAPI/Swagger discovery implemented
- [x] Security measures in place (calculator safety)
- [x] Test suite created and passing (7/7)
- [x] Diagnostic tool created
- [x] Documentation complete
**Phase 2 Status:** ✅ **COMPLETE AND FULLY TESTED**
---
## Resources
- [Google ADK Tool Documentation](https://google.github.io/adk-docs/python/tools/)
- [OpenAPI Specification](https://swagger.io/specification/)
- [FastAPI OpenAPI Support](https://fastapi.tiangolo.com/how-to/extending-openapi/)
---
**Last Updated:** 2025-11-27
**Next Phase:** Core-API Integration (Phase 3) or API Routes (Phase 4)
**Recommendation:** Add OpenAPI docs to core-api, then test full tool integration
services/core-ai/PHASE4_COMPLETE.md
-424
View File
@@ -1,424 +0,0 @@
# Phase 4: API Routes - COMPLETE ✓
**Date:** 2025-11-27
**Status:** COMPLETE WITH KNOWN ISSUES
---
## 🎉 Achievement
**Core-AI now has complete HTTP API endpoints:**
1. ✅ `/v1/chat/completions` - Default endpoint (simple agent)
2. ✅ `/v1/chat/simple` - Explicit simple agent (no tools)
3. ✅ `/v1/chat/adk` - ADK agent with tools
4. ✅ `/v1/tools` - List all available tools
5. ✅ `/health` - Enhanced health check with agent status
**Test Results:** 7/8 tests passing (87.5% pass rate)
---
## API Endpoints
### POST /v1/chat/completions
**Description:** Default chat endpoint (uses SimpleLiteLLMAgent)
**Status:** ✅ Working
**Request:**
```json
{
"messages": [
{"role": "user", "content": "What is 2+2?"}
],
"stream": false
}
```
**Response:**
```json
{
"id": "chatcmpl-abc123",
"object": "chat.completion",
"created": 1701234567,
"model": "default_model",
"choices": [{
"index": 0,
"message": {"role": "assistant", "content": "4"},
"finish_reason": "stop"
}]
}
```
### POST /v1/chat/simple
**Description:** Explicit simple agent endpoint (no tools)
**Status:** ✅ Working
**Request:**
```json
{
"messages": [{"role": "user", "content": "Hello"}],
"stream": false
}
```
**Response:** Same format as `/v1/chat/completions` with `"model": "simple"`
### POST /v1/chat/adk
**Description:** ADK agent endpoint with tool support
**Status:** ⚠️ Working but tool execution needs improvement
**Request:**
```json
{
"messages": [{"role": "user", "content": "What is the current date?"}],
"stream": false,
"enable_tools": true
}
```
**Response:**
```json
{
"id": "chatcmpl-xyz789",
"object": "chat.completion",
"created": 1701234567,
"model": "adk",
"choices": [{
"index": 0,
"message": {"role": "assistant", "content": "..."},
"finish_reason": "stop"
}],
"tools_enabled": true,
"tools_count": 5
}
```
**Parameters:**
- `enable_tools` (boolean, default: true) - Enable/disable tool usage
- `stream` (boolean, default: false) - Enable streaming responses
### GET /v1/tools
**Description:** List all available tools
**Status:** ✅ Working
**Response:**
```json
{
"tools": [
{
"name": "get_current_time",
"description": "Get the current time in UTC timezone...",
"type": "local"
},
...
],
"count": 5,
"adk_available": true
}
```
### GET /health
**Description:** Enhanced health check
**Status:** ✅ Working
**Response:**
```json
{
"status": "ok",
"service": "core-ai",
"agents": {
"simple": true,
"adk": true
},
"tools_count": 5
}
```
---
## Test Results
### Layer 10: API Integration Tests
```bash
docker exec core-ai pytest tests/test_10_adk_api.py -v
```
**Results:** ✅ **7/8 PASSED** (87.5%)
| Test | Status | Description |
|------|--------|-------------|
| `test_health_check` | ✅ PASS | Health endpoint returns correct status |
| `test_list_tools` | ✅ PASS | Tools listing endpoint works |
| `test_chat_completions_simple` | ✅ PASS | Default endpoint works |
| `test_chat_simple_endpoint` | ✅ PASS | Simple agent endpoint works |
| `test_chat_adk_endpoint` | ❌ FAIL | ADK endpoint timeout (30s) |
| `test_chat_adk_with_calculator` | ✅ PASS | ADK with calculator works |
| `test_streaming_simple` | ✅ PASS | Streaming responses work |
| `test_adk_without_tools` | ✅ PASS | ADK without tools works |
---
## What Was Implemented
### 1. HTTP Endpoints ✓
**File:** `main.py`
**New Handlers:**
- `chat_simple()` - SimpleLiteLLMAgent endpoint
- `chat_adk()` - ADKAgent endpoint with tool support
- `list_tools()` - Tool listing endpoint
- Enhanced `health_check()` - Shows agent and tool status
**Features:**
- OpenAI-compatible response format
- Streaming and non-streaming support
- Tool enable/disable control
- Proper error handling and logging
- Request logging with agent identifiers
### 2. Test Suite ✓
**File:** `tests/test_10_adk_api.py`
**Tests Created:**
- Health check validation
- Tools listing validation
- Simple agent endpoint testing
- ADK agent endpoint testing
- Streaming response testing
- Tool execution testing
- Error handling testing
---
## Architecture
### Request Flow
```
┌─────────────────────────────────────────────────────────────┐
│ HTTP Client │
└────────────┬────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ aiohttp Server │
│ (main.py) │
│ │
│ ┌──────────────────┐ ┌──────────────────────────────┐ │
│ │ /v1/chat/ │ │ /v1/chat/adk │ │
│ │ completions │ │ │ │
│ │ /v1/chat/simple │ │ • enable_tools param │ │
│ │ │ │ • Tool discovery │ │
│ │ → Simple Agent │ │ → ADK Agent │ │
│ └──────────────────┘ └──────────────────────────────┘ │
│ │
│ ┌──────────────────┐ ┌──────────────────────────────┐ │
│ │ /v1/tools │ │ /health │ │
│ │ → List tools │ │ → Status check │ │
│ └──────────────────┘ └──────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
```
### Endpoint Comparison
| Feature | /v1/chat/simple | /v1/chat/adk |
|---------|----------------|--------------|
| Agent | SimpleLiteLLMAgent | ADKAgent |
| Tools | ❌ No | ✅ Yes (optional) |
| Performance | Fast | Slower (with tools) |
| Streaming | ✅ Yes | ✅ Yes |
| Use Case | Quick Q&A | Complex tasks with tools |
---
## Usage Examples
### Simple Query (No Tools)
```bash
curl -X POST http://localhost:8086/v1/chat/simple \
-H "Content-Type: application/json" \
-d '{
"messages": [{"role": "user", "content": "What is 2+2?"}],
"stream": false
}'
```
### ADK Query (With Tools)
```bash
curl -X POST http://localhost:8086/v1/chat/adk \
-H "Content-Type: application/json" \
-d '{
"messages": [{"role": "user", "content": "What is the current date?"}],
"stream": false,
"enable_tools": true
}'
```
### List Available Tools
```bash
curl -X GET http://localhost:8086/v1/tools
```
### Streaming Request
```bash
curl -N -X POST http://localhost:8086/v1/chat/simple \
-H "Content-Type: application/json" \
-d '{
"messages": [{"role": "user", "content": "Count to 5"}],
"stream": true
}'
```
---
## Known Issues & Limitations
### 1. ADK Tool Execution Timeout
**Issue:** `test_chat_adk_endpoint` times out after 30 seconds
**Impact:** Medium - ADK agent with tool discovery takes too long for some queries
**Symptoms:**
- Request times out waiting for response
- Happens when ADK tries to determine which tool to use
- Works fine when tools are disabled
**Possible Causes:**
- ADK runner processing all events before returning final response
- Tool call event handling incomplete
- Model taking too long to decide on tool usage
**Workaround:**
- Increase timeout to 60 seconds
- Disable tools for simple queries
- Use `/v1/chat/simple` for basic Q&A
**TODO:** Investigate ADK event loop and tool execution flow
### 2. Tool Call Format
**Issue:** ADK sometimes returns tool call JSON instead of executing tools
**Impact:** Low - Appears to be intermittent
**Symptoms:**
```json
{
"toolCalls": [{
"id": "call_xxx",
"type": "function",
"function": {"name": "get_current_date", "arguments": {}}
}]
}
```
**Possible Causes:**
- ADK runner not processing all events
- Breaking out of event loop too early
- Missing event type handling
**TODO:** Review adk_agent.py event processing logic
### 3. No REST Tool Discovery Yet
**Status:** Not implemented in this phase
**Impact:** Low - Phase 2 implemented the framework, Phase 3 will test it
**Next Steps:** Test with real core-api OpenAPI documentation
---
## Files Modified/Created
### Modified Files
- ✏️ `main.py` - Added 4 new endpoints and enhanced health check
### New Files
- 📄 `tests/test_10_adk_api.py` - API integration tests
- 📄 `PHASE4_COMPLETE.md` - This file
---
## Performance Metrics
### Response Times (Approximate)
- `/health`: < 50ms
- `/v1/tools`: < 100ms
- `/v1/chat/simple`: 1-5 seconds (depends on model)
- `/v1/chat/adk` (no tools): 2-8 seconds
- `/v1/chat/adk` (with tools): 5-30+ seconds
### Concurrent Requests
- Simple endpoint: Handles multiple concurrent requests well
- ADK endpoint: One request at a time recommended (caching helps)
---
## Next Steps
### Immediate Fixes
- [ ] Investigate and fix ADK tool execution timeout
- [ ] Improve ADK event processing to handle tool calls properly
- [ ] Add request timeout configuration
### Phase 5 (Future)
- [ ] Add authentication/authorization
- [ ] Add rate limiting
- [ ] Add request/response logging to database
- [ ] Add metrics/monitoring endpoints
- [ ] Implement conversation history persistence
### Phase 3 (Revisit)
- [ ] Test REST tool discovery with real core-api
- [ ] Add core-api OpenAPI documentation
- [ ] Verify cross-service tool calling
---
## Success Criteria
- [x] `/v1/chat/completions` endpoint working
- [x] `/v1/chat/simple` endpoint working
- [x] `/v1/chat/adk` endpoint working (with known issues)
- [x] `/v1/tools` endpoint working
- [x] Enhanced `/health` endpoint
- [x] Streaming support for all chat endpoints
- [x] OpenAI-compatible response format
- [x] Test suite created (8 tests)
- [x] 87.5% test pass rate (7/8 passing)
- [ ] 100% test pass rate (pending timeout fix)
**Phase 4 Status:** ✅ **COMPLETE WITH KNOWN ISSUES**
---
## Testing
### Quick Manual Tests
```bash
# Health check
curl -s http://localhost:8086/health | jq .
# List tools
curl -s http://localhost:8086/v1/tools | jq .tools[].name
# Simple chat
curl -s -X POST http://localhost:8086/v1/chat/simple \
-H "Content-Type: application/json" \
-d '{"messages": [{"role": "user", "content": "Hello"}], "stream": false}' \
| jq .choices[0].message.content
# ADK chat (no tools)
curl -s -X POST http://localhost:8086/v1/chat/adk \
-H "Content-Type: application/json" \
-d '{"messages": [{"role": "user", "content": "Hello"}], "stream": false, "enable_tools": false}' \
| jq .choices[0].message.content
```
### Full Test Suite
```bash
docker exec core-ai pytest tests/test_10_adk_api.py -v -s
```
---
**Last Updated:** 2025-11-27
**Next Phase:** Fix tool execution issues, then proceed to Phase 3 (Core-API integration)
**Recommendation:** Address ADK timeout issue before production use
services/core-ai/README.md
+430 -208
View File
@@ -1,23 +1,241 @@
# Core-AI Service
Simplified AI service for testing LiteLLM → Ollama → Model integration without ADK complexity.
## Purpose
This service strips away the ADK layer to isolate and debug the fundamental LiteLLM/Ollama integration. It provides:
- **Direct LiteLLM integration** - No ADK overhead
- **OpenAI-compatible API** - Drop-in replacement for testing
- **Comprehensive diagnostics** - Layered testing to identify issues
- **Minimal complexity** - Easy to understand and debug
AI agent service built on PydanticAI for infrastructure management and automation.
## Architecture
Core-AI provides two agents with distinct capabilities:
```
HTTP Request → SimpleLiteLLMAgent → LiteLLM → Ollama → Model → Response
┌─────────────────────────────────────────────────────────────┐
│ Core-AI Service │
├─────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────────┐ ┌──────────────────────┐ │
│ │ PydanticAgent │ │ SimpleLiteLLMAgent │ │
│ │ (Primary) │ │ (Fallback) │ │
│ │ │ │ │ │
│ │ • Tool calling │ │ • No tools │ │
│ │ • Memory (3-tier)│ │ • Direct LiteLLM │ │
│ │ • OpenAPI tools │ │ • Minimal overhead │ │
│ └────────┬─────────┘ └──────────┬───────────┘ │
│ │ │ │
│ └──────────┬───────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────┐ │
│ │ PydanticAI Runtime │ │
│ │ (Ollama backend) │ │
│ └──────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
```
**Bypassed:** Google ADK, tool calling, complex orchestration
### PydanticAgent (Primary)
**Endpoint:** `/v1/chat/completions` (default)
Advanced agent using the PydanticAI framework with:
- **Tool Calling:** Automatic function calling with proper validation
- **Memory System:** 3-tier conversation memory (buffer + Qdrant)
- **Local Tools:** Time, calculations, web search (SearXNG)
- **OpenAPI Tools:** Auto-discovered from core-api infrastructure endpoints
- **Streaming Support:** Server-sent events for real-time responses
### SimpleLiteLLMAgent (Fallback)
**Endpoint:** `/v1/chat/simple`
Lightweight agent for direct LLM interaction:
- **No Tools:** Pure conversational mode
- **Direct LiteLLM:** Minimal abstraction layer
- **No Memory:** Stateless request/response
- **Low Latency:** Fastest response times
## Tool System
### Local Tools
Built-in utilities available immediately (defined in `src/tools/local.py`):
- `get_current_time(timezone)` - Timezone-aware time with IANA timezone support
- `get_current_date()` - Current date in ISO format
- `calculate(expression)` - Safe mathematical calculations
- `calculate_date_difference(date1, date2)` - Date arithmetic
- `add_days_to_date(date, days)` - Date manipulation
- `web_search(query, category, max_results)` - SearXNG metasearch integration
### OpenAPI Discovery
Dynamically discovers infrastructure tools from core-api's OpenAPI spec:
- **Auto-Discovery:** Fetches `/openapi.json` on startup
- **REST Mapping:** Converts endpoints to callable functions
- **Prefixed Names:** Tools prefixed with service name (e.g., `core-api__list_containers`)
- **Type Safety:** Preserves parameter types and validation
**Configuration:**
```bash
OPENAPI_ENABLED=true
OPENAPI_ENDPOINTS=http://core-api:8083/openapi.json
```
**List Available Tools:**
```bash
curl http://localhost:8086/v1/tools
```
## Memory System
3-tier multi-tenant memory with per-user data isolation:
### Tier 1: Conversation Buffer (RAM)
- **Storage:** In-memory per-user buffers
- **Scope:** Recent N turns (configurable, default: 10)
- **Speed:** Instant access
- **Purpose:** Fast context for ongoing conversations
### Tier 2: Persistent Storage (Qdrant)
- **Storage:** Per-user Qdrant collections
- **Scope:** Complete conversation history
- **Speed:** Fast retrieval by conversation ID
- **Purpose:** Conversation continuity across sessions
### Tier 3: Semantic Search (Qdrant)
- **Storage:** Same as Tier 2 with vector embeddings
- **Scope:** Cross-conversation semantic search
- **Speed:** Sub-second similarity search
- **Purpose:** Contextual recall across all user conversations
### Multi-Tenancy
- **Per-User Collections:** Each user gets isolated Qdrant collection
- **User ID Format:** Sanitized email (`username_at_domain_com`)
- **GDPR Compliance:** Complete user data deletion support
- **Automatic Isolation:** No cross-user data leakage
**Memory Configuration:**
```bash
MEMORY_ENABLED=true
MEMORY_TIER1_SIZE=10
QDRANT_URL=http://qdrant:6333
EMBEDDING_MODEL=nomic-embed-text
DEFAULT_USER_ID=llmdefault_at_schweitz_net
```
## API Endpoints
### Chat Completions
**POST /v1/chat/completions** (Default: PydanticAI)
OpenAI-compatible chat endpoint using PydanticAgent.
**Request:**
```json
{
"messages": [
{"role": "user", "content": "What containers are running?"}
],
"conversation_id": "optional-conversation-id",
"enable_tools": true,
"stream": false
}
```
**Response:**
```json
{
"choices": [{
"index": 0,
"message": {
"role": "assistant",
"content": "I found 5 running containers..."
},
"finish_reason": "stop"
}],
"model": "pydantic",
"tools_enabled": true,
"tools_count": 12
}
```
**Streaming:** Set `"stream": true` for SSE response
### Simple Chat
**POST /v1/chat/simple**
No-tools fallback endpoint using SimpleLiteLLMAgent.
Same request/response format as above, but `tools_enabled` will be `false`.
### List Models
**GET /v1/models**
Returns available agent types:
- `pydantic` - PydanticAgent (primary)
- `simple` - SimpleLiteLLMAgent (fallback)
### List Tools
**GET /v1/tools**
Returns all available tools (local + discovered OpenAPI tools).
### Health Check
**GET /health**
Service health status with agent availability.
## Configuration
All configuration via environment variables (see `src/config.py`):
### Core Settings
| Variable | Default | Description |
|----------|---------|-------------|
| `HOST` | `0.0.0.0` | Server host |
| `PORT` | `8086` | Server port |
| `LOG_LEVEL` | `INFO` | Logging level |
### Ollama Integration
| Variable | Default | Description |
|----------|---------|-------------|
| `OLLAMA_BASE_URL` | `http://ollama:11434` | Ollama API URL |
| `AGENT_MODEL` | `mistral-nemo:latest` | Primary model (tool-calling optimized) |
| `OLLAMA_TIMEOUT` | `300` | Request timeout (seconds) |
### System Prompts
| Variable | Default | Description |
|----------|---------|-------------|
| `SYSTEM_PROMPT_VARIANT` | `minimal_agent` | Prompt for SimpleLiteLLMAgent |
| `PYDANTIC_SYSTEM_PROMPT_VARIANT` | `pydantic_agent` | Prompt for PydanticAgent |
### Tool Discovery
| Variable | Default | Description |
|----------|---------|-------------|
| `OPENAPI_ENABLED` | `true` | Enable OpenAPI tool discovery |
| `OPENAPI_ENDPOINTS` | `http://core-api:8083/openapi.json` | OpenAPI spec URLs (comma-separated) |
### Memory System
| Variable | Default | Description |
|----------|---------|-------------|
| `MEMORY_ENABLED` | `true` | Enable conversation memory |
| `MEMORY_TIER1_SIZE` | `10` | Max turns in RAM buffer |
| `QDRANT_URL` | `http://qdrant:6333` | Qdrant vector DB URL |
| `QDRANT_COLLECTION_PREFIX` | `core_ai_user` | Prefix for user collections |
| `EMBEDDING_MODEL` | `nomic-embed-text` | Ollama embedding model |
| `EMBEDDING_DIMENSION` | `768` | Embedding vector size |
| `DEFAULT_USER_ID` | `llmdefault_at_schweitz_net` | Default user (until auth integration) |
## Quick Start
@@ -29,158 +247,83 @@ pip install -r requirements.txt
### 2. Configure Environment
Create `.env` file or set environment variables:
Create `.env` file:
```bash
OLLAMA_BASE_URL=http://ollama:11434
AGENT_MODEL=gemma2:9b-instruct-q5_K_M
SYSTEM_PROMPT_VARIANT=minimal_agent
HOST=0.0.0.0
PORT=8086
AGENT_MODEL=mistral-nemo:latest
QDRANT_URL=http://qdrant:6333
MEMORY_ENABLED=true
OPENAPI_ENABLED=true
```
### 3. Run Diagnostics
```bash
# Check Ollama connectivity
python diagnostics/check_ollama.py
# Test direct LiteLLM
python diagnostics/test_litellm_direct.py
# Run full test suite
bash tests/run_all_tests.sh
```
### 4. Start Service
### 3. Start Service
```bash
python main.py
```
Service will be available at `http://localhost:8086`
Service available at `http://localhost:8086`
### 5. Test It
### 4. Test Chat
```bash
curl -X POST http://localhost:8086/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"messages": [
{"role": "user", "content": "What is the capital of France?"}
]
{"role": "user", "content": "What time is it in Amsterdam?"}
],
"enable_tools": true
}'
```
## API Endpoints
### `GET /health`
Health check endpoint
**Response:**
```json
{
"status": "ok",
"service": "core-ai"
}
```
### `POST /v1/chat/completions`
OpenAI-compatible chat completions endpoint
**Request:**
```json
{
"model": "test",
"messages": [
{"role": "user", "content": "Your question here"}
],
"stream": false
}
```
**Response (non-streaming):**
```json
{
"id": "chatcmpl-...",
"object": "chat.completion",
"created": 1234567890,
"model": "test",
"choices": [{
"index": 0,
"message": {
"role": "assistant",
"content": "Response here"
},
"finish_reason": "stop"
}],
"usage": {
"prompt_tokens": 0,
"completion_tokens": 0,
"total_tokens": 0
}
}
```
**Streaming:** Set `"stream": true` for Server-Sent Events response
## Project Structure
```
services/core-ai/
├── main.py # HTTP server (aiohttp)
├── src/
│ ├── agent.py # SimpleLiteLLMAgent
│ ├── config.py # Configuration (Pydantic)
│ ├── prompts.py # System prompts
│ └── tools.py # (Unused in this version)
├── diagnostics/
│ ├── check_ollama.py # Ollama connectivity check
│ └── test_litellm_direct.py # Direct LiteLLM test
├── tests/
│ ├── test_01_environment.py # Config tests
│ ├── test_02_litellm_raw.py # Raw LiteLLM tests
│ ├── test_03_message_format.py # Message formatting
│ ├── test_04_agent.py # Agent logic tests
│ ├── test_05_api.py # API endpoint tests
│ ├── run_all_tests.sh # Run all tests
│ └── README.md # Test documentation
├── requirements.txt
├── Dockerfile
└── README.md (this file)
```
## Configuration
Configuration is managed via `src/config.py` using Pydantic Settings.
### Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `HOST` | `0.0.0.0` | Server host |
| `PORT` | `8086` | Server port |
| `OLLAMA_BASE_URL` | `http://ollama:11434` | Ollama API URL |
| `AGENT_MODEL` | `gemma2:9b-instruct-q5_K_M` | Model name |
| `SYSTEM_PROMPT_VARIANT` | `minimal_agent` | Prompt variant to use |
| `DEBUG` | `false` | Enable debug mode |
| `LOG_LEVEL` | `INFO` | Logging level |
The agent will automatically use the `get_current_time` tool.
## Testing
See [tests/README.md](tests/README.md) for comprehensive testing documentation.
### Unit Tests
**Quick test:**
```bash
bash tests/run_all_tests.sh
# Run all tests
pytest tests/ -v
# Run specific test suite
pytest tests/test_ai_flow_quality.py -v
# Run with coverage
pytest tests/ --cov=src --cov-report=html
```
This runs 5 layers of tests to isolate issues:
1. Environment & Configuration
2. Raw LiteLLM Connection
3. Message Formatting
4. Agent Logic
5. API Integration
### Integration Tests
Quality tests for end-to-end AI flows:
```bash
pytest tests/test_ai_flow_quality.py -v
```
See `tests/QUALITY_TESTS.md` for test documentation.
### Manual Testing
```bash
# Test PydanticAgent (with tools)
curl -X POST http://localhost:8086/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"messages": [{"role": "user", "content": "Calculate 123 * 456"}]}'
# Test SimpleLiteLLMAgent (no tools)
curl -X POST http://localhost:8086/v1/chat/simple \
-H 'Content-Type: application/json' \
-d '{"messages": [{"role": "user", "content": "Hello!"}]}'
# List available tools
curl http://localhost:8086/v1/tools
# Health check
curl http://localhost:8086/health
```
## Docker Deployment
@@ -197,7 +340,8 @@ docker run -d \
--name core-ai \
-p 8086:8086 \
-e OLLAMA_BASE_URL=http://ollama:11434 \
-e AGENT_MODEL=gemma2:9b-instruct-q5_K_M \
-e QDRANT_URL=http://qdrant:6333 \
-e AGENT_MODEL=mistral-nemo:latest \
--network docker-dataplane \
core-ai:latest
```
@@ -208,106 +352,184 @@ docker run -d \
docker-compose -f ../../stacks/core-ai.yml up
```
## Troubleshooting
## Project Structure
### Service won't start
1. Check logs: `docker logs core-ai`
2. Verify Ollama is running: `docker ps | grep ollama`
3. Run diagnostics: `python diagnostics/check_ollama.py`
### No response or timeout
1. Check Ollama logs: `docker logs ollama`
2. Model may be loading (first run takes 30-60s)
3. Verify model exists: `docker exec ollama ollama list`
4. Test directly: `docker exec ollama ollama run gemma2:9b-instruct-q5_K_M "test"`
### Wrong or empty responses
1. Check system prompt is loaded (see agent logs)
2. Verify prompt variant exists in `src/prompts.py`
3. Run Layer 3 tests: `pytest tests/test_03_message_format.py -v`
### Connection refused
1. Check network: `docker network inspect docker-dataplane`
2. Verify both services are on the same network
3. Try using container IP instead of hostname
```
services/core-ai/
├── main.py # HTTP server (aiohttp)
├── src/
│ ├── agents/
│ │ ├── __init__.py # Agent exports
│ │ ├── pydantic_agent.py # PydanticAgent (primary)
│ │ └── simple.py # SimpleLiteLLMAgent (fallback)
│ ├── memory/
│ │ ├── manager.py # Multi-tenant memory manager
│ │ ├── tier1_buffer.py # RAM conversation buffer
│ │ ├── qdrant_memory.py # Qdrant persistent + semantic
│ │ ├── base.py # Base memory interfaces
│ │ └── schemas.py # Memory data schemas
│ ├── tools/
│ │ ├── local.py # Local utility tools
│ │ ├── openapi_discovery.py # OpenAPI tool discovery
│ │ └── registry.py # Tool registration system
│ ├── config.py # Configuration (Pydantic Settings)
│ ├── prompts.py # System prompts
│ └── utils.py # Utilities
├── tests/
│ ├── test_ai_flow_quality.py # End-to-end AI quality tests
│ └── QUALITY_TESTS.md # Test documentation
├── requirements.txt
├── Dockerfile
└── README.md (this file)
```
## Development
### Adding New Prompts
### Adding Local Tools
Edit `src/tools/local.py`:
```python
from src.tools.registry import register_tool
@register_tool
async def my_new_tool(param: str) -> str:
"""
Tool description for LLM.
Args:
param: Parameter description
Returns:
Result description
"""
# Implementation
return f"Result: {param}"
```
Tool automatically available to PydanticAgent.
### Adding OpenAPI Sources
Add endpoints to configuration:
```bash
OPENAPI_ENDPOINTS=http://core-api:8083/openapi.json,http://automation:8080/openapi.json
```
Tools auto-discovered on startup with service prefix:
- `core-api__list_containers`
- `automation__deploy_stack`
### Modifying System Prompts
Edit `src/prompts.py`:
```python
PROMPTS = {
"minimal_agent": "You are a helpful assistant.",
"my_new_prompt": "Your custom system prompt here."
"pydantic_agent": "Your custom PydanticAgent prompt...",
"minimal_agent": "Your custom SimpleLiteLLMAgent prompt..."
}
```
Update environment variable:
Update environment:
```bash
SYSTEM_PROMPT_VARIANT=my_new_prompt
PYDANTIC_SYSTEM_PROMPT_VARIANT=pydantic_agent
```
### Modifying Agent Behavior
### Memory System Usage
Edit `src/agent.py` - specifically the `SimpleLiteLLMAgent` class.
Memory automatically managed per user:
**Key methods:**
- `__init__()` - Initialization and configuration
- `chat()` - Streaming chat handler
- `chat_completion()` - Non-streaming completion handler
```python
from src.memory import get_memory_manager_for_user
### Adding Tests
# Get user's memory manager
memory = get_memory_manager_for_user(user_id="user_at_example_com")
Add to appropriate test layer in `tests/`:
- Configuration changes → `test_01_environment.py`
- LiteLLM behavior → `test_02_litellm_raw.py`
- Message formatting → `test_03_message_format.py`
- Agent logic → `test_04_agent.py`
- API changes → `test_05_api.py`
# Memory automatically used by PydanticAgent when conversation_id provided
# See: src/agents/pydantic_agent.py
```
## Comparison with Core-API
## Troubleshooting
| Feature | Core-AI | Core-API |
|---------|---------|----------|
| **ADK Integration** | ❌ No | ✅ Yes |
| **Tool Calling** | ❌ No | ✅ Yes |
| **System Orchestration** | ❌ No | ✅ Yes |
| **Complexity** | Low | High |
| **Purpose** | Debugging | Production |
| **Direct LiteLLM** | ✅ Yes | ❌ No |
| **Diagnostics** | ✅ Comprehensive | Limited |
### PydanticAI Not Available
## Next Steps
**Error:** `PydanticAI not available. Install with: pip install pydantic-ai`
### If Tests Pass
**Solution:**
```bash
pip install pydantic-ai
```
1. ✅ Foundation is solid
2. Consider migrating fixes to core-api
3. Add ADK layer back in phases
4. Test tool calling integration
### Tools Not Discovered
### If Tests Fail
**Issue:** `/v1/tools` returns empty list or only local tools
1. Run diagnostics to identify layer
2. Fix that specific layer
3. Re-run tests
4. Proceed once all pass
**Check:**
1. Verify `OPENAPI_ENABLED=true`
2. Check core-api is running: `curl http://core-api:8083/openapi.json`
3. Review logs for discovery errors: `docker logs core-ai`
### Memory Errors
**Issue:** Memory operations failing
**Check:**
1. Verify Qdrant running: `curl http://qdrant:6333/collections`
2. Check embedding model available: `docker exec ollama ollama list | grep nomic-embed-text`
3. Review logs for initialization errors
### Model Timeouts
**Issue:** Requests timing out
**Solutions:**
1. Increase timeout: `OLLAMA_TIMEOUT=600`
2. Use smaller model: `AGENT_MODEL=mistral-tools:7b`
3. Check GPU access: `docker exec ollama nvidia-smi`
### Tool Calling Failures
**Issue:** Agent not using tools correctly
**Check:**
1. Verify model supports tool calling: `mistral-nemo`, `mistral-tools:7b`
2. Test with `enable_tools=false` to isolate issue
3. Review tool logs: Look for `🔧 TOOL CALL:` in logs
## Model Recommendations
### For Tool Calling (PydanticAgent)
- **mistral-nemo:latest** (default) - Best balance
- **mistral-tools:7b** - Faster, less accurate
- **llama3.1:8b** - Good alternative
### For Simple Chat (SimpleLiteLLMAgent)
- **gemma2:9b** - Fast conversational
- **llama3.2:3b** - Minimal resources
- Any model works (no tool calling required)
## Migration Notes
This service has migrated from:
- **ADK (Agent Development Kit)** → PydanticAI
- **LangChain/LangGraph** → PydanticAI native
- **OllamaNativeAgent** → Removed (superseded by PydanticAgent)
All references to these frameworks have been removed. The codebase now exclusively uses PydanticAI for agent orchestration.
## Contributing
When making changes:
1. Run diagnostics first
2. Make changes
3. Run full test suite
4. Update relevant documentation
5. Test in Docker environment
1. Add tests in `tests/`
2. Update docstrings
3. Test with both agents (`/v1/chat/completions` and `/v1/chat/simple`)
4. Verify tool discovery works
5. Test memory persistence
## License
Part of the tower-of-joy project.
Part of the portainer-core project.
services/core-ai/diagnostics/test_litellm_direct.py
-144
View File
@@ -1,144 +0,0 @@
#!/usr/bin/env python3
"""
Diagnostic tool to test direct LiteLLM → Ollama communication.
This bypasses all abstractions and tests the raw integration.
Usage:
python diagnostics/test_litellm_direct.py
"""
import asyncio
import sys
import os
from pathlib import Path
# Add parent directory to path to import from src
sys.path.insert(0, str(Path(__file__).parent.parent))
from src.config import get_settings
# Import LiteLLM
try:
import litellm
litellm.set_verbose = True
except ImportError:
print("✗ LiteLLM not installed. Run: pip install litellm")
sys.exit(1)
async def test_litellm_direct():
"""Test direct LiteLLM completion with Ollama"""
settings = get_settings()
print("=" * 70)
print("LITELLM DIRECT TEST")
print("=" * 70)
# Test configuration
model_name = settings.agent_model
litellm_model = f"ollama/{model_name}"
api_base = settings.ollama_base_url
print(f"\n1. Configuration")
print(f" LiteLLM Model: {litellm_model}")
print(f" API Base: {api_base}")
print(f" Temperature: 0.1")
# Test messages
test_cases = [
{
"name": "Simple question (no system prompt)",
"messages": [
{"role": "user", "content": "What is the capital of France? Answer in one word."}
]
},
{
"name": "Simple question (with system prompt)",
"messages": [
{"role": "system", "content": "You are a helpful assistant. Answer questions concisely."},
{"role": "user", "content": "What is the capital of France? Answer in one word."}
]
},
{
"name": "Math problem",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is 2 + 2? Answer with just the number."}
]
}
]
# Run tests
for i, test_case in enumerate(test_cases, 1):
print(f"\n{'-' * 70}")
print(f"Test {i}/{len(test_cases)}: {test_case['name']}")
print(f"{'-' * 70}")
# Log messages being sent
print("\nMessages being sent:")
for j, msg in enumerate(test_case['messages']):
content_preview = msg['content'][:60] + "..." if len(msg['content']) > 60 else msg['content']
print(f" [{j}] {msg['role']}: {content_preview}")
try:
# Test non-streaming first
print("\n→ Testing non-streaming mode...")
response = await litellm.acompletion(
model=litellm_model,
messages=test_case['messages'],
api_base=api_base,
temperature=0.1,
stream=False
)
content = response.choices[0].message.content
finish_reason = response.choices[0].finish_reason
print(f"\n✓ Non-streaming response received:")
print(f" Content: {content}")
print(f" Finish reason: {finish_reason}")
print(f" Model: {response.model}")
# Test streaming
print("\n→ Testing streaming mode...")
stream_response = await litellm.acompletion(
model=litellm_model,
messages=test_case['messages'],
api_base=api_base,
temperature=0.1,
stream=True
)
chunks = []
chunk_count = 0
async for chunk in stream_response:
chunk_count += 1
if chunk.choices[0].delta.content:
chunks.append(chunk.choices[0].delta.content)
full_content = "".join(chunks)
print(f"\n✓ Streaming response received:")
print(f" Content: {full_content}")
print(f" Chunks: {chunk_count}")
print(f"\n✓ Test {i} PASSED")
except Exception as e:
print(f"\n✗ Test {i} FAILED")
print(f" Error: {type(e).__name__}: {e}")
import traceback
traceback.print_exc()
return False
print("\n" + "=" * 70)
print("✓ ALL LITELLM TESTS PASSED!")
print("=" * 70)
print("\nNext steps:")
print(" 1. If this works, the LiteLLM → Ollama connection is solid")
print(" 2. Any issues are likely in the agent wrapper or API layer")
print(" 3. Run the full test suite: bash diagnostics/run_all_tests.sh")
return True
if __name__ == "__main__":
result = asyncio.run(test_litellm_direct())
sys.exit(0 if result else 1)
services/core-ai/main.py
+178 -331
View File
@@ -1,7 +1,7 @@
import os
import logging
import json
import time # Import time module
import time
from aiohttp import web
from aiohttp_cors import setup as cors_setup, ResourceOptions
from dotenv import load_dotenv
@@ -16,8 +16,6 @@ logger = logging.getLogger(__name__)
# Import the agent logic
from src.agents import (
get_simple_litellm_agent,
get_ollama_native_agent,
OLLAMA_NATIVE_AVAILABLE,
get_pydantic_agent,
PYDANTIC_AI_AVAILABLE
)
@@ -26,182 +24,8 @@ from src.utils import extract_user_id_from_request
async def chat_completions(request):
"""
Handles OpenAI-compatible chat completion requests using Ollama Native agent.
Default endpoint - uses Ollama Native Agent with tools enabled.
"""
if not OLLAMA_NATIVE_AVAILABLE:
return web.json_response({
"error": {"message": "Ollama Native agent not available"}
}, status=503)
try:
data = await request.json()
logger.info(f"[DEFAULT/OLLAMA_NATIVE] Received chat request")
# Extract relevant fields from the request
messages = data.get("messages")
model = data.get("model", "ollama-native")
stream = data.get("stream", False)
conversation_id = data.get("conversation_id")
enable_tools = data.get("enable_tools", True) # Tools enabled by default
if not messages:
raise web.HTTPBadRequest(reason="'messages' field is required")
# Get the agent instance (Ollama Native with working tool calling)
agent = get_ollama_native_agent(discover_tools=enable_tools)
# For non-streaming requests, collect the full response
if not stream:
response_content = await agent.chat_completion(
messages=messages,
conversation_id=conversation_id
)
return web.json_response({
"id": f"chatcmpl-{os.urandom(12).hex()}",
"object": "chat.completion",
"created": int(time.time()),
"model": "pydantic",
"choices": [{
"index": 0,
"message": {"role": "assistant", "content": response_content},
"finish_reason": "stop"
}],
"usage": {
"prompt_tokens": 0,
"completion_tokens": 0,
"total_tokens": 0
},
"tools_enabled": enable_tools,
"tools_count": len(agent.tools_dict) if enable_tools else 0
})
else:
# Handle streaming response
response = web.StreamResponse(
status=200,
headers={'Content-Type': 'text/event-stream', 'Cache-Control': 'no-cache', 'Connection': 'keep-alive'}
)
await response.prepare(request)
async for chunk in agent.chat(messages=messages, conversation_id=conversation_id, stream=True):
chunk_type = chunk.get("type", "content")
if chunk_type == "content":
json_chunk = {
"id": f"chatcmpl-{os.urandom(12).hex()}",
"object": "chat.completion.chunk",
"created": int(time.time()),
"model": "pydantic",
"choices": [{
"index": 0,
"delta": {"content": chunk.get("content", "")},
"finish_reason": chunk.get("finish_reason")
}]
}
await response.write(f"data: {json.dumps(json_chunk)}\n\n".encode())
if chunk.get("finish_reason") == "stop":
break
elif chunk_type == "error":
error_chunk = {
"error": {"message": chunk.get("content", "Unknown error")}
}
await response.write(f"data: {json.dumps(error_chunk)}\n\n".encode())
break
await response.write(b"data: [DONE]\n\n")
await response.write_eof()
return response
except web.HTTPBadRequest as e:
logger.warning(f"Bad request: {e.reason}")
return web.json_response({"error": {"message": e.reason}}, status=400)
except Exception as e:
logger.exception("[DEFAULT/PYDANTIC_AI] Error during chat completion:")
return web.json_response({"error": {"message": str(e)}}, status=500)
async def chat_simple(request):
"""
Handles chat requests using SimpleLiteLLMAgent (no tools).
Endpoint: POST /v1/chat/simple
"""
try:
data = await request.json()
logger.info(f"[SIMPLE] Received chat request")
messages = data.get("messages")
model = data.get("model", "simple")
stream = data.get("stream", False)
conversation_id = data.get("conversation_id")
if not messages:
raise web.HTTPBadRequest(reason="'messages' field is required")
# Get SimpleLiteLLM agent
agent = get_simple_litellm_agent()
# Non-streaming response
if not stream:
response_content = await agent.chat_completion(
messages=messages,
conversation_id=conversation_id
)
return web.json_response({
"id": f"chatcmpl-{os.urandom(12).hex()}",
"object": "chat.completion",
"created": int(time.time()),
"model": "simple",
"choices": [{
"index": 0,
"message": {"role": "assistant", "content": response_content},
"finish_reason": "stop"
}],
"usage": {
"prompt_tokens": 0,
"completion_tokens": 0,
"total_tokens": 0
}
})
else:
# Streaming response
response = web.StreamResponse(
status=200,
headers={'Content-Type': 'text/event-stream', 'Cache-Control': 'no-cache', 'Connection': 'keep-alive'}
)
await response.prepare(request)
async for chunk in agent.chat(messages=messages, conversation_id=conversation_id, stream=True):
json_chunk = {
"id": f"chatcmpl-{os.urandom(12).hex()}",
"object": "chat.completion.chunk",
"created": int(time.time()),
"model": "simple",
"choices": [{
"index": 0,
"delta": {"content": chunk.get("content", "")},
"finish_reason": chunk.get("finish_reason")
}]
}
await response.write(f"data: {json.dumps(json_chunk)}\n\n".encode())
if chunk.get("finish_reason") == "stop":
break
await response.write(b"data: [DONE]\n\n")
await response.write_eof()
return response
except web.HTTPBadRequest as e:
logger.warning(f"Bad request: {e.reason}")
return web.json_response({"error": {"message": e.reason}}, status=400)
except Exception as e:
logger.exception("[SIMPLE] Error during chat completion:")
return web.json_response({"error": {"message": str(e)}}, status=500)
async def chat_pydantic(request):
"""
Handles chat requests using PydanticAI Agent with tools.
Endpoint: POST /v1/chat/pydantic
Handles OpenAI-compatible chat completion requests using PydanticAI.
Default endpoint - uses PydanticAI with tools enabled.
"""
if not PYDANTIC_AI_AVAILABLE:
return web.json_response({
@@ -210,8 +34,9 @@ async def chat_pydantic(request):
try:
data = await request.json()
logger.info(f"[PYDANTIC_AI] Received chat request")
logger.info(f"[DEFAULT/PYDANTIC_AI] Received chat request")
# Extract relevant fields from the request
messages = data.get("messages")
model = data.get("model", "pydantic")
stream = data.get("stream", False)
@@ -224,25 +49,25 @@ async def chat_pydantic(request):
if not messages:
raise web.HTTPBadRequest(reason="'messages' field is required")
# Get PydanticAI agent with or without tools
# Get the agent instance
agent = get_pydantic_agent(discover_tools=enable_tools, user_id=user_id)
# Non-streaming response
# For non-streaming requests, collect the full response
if not stream:
response_content = await agent.chat_completion(
messages=messages,
conversation_id=conversation_id
)
return web.json_response({
"id": f"chatcmpl-{os.urandom(12).hex()}",
"object": "chat.completion",
"created": int(time.time()),
"model": "pydantic",
"choices": [{
"index": 0,
"message": {"role": "assistant", "content": response_content},
"message": {
"role": "assistant",
"content": response_content
},
"finish_reason": "stop"
}],
"model": model,
"usage": {
"prompt_tokens": 0,
"completion_tokens": 0,
@@ -252,182 +77,203 @@ async def chat_pydantic(request):
"tools_count": len(agent.tools_dict) if enable_tools else 0
})
else:
# Streaming response
# Handle streaming response
response = web.StreamResponse(
status=200,
headers={'Content-Type': 'text/event-stream', 'Cache-Control': 'no-cache', 'Connection': 'keep-alive'}
reason='OK',
headers={
'Content-Type': 'text/event-stream',
'Cache-Control': 'no-cache',
'Connection': 'keep-alive',
}
)
await response.prepare(request)
async for chunk in agent.chat(messages=messages, conversation_id=conversation_id, stream=True):
chunk_type = chunk.get("type", "content")
try:
async for chunk in agent.chat(messages=messages, conversation_id=conversation_id, stream=True):
if chunk["type"] == "content":
chunk_data = {
"choices": [{
"index": 0,
"delta": {"content": chunk["content"]},
"finish_reason": chunk.get("finish_reason")
}],
"model": model
}
await response.write(f"data: {json.dumps(chunk_data)}\n\n".encode('utf-8'))
if chunk_type == "content":
json_chunk = {
"id": f"chatcmpl-{os.urandom(12).hex()}",
"object": "chat.completion.chunk",
"created": int(time.time()),
"model": "pydantic",
"choices": [{
"index": 0,
"delta": {"content": chunk.get("content", "")},
"finish_reason": chunk.get("finish_reason")
}]
}
await response.write(f"data: {json.dumps(json_chunk)}\n\n".encode())
await response.write(b"data: [DONE]\n\n")
finally:
await response.write_eof()
if chunk.get("finish_reason") == "stop":
break
elif chunk_type == "error":
error_chunk = {
"error": {"message": chunk.get("content", "Unknown error")}
}
await response.write(f"data: {json.dumps(error_chunk)}\n\n".encode())
break
await response.write(b"data: [DONE]\n\n")
await response.write_eof()
return response
except web.HTTPBadRequest as e:
logger.warning(f"Bad request: {e.reason}")
return web.json_response({"error": {"message": e.reason}}, status=400)
except web.HTTPBadRequest:
raise
except Exception as e:
logger.exception("[PYDANTIC_AI] Error during chat completion:")
return web.json_response({"error": {"message": str(e)}}, status=500)
logger.exception(f"Error in chat_completions: {e}")
return web.json_response({
"error": {"message": f"Internal server error: {str(e)}"}
}, status=500)
async def list_models(request):
async def chat_simple(request):
"""
Lists available models (OpenAI-compatible endpoint).
Endpoint: GET /v1/models
"""
models = [
{
"id": "Tatlock",
"object": "model",
"created": int(time.time()),
"owned_by": "core-ai",
"permission": [],
"root": "tatlock",
"parent": None,
},
{
"id": "simple",
"object": "model",
"created": int(time.time()),
"owned_by": "core-ai",
"permission": [],
"root": "simple",
"parent": None,
}
]
return web.json_response({
"object": "list",
"data": models
})
async def list_tools(request):
"""
Lists all available tools.
Endpoint: GET /v1/tools
Handles chat requests using SimpleLiteLLMAgent (fallback, no tools).
Endpoint: /v1/chat/simple
"""
try:
data = await request.json()
logger.info(f"[SIMPLE/LITELLM] Received chat request")
# Extract relevant fields from the request
messages = data.get("messages")
model = data.get("model", "simple")
stream = data.get("stream", False)
conversation_id = data.get("conversation_id")
if not messages:
raise web.HTTPBadRequest(reason="'messages' field is required")
# Get simple agent instance (no tools)
agent = get_simple_litellm_agent()
# For non-streaming requests
if not stream:
response_content = await agent.chat_completion(
messages=messages,
conversation_id=conversation_id
)
return web.json_response({
"choices": [{
"index": 0,
"message": {
"role": "assistant",
"content": response_content
},
"finish_reason": "stop"
}],
"model": model,
"usage": {
"prompt_tokens": 0,
"completion_tokens": 0,
"total_tokens": 0
}
})
else:
# Streaming response
response = web.StreamResponse(
status=200,
reason='OK',
headers={
'Content-Type': 'text/event-stream',
'Cache-Control': 'no-cache',
'Connection': 'keep-alive',
}
)
await response.prepare(request)
try:
async for chunk in agent.chat(messages=messages, conversation_id=conversation_id, stream=True):
if chunk["type"] == "content":
chunk_data = {
"choices": [{
"index": 0,
"delta": {"content": chunk["content"]},
"finish_reason": chunk.get("finish_reason")
}],
"model": model
}
await response.write(f"data: {json.dumps(chunk_data)}\n\n".encode('utf-8'))
await response.write(b"data: [DONE]\n\n")
finally:
await response.write_eof()
return response
except web.HTTPBadRequest:
raise
except Exception as e:
logger.exception(f"Error in chat_simple: {e}")
return web.json_response({
"error": {"message": f"Internal server error: {str(e)}"}
}, status=500)
async def list_models(request):
"""List available models"""
return web.json_response({
"object": "list",
"data": [
{
"id": "pydantic",
"object": "model",
"created": int(time.time()),
"owned_by": "core-ai"
},
{
"id": "simple",
"object": "model",
"created": int(time.time()),
"owned_by": "core-ai"
}
]
})
async def list_tools(request):
"""List all available tools"""
try:
tools = get_all_tools()
tool_list = []
tools_info = []
for name, func in tools.items():
tools_info.append({
import inspect
doc = inspect.getdoc(func) or "No description"
tool_list.append({
"name": name,
"description": func.__doc__.strip() if func.__doc__ else "No description available",
"type": "local"
"description": doc.split('\n')[0],
"type": "local" if not name.startswith("core-api__") else "openapi"
})
return web.json_response({
"tools": tools_info,
"count": len(tools_info),
"pydantic_ai_available": PYDANTIC_AI_AVAILABLE
"tools": tool_list,
"tools_count": len(tool_list)
})
except Exception as e:
logger.exception("Error listing tools:")
return web.json_response({"error": {"message": str(e)}}, status=500)
logger.exception(f"Error listing tools: {e}")
return web.json_response({
"error": {"message": str(e)}
}, status=500)
async def health_check(request):
"""Simple health check endpoint."""
"""Health check endpoint"""
return web.json_response({
"status": "ok",
"service": "core-ai",
"agents": {
"simple": True,
"ollama-native": OLLAMA_NATIVE_AVAILABLE,
"pydantic": PYDANTIC_AI_AVAILABLE
},
"default_agent": "ollama-native" if OLLAMA_NATIVE_AVAILABLE else "simple",
"default_agent": "pydantic" if PYDANTIC_AI_AVAILABLE else "simple",
"tools_count": len(get_all_tools())
})
async def test_ollama_tools(request):
"""Test Ollama tool calling directly"""
import httpx
try:
tool_def = {
"type": "function",
"function": {
"name": "web_search",
"description": "Search the web using SearXNG",
"parameters": {
"type": "object",
"properties": {
"query": {"type": "string", "description": "Search query"}
},
"required": ["query"]
}
}
}
payload = {
"model": "mistral-nemo:latest",
"messages": [{"role": "user", "content": "Search for Python 3.13 features"}],
"tools": [tool_def],
"stream": False
}
async with httpx.AsyncClient(timeout=30.0) as client:
response = await client.post('http://ollama:11434/api/chat', json=payload)
result = response.json()
return web.json_response({
"status_code": response.status_code,
"has_tool_calls": 'tool_calls' in result.get('message', {}),
"response": result
})
except Exception as e:
logger.exception("Test error:")
return web.json_response({"error": str(e)}, status=500)
async def setup_routes(app):
# Chat endpoints
app.router.add_post("/chat/completions", chat_completions) # Alias without /v1 for compatibility
app.router.add_post("/v1/chat/completions", chat_completions) # Default (PydanticAI)
app.router.add_post("/v1/chat/simple", chat_simple) # Simple agent (no tools)
app.router.add_post("/v1/chat/pydantic", chat_pydantic) # Alias for default
app.router.add_post("/chat/completions", chat_completions) # Alias without /v1
app.router.add_post("/v1/chat/completions", chat_completions) # Default: PydanticAI
app.router.add_post("/v1/chat/simple", chat_simple) # Fallback: Simple agent
# OpenAI-compatible endpoints
app.router.add_get("/v1/models", list_models) # List available models
app.router.add_get("/models", list_models) # Alias without /v1 prefix
# Models endpoint
app.router.add_get("/models", list_models)
app.router.add_get("/v1/models", list_models)
# Tool management
app.router.add_get("/v1/tools", list_tools) # List available tools
# Tools endpoint
app.router.add_get("/tools", list_tools)
app.router.add_get("/v1/tools", list_tools)
# Health check
app.router.add_get("/health", health_check)
app.router.add_get("/test/ollama-tools", test_ollama_tools)
# Setup CORS
cors = cors_setup(app, defaults={
@@ -439,20 +285,21 @@ async def setup_routes(app):
)
})
# Configure CORS on all routes
# Apply CORS to all routes
for route in list(app.router.routes()):
cors.add(route)
def main():
app = web.Application()
app.on_startup.append(setup_routes) # Register routes on startup
# Configuration
host = os.getenv("HOST", "0.0.0.0")
port = int(os.getenv("PORT", 8086)) # Use 8086 to avoid conflict with core-ai
logger.info(f"Starting core-ai service on http://{host}:{port}")
web.run_app(app, host=host, port=port)
if __name__ == "__main__":
main()
# Set up routes
import asyncio
loop = asyncio.get_event_loop()
loop.run_until_complete(setup_routes(app))
# Run the application
logger.info("Starting core-ai service on http://0.0.0.0:8086")
web.run_app(app, host='0.0.0.0', port=8086)
if __name__ == '__main__':
main()
services/core-ai/src/agents/__init__.py
-6
View File
@@ -1,9 +1,6 @@
"""Agent implementations for core-ai service"""
from .simple import SimpleLiteLLMAgent, get_simple_litellm_agent
from .ollama_native_agent import OllamaNativeAgent, get_ollama_native_agent
OLLAMA_NATIVE_AVAILABLE = True
try:
from .pydantic_agent import PydanticAgent, get_pydantic_agent
@@ -16,9 +13,6 @@ except ImportError:
__all__ = [
'SimpleLiteLLMAgent',
'get_simple_litellm_agent',
'OllamaNativeAgent',
'get_ollama_native_agent',
'OLLAMA_NATIVE_AVAILABLE',
'PydanticAgent',
'get_pydantic_agent',
'PYDANTIC_AI_AVAILABLE',
services/core-ai/src/agents/ollama_native_agent.py
-297
View File
@@ -1,297 +0,0 @@
"""
Native Ollama Agent - Uses Ollama's native API with tool calling support.
This agent bypasses PydanticAI's OpenAI-compatible approach and uses
Ollama's native /api/chat endpoint which has better tool calling support.
"""
import logging
import httpx
import json
from typing import List, Dict, Any, AsyncIterator
from functools import lru_cache
from src.config import get_settings
from src.prompts import get_prompt
from src.tools.registry import get_all_tools
logger = logging.getLogger(__name__)
class OllamaNativeAgent:
"""
Agent using Ollama's native API with tool calling support.
Unlike PydanticAI which uses Ollama's OpenAI-compatible API,
this uses the native /api/chat endpoint which has proper tool support.
"""
def __init__(self, tools: List = None, discover_tools: bool = False, include_openapi: bool = True):
logger.info("OllamaNativeAgent: Initializing...")
self.settings = get_settings()
self.model = self.settings.agent_model
self.include_openapi = include_openapi
self._tools_loaded = False
# Load system prompt
from datetime import datetime
base_prompt = get_prompt("pydantic_agent")
current_date = datetime.now().strftime("%A, %B %d, %Y")
self.system_prompt = f"Today is {current_date}.\n\n{base_prompt}"
# Get tools (sync part only)
if tools is not None:
self.tools_dict = {func.__name__: func for func in tools}
self._tools_loaded = True
elif discover_tools:
# Get core tools (local) - sync
self.tools_dict = get_all_tools()
# OpenAPI tools will be loaded async on first use
else:
self.tools_dict = {}
self._tools_loaded = True
logger.info(f"OllamaNativeAgent: {len(self.tools_dict)} core tools loaded")
logger.info(f"OllamaNativeAgent: Model: {self.model}")
logger.info("✓ OllamaNativeAgent: Initialization complete")
async def _ensure_tools_loaded(self):
"""Load OpenAPI tools asynchronously (called on first use)"""
if self._tools_loaded:
return
if self.include_openapi and self.settings.openapi_enabled:
try:
from src.tools.openapi_discovery import get_openapi_tools
# Parse OpenAPI endpoints from config
endpoints = [e.strip() for e in self.settings.openapi_endpoints.split(",")]
# Fetch OpenAPI tools (async)
openapi_tools = await get_openapi_tools(endpoints=endpoints)
self.tools_dict.update(openapi_tools)
logger.info(f"OllamaNativeAgent: Added {len(openapi_tools)} OpenAPI tools")
except Exception as e:
logger.warning(f"OllamaNativeAgent: Failed to load OpenAPI tools: {e}")
self._tools_loaded = True
logger.info(f"OllamaNativeAgent: Total tools available: {len(self.tools_dict)}")
def _format_tools_for_ollama(self) -> List[Dict[str, Any]]:
"""
Convert Python functions to Ollama tool format.
Ollama expects:
{
"type": "function",
"function": {
"name": "function_name",
"description": "...",
"parameters": {...JSON Schema...}
}
}
"""
tools = []
for name, func in self.tools_dict.items():
# Extract function signature and docstring
import inspect
sig = inspect.signature(func)
doc = inspect.getdoc(func) or "No description"
# Build parameters schema
properties = {}
required = []
for param_name, param in sig.parameters.items():
if param_name in ['self', 'cls']:
continue
# Determine type
param_type = "string" # default
if param.annotation != inspect.Parameter.empty:
if param.annotation == int:
param_type = "integer"
elif param.annotation == float:
param_type = "number"
elif param.annotation == bool:
param_type = "boolean"
properties[param_name] = {
"type": param_type,
"description": f"Parameter {param_name}"
}
# Required if no default value
if param.default == inspect.Parameter.empty:
required.append(param_name)
tool_def = {
"type": "function",
"function": {
"name": name,
"description": doc.split('\n')[0], # First line of docstring
"parameters": {
"type": "object",
"properties": properties,
"required": required
}
}
}
tools.append(tool_def)
return tools
async def chat(
self,
messages: List[Dict[str, str]],
conversation_id: str = None,
stream: bool = True
) -> AsyncIterator[Dict[str, Any]]:
"""
Process chat messages with tool calling support.
Args:
messages: List of message dicts with 'role' and 'content'
conversation_id: Optional conversation ID
stream: Whether to stream responses
Yields:
Dict with 'type' and content
"""
# Ensure OpenAPI tools are loaded (async, called once)
await self._ensure_tools_loaded()
logger.info(f"OllamaNativeAgent: Processing message: {messages[-1]['content'][:50]}...")
try:
# Extract user message
user_messages = [m for m in messages if m["role"] != "system"]
if not user_messages:
raise ValueError("No user messages provided")
# Build Ollama messages format
ollama_messages = [
{"role": "system", "content": self.system_prompt}
]
ollama_messages.extend(user_messages)
# Format tools
tools = self._format_tools_for_ollama() if self.tools_dict else None
# Make request to Ollama
payload = {
"model": self.model,
"messages": ollama_messages,
"stream": False # Handle streaming separately if needed
}
if tools:
payload["tools"] = tools
async with httpx.AsyncClient(timeout=60.0) as client:
response = await client.post(
f"{self.settings.ollama_base_url}/api/chat",
json=payload
)
response.raise_for_status()
result = response.json()
message = result.get("message", {})
# Check if model wants to call tools
if "tool_calls" in message and message["tool_calls"]:
logger.info(f"Tool calls requested: {len(message['tool_calls'])}")
# Execute tools
tool_results = []
for tool_call in message["tool_calls"]:
func_name = tool_call["function"]["name"]
func_args = tool_call["function"]["arguments"]
logger.info(f"Executing tool: {func_name}({func_args})")
if func_name in self.tools_dict:
try:
tool_func = self.tools_dict[func_name]
# Call tool (handle both sync and async)
import asyncio
if asyncio.iscoroutinefunction(tool_func):
tool_result = await tool_func(**func_args)
else:
tool_result = tool_func(**func_args)
tool_results.append({
"role": "tool",
"content": str(tool_result)
})
logger.info(f"Tool result: {str(tool_result)[:100]}...")
except Exception as e:
error_msg = f"Tool execution error: {str(e)}"
logger.error(error_msg)
tool_results.append({
"role": "tool",
"content": error_msg
})
else:
logger.warning(f"Tool {func_name} not found")
tool_results.append({
"role": "tool",
"content": f"Error: Tool {func_name} not available"
})
# Send tool results back to model
ollama_messages.append(message)
ollama_messages.extend(tool_results)
payload["messages"] = ollama_messages
payload.pop("tools", None) # Don't send tools again
async with httpx.AsyncClient(timeout=60.0) as client:
response = await client.post(
f"{self.settings.ollama_base_url}/api/chat",
json=payload
)
response.raise_for_status()
final_result = response.json()
final_content = final_result.get("message", {}).get("content", "")
logger.info(f"Final response: {final_content[:100]}...")
yield {"type": "content", "content": final_content, "finish_reason": "stop"}
else:
# No tool calls, return response directly
content = message.get("content", "")
logger.info(f"Direct response: {content[:100]}...")
yield {"type": "content", "content": content, "finish_reason": "stop"}
except Exception as e:
logger.error(f"OllamaNativeAgent error: {e}", exc_info=True)
yield {
"type": "error",
"content": f"Error: {str(e)}",
"finish_reason": "error"
}
async def chat_completion(
self,
messages: List[Dict[str, str]],
conversation_id: str = None
) -> str:
"""Non-streaming chat completion."""
final_content = ""
async for chunk in self.chat(messages=messages, conversation_id=conversation_id, stream=False):
if chunk["type"] == "content":
final_content += chunk["content"]
return final_content if final_content else "I couldn't generate a response."
@lru_cache()
def get_ollama_native_agent(discover_tools: bool = True) -> OllamaNativeAgent:
"""Get cached Ollama native agent instance."""
return OllamaNativeAgent(discover_tools=discover_tools)
services/core-ai/src/prompts.py
+26 -9
View File
@@ -13,18 +13,35 @@ Your core responsibility: Verify facts before presenting them as truth.
You have access to two categories of tools:
**Core Tools** (essential utilities):
- web_search: Latest/current/recent information (always verify facts, sir)
**Core Tools** (always available):
- web_search: ONLY for current events, news, research, or information not available through other tools
- calculate: Mathematical operations (precision is paramount)
- get_current_time/get_current_date: Time/date queries
- add_days_to_date/calculate_date_difference: Date calculations
**Infrastructure Tools** (prefixed with "core_api__"):
When managing sir's home infrastructure, use these tools:
- Services: List, start, stop Docker services
- Domains: List configured domains
- Proxy: Manage reverse proxy configurations
- Monitors: Health monitoring (Uptime Kuma integration)
- Ports: Check allocated ports
**Infrastructure Tools** (discovered from core-api, prefixed with "core_api__"):
These tools provide DIRECT system access. PREFER these over web searches when available:
- DNS queries: core_api__dns_lookup_tools_dns_lookup_post
* Use for: A, AAAA, MX, TXT, CNAME, NS, SOA, PTR records
* Example: "lookup A records for github.com" → use DNS tool, NOT web_search
- Web scraping: core_api__scrape_website_tools_scrape_post
* Use for: Extract content from specific URLs
- Docker services: core_api__list_services*, core_api__get_service*, core_api__start_service*, core_api__stop_service*
* Use for: Manage sir's containerized services
- Network management: core_api__list_domains*, core_api__list_ports*, core_api__get_proxy_host*
* Use for: Infrastructure configuration
- Monitoring: core_api__list_monitors*, core_api__get_monitor*, core_api__create_monitor*
* Use for: System health checks
**Tool Selection Priority:**
1. If an infrastructure tool exists for the task → use it (more reliable than web search)
2. If no infrastructure tool exists → use web_search
3. For general knowledge → answer directly (no tool needed)
Be concise unless details are specifically requested. When using tools, acknowledge them naturally in your dignified manner."""
}
services/core-ai/src/tools.py
-101
View File
@@ -1,101 +0,0 @@
"""
Agent Tools - Tools for the Core AI agent
These tools make REST API calls to the Core API service.
"""
from typing import List, Dict, Any
import logging
import functools
import inspect
import httpx # For making asynchronous HTTP requests
# Adjusted import path for the new core-ai service structure
from src.config import get_settings
logger = logging.getLogger(__name__)
# Initialize settings once
settings = get_settings()
CORE_API_BASE_URL = settings.core_api_base_url
# ============================================================================
# Decorator for logging tool calls
# ============================================================================
def log_tool_call(func):
"""Decorator to log tool calls with their parameters"""
@functools.wraps(func)
async def wrapper(*args, **kwargs):
params_str = ", ".join(
[f"{arg}" for arg in args] +
[f"{k}={repr(v)}" for k, v in kwargs.items()]
)
logger.info(f"🔧 TOOL CALL: {func.__name__}({params_str})")
try:
sig = inspect.signature(func)
valid_kwargs = {
key: value for key, value in kwargs.items()
if key in sig.parameters
}
result = await func(*args, **valid_kwargs)
result_preview = str(result)[:200] if result else "None"
logger.info(f"✅ TOOL RESULT: {func.__name__} → {result_preview}...")
return result
except Exception as e:
logger.error(f"❌ TOOL ERROR: {func.__name__} failed with {type(e).__name__}: {e}", exc_info=True)
raise
return wrapper
# ============================================================================
# HTTP Client
# ============================================================================
# Use a single httpx client for performance
# It's important to close the client when the application shuts down
http_client = httpx.AsyncClient()
# ============================================================================
# Knowledge & Search Tools
# ============================================================================
@log_tool_call
async def web_search(query: str, num_results: int) -> str:
"""
Search the web. (Neutered for testing purposes).
"""
logger.info(f"--- NEUTERED WEB SEARCH CALLED FOR: {query} ---")
if "capital of france" in query.lower():
return "Search results for 'Capital of France':\n\n1. **Paris - Wikipedia**\n URL: https://en.wikipedia.org/wiki/Paris\n Paris is the capital and most populous city of France."
else:
return f"Search results for '{query}':\n\n1. No specific results for this neutered test. Try 'capital of France'."
# ============================================================================
# Special Tools (Response tool is here for consistency, but will be removed for initial test)
# ============================================================================
@log_tool_call
async def response(answer: str) -> None:
"""
Deliver your final response to the user.
"""
logger.info("`response` tool called. Returning None to terminate agent loop.")
return None
# ============================================================================
# Tool Registry - Legacy (deprecated, use src/tools/registry.py instead)
# ============================================================================
try:
from google.adk.tools import FunctionTool
ADK_AVAILABLE = True
except ImportError:
ADK_AVAILABLE = False
FunctionTool = None
def get_agent_tools() -> List:
"""DEPRECATED: Get all tools available to the agent. Use src/tools/registry.py instead."""
logger.info("--- DIAGNOSTIC MODE (Phase 1): Agent has NO tools. ---")
return []
services/core-ai/tests/QUALITY_TESTS.md
-370
View File
@@ -1,370 +0,0 @@
# Core-AI Quality Test Suite
Comprehensive test suite for benchmarking AI agent performance and detecting regressions across code changes.
## Purpose
This test suite validates:
- **Tool calling decision-making** - Does the agent choose the right tools?
- **Response quality** - Are responses accurate and complete?
- **Performance** - Are responses delivered within acceptable timeframes?
- **Regression detection** - Has quality degraded since the last version?
## Current Implementation
- **Agent**: OllamaNativeAgent (Ollama native API with tool calling)
- **Model**: mistral-nemo:latest
- **Framework**: PydanticAI
- **Tools**: web_search, calculate, date/time operations
## Quick Start
### Run All Tests
```bash
# From services/core-ai directory
python tests/test_ai_flow_quality.py
```
This will:
1. Run all test scenarios
2. Generate a comprehensive report
3. Save reports to `tests/reports/` with timestamp and git tag
4. Output results to console
### Run with pytest
```bash
# Run all tests
pytest tests/test_ai_flow_quality.py -v
# Run specific scenario
pytest tests/test_ai_flow_quality.py::test_scenario2_web_search -v
# Run regression tests only
pytest tests/test_ai_flow_quality.py -k regression -v
```
## Test Scenarios
### Scenario 1: Simple Knowledge Query
- **Query**: "What is Docker?"
- **Expected**: Direct answer without tools
- **Performance Target**: < 10s
### Scenario 2: Web Search
- **Query**: "What are the latest Kubernetes security best practices?"
- **Expected**: Uses web_search tool, synthesizes results
- **Performance Target**: < 30s
### Scenario 3: Mathematical Calculation
- **Query**: "Calculate 2847 * 1923 + 5612 - 999"
- **Expected**: Uses calculate tool for precision
- **Performance Target**: < 15s
- **Expected Result**: 5,479,394
### Scenario 4: Date/Time Operations
- **Query**: "What is the current date and what will it be in 30 days?"
- **Expected**: Uses get_current_date and add_days_to_date tools
- **Performance Target**: < 15s
### Scenario 5: Multi-Tool Complex Query
- **Query**: "Get current time in NYC and Tokyo, calculate difference"
- **Expected**: Multiple tool calls (get_current_time × 2, synthesis)
- **Performance Target**: < 30s
### Scenario 6: DNS Lookup (OpenAPI Discovery)
- **Query**: "What are the A records for github.com? Use the core-api DNS lookup tool."
- **Expected**: Uses DNS lookup tool discovered via OpenAPI from core-api
- **Performance Target**: < 15s
- **Purpose**: Tests OpenAPI tool discovery and infrastructure integration
## Report Format
Reports are saved in two formats:
### 1. Text Report (`quality-report-YYYYMMDD-HHMMSS.txt`)
```
================================================================================
CORE-AI QUALITY REPORT
Generated: 2025-12-01T10:30:45
Git Tag: v2.1.0
Git Commit: a3b2c1d
Git Branch: main
Implementation:
Agent: OllamaNativeAgent
Model: mistral-nemo:latest
Framework: PydanticAI
API: Ollama native (/api/chat)
================================================================================
Total Tests: 5
Passed: 5 (100.0%)
Failed: 0
Performance:
Average response time: 8.45s
Fastest response: 3.21s
Slowest response: 15.67s
Test Details:
--------------------------------------------------------------------------------
1. Scenario 1: Simple Knowledge: ✓ PASS
Query: What is Docker?...
Response time: 3.21s
Tools available: 6
Response length: 245 chars
...
================================================================================
REVERT INSTRUCTIONS:
If quality has degraded, revert to: v2.1.0
git checkout v2.1.0
================================================================================
```
### 2. JSON Report (`quality-report-YYYYMMDD-HHMMSS.json`)
Machine-readable format for programmatic analysis and trend tracking:
```json
{
"timestamp": "2025-12-01T10:30:45",
"git_info": {
"tag": "v2.1.0",
"commit": "a3b2c1d",
"branch": "main"
},
"summary": {
"total": 5,
"passed": 5,
"failed": 0
},
"results": [...]
}
```
## Workflow: Before Making Changes
### 1. Establish Baseline
Before making any code changes, run the test suite to establish a quality baseline:
```bash
cd /home/jpmschweitzer/Projects/portainer-core/services/core-ai
python tests/test_ai_flow_quality.py
```
**Save the report location** - you'll compare against this later.
### 2. Make Your Changes
Edit agent code, tools, prompts, etc.
### 3. Run Tests Again
```bash
python tests/test_ai_flow_quality.py
```
### 4. Compare Reports
Compare the new report against the baseline:
```bash
# List recent reports
ls -lh tests/reports/
# Compare two reports
diff tests/reports/quality-report-20251201-103045.txt \
tests/reports/quality-report-20251201-115522.txt
```
**Key metrics to watch**:
- Pass rate (should stay 100%)
- Average response time (should not significantly increase)
- Individual test failures (investigate immediately)
### 5. Revert if Quality Degrades
If tests fail or performance degrades significantly:
```bash
# Check the git tag from the failing report
cat tests/reports/quality-report-20251201-115522.txt | grep "Git Tag"
# Revert to that tag
git checkout v2.1.0
```
## Benchmarking Models
To compare different models:
### 1. Run baseline with current model
```bash
python tests/test_ai_flow_quality.py
# Save this as baseline
```
### 2. Change model in config
Edit `services/core-ai/src/config.py` or environment variable:
```python
# Change from mistral-nemo:latest to gemma2:9b
AGENT_MODEL = "gemma2:9b"
```
Restart core-ai:
```bash
cd /home/jpmschweitzer/Projects/portainer-core/stacks
docker restart core-ai
```
### 3. Run tests with new model
```bash
python tests/test_ai_flow_quality.py
```
### 4. Compare results
```bash
# Check both JSON reports for performance comparison
cat tests/reports/quality-report-BASELINE.json | jq '.summary'
cat tests/reports/quality-report-NEW_MODEL.json | jq '.summary'
```
Look for:
- **Pass rate changes** - Did the new model fail any tests?
- **Response time changes** - Is it faster or slower?
- **Response quality** - Are answers as good?
## Troubleshooting
### Tests Fail: "Connection refused"
**Problem**: core-ai service not running
**Solution**:
```bash
cd /home/jpmschweitzer/Projects/portainer-core/stacks
docker restart core-ai
docker logs core-ai # Check for startup errors
```
### Tests Timeout
**Problem**: Model too slow or stuck
**Solution**:
1. Check Ollama GPU usage: `nvidia-smi`
2. Check model is loaded: `docker exec ollama ollama list`
3. Increase timeout in test file if needed
### Web Search Tests Fail
**Problem**: SearXNG not available
**Solution**:
```bash
docker restart searxng
curl "http://localhost:8087/search?q=test&format=json"
```
### Calculation Tests Fail
**Problem**: Agent not using calculate tool
**Solution**: Check tool registration:
```bash
curl http://localhost:8086/v1/tools | jq '.tools[] | .name'
```
## Adding New Test Scenarios
### 1. Add test function
```python
@pytest.mark.asyncio
async def test_my_new_scenario():
"""
Scenario: My New Feature
Expected: Describe expected behavior
Performance target: < Xs
"""
async with AIFlowTester() as tester:
result = await tester.chat("My test query")
tester.assert_response_quality(
result,
expected_keywords=["keyword1", "keyword2"],
min_length=50,
max_time=15.0
)
# Custom assertions
assert "expected result" in result["response"]
print(f"✓ My scenario: {result['total_time']:.2f}s")
```
### 2. Add to scenario list
In `run_full_quality_check()`:
```python
test_scenarios = [
# ... existing scenarios ...
{
"name": "Scenario X: My New Feature",
"query": "My test query",
"test": test_my_new_scenario
},
]
```
### 3. Run to verify
```bash
pytest tests/test_ai_flow_quality.py::test_my_new_scenario -v
```
## Best Practices
### ✅ DO:
- Run tests before committing major changes
- Compare reports to detect regressions
- Save baseline reports for each release
- Document expected behavior in test docstrings
- Use meaningful git tags for easy reversion
### ❌ DON'T:
- Skip tests when making agent changes
- Ignore performance degradation warnings
- Delete old reports (keep for trend analysis)
- Change test expectations to make tests pass
- Commit without running tests first
## Report Retention
Keep reports organized:
```bash
# Keep last 30 days of reports
find tests/reports/ -name "*.txt" -mtime +30 -delete
find tests/reports/ -name "*.json" -mtime +30 -delete
# Archive reports by month
mkdir -p tests/reports/archive/2025-12/
mv tests/reports/quality-report-202512*.* tests/reports/archive/2025-12/
```
---
**Remember**: These tests protect quality. If they fail, investigate before proceeding!
services/core-ai/tests/test_02_litellm_raw.py
-137
View File
@@ -1,137 +0,0 @@
#!/usr/bin/env python3
"""
Layer 2: Raw LiteLLM Connection Tests
Tests direct LiteLLM → Ollama communication without any wrappers.
"""
import pytest
import sys
from pathlib import Path
import time
# Add parent directory to path
sys.path.insert(0, str(Path(__file__).parent.parent))
from src.config import get_settings
try:
import litellm
except ImportError:
pytest.skip("LiteLLM not installed", allow_module_level=True)
@pytest.mark.asyncio
async def test_litellm_simple_completion():
"""Test a simple LiteLLM completion"""
settings = get_settings()
model = f"ollama/{settings.agent_model}"
messages = [{"role": "user", "content": "Say 'test' and nothing else."}]
print(f"\n→ Testing LiteLLM with model: {model}")
print(f"→ API base: {settings.ollama_base_url}")
start_time = time.time()
response = await litellm.acompletion(
model=model,
messages=messages,
api_base=settings.ollama_base_url,
temperature=0.1,
stream=False
)
elapsed = time.time() - start_time
content = response.choices[0].message.content
assert content is not None, "No content in response"
assert len(content) > 0, "Empty content"
print(f"✓ Response received in {elapsed:.2f}s: {content}")
@pytest.mark.asyncio
async def test_litellm_with_system_prompt():
"""Test LiteLLM completion with system prompt"""
settings = get_settings()
model = f"ollama/{settings.agent_model}"
messages = [
{"role": "system", "content": "You are a helpful assistant. Be concise."},
{"role": "user", "content": "What is 2+2? Answer with just the number."}
]
response = await litellm.acompletion(
model=model,
messages=messages,
api_base=settings.ollama_base_url,
temperature=0.1,
stream=False
)
content = response.choices[0].message.content
assert content is not None, "No content in response"
print(f"✓ Response with system prompt: {content}")
@pytest.mark.asyncio
async def test_litellm_streaming():
"""Test LiteLLM streaming mode"""
settings = get_settings()
model = f"ollama/{settings.agent_model}"
messages = [{"role": "user", "content": "Count from 1 to 3. Just numbers."}]
response = await litellm.acompletion(
model=model,
messages=messages,
api_base=settings.ollama_base_url,
temperature=0.1,
stream=True
)
chunks = []
chunk_count = 0
async for chunk in response:
chunk_count += 1
if chunk.choices[0].delta.content:
chunks.append(chunk.choices[0].delta.content)
full_content = "".join(chunks)
assert chunk_count > 0, "No chunks received"
assert len(full_content) > 0, "No content in chunks"
print(f"✓ Received {chunk_count} chunks: {full_content}")
@pytest.mark.asyncio
async def test_litellm_capital_of_france():
"""Test the actual failing case: 'What is the capital of France?'"""
settings = get_settings()
model = f"ollama/{settings.agent_model}"
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France?"}
]
print(f"\n→ Testing the actual failing query...")
response = await litellm.acompletion(
model=model,
messages=messages,
api_base=settings.ollama_base_url,
temperature=0.1,
stream=False
)
content = response.choices[0].message.content
assert content is not None, "No content in response"
assert len(content) > 0, "Empty response"
# Check if the answer is reasonable
content_lower = content.lower()
assert "paris" in content_lower, f"Expected 'Paris' in answer, got: {content}"
print(f"✓ Correct answer received: {content}")
if __name__ == "__main__":
pytest.main([__file__, "-v", "-s"])
services/core-ai/tests/test_03_message_format.py
-113
View File
@@ -1,113 +0,0 @@
#!/usr/bin/env python3
"""
Layer 3: Message Formatting & Prompts Tests
Tests that system prompts are correctly injected and messages are formatted properly.
"""
import pytest
import sys
from pathlib import Path
# Add parent directory to path
sys.path.insert(0, str(Path(__file__).parent.parent))
from src.config import get_settings
from src.prompts import get_prompt, PROMPTS
def test_prompts_defined():
"""Test that prompts are defined"""
assert len(PROMPTS) > 0, "No prompts defined"
print(f"✓ {len(PROMPTS)} prompt variant(s) defined")
def test_get_prompt_default():
"""Test getting default prompt"""
prompt = get_prompt()
assert prompt is not None, "Default prompt is None"
assert len(prompt) > 0, "Default prompt is empty"
print(f"✓ Default prompt: {prompt[:80]}...")
def test_get_prompt_specific():
"""Test getting specific prompt variant"""
settings = get_settings()
prompt = get_prompt(settings.system_prompt_variant)
assert prompt is not None, f"Prompt '{settings.system_prompt_variant}' is None"
assert len(prompt) > 0, f"Prompt '{settings.system_prompt_variant}' is empty"
print(f"✓ Prompt '{settings.system_prompt_variant}': {prompt[:80]}...")
def test_message_structure():
"""Test that message structure is valid"""
test_messages = [
{"role": "user", "content": "Hello"}
]
# Simulate system prompt injection
from src.prompts import get_prompt
system_prompt = get_prompt()
formatted_messages = [
{"role": "system", "content": system_prompt},
*test_messages
]
# Validate structure
assert len(formatted_messages) == 2, "Expected 2 messages after injection"
assert formatted_messages[0]["role"] == "system", "First message should be system"
assert formatted_messages[1]["role"] == "user", "Second message should be user"
print(f"✓ Message structure correct")
for i, msg in enumerate(formatted_messages):
content_preview = msg['content'][:50] + "..." if len(msg['content']) > 50 else msg['content']
print(f" [{i}] {msg['role']}: {content_preview}")
def test_system_prompt_not_duplicated():
"""Test that system prompt is not duplicated if already present"""
from src.prompts import get_prompt
system_prompt = get_prompt()
# Messages already have system prompt
messages = [
{"role": "system", "content": "Custom system prompt"},
{"role": "user", "content": "Hello"}
]
# Simulate the check in agent
if messages and messages[0]["role"] == "system":
# Should not inject
formatted_messages = messages
else:
# Would inject
formatted_messages = [{"role": "system", "content": system_prompt}, *messages]
# Should still have only 2 messages
assert len(formatted_messages) == 2, "System prompt was duplicated"
assert formatted_messages[0]["role"] == "system"
assert formatted_messages[0]["content"] == "Custom system prompt"
print(f"✓ System prompt not duplicated when already present")
def test_empty_messages_handling():
"""Test handling of empty messages list"""
from src.prompts import get_prompt
system_prompt = get_prompt()
messages = []
# Simulate injection
if not messages or messages[0]["role"] != "system":
formatted_messages = [{"role": "system", "content": system_prompt}, *messages]
else:
formatted_messages = messages
assert len(formatted_messages) == 1, "Should have system prompt only"
assert formatted_messages[0]["role"] == "system"
print(f"✓ Empty messages handled correctly")
if __name__ == "__main__":
pytest.main([__file__, "-v", "-s"])
services/core-ai/tests/test_ai_flow_quality.py
-634
View File
@@ -1,634 +0,0 @@
"""
AI Flow Quality Test Suite
Purpose: Benchmark core-ai agent behavior for regression detection
and performance tracking over time.
This test suite validates:
- Tool calling decision-making (OllamaNativeAgent)
- Multi-step reasoning capability
- Response quality and accuracy
- Performance characteristics
- Regression detection across code changes
Current Implementation:
- Agent: OllamaNativeAgent (Ollama native API with tool calling)
- Model: mistral-nemo:latest (primary reasoning model)
- Tools: Local tools (web_search, calculate, date/time operations)
- Framework: PydanticAI
Usage:
pytest tests/test_ai_flow_quality.py -v
pytest tests/test_ai_flow_quality.py::test_simple_knowledge -v
python tests/test_ai_flow_quality.py # Run and generate report
Reports saved to: tests/reports/
"""
import asyncio
import time
import json
import httpx
import subprocess
from pathlib import Path
from typing import Dict, List, Any, Optional
from datetime import datetime
import pytest
# Test Configuration
BASE_URL = "http://localhost:8086"
TIMEOUT = 60.0
REPORTS_DIR = Path(__file__).parent / "reports"
def get_git_info() -> Dict[str, str]:
"""Get current git tag and commit hash"""
try:
# Get current tag
tag = subprocess.check_output(
["git", "describe", "--tags", "--exact-match"],
stderr=subprocess.DEVNULL,
text=True
).strip()
except subprocess.CalledProcessError:
# No exact tag, get latest tag + commit
try:
tag = subprocess.check_output(
["git", "describe", "--tags", "--always"],
text=True
).strip()
except subprocess.CalledProcessError:
tag = "unknown"
try:
commit = subprocess.check_output(
["git", "rev-parse", "--short", "HEAD"],
text=True
).strip()
except subprocess.CalledProcessError:
commit = "unknown"
try:
branch = subprocess.check_output(
["git", "rev-parse", "--abbrev-ref", "HEAD"],
text=True
).strip()
except subprocess.CalledProcessError:
branch = "unknown"
return {"tag": tag, "commit": commit, "branch": branch}
class AIFlowTester:
"""Test harness for AI flow quality checks"""
def __init__(self):
self.results = []
self.client = None
self.git_info = get_git_info()
async def __aenter__(self):
self.client = httpx.AsyncClient(timeout=TIMEOUT)
return self
async def __aexit__(self, exc_type, exc_val, exc_tb):
if self.client:
await self.client.aclose()
async def chat(
self,
message: str,
enable_tools: bool = True,
stream: bool = False
) -> Dict[str, Any]:
"""
Send a chat request and measure performance
Returns:
{
"response": str,
"time_to_first_token": float,
"total_time": float,
"tools_count": int,
"success": bool,
"error": str | None
}
"""
start_time = time.time()
time_to_first_token = None
response_text = ""
try:
payload = {
"messages": [{"role": "user", "content": message}],
"stream": stream,
"enable_tools": enable_tools
}
response = await self.client.post(
f"{BASE_URL}/v1/chat/completions",
json=payload
)
response.raise_for_status()
if not stream:
time_to_first_token = time.time() - start_time
data = response.json()
response_text = data["choices"][0]["message"]["content"]
tools_count = data.get("tools_count", 0)
tools_enabled = data.get("tools_enabled", False)
else:
# Handle streaming response
# TODO: Implement streaming support
raise NotImplementedError("Streaming not yet implemented")
total_time = time.time() - start_time
return {
"response": response_text,
"time_to_first_token": time_to_first_token,
"total_time": total_time,
"tools_count": tools_count if tools_enabled else 0,
"success": True,
"error": None
}
except Exception as e:
total_time = time.time() - start_time
return {
"response": "",
"time_to_first_token": None,
"total_time": total_time,
"tools_count": 0,
"success": False,
"error": str(e)
}
def assert_response_quality(
self,
result: Dict[str, Any],
expected_keywords: List[str] = None,
min_length: int = 20,
max_time: float = 30.0
):
"""
Validate response quality
Args:
result: Test result from chat()
expected_keywords: Keywords that should appear in response
min_length: Minimum response length
max_time: Maximum acceptable response time
"""
assert result["success"], f"Request failed: {result['error']}"
assert len(result["response"]) >= min_length, \
f"Response too short: {len(result['response'])} < {min_length}"
assert result["total_time"] <= max_time, \
f"Response too slow: {result['total_time']:.2f}s > {max_time}s"
if expected_keywords:
response_lower = result["response"].lower()
for keyword in expected_keywords:
assert keyword.lower() in response_lower, \
f"Missing keyword '{keyword}' in response"
def generate_report(self) -> str:
"""Generate a quality report from test results"""
if not self.results:
return "No test results to report"
report = []
report.append("=" * 80)
report.append("CORE-AI QUALITY REPORT")
report.append(f"Generated: {datetime.now().isoformat()}")
report.append(f"Git Tag: {self.git_info['tag']}")
report.append(f"Git Commit: {self.git_info['commit']}")
report.append(f"Git Branch: {self.git_info['branch']}")
report.append("")
report.append("Implementation:")
report.append(" Agent: OllamaNativeAgent")
report.append(" Model: mistral-nemo:latest")
report.append(" Framework: PydanticAI")
report.append(" API: Ollama native (/api/chat)")
report.append("=" * 80)
report.append("")
# Summary statistics
total_tests = len(self.results)
passed_tests = sum(1 for r in self.results if r.get("passed", False))
failed_tests = total_tests - passed_tests
report.append(f"Total Tests: {total_tests}")
report.append(f"Passed: {passed_tests} ({100*passed_tests/total_tests:.1f}%)")
report.append(f"Failed: {failed_tests}")
report.append("")
# Performance metrics
response_times = [r["result"]["total_time"] for r in self.results
if r["result"]["success"]]
if response_times:
avg_time = sum(response_times) / len(response_times)
min_time = min(response_times)
max_time = max(response_times)
report.append("Performance:")
report.append(f" Average response time: {avg_time:.2f}s")
report.append(f" Fastest response: {min_time:.2f}s")
report.append(f" Slowest response: {max_time:.2f}s")
report.append("")
# Individual test results
report.append("Test Details:")
report.append("-" * 80)
for i, test in enumerate(self.results, 1):
status = "✓ PASS" if test.get("passed", False) else "✗ FAIL"
report.append(f"\n{i}. {test['name']}: {status}")
report.append(f" Query: {test['query'][:60]}...")
result = test["result"]
if result["success"]:
report.append(f" Response time: {result['total_time']:.2f}s")
report.append(f" Tools available: {result['tools_count']}")
report.append(f" Response length: {len(result['response'])} chars")
if test.get("error"):
report.append(f" ⚠ Assertion failed: {test['error']}")
else:
report.append(f" ✗ Error: {result['error']}")
report.append("")
report.append("=" * 80)
report.append("REVERT INSTRUCTIONS:")
report.append(f"If quality has degraded, revert to: {self.git_info['tag']}")
report.append(f" git checkout {self.git_info['tag']}")
report.append("=" * 80)
return "\n".join(report)
# ============================================================================
# TEST SCENARIOS
# ============================================================================
@pytest.mark.asyncio
async def test_scenario1_simple_knowledge():
"""
Scenario 1: Simple Knowledge Query (No Tools Needed)
Expected: Direct answer without requiring tools
Performance target: < 10s
"""
async with AIFlowTester() as tester:
result = await tester.chat("What is Docker?")
tester.assert_response_quality(
result,
expected_keywords=["container", "platform"],
min_length=50,
max_time=10.0
)
assert result["success"], "Request failed"
print(f"✓ Simple knowledge query: {result['total_time']:.2f}s")
@pytest.mark.asyncio
async def test_scenario2_web_search():
"""
Scenario 2: Web Search Required
Expected: Uses web_search tool via SearXNG, synthesizes results
Performance target: < 30s
"""
async with AIFlowTester() as tester:
result = await tester.chat(
"What are the latest Kubernetes security best practices?"
)
tester.assert_response_quality(
result,
expected_keywords=["kubernetes", "security"],
min_length=100,
max_time=30.0
)
assert result["success"], "Request failed"
print(f"✓ Web search query: {result['total_time']:.2f}s")
print(f" Tools available: {result['tools_count']}")
@pytest.mark.asyncio
async def test_scenario3_calculation():
"""
Scenario 3: Mathematical Calculation
Expected: Uses calculate tool for accurate results
Performance target: < 15s
"""
async with AIFlowTester() as tester:
result = await tester.chat(
"Calculate 2847 * 1923 + 5612 - 999"
)
tester.assert_response_quality(
result,
min_length=20,
max_time=15.0
)
assert result["success"], "Request failed"
# Verify correct answer: 5,479,394
response_clean = result["response"].replace(",", "").replace(" ", "")
assert "5479394" in response_clean, \
"Calculation result not found in response"
print(f"✓ Calculation query: {result['total_time']:.2f}s")
@pytest.mark.asyncio
async def test_scenario4_date_operations():
"""
Scenario 4: Date/Time Operations
Expected: Uses date/time tools for accurate results
Performance target: < 15s
"""
async with AIFlowTester() as tester:
result = await tester.chat(
"What is the current date and what will the date be 30 days from now?"
)
tester.assert_response_quality(
result,
expected_keywords=["date", "2025"],
min_length=50,
max_time=15.0
)
assert result["success"], "Request failed"
print(f"✓ Date operation query: {result['total_time']:.2f}s")
@pytest.mark.asyncio
async def test_scenario5_multi_tool_reasoning():
"""
Scenario 5: Multi-Tool Complex Query
Expected: Multiple tool calls, synthesizes results
Performance target: < 30s
"""
async with AIFlowTester() as tester:
result = await tester.chat(
"Get the current time in New York and Tokyo, then calculate the time difference in hours"
)
tester.assert_response_quality(
result,
expected_keywords=["time", "hour"],
min_length=80,
max_time=30.0
)
assert result["success"], "Request failed"
print(f"✓ Multi-tool query: {result['total_time']:.2f}s")
@pytest.mark.asyncio
async def test_scenario6_dns_lookup():
"""
Scenario 6: DNS Lookup
Expected: Uses DNS lookup tool from core-api (discovered via OpenAPI)
Performance target: < 15s
"""
async with AIFlowTester() as tester:
result = await tester.chat(
"What are the A records for github.com? Use the core-api DNS lookup tool."
)
tester.assert_response_quality(
result,
expected_keywords=["github", "record"],
min_length=30,
max_time=15.0
)
assert result["success"], "Request failed"
print(f"✓ DNS lookup query: {result['total_time']:.2f}s")
@pytest.mark.asyncio
async def test_tools_disabled():
"""
Test: Agent with Tools Disabled
Expected: Works without tool access, generates knowledge-based response
"""
async with AIFlowTester() as tester:
result = await tester.chat(
"What is Python?",
enable_tools=False
)
tester.assert_response_quality(
result,
expected_keywords=["python", "programming"],
min_length=30,
max_time=10.0
)
assert result["tools_count"] == 0, "Tools should be disabled"
print(f"✓ No-tools query: {result['total_time']:.2f}s")
# ============================================================================
# PERFORMANCE BENCHMARKS
# ============================================================================
@pytest.mark.asyncio
async def test_performance_baseline():
"""
Performance Baseline Test
Establishes baseline metrics for comparison across versions
"""
async with AIFlowTester() as tester:
queries = [
"What is containerization?",
"Explain Kubernetes in one sentence",
"What does API stand for?",
]
times = []
for query in queries:
result = await tester.chat(query)
assert result["success"], f"Query failed: {query}"
times.append(result["total_time"])
avg_time = sum(times) / len(times)
print(f"\n✓ Performance baseline:")
print(f" Average response time: {avg_time:.2f}s")
print(f" Range: {min(times):.2f}s - {max(times):.2f}s")
# Assert reasonable performance
assert avg_time < 10.0, f"Average response too slow: {avg_time:.2f}s"
# ============================================================================
# REGRESSION TESTS
# ============================================================================
@pytest.mark.asyncio
async def test_regression_tool_availability():
"""
Regression: Verify all expected tools are available
"""
async with AIFlowTester() as tester:
response = await tester.client.get(f"{BASE_URL}/v1/tools")
response.raise_for_status()
data = response.json()
tools = {tool["name"] for tool in data["tools"]}
# Expected tools from local.py
expected_tools = {
"get_current_time",
"get_current_date",
"calculate_date_difference",
"add_days_to_date",
"calculate",
"web_search"
}
for tool in expected_tools:
assert tool in tools, f"Missing tool: {tool}"
print(f"✓ Tool availability: {len(tools)} tools registered")
@pytest.mark.asyncio
async def test_regression_health_check():
"""
Regression: Health check endpoint works
"""
async with AIFlowTester() as tester:
response = await tester.client.get(f"{BASE_URL}/health")
response.raise_for_status()
data = response.json()
assert data["status"] == "ok", "Service not healthy"
assert data["service"] == "core-ai"
assert data["agents"]["ollama-native"] is True, \
"OllamaNativeAgent not available"
print(f"✓ Health check passed")
print(f" Default agent: {data['default_agent']}")
print(f" Tools count: {data['tools_count']}")
# ============================================================================
# MAIN RUNNER (for standalone execution)
# ============================================================================
async def run_full_quality_check():
"""Run all tests and generate comprehensive report"""
tester = AIFlowTester()
test_scenarios = [
{
"name": "Scenario 1: Simple Knowledge",
"query": "What is Docker?",
"test": test_scenario1_simple_knowledge
},
{
"name": "Scenario 2: Web Search",
"query": "What are the latest Python 3.13 features?",
"test": test_scenario2_web_search
},
{
"name": "Scenario 3: Calculation",
"query": "Calculate 2847 * 1923 + 5612 - 999",
"test": test_scenario3_calculation
},
{
"name": "Scenario 4: Date Operations",
"query": "What is the current date and what will it be in 30 days?",
"test": test_scenario4_date_operations
},
{
"name": "Scenario 5: Multi-tool Reasoning",
"query": "Get current time in NYC and Tokyo, calculate difference",
"test": test_scenario5_multi_tool_reasoning
},
{
"name": "Scenario 6: DNS Lookup",
"query": "What are the A records for github.com? Use the core-api DNS lookup tool.",
"test": test_scenario6_dns_lookup
},
]
print("\n" + "=" * 80)
print("CORE-AI QUALITY CHECK")
print(f"Started: {datetime.now().isoformat()}")
print(f"Git Tag: {tester.git_info['tag']}")
print(f"Git Commit: {tester.git_info['commit']}")
print("=" * 80 + "\n")
async with tester:
for scenario in test_scenarios:
print(f"\nRunning: {scenario['name']}")
print(f"Query: {scenario['query']}")
print("-" * 80)
try:
await scenario["test"]()
tester.results.append({
"name": scenario["name"],
"query": scenario["query"],
"result": {"success": True, "total_time": 0, "tools_count": 0, "response": ""},
"passed": True
})
except Exception as e:
tester.results.append({
"name": scenario["name"],
"query": scenario["query"],
"result": {"success": False, "error": str(e), "total_time": 0, "tools_count": 0, "response": ""},
"passed": False,
"error": str(e)
})
print(f"✗ FAILED: {e}")
# Generate and print report
report = tester.generate_report()
print("\n" + report)
# Ensure reports directory exists
REPORTS_DIR.mkdir(parents=True, exist_ok=True)
# Save report to file
timestamp = datetime.now().strftime('%Y%m%d-%H%M%S')
report_file = REPORTS_DIR / f"quality-report-{timestamp}.txt"
with open(report_file, "w") as f:
f.write(report)
print(f"\nReport saved to: {report_file}")
# Also save as JSON for programmatic analysis
json_file = REPORTS_DIR / f"quality-report-{timestamp}.json"
json_data = {
"timestamp": datetime.now().isoformat(),
"git_info": tester.git_info,
"summary": {
"total": len(tester.results),
"passed": sum(1 for r in tester.results if r.get("passed", False)),
"failed": sum(1 for r in tester.results if not r.get("passed", False))
},
"results": tester.results
}
with open(json_file, "w") as f:
json.dump(json_data, f, indent=2)
print(f"JSON report saved to: {json_file}")
if __name__ == "__main__":
asyncio.run(run_full_quality_check())