portainer-core/docs/sessions/2025-11-24-vram-optimization-strategy.md

VRAM Optimization Strategy for Model Orchestration

Date: 2025-11-24 Context: Multi-model architecture with always-loaded orchestrator + expert models Hardware: RTX 2080 Ti (11GB VRAM)

Problem Statement

Goal: Keep orchestrator model always loaded in VRAM to prevent cold starts, while maximizing VRAM availability for expert models.

Current State:

• Orchestrator: mistral:7b (5.1GB VRAM)
• Free VRAM: 4.7GB
• Use case: Orchestrator decides → routes to expert models (codestral, etc.)

Challenge: mistral:7b consumes 45% of available VRAM, limiting expert model options.

VRAM Budget Analysis

Current Configuration

Total VRAM:        11.0 GB
├─ mistral:7b:      5.1 GB (46.4%) - Orchestrator
├─ Overhead:        1.2 GB (10.6%) - System/Ollama
└─ Available:       4.7 GB (42.7%) - For expert models

Desired Configuration

Total VRAM:        11.0 GB
├─ Orchestrator:    ??? GB (minimize)
├─ Expert Model:    ??? GB (maximize)
└─ Overhead:        1.2 GB

Solution Options

Option 1: Accept gemma3-tools:1b Limitations ⚠️

VRAM Savings: 3.8GB (5.1GB → 1.3GB)

Orchestrator: gemma3-tools:1b (1.3GB)
Free for experts: 8.5GB

Pros:

• ✅ Massive VRAM savings (74% reduction)
• ✅ Leaves 8.5GB for expert models
• ✅ Can load codestral:22b (full size) + orchestrator simultaneously

Cons:

• ❌ 33% tool calling reliability
• ❌ Wrong tool selection
• ❌ Erratic responses (raw JSON output)
• ❌ Poor user experience

Verdict: ❌ Not recommended - Unreliability hurts more than VRAM savings help

---

Option 2: Use Smaller Quantization of mistral:7b ✅ RECOMMENDED

Ollama supports multiple quantization levels. You're currently using Q4_K_M, but Q2 or Q3 exist.

Available Quantizations:

• Q2_K: ~2.5GB VRAM (70% quality retention, aggressive)
• Q3_K_M: ~3.2GB VRAM (80% quality, good balance)
• Q4_K_M: ~5.1GB VRAM (90% quality, current)
• Q5_K_M: ~6.2GB VRAM (95% quality)
• Q8: ~7.7GB VRAM (99% quality, near full precision)

Recommended: Pull mistral:7b-instruct-q3_K_M

# Pull lower quantization
ollama pull mistral:7b-instruct-q3_K_M

# Update Core API config
# services/core-api/.env
AGENT_MODEL=mistral:7b-instruct-q3_K_M

New VRAM Budget:

Orchestrator: mistral:7b Q3_K_M (3.2GB)
Free for experts: 6.6GB
Savings: 1.9GB (37% reduction)

Pros:

• ✅ 100% tool calling compatibility (same model architecture)
• ✅ 1.9GB VRAM savings
• ✅ Minimal quality loss (80% of full precision is fine for routing)
• ✅ Proven reliability maintained

Cons:

• ⚠️ Slightly lower response quality (acceptable for orchestration)
• ⚠️ May need testing to verify tool calling still works

Verdict: ✅ Best option - Balanced approach

---

Option 3: Hybrid Orchestrator (Simple Router + mistral:7b) 🔮 ADVANCED

Use a two-tier routing system:

1. Lightweight classifier (gemma3-tools:1b) - Always loaded
2. Full orchestrator (mistral:7b) - Loaded on demand for complex queries

Architecture:

# Tier 1: Fast classifier (always loaded)
if query_is_simple(message):
    # Direct routing: "list services" → list_services tool
    # Load time: 0ms (always in VRAM)
    use_simple_router(gemma3-tools:1b)
else:
    # Complex routing: multi-tool, reasoning needed
    # Load time: ~2s (load mistral:7b)
    use_full_orchestrator(mistral:7b)

VRAM Budget:

Tier 1 (always): gemma3-tools:1b (1.3GB)
Tier 2 (on-demand): mistral:7b (5.1GB, loaded when needed)
Free when Tier 1 only: 8.5GB
Free when both loaded: 3.4GB

Pros:

• ✅ 8.5GB free for expert models most of the time
• ✅ Only loads mistral:7b when truly needed
• ✅ Simple queries stay fast (no model swap)

Cons:

• ❌ Complex implementation (need query classifier)
• ❌ 2s latency spike when switching to Tier 2
• ❌ More failure modes (what if Tier 1 misclassifies?)

Verdict: 🔮 Future enhancement - Interesting but complex

---

Option 4: Use Different Base Model 🔍 RESEARCH NEEDED

Look for other tool-capable models with better size/quality trade-offs.

Candidates to research:

• qwen2.5:7b-instruct-q3 - Alibaba's model, claimed good tool support
• llama3.2:3b-instruct - Meta's latest, check if tool-capable
• hermes3:3b - Nous Research, specifically trained for function calling

Action: Test these if available in Ollama registry.

---

Recommended Implementation: Option 2

Step 1: Pull Q3 Quantization

# Check if Q3 variant exists
ollama list | grep mistral

# Pull Q3 quantization (if available)
ollama pull mistral:7b-instruct-q3_K_M

# OR manually create Q3 from modelfile
cat > /tmp/mistral-q3.Modelfile << 'EOF'
FROM mistral:7b
PARAMETER quantization Q3_K_M
EOF

ollama create mistral:7b-q3 -f /tmp/mistral-q3.Modelfile

Step 2: Test Tool Calling with Q3

# Run our test script with Q3 variant
source .venv/bin/activate
python3 << 'PYEOF'
import asyncio
import httpx

async def test():
    payload = {
        "model": "mistral:7b-q3",
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "List all services"}
        ],
        "tools": [{
            "type": "function",
            "function": {
                "name": "list_services",
                "description": "List all running services",
                "parameters": {"type": "object", "properties": {}}
            }
        }]
    }

    async with httpx.AsyncClient(timeout=60) as client:
        r = await client.post("http://localhost:11434/api/chat", json=payload)
        data = r.json()
        message = data.get("message", {})

        if "tool_calls" in message:
            print("✅ Q3 quantization: Tool calling WORKS")
            print(f"   Called: {message['tool_calls'][0]['function']['name']}")
        else:
            print("❌ Q3 quantization: Tool calling BROKEN")
            print(f"   Response: {message.get('content', '')[:100]}")

asyncio.run(test())
PYEOF

Step 3: Update Core API Configuration

# services/core-api/.env
AGENT_MODEL=mistral:7b-q3

# Restart core-api to pick up new model
docker restart core-api

Step 4: Verify VRAM Usage

# Check new VRAM allocation
curl -s http://localhost:11434/api/ps | jq '.models[] | {name, size_vram_gb: (.size_vram / 1024 / 1024 / 1024)}'

Expected Result:

{
  "name": "mistral:7b-q3",
  "size_vram_gb": 3.2
}

---

Expert Model Strategy

With ~6.6GB available after Q3 orchestrator, you can now fit:

Option A: Single Large Expert

Orchestrator: mistral:7b-q3 (3.2GB)
Expert:       codestral:22b-q2 (6GB)
Total:        9.2GB / 11GB

Option B: Multiple Smaller Experts

Orchestrator: mistral:7b-q3 (3.2GB)
Expert 1:     codegemma:7b (4GB) - Code generation
Expert 2:     gemma3:4b (3GB) - General knowledge
Total:        10.2GB / 11GB (near full capacity)

Option C: Dynamic Loading (Current Behavior)

Orchestrator: mistral:7b-q3 (3.2GB) - Always loaded
Expert:       Load on demand (6.6GB available)
              - codestral for code
              - gemma3:12b for general
              - Model swaps as needed

---

Advanced: Ollama Keep Alive Configuration

Control how long models stay in VRAM:

# Keep orchestrator always loaded (never unload)
curl -X POST http://localhost:11434/api/generate \
  -d '{
    "model": "mistral:7b-q3",
    "keep_alive": -1,
    "prompt": "warm up"
  }'

# Expert models: unload after 5 minutes idle
curl -X POST http://localhost:11434/api/generate \
  -d '{
    "model": "codestral:latest",
    "keep_alive": "5m",
    "prompt": "warm up"
  }'

Configuration in Core API:

# services/core-api/src/agent/orchestrator.py

self.llm = ChatOllama(
    model=self.settings.agent_model,  # mistral:7b-q3
    base_url=self.settings.ollama_base_url,
    temperature=0.7,
    keep_alive=-1,  # Never unload orchestrator
)

---

Testing Checklist

Before switching to Q3 quantization:

• Pull or create Q3 variant
• Test tool calling functionality
• Test tool selection accuracy (list_services vs get_service_details)
• Test multi-tool workflows
• Compare response quality vs Q4
• Verify VRAM usage reduction
• Test with Open WebUI
• Monitor for any degradation

If Q3 shows issues:

• Try Q4_K_S (slightly smaller than Q4_K_M)
• Fall back to current Q4_K_M if necessary

---

Alternative Models Research

If mistral Q3 proves insufficient, test these:

qwen2.5:7b (Alibaba Cloud)

• Similar size to mistral
• Claimed excellent tool calling
• May have Q3/Q4 variants available

ollama pull qwen2.5:7b-instruct
# Test with our tool calling script

hermes3:3b (Nous Research)

• Specifically trained for function calling
• 3B parameters (smaller than mistral)
• Check Ollama availability

ollama search hermes3

---

Summary

Immediate Action: Pull mistral:7b with Q3_K_M quantization

# Check available quantizations
ollama show mistral:7b --modelfile

# Pull Q3 if available, or create from Q4
ollama pull mistral:7b-instruct-q3_K_M

Expected Outcome:

• VRAM savings: 1.9GB (5.1GB → 3.2GB)
• Tool calling: Should work (same architecture)
• Quality: 80% of Q4 (acceptable for routing logic)
• Expert model budget: 6.6GB (up from 4.7GB)

Risk Mitigation:

• Test thoroughly before production
• Keep Q4 variant as backup
• Monitor for quality degradation

Long-term:

• Research newer models (qwen2.5, hermes3)
• Consider hybrid routing if complexity justified
• Revisit when Ollama adds model multiplexing features

---

Status: Research complete, awaiting quantization testing Next Steps: User decision on Q3 testing approach