portainer-core/docs/sessions/2025-11-24-vram-optimization-analysis.md

VRAM Budget Analysis - Multi-Model Strategy

Hardware: RTX 2080 Ti (11GB VRAM) Goal: Keep orchestrator loaded + room for expert models

Current Model Inventory

Model	Size on Disk	VRAM When Loaded	Quantization
mistral:7b	4.4GB	~5.1GB	Q4_K_M
mistral:7b Q4_K_S	4.1GB	~4.7GB	Q4_K_S
mistral:7b Q3_K_M	3.5GB	~4.0GB	Q3_K_M
codegemma:latest	5.0GB	~5.8GB	Unknown
codestral:latest	12GB	~13GB	Too large!

Key Finding: Q3 Removes Tool Support ❌

Critical Issue: The Q3_K_M quantization removes tool calling capability.

mistral:7b Q4_K_M:
  Capabilities: completion, tools  ✅

mistral:7b Q3_K_M:
  Capabilities: completion          ❌ No tools!

This means: You cannot use Q3 for the orchestrator. Tool calling requires Q4 or higher.

---

Scenario Analysis

Scenario 1: Current Setup (mistral:7b Q4_K_M)

Total VRAM:        11.0 GB
├─ mistral:7b Q4:   5.1 GB (46%) ← Orchestrator (always loaded)
├─ Overhead:        1.2 GB (11%)
└─ Available:       4.7 GB (43%) ← For expert models

What fits in 4.7GB free space:

• ✅ codegemma:latest (5.8GB) - Does NOT fit (need 5.8GB, have 4.7GB)
• ❌ codestral:latest (13GB) - Does NOT fit (way too large)
• ✅ gemma3:4b (4.5GB) - Barely fits (general purpose)
• ✅ qwen2.5:3b (3.5GB) - Fits comfortably (if available)

Reality Check: You cannot load codegemma or codestral alongside mistral:7b Q4.

---

Scenario 2: Slightly Smaller Q4 (mistral:7b-instruct-q4_K_S)

Total VRAM:        11.0 GB
├─ mistral:7b Q4_K_S: 4.7 GB (43%) ← Orchestrator (slightly smaller)
├─ Overhead:        1.2 GB (11%)
└─ Available:       5.1 GB (46%) ← For expert models

Savings: 400MB (5.1GB → 4.7GB)

What fits now:

• ⚠️ codegemma:latest (5.8GB) - Still doesn't fit (need 5.8GB, have 5.1GB)
• ❌ codestral:latest (13GB) - No chance
• ✅ gemma3:4b (4.5GB) - Fits with room to spare

Benefit: Not enough to matter. Still can't fit codegemma.

---

Scenario 3: Dynamic Loading (Current Ollama Behavior)

This is what Ollama already does by default!

Step 1: Only orchestrator loaded
├─ mistral:7b Q4:   5.1 GB
├─ Overhead:        1.2 GB
└─ Available:       4.7 GB

Step 2: User requests code generation
├─ Unload mistral:7b   (-5.1GB)
├─ Load codestral      (+13GB)   ← Swaps automatically
└─ Available:       0 GB (codestral fills VRAM)

Step 3: Codestral finishes, times out
├─ Unload codestral    (-13GB)
├─ Load mistral:7b     (+5.1GB)  ← Swaps back
└─ Back to Step 1

How it works:

• Ollama has a keep_alive timer (default: 5 minutes)
• When a model isn't used for 5min, it's unloaded from VRAM
• When you request a different model, Ollama swaps them automatically

Cold start times:

• Loading mistral:7b: ~2-3 seconds
• Loading codestral:22b: ~8-10 seconds
• Loading codegemma:9b: ~3-4 seconds

---

The Math: Why Expert Models Don't Fit

Your 11GB VRAM budget breaks down like this:

11GB total VRAM
- 5.1GB orchestrator (mistral:7b Q4)
- 1.2GB system overhead
━━━━━━━━━━━━━━━━━━━━━━
= 4.7GB available

But your expert models need:
- codestral:22b = 13GB  ❌ (needs 8GB more than you have)
- codegemma:9b  = 5.8GB ❌ (needs 1GB more than available)

Even if you use the smallest possible orchestrator:

11GB total VRAM
- 3.8GB orchestrator (gemma3-tools:1b, unreliable!)
- 1.2GB system overhead
━━━━━━━━━━━━━━━━━━━━━━
= 6.0GB available

Still not enough for:
- codestral:22b = 13GB  ❌ (needs 7GB more)
- codegemma:9b  = 5.8GB ✅ (fits, but orchestrator is unreliable)

---

Reality: You Need Dynamic Loading

Conclusion: With 11GB VRAM, you cannot keep both:

1. A reliable orchestrator (min 4.7GB for mistral Q4_K_S)
2. Large expert models (5.8GB+ for code models)

loaded simultaneously.

Option A: Accept Dynamic Loading (Recommended)

Keep orchestrator loaded with keep_alive=-1, but expert models swap in/out:

# In Core API orchestrator.py
self.llm = ChatOllama(
    model="mistral:7b",  # Use Q4_K_M or Q4_K_S
    keep_alive=-1,  # Never unload orchestrator
)

# When calling expert models:
codestral_llm = ChatOllama(
    model="codestral:latest",
    keep_alive="5m",  # Auto-unload after 5 min idle
)

How it works in practice:

1. Orchestrator queries (~80% of requests):

   • mistral:7b always in VRAM
   • Instant response (~0ms cold start)
   • Uses 5.1GB VRAM

2. Code generation (~20% of requests):

   • mistral:7b stays loaded initially
   • Ollama sees codestral request
   • Unloads mistral automatically
   • Loads codestral (8-10s cold start)
   • Codestral generates code
   • After 5min idle: unloads codestral, reloads mistral

Trade-offs:

• ✅ Orchestrator instant most of the time
• ⚠️ 8-10s cold start when switching to codestral (first code request)
• ⚠️ 2-3s cold start when switching back to orchestrator (after codestral timeout)
• ✅ Can use full-size expert models (codestral:22b, etc.)

---

Option B: Use Smaller Expert Models

If cold starts are unacceptable, use smaller expert models that fit alongside orchestrator:

Orchestrator: mistral:7b Q4_K_S (4.7GB)
Expert:       qwen2.5-coder:3b  (3.5GB)  ← Smaller code model
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Total:        8.2GB + 1.2GB overhead = 9.4GB
Available:    1.6GB buffer

Smaller code model options:

• qwen2.5-coder:3b (3.5GB) - Good for simple code tasks
• starcoder2:3b (3.2GB) - Focused on code completion
• deepseek-coder:1.3b (1.5GB) - Very small, lower quality

Trade-offs:

• ✅ Both models always loaded (no cold starts)
• ✅ Instant switching
• ❌ Smaller models = lower code quality
• ❌ Can't use top-tier models like codestral

---

Option C: Upgrade GPU (Future)

If you want both instant orchestrator AND large expert models:

RTX 4070 Ti (16GB VRAM):

Total VRAM:         16GB
├─ mistral:7b Q4:    5.1GB (32%)
├─ codestral:22b:    8.0GB (50%)  ← Quantized version
├─ Overhead:         1.5GB (9%)
└─ Available:        1.4GB (9%)

With 16GB, you can fit:

• Orchestrator + codestral Q4 (13GB total)
• Orchestrator + codegemma (11GB total)
• Orchestrator + multiple small experts

---

Video/Image Models: The Situation

Video and image models are MUCH larger than text models:

Image Generation Models:

• SDXL (Stable Diffusion XL): 6-7GB VRAM
• Flux.1: 16-24GB VRAM (dev/schnell variants)
• SD 1.5: 3-4GB VRAM (older, lower quality)

Video Models:

• AnimateDiff: 8-12GB VRAM
• Stable Video Diffusion: 10-14GB VRAM
• CogVideoX: 16-48GB VRAM

Vision Models (Image Understanding):

• LLaVA 7B: 6-7GB VRAM
• LLaVA 13B: 10-12GB VRAM
• GPT-4V equivalent: 12-16GB VRAM

Reality Check for 11GB VRAM:

Scenario: Orchestrator + Vision Model
├─ mistral:7b Q4:   5.1GB
├─ LLaVA 7B:        6.5GB
━━━━━━━━━━━━━━━━━━━━━━━
Total needed:       11.6GB  ❌ Doesn't fit!

Even the smallest vision model (LLaVA 7B) won't fit alongside your orchestrator.

For image/video generation: You'd need to fully unload the orchestrator to make room.

---

Recommendation: Hybrid Strategy

For your 11GB VRAM constraint, I recommend:

1. Keep Orchestrator Always Loaded

# mistral:7b Q4_K_M (5.1GB) with keep_alive=-1
# Current setup, no changes needed

2. Accept Dynamic Loading for Experts

• Code models: Load on-demand (codestral, codegemma)
• Vision models: Load on-demand (LLaVA)
• Image gen: Load on-demand (SDXL)

3. Optimize with keep_alive Tuning

# Orchestrator: Never unload
orchestrator = ChatOllama(model="mistral:7b", keep_alive=-1)

# Frequently used expert: Keep for 30min
code_expert = ChatOllama(model="codegemma:9b", keep_alive="30m")

# Rarely used expert: Keep for 5min only
vision_expert = ChatOllama(model="llava:7b", keep_alive="5m")

Result:

• Orchestrator: Always instant
• Frequent code requests: 1st request has 3-4s cold start, then instant for 30min
• Rare vision requests: 6-8s cold start each time

4. Monitor and Adjust

Track which expert models you use most:

• If you do a LOT of coding → Keep codegemma loaded longer (keep_alive="1h")
• If coding is rare → Accept the cold start (keep_alive="5m")

---

Future-Proofing

If you want to add image/video in the future:

Option 1: Offload to CPU (Slow)

# Run image generation on CPU (very slow, 5-10min per image)
OLLAMA_NUM_GPU=0 ollama run stable-diffusion

Option 2: Dedicated GPU

• Keep RTX 2080 Ti for text models (orchestrator + code)
• Add second GPU for image/video (RTX 3060 12GB, ~$250 used)

Option 3: Cloud Hybrid

• Local: Text models (orchestrator, code, chat)
• Cloud: Image/video generation (Replicate API, RunPod, etc.)
• Cost: ~$0.002-0.01 per image

---

Bottom Line

Your VRAM situation:

Capability	Status	Notes
Keep orchestrator loaded	✅ Yes	5.1GB with mistral:7b Q4
+ codegemma simultaneously	❌ No	Need 5.8GB, have 4.7GB free
+ codestral simultaneously	❌ No	Need 13GB, have 4.7GB free
+ vision model simultaneously	❌ No	Need 6GB+, have 4.7GB free
Dynamic loading (swap models)	✅ Yes	2-10s cold starts
Smaller experts simultaneously	✅ Maybe	With 3-4GB models only

Verdict:

• ✅ You CAN keep orchestrator always loaded
• ⚠️ You CANNOT keep large experts loaded simultaneously
• ✅ Dynamic loading works fine with acceptable cold start times
• ❌ Image/video models won't fit even with dynamic loading (need GPU upgrade)

Best approach: Keep current setup (mistral:7b Q4 always loaded), accept dynamic swapping for expert models. It's what Ollama is designed to do, and 3-8s cold starts are acceptable for occasional expert model use.