portainer-core/docs/sessions/2025-11-24-vram-optimization-analysis.md

# VRAM Budget Analysis - Multi-Model Strategy

**Hardware**: RTX 2080 Ti (11GB VRAM)
**Goal**: Keep orchestrator loaded + room for expert models

## Current Model Inventory

| Model | Size on Disk | VRAM When Loaded | Quantization |
|-------|--------------|------------------|--------------|
| **mistral:7b** | 4.4GB | ~5.1GB | Q4_K_M |
| **mistral:7b Q4_K_S** | 4.1GB | ~4.7GB | Q4_K_S |
| **mistral:7b Q3_K_M** | 3.5GB | ~4.0GB | Q3_K_M |
| **codegemma:latest** | 5.0GB | ~5.8GB | Unknown |
| **codestral:latest** | 12GB | ~13GB | Too large! |

## Key Finding: Q3 Removes Tool Support ❌

**Critical Issue**: The Q3_K_M quantization **removes tool calling capability**.

```
mistral:7b Q4_K_M:
  Capabilities: completion, tools  ✅

mistral:7b Q3_K_M:
  Capabilities: completion          ❌ No tools!
```

**This means**: You cannot use Q3 for the orchestrator. Tool calling requires Q4 or higher.

---

## Scenario Analysis

### Scenario 1: Current Setup (mistral:7b Q4_K_M)

```
Total VRAM:        11.0 GB
├─ mistral:7b Q4:   5.1 GB (46%) ← Orchestrator (always loaded)
├─ Overhead:        1.2 GB (11%)
└─ Available:       4.7 GB (43%) ← For expert models
```

**What fits in 4.7GB free space:**
- ✅ codegemma:latest (5.8GB) - **Does NOT fit** (need 5.8GB, have 4.7GB)
- ❌ codestral:latest (13GB) - **Does NOT fit** (way too large)
- ✅ gemma3:4b (4.5GB) - **Barely fits** (general purpose)
- ✅ qwen2.5:3b (3.5GB) - **Fits comfortably** (if available)

**Reality Check**: You **cannot** load codegemma or codestral alongside mistral:7b Q4.

---

### Scenario 2: Slightly Smaller Q4 (mistral:7b-instruct-q4_K_S)

```
Total VRAM:        11.0 GB
├─ mistral:7b Q4_K_S: 4.7 GB (43%) ← Orchestrator (slightly smaller)
├─ Overhead:        1.2 GB (11%)
└─ Available:       5.1 GB (46%) ← For expert models
```

**Savings**: 400MB (5.1GB → 4.7GB)

**What fits now:**
- ⚠️ codegemma:latest (5.8GB) - **Still doesn't fit** (need 5.8GB, have 5.1GB)
- ❌ codestral:latest (13GB) - **No chance**
- ✅ gemma3:4b (4.5GB) - **Fits with room to spare**

**Benefit**: Not enough to matter. Still can't fit codegemma.

---

### Scenario 3: Dynamic Loading (Current Ollama Behavior)

**This is what Ollama already does by default!**

```
Step 1: Only orchestrator loaded
├─ mistral:7b Q4:   5.1 GB
├─ Overhead:        1.2 GB
└─ Available:       4.7 GB

Step 2: User requests code generation
├─ Unload mistral:7b   (-5.1GB)
├─ Load codestral      (+13GB)   ← Swaps automatically
└─ Available:       0 GB (codestral fills VRAM)

Step 3: Codestral finishes, times out
├─ Unload codestral    (-13GB)
├─ Load mistral:7b     (+5.1GB)  ← Swaps back
└─ Back to Step 1
```

**How it works:**
- Ollama has a `keep_alive` timer (default: 5 minutes)
- When a model isn't used for 5min, it's unloaded from VRAM
- When you request a different model, Ollama swaps them automatically

**Cold start times:**
- Loading mistral:7b: ~2-3 seconds
- Loading codestral:22b: ~8-10 seconds
- Loading codegemma:9b: ~3-4 seconds

---

## The Math: Why Expert Models Don't Fit

Your 11GB VRAM budget breaks down like this:

```
11GB total VRAM
- 5.1GB orchestrator (mistral:7b Q4)
- 1.2GB system overhead
━━━━━━━━━━━━━━━━━━━━━━
= 4.7GB available

But your expert models need:
- codestral:22b = 13GB  ❌ (needs 8GB more than you have)
- codegemma:9b  = 5.8GB ❌ (needs 1GB more than available)
```

**Even if you use the smallest possible orchestrator:**
```
11GB total VRAM
- 3.8GB orchestrator (gemma3-tools:1b, unreliable!)
- 1.2GB system overhead
━━━━━━━━━━━━━━━━━━━━━━
= 6.0GB available

Still not enough for:
- codestral:22b = 13GB  ❌ (needs 7GB more)
- codegemma:9b  = 5.8GB ✅ (fits, but orchestrator is unreliable)
```

---

## Reality: You Need Dynamic Loading

**Conclusion**: With 11GB VRAM, you **cannot** keep both:
1. A reliable orchestrator (min 4.7GB for mistral Q4_K_S)
2. Large expert models (5.8GB+ for code models)

**loaded simultaneously**.

### Option A: Accept Dynamic Loading (Recommended)

**Keep orchestrator loaded** with `keep_alive=-1`, but expert models swap in/out:

```python
# In Core API orchestrator.py
self.llm = ChatOllama(
    model="mistral:7b",  # Use Q4_K_M or Q4_K_S
    keep_alive=-1,  # Never unload orchestrator
)

# When calling expert models:
codestral_llm = ChatOllama(
    model="codestral:latest",
    keep_alive="5m",  # Auto-unload after 5 min idle
)
```

**How it works in practice:**

1. **Orchestrator queries** (~80% of requests):
   - mistral:7b always in VRAM
   - Instant response (~0ms cold start)
   - Uses 5.1GB VRAM

2. **Code generation** (~20% of requests):
   - mistral:7b stays loaded initially
   - Ollama sees codestral request
   - **Unloads mistral** automatically
   - **Loads codestral** (8-10s cold start)
   - Codestral generates code
   - After 5min idle: **unloads codestral, reloads mistral**

**Trade-offs:**
- ✅ Orchestrator instant most of the time
- ⚠️ 8-10s cold start when switching to codestral (first code request)
- ⚠️ 2-3s cold start when switching back to orchestrator (after codestral timeout)
- ✅ Can use full-size expert models (codestral:22b, etc.)

---

### Option B: Use Smaller Expert Models

If cold starts are unacceptable, use smaller expert models that fit alongside orchestrator:

```
Orchestrator: mistral:7b Q4_K_S (4.7GB)
Expert:       qwen2.5-coder:3b  (3.5GB)  ← Smaller code model
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Total:        8.2GB + 1.2GB overhead = 9.4GB
Available:    1.6GB buffer
```

**Smaller code model options:**
- `qwen2.5-coder:3b` (3.5GB) - Good for simple code tasks
- `starcoder2:3b` (3.2GB) - Focused on code completion
- `deepseek-coder:1.3b` (1.5GB) - Very small, lower quality

**Trade-offs:**
- ✅ Both models always loaded (no cold starts)
- ✅ Instant switching
- ❌ Smaller models = lower code quality
- ❌ Can't use top-tier models like codestral

---

### Option C: Upgrade GPU (Future)

If you want both instant orchestrator AND large expert models:

**RTX 4070 Ti (16GB VRAM):**
```
Total VRAM:         16GB
├─ mistral:7b Q4:    5.1GB (32%)
├─ codestral:22b:    8.0GB (50%)  ← Quantized version
├─ Overhead:         1.5GB (9%)
└─ Available:        1.4GB (9%)
```

**With 16GB, you can fit:**
- Orchestrator + codestral Q4 (13GB total)
- Orchestrator + codegemma (11GB total)
- Orchestrator + multiple small experts

---

## Video/Image Models: The Situation

Video and image models are **MUCH larger** than text models:

### Image Generation Models:
- **SDXL (Stable Diffusion XL)**: 6-7GB VRAM
- **Flux.1**: 16-24GB VRAM (dev/schnell variants)
- **SD 1.5**: 3-4GB VRAM (older, lower quality)

### Video Models:
- **AnimateDiff**: 8-12GB VRAM
- **Stable Video Diffusion**: 10-14GB VRAM
- **CogVideoX**: 16-48GB VRAM

### Vision Models (Image Understanding):
- **LLaVA 7B**: 6-7GB VRAM
- **LLaVA 13B**: 10-12GB VRAM
- **GPT-4V equivalent**: 12-16GB VRAM

**Reality Check for 11GB VRAM:**

```
Scenario: Orchestrator + Vision Model
├─ mistral:7b Q4:   5.1GB
├─ LLaVA 7B:        6.5GB
━━━━━━━━━━━━━━━━━━━━━━━
Total needed:       11.6GB  ❌ Doesn't fit!
```

Even the smallest vision model (LLaVA 7B) won't fit alongside your orchestrator.

**For image/video generation**: You'd need to fully unload the orchestrator to make room.

---

## Recommendation: Hybrid Strategy

**For your 11GB VRAM constraint, I recommend:**

### 1. Keep Orchestrator Always Loaded
```bash
# mistral:7b Q4_K_M (5.1GB) with keep_alive=-1
# Current setup, no changes needed
```

### 2. Accept Dynamic Loading for Experts
- **Code models**: Load on-demand (codestral, codegemma)
- **Vision models**: Load on-demand (LLaVA)
- **Image gen**: Load on-demand (SDXL)

### 3. Optimize with `keep_alive` Tuning

```python
# Orchestrator: Never unload
orchestrator = ChatOllama(model="mistral:7b", keep_alive=-1)

# Frequently used expert: Keep for 30min
code_expert = ChatOllama(model="codegemma:9b", keep_alive="30m")

# Rarely used expert: Keep for 5min only
vision_expert = ChatOllama(model="llava:7b", keep_alive="5m")
```

**Result:**
- Orchestrator: Always instant
- Frequent code requests: 1st request has 3-4s cold start, then instant for 30min
- Rare vision requests: 6-8s cold start each time

### 4. Monitor and Adjust

Track which expert models you use most:
- If you do a LOT of coding → Keep codegemma loaded longer (`keep_alive="1h"`)
- If coding is rare → Accept the cold start (`keep_alive="5m"`)

---

## Future-Proofing

**If you want to add image/video in the future:**

### Option 1: Offload to CPU (Slow)
```bash
# Run image generation on CPU (very slow, 5-10min per image)
OLLAMA_NUM_GPU=0 ollama run stable-diffusion
```

### Option 2: Dedicated GPU
- Keep RTX 2080 Ti for text models (orchestrator + code)
- Add second GPU for image/video (RTX 3060 12GB, ~$250 used)

### Option 3: Cloud Hybrid
- Local: Text models (orchestrator, code, chat)
- Cloud: Image/video generation (Replicate API, RunPod, etc.)
- Cost: ~$0.002-0.01 per image

---

## Bottom Line

**Your VRAM situation:**

| Capability | Status | Notes |
|------------|--------|-------|
| **Keep orchestrator loaded** | ✅ Yes | 5.1GB with mistral:7b Q4 |
| **+ codegemma simultaneously** | ❌ No | Need 5.8GB, have 4.7GB free |
| **+ codestral simultaneously** | ❌ No | Need 13GB, have 4.7GB free |
| **+ vision model simultaneously** | ❌ No | Need 6GB+, have 4.7GB free |
| **Dynamic loading (swap models)** | ✅ Yes | 2-10s cold starts |
| **Smaller experts simultaneously** | ✅ Maybe | With 3-4GB models only |

**Verdict**:
- ✅ You CAN keep orchestrator always loaded
- ⚠️ You CANNOT keep large experts loaded simultaneously
- ✅ Dynamic loading works fine with acceptable cold start times
- ❌ Image/video models won't fit even with dynamic loading (need GPU upgrade)

**Best approach**: Keep current setup (mistral:7b Q4 always loaded), accept dynamic swapping for expert models. It's what Ollama is designed to do, and 3-8s cold starts are acceptable for occasional expert model use.