# Housekeeper Agent Optimization Findings

## Background

Research with Gemini identified key issues with mistral-nemo and tool calling:
- "Pre-computation Hallucination" - model answers before using tools
- High default temperature (0.7-0.8) causes wandering
- Model is "chatty and confident" - needs explicit constraints

## Key Recommendations from Gemini Research

1. **Temperature 0.0** for tool-calling agents (deterministic, follows schema)
2. **Chain of Thought (CoT)** - force step-by-step reasoning
3. **Negative constraints** - tell model what NOT to do (Nemo responds better)
4. **Explicit tool descriptions** - verbose docstrings with "never estimate yourself"
5. **"Strictly tool-based assistant"** pattern - NO internal knowledge claim

---

## Experiment Log

### Baseline (v1.8.6)
- **Date**: 2025-12-17
- **Configuration**: Default temperature, improved prompt requiring list_devices first
- **Results**:
  - Called list_devices first ✓
  - Still hallucinated `light.study_desk` despite seeing list with only `light.study` and `light.study_main`
  - Partial success: turned off `light.study_main`, failed on hallucinated entity
- **Success rate**: ~50% (1 of 2 study lights controlled correctly)

---

### Experiment 1: Temperature 0.0
- **Date**: 2025-12-18
- **Change**: Set `model_settings=ModelSettings(temperature=0.0)` for Housekeeper
- **Hypothesis**: Deterministic output will force model to use exact entity IDs from tool results
- **Results**:

  **Study lights test:**
  - Called `list_devices()` first ✓ (but no domain filter)
  - Used wrong parameter `device_id` instead of `entity_id` (recovered after validation error)
  - Only identified `light.studeerlamp` as "study" related (Dutch name)
  - **Missed `light.study` and `light.study_main`** - didn't match English "study"
  - Turned off 1 wrong light, missed 2 actual study lights

  **Kitchen lights test:**
  - Called `list_devices()` first ✓ (no domain filter)
  - Saw full device list including `light.kitchen`
  - Used wrong parameter `device_id` instead of `entity_id` (recovered after validation)
  - After correction, dropped domain prefix: used `kitchen` instead of `light.kitchen`
  - 404 error - device not found

- **Success rate**: 0% (no target lights successfully controlled)
- **Observations**:
  - Temperature 0.0 alone is insufficient
  - Model consistently confuses `device_id` vs `entity_id` parameter name
  - After validation error correction, model truncates entity_id (drops domain prefix)
  - Semantic matching of room names to devices is weak
  - Model doesn't understand entity_id format: `domain.name`

---

### Experiment 2: Negative Constraints + CoT
- **Date**: 2025-12-18
- **Change**: Complete prompt rewrite with:
  - "You have NO Internal Knowledge" - negative framing
  - Explicit entity_id format with WRONG/RIGHT examples
  - Step-by-step process (ALWAYS FOLLOW)
  - Explicit parameter names section
  - "What NOT To Do" negative constraints
- **Hypothesis**: Negative constraints work better with Mistral-Nemo
- **Results**:

  **Study lights test:**
  - Called `list_devices(domain="light")` ✓ with domain filter (improvement!)
  - Still used `device_id` first, recovered to `entity_id` after validation error
  - After recovery, used correct full format: `light.studeerlamp`
  - **Still only matched `studeerlamp` not `light.study` or `light.study_main`**

  **Kitchen lights test:**
  - Called `list_devices(domain="light")` ✓
  - Called `turn_off(entity_id="light.kitchen")` ✓ correct format!
  - All 4 kitchen lights turned off (light.kitchen is a group)
  - **100% success for kitchen!**

- **Success rate**:
  - Study: 0% (wrong semantic match)
  - Kitchen: 100% (4/4 lights off)
  - Combined: ~50% (1 of 2 tests successful)
- **Observations**:
  - Domain filter now consistently used ✓
  - Entity_id format correct after recovery ✓
  - Semantic matching still fails for "study" → prefers Dutch "studeerlamp" over English "study"
  - Parameter name confusion persists (`device_id` vs `entity_id`)
  - Simple room names (kitchen) work; mixed language fails (study/studeerlamp)

---

### Experiment 3: Temperature 0.1 + Explicit Tool Docstrings
- **Date**: 2025-12-18
- **Change**:
  - Temperature 0.1
  - Updated turn_on/turn_off docstrings with explicit `entity_id=` in examples
- **Results**:
  - Still uses `device_id` first, recovers to `entity_id` after validation
  - Still picks wrong entity (studeerlamp over study)
- **Success rate**: 0%

---

### Experiment 4: Room Group Priority (with explicit examples)
- **Date**: 2025-12-18
- **Change**: Updated prompt with:
  - Explicit instruction: "Look for EXACT match `light.<room_name>` first!"
  - Concrete examples: "For 'study lights' → look for `light.study`"
  - Working example showing `turn_off(entity_id="light.study")`
- **Hypothesis**: Explicit examples will guide model to use room groups
- **Results**:

  **Test 1 & 2 (consecutive):**
  - Called `list_devices(domain="light")` ✓
  - Device list clearly shows `light.study` at the bottom
  - First call: `turn_off({"devices":["studeerlamp"]})` - wrong param AND wrong device
  - After validation error: `turn_off(entity_id="light.studeerlamp")` - correct param, still wrong device
  - **Completely ignored `light.study` despite prompt explicitly saying to use it**

- **Success rate**: 0% (wrong device controlled)
- **Observations**:
  - Model ignores explicit step-by-step instructions in favor of substring matching
  - Dutch "studeerlamp" contains "studer" which the model prefers over exact "study" match
  - Even when prompt has a literal example `turn_off(entity_id="light.study")`, model uses `light.studeerlamp`
  - Positional bias possible - `light.study` appears at end of 21-item list
  - **Fundamental limitation**: Mistral-Nemo cannot follow explicit matching rules

---

### Experiment 5: Room Groups First (Tool Output Ordering)
- **Date**: 2025-12-18
- **Change**: Modified `list_devices` to sort room groups to top of list using HA attributes (`is_hue_group`, `hue_type="room"`)
- **Hypothesis**: Positional bias - model focuses on items earlier in list
- **Results**:
  - Room groups (`light.study`, `light.kitchen`, etc.) now appear first in device list
  - Combined with improved prompt, model now consistently uses room groups
  - **70% success rate** (7/10 tests) with default q4 quantization

---

### Experiment 6: Model Quantization (q5_1)
- **Date**: 2025-12-18
- **Change**: Upgraded from default Mistral-Nemo quantization (q4) to `mistral-nemo:12b-instruct-2407-q5_1`
- **Hypothesis**: Higher precision weights improve tool calling accuracy
- **Results**:

  | Test | Action | Result |
  |------|--------|--------|
  | 1 | Turn off study | PASS |
  | 2 | Turn on study | PASS |
  | 3 | Toggle study | PASS |
  | 4 | Turn off kitchen | PASS |
  | 5 | Turn on kitchen | PASS |
  | 6 | Toggle kitchen | PASS |
  | 7 | Turn off bedroom | PASS |
  | 8 | Turn on bedroom | PASS |
  | 9 | Turn off living room | PASS |
  | 10 | Turn on living room | PASS |

- **Success rate**: **100%** (10/10 tests)
- **Observations**:
  - q5_1 quantization dramatically improves tool calling accuracy
  - All room groups correctly identified and used
  - No parameter confusion (`entity_id` used correctly)
  - No entity_id truncation issues
  - Toggle operations now work reliably
  - Model fits within 10GB VRAM (q6 did not)

---

### Experiment 7: Device List in System Prompt (Context Injection)
- **Date**: [PENDING]
- **Change**: Store device list in database (per user/household) and inject into system prompt
- **Approach**:
  1. Periodically sync device list from Home Assistant to PostgreSQL
  2. On each Housekeeper invocation, fetch device list and include in prompt
  3. Remove need for model to call list_devices() - just match from context
- **Hypothesis**:
  - Eliminates tool call step where errors occur
  - Reduces context size by not returning full device list as tool output
  - Makes entity matching a language task (in prompt) rather than tool result parsing
- **Trade-offs**:
  - Stale data if sync is infrequent
  - Prompt size increase (but less than tool call response)
  - Need sync mechanism and storage
- **Results**: [TO BE RECORDED]
- **Success rate**: [TO BE RECORDED]

---

## Key Problem Identified (Solved)

The model struggled with:
1. **Parameter schema adherence** - uses `device_id` when schema requires `entity_id`
2. **Value preservation** - truncates values after validation errors (drops `light.` prefix)
3. **Semantic matching** - prefers substring matches ("studeerlamp" contains "studer") over exact matches (`light.study`)
4. **Following explicit instructions** - ignores step-by-step processes even when examples are provided
5. **Positional bias** - may not "see" items at the end of long lists

**Solution**: These issues were resolved by:
1. Using q5_1 quantization instead of default q4 (higher precision weights)
2. Sorting room groups to top of device list (address positional bias)
3. Explicit prompt guidance with negative constraints and examples

---

## Potential Next Experiments

### Experiment 5: Room Groups First (List Ordering)
- **Hypothesis**: Positional bias - model focuses on items earlier in list
- **Change**: Sort device list to put room groups (entities matching `light.<single_word>`) at the TOP
- **Effort**: Low - modify list_devices output formatting
- **Risk**: May affect other use cases where individual devices are needed

### Experiment 6: Simplified Device List Format
- **Hypothesis**: Markdown formatting adds noise that confuses the model
- **Change**: Return simple list: `light.study (Study - GROUP), light.study_main (Ceiling light), ...`
- **Effort**: Low - modify list_devices output
- **Risk**: Less human-readable responses

---

## Learnings to Apply Elsewhere

1. **Quantization matters** - q5_1 dramatically outperforms q4 for tool calling (100% vs 70%)
2. **Positional bias is real** - sort important items to top of lists
3. **Smaller models need simpler workflows** - fewer tool calls, more context injection
4. **Validation errors don't teach** - model often makes worse mistakes on retry
5. **Entity IDs are hard** - domain.name format confuses the model
6. **Consider pre-computation** - move matching logic to code, not LLM
7. **Use explicit negative constraints** - "NEVER do X" works better than "always do Y"

---

## Notes

- Librarian may need higher temperature for creative synthesis
- All "action" agents (Housekeeper, future agents) should use low temperature
- Consider testing with Gemma 2 9B for better function calling (Google, open weights)