tatlock/docs/housekeeper-optimization-findings.md

Housekeeper Agent Optimization Findings

Background

Research with Gemini identified key issues with mistral-nemo and tool calling:

• "Pre-computation Hallucination" - model answers before using tools
• High default temperature (0.7-0.8) causes wandering
• Model is "chatty and confident" - needs explicit constraints

Key Recommendations from Gemini Research

1. Temperature 0.0 for tool-calling agents (deterministic, follows schema)
2. Chain of Thought (CoT) - force step-by-step reasoning
3. Negative constraints - tell model what NOT to do (Nemo responds better)
4. Explicit tool descriptions - verbose docstrings with "never estimate yourself"
5. "Strictly tool-based assistant" pattern - NO internal knowledge claim

---

Experiment Log

Baseline (v1.8.6)

• Date: 2025-12-17
• Configuration: Default temperature, improved prompt requiring list_devices first
• Results:
  • Called list_devices first ✓
  • Still hallucinated light.study_desk despite seeing list with only light.study and light.study_main
  • Partial success: turned off light.study_main, failed on hallucinated entity
• Success rate: ~50% (1 of 2 study lights controlled correctly)

---

Experiment 1: Temperature 0.0

• Date: 2025-12-18

• Change: Set model_settings=ModelSettings(temperature=0.0) for Housekeeper

• Hypothesis: Deterministic output will force model to use exact entity IDs from tool results

• Results:

  Study lights test:

  • Called list_devices() first ✓ (but no domain filter)
  • Used wrong parameter device_id instead of entity_id (recovered after validation error)
  • Only identified light.studeerlamp as "study" related (Dutch name)
  • Missed light.study and light.study_main - didn't match English "study"
  • Turned off 1 wrong light, missed 2 actual study lights

  Kitchen lights test:

  • Called list_devices() first ✓ (no domain filter)
  • Saw full device list including light.kitchen
  • Used wrong parameter device_id instead of entity_id (recovered after validation)
  • After correction, dropped domain prefix: used kitchen instead of light.kitchen
  • 404 error - device not found

• Success rate: 0% (no target lights successfully controlled)

• Observations:

  • Temperature 0.0 alone is insufficient
  • Model consistently confuses device_id vs entity_id parameter name
  • After validation error correction, model truncates entity_id (drops domain prefix)
  • Semantic matching of room names to devices is weak
  • Model doesn't understand entity_id format: domain.name

---

Experiment 2: Negative Constraints + CoT

• Date: 2025-12-18

• Change: Complete prompt rewrite with:

  • "You have NO Internal Knowledge" - negative framing
  • Explicit entity_id format with WRONG/RIGHT examples
  • Step-by-step process (ALWAYS FOLLOW)
  • Explicit parameter names section
  • "What NOT To Do" negative constraints

• Hypothesis: Negative constraints work better with Mistral-Nemo

• Results:

  Study lights test:

  • Called list_devices(domain="light") ✓ with domain filter (improvement!)
  • Still used device_id first, recovered to entity_id after validation error
  • After recovery, used correct full format: light.studeerlamp
  • Still only matched studeerlamp not light.study or light.study_main

  Kitchen lights test:

  • Called list_devices(domain="light") ✓
  • Called turn_off(entity_id="light.kitchen") ✓ correct format!
  • All 4 kitchen lights turned off (light.kitchen is a group)
  • 100% success for kitchen!

• Success rate:

  • Study: 0% (wrong semantic match)
  • Kitchen: 100% (4/4 lights off)
  • Combined: ~50% (1 of 2 tests successful)

• Observations:

  • Domain filter now consistently used ✓
  • Entity_id format correct after recovery ✓
  • Semantic matching still fails for "study" → prefers Dutch "studeerlamp" over English "study"
  • Parameter name confusion persists (device_id vs entity_id)
  • Simple room names (kitchen) work; mixed language fails (study/studeerlamp)

---

Experiment 3: Temperature 0.1 + Explicit Tool Docstrings

• Date: 2025-12-18
• Change:
  • Temperature 0.1
  • Updated turn_on/turn_off docstrings with explicit entity_id= in examples
• Results:
  • Still uses device_id first, recovers to entity_id after validation
  • Still picks wrong entity (studeerlamp over study)
• Success rate: 0%

---

Experiment 4: Room Group Priority (with explicit examples)

• Date: 2025-12-18

• Change: Updated prompt with:

  • Explicit instruction: "Look for EXACT match light.<room_name> first!"
  • Concrete examples: "For 'study lights' → look for light.study"
  • Working example showing turn_off(entity_id="light.study")

• Hypothesis: Explicit examples will guide model to use room groups

• Results:

  Test 1 & 2 (consecutive):

  • Called list_devices(domain="light") ✓
  • Device list clearly shows light.study at the bottom
  • First call: turn_off({"devices":["studeerlamp"]}) - wrong param AND wrong device
  • After validation error: turn_off(entity_id="light.studeerlamp") - correct param, still wrong device
  • Completely ignored light.study despite prompt explicitly saying to use it

• Success rate: 0% (wrong device controlled)

• Observations:

  • Model ignores explicit step-by-step instructions in favor of substring matching
  • Dutch "studeerlamp" contains "studer" which the model prefers over exact "study" match
  • Even when prompt has a literal example turn_off(entity_id="light.study"), model uses light.studeerlamp
  • Positional bias possible - light.study appears at end of 21-item list
  • Fundamental limitation: Mistral-Nemo cannot follow explicit matching rules

---

Experiment 5: Room Groups First (Tool Output Ordering)

• Date: 2025-12-18
• Change: Modified list_devices to sort room groups to top of list using HA attributes (is_hue_group, hue_type="room")
• Hypothesis: Positional bias - model focuses on items earlier in list
• Results:
  • Room groups (light.study, light.kitchen, etc.) now appear first in device list
  • Combined with improved prompt, model now consistently uses room groups
  • 70% success rate (7/10 tests) with default q4 quantization

---

Experiment 6: Model Quantization (q5_1)

• Date: 2025-12-18

• Change: Upgraded from default Mistral-Nemo quantization (q4) to mistral-nemo:12b-instruct-2407-q5_1

• Hypothesis: Higher precision weights improve tool calling accuracy

• Results:

  Test	Action	Result
  1	Turn off study	PASS
  2	Turn on study	PASS
  3	Toggle study	PASS
  4	Turn off kitchen	PASS
  5	Turn on kitchen	PASS
  6	Toggle kitchen	PASS
  7	Turn off bedroom	PASS
  8	Turn on bedroom	PASS
  9	Turn off living room	PASS
  10	Turn on living room	PASS

• Success rate: 100% (10/10 tests)

• Observations:

  • q5_1 quantization dramatically improves tool calling accuracy
  • All room groups correctly identified and used
  • No parameter confusion (entity_id used correctly)
  • No entity_id truncation issues
  • Toggle operations now work reliably
  • Model fits within 10GB VRAM (q6 did not)

---

Experiment 7: Device List in System Prompt (Context Injection)

• Date: [PENDING]
• Change: Store device list in database (per user/household) and inject into system prompt
• Approach:
  1. Periodically sync device list from Home Assistant to PostgreSQL
  2. On each Housekeeper invocation, fetch device list and include in prompt
  3. Remove need for model to call list_devices() - just match from context
• Hypothesis:
  • Eliminates tool call step where errors occur
  • Reduces context size by not returning full device list as tool output
  • Makes entity matching a language task (in prompt) rather than tool result parsing
• Trade-offs:
  • Stale data if sync is infrequent
  • Prompt size increase (but less than tool call response)
  • Need sync mechanism and storage
• Results: [TO BE RECORDED]
• Success rate: [TO BE RECORDED]

---

Key Problem Identified (Solved)

The model struggled with:

1. Parameter schema adherence - uses device_id when schema requires entity_id
2. Value preservation - truncates values after validation errors (drops light. prefix)
3. Semantic matching - prefers substring matches ("studeerlamp" contains "studer") over exact matches (light.study)
4. Following explicit instructions - ignores step-by-step processes even when examples are provided
5. Positional bias - may not "see" items at the end of long lists

Solution: These issues were resolved by:

1. Using q5_1 quantization instead of default q4 (higher precision weights)
2. Sorting room groups to top of device list (address positional bias)
3. Explicit prompt guidance with negative constraints and examples

---

Potential Next Experiments

Experiment 5: Room Groups First (List Ordering)

• Hypothesis: Positional bias - model focuses on items earlier in list
• Change: Sort device list to put room groups (entities matching light.<single_word>) at the TOP
• Effort: Low - modify list_devices output formatting
• Risk: May affect other use cases where individual devices are needed

Experiment 6: Simplified Device List Format

• Hypothesis: Markdown formatting adds noise that confuses the model
• Change: Return simple list: light.study (Study - GROUP), light.study_main (Ceiling light), ...
• Effort: Low - modify list_devices output
• Risk: Less human-readable responses

---

Learnings to Apply Elsewhere

1. Quantization matters - q5_1 dramatically outperforms q4 for tool calling (100% vs 70%)
2. Positional bias is real - sort important items to top of lists
3. Smaller models need simpler workflows - fewer tool calls, more context injection
4. Validation errors don't teach - model often makes worse mistakes on retry
5. Entity IDs are hard - domain.name format confuses the model
6. Consider pre-computation - move matching logic to code, not LLM
7. Use explicit negative constraints - "NEVER do X" works better than "always do Y"

---

Notes

• Librarian may need higher temperature for creative synthesis
• All "action" agents (Housekeeper, future agents) should use low temperature
• Consider testing with Gemma 2 9B for better function calling (Google, open weights)