jpmschweitzer/tatlock
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 5 Wiki Activity

feat: optimize Housekeeper for Mistral-Nemo tool calling

Browse Source 
- Rewrite system prompt with negative constraints and step-by-step process
- Set temperature to 0.1 for deterministic tool calling
- Sort room groups to top of device list (address positional bias)
- Add [ROOM GROUP] marker in list_devices output
- Update tool docstrings with explicit entity_id= parameter examples
- Add optimization findings doc (experiment log: 0% → 100% success)
- Add test script for room group detection regression testing

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
jpmschweitzer
2025-12-18 20:58:39 +01:00
co-authored by Claude Opus 4.5
parent 43c09f9922
commit 363ab378af
5 changed files with 488 additions and 86 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/housekeeper-optimization-findings.md
+246
View File
@@ -0,0 +1,246 @@
# Housekeeper Agent Optimization Findings
## Background
Research with Gemini identified key issues with mistral-nemo and tool calling:
- "Pre-computation Hallucination" - model answers before using tools
- High default temperature (0.7-0.8) causes wandering
- Model is "chatty and confident" - needs explicit constraints
## Key Recommendations from Gemini Research
1. **Temperature 0.0** for tool-calling agents (deterministic, follows schema)
2. **Chain of Thought (CoT)** - force step-by-step reasoning
3. **Negative constraints** - tell model what NOT to do (Nemo responds better)
4. **Explicit tool descriptions** - verbose docstrings with "never estimate yourself"
5. **"Strictly tool-based assistant"** pattern - NO internal knowledge claim
---
## Experiment Log
### Baseline (v1.8.6)
- **Date**: 2025-12-17
- **Configuration**: Default temperature, improved prompt requiring list_devices first
- **Results**:
- Called list_devices first ✓
- Still hallucinated `light.study_desk` despite seeing list with only `light.study` and `light.study_main`
- Partial success: turned off `light.study_main`, failed on hallucinated entity
- **Success rate**: ~50% (1 of 2 study lights controlled correctly)
---
### Experiment 1: Temperature 0.0
- **Date**: 2025-12-18
- **Change**: Set `model_settings=ModelSettings(temperature=0.0)` for Housekeeper
- **Hypothesis**: Deterministic output will force model to use exact entity IDs from tool results
- **Results**:
**Study lights test:**
- Called `list_devices()` first ✓ (but no domain filter)
- Used wrong parameter `device_id` instead of `entity_id` (recovered after validation error)
- Only identified `light.studeerlamp` as "study" related (Dutch name)
- **Missed `light.study` and `light.study_main`** - didn't match English "study"
- Turned off 1 wrong light, missed 2 actual study lights
**Kitchen lights test:**
- Called `list_devices()` first ✓ (no domain filter)
- Saw full device list including `light.kitchen`
- Used wrong parameter `device_id` instead of `entity_id` (recovered after validation)
- After correction, dropped domain prefix: used `kitchen` instead of `light.kitchen`
- 404 error - device not found
- **Success rate**: 0% (no target lights successfully controlled)
- **Observations**:
- Temperature 0.0 alone is insufficient
- Model consistently confuses `device_id` vs `entity_id` parameter name
- After validation error correction, model truncates entity_id (drops domain prefix)
- Semantic matching of room names to devices is weak
- Model doesn't understand entity_id format: `domain.name`
---
### Experiment 2: Negative Constraints + CoT
- **Date**: 2025-12-18
- **Change**: Complete prompt rewrite with:
- "You have NO Internal Knowledge" - negative framing
- Explicit entity_id format with WRONG/RIGHT examples
- Step-by-step process (ALWAYS FOLLOW)
- Explicit parameter names section
- "What NOT To Do" negative constraints
- **Hypothesis**: Negative constraints work better with Mistral-Nemo
- **Results**:
**Study lights test:**
- Called `list_devices(domain="light")` ✓ with domain filter (improvement!)
- Still used `device_id` first, recovered to `entity_id` after validation error
- After recovery, used correct full format: `light.studeerlamp`
- **Still only matched `studeerlamp` not `light.study` or `light.study_main`**
**Kitchen lights test:**
- Called `list_devices(domain="light")` ✓
- Called `turn_off(entity_id="light.kitchen")` ✓ correct format!
- All 4 kitchen lights turned off (light.kitchen is a group)
- **100% success for kitchen!**
- **Success rate**:
- Study: 0% (wrong semantic match)
- Kitchen: 100% (4/4 lights off)
- Combined: ~50% (1 of 2 tests successful)
- **Observations**:
- Domain filter now consistently used ✓
- Entity_id format correct after recovery ✓
- Semantic matching still fails for "study" → prefers Dutch "studeerlamp" over English "study"
- Parameter name confusion persists (`device_id` vs `entity_id`)
- Simple room names (kitchen) work; mixed language fails (study/studeerlamp)
---
### Experiment 3: Temperature 0.1 + Explicit Tool Docstrings
- **Date**: 2025-12-18
- **Change**:
- Temperature 0.1
- Updated turn_on/turn_off docstrings with explicit `entity_id=` in examples
- **Results**:
- Still uses `device_id` first, recovers to `entity_id` after validation
- Still picks wrong entity (studeerlamp over study)
- **Success rate**: 0%
---
### Experiment 4: Room Group Priority (with explicit examples)
- **Date**: 2025-12-18
- **Change**: Updated prompt with:
- Explicit instruction: "Look for EXACT match `light.<room_name>` first!"
- Concrete examples: "For 'study lights' → look for `light.study`"
- Working example showing `turn_off(entity_id="light.study")`
- **Hypothesis**: Explicit examples will guide model to use room groups
- **Results**:
**Test 1 & 2 (consecutive):**
- Called `list_devices(domain="light")` ✓
- Device list clearly shows `light.study` at the bottom
- First call: `turn_off({"devices":["studeerlamp"]})` - wrong param AND wrong device
- After validation error: `turn_off(entity_id="light.studeerlamp")` - correct param, still wrong device
- **Completely ignored `light.study` despite prompt explicitly saying to use it**
- **Success rate**: 0% (wrong device controlled)
- **Observations**:
- Model ignores explicit step-by-step instructions in favor of substring matching
- Dutch "studeerlamp" contains "studer" which the model prefers over exact "study" match
- Even when prompt has a literal example `turn_off(entity_id="light.study")`, model uses `light.studeerlamp`
- Positional bias possible - `light.study` appears at end of 21-item list
- **Fundamental limitation**: Mistral-Nemo cannot follow explicit matching rules
---
### Experiment 5: Room Groups First (Tool Output Ordering)
- **Date**: 2025-12-18
- **Change**: Modified `list_devices` to sort room groups to top of list using HA attributes (`is_hue_group`, `hue_type="room"`)
- **Hypothesis**: Positional bias - model focuses on items earlier in list
- **Results**:
- Room groups (`light.study`, `light.kitchen`, etc.) now appear first in device list
- Combined with improved prompt, model now consistently uses room groups
- **70% success rate** (7/10 tests) with default q4 quantization
---
### Experiment 6: Model Quantization (q5_1)
- **Date**: 2025-12-18
- **Change**: Upgraded from default Mistral-Nemo quantization (q4) to `mistral-nemo:12b-instruct-2407-q5_1`
- **Hypothesis**: Higher precision weights improve tool calling accuracy
- **Results**:
| Test | Action | Result |
|------|--------|--------|
| 1 | Turn off study | PASS |
| 2 | Turn on study | PASS |
| 3 | Toggle study | PASS |
| 4 | Turn off kitchen | PASS |
| 5 | Turn on kitchen | PASS |
| 6 | Toggle kitchen | PASS |
| 7 | Turn off bedroom | PASS |
| 8 | Turn on bedroom | PASS |
| 9 | Turn off living room | PASS |
| 10 | Turn on living room | PASS |
- **Success rate**: **100%** (10/10 tests)
- **Observations**:
- q5_1 quantization dramatically improves tool calling accuracy
- All room groups correctly identified and used
- No parameter confusion (`entity_id` used correctly)
- No entity_id truncation issues
- Toggle operations now work reliably
- Model fits within 10GB VRAM (q6 did not)
---
### Experiment 7: Device List in System Prompt (Context Injection)
- **Date**: [PENDING]
- **Change**: Store device list in database (per user/household) and inject into system prompt
- **Approach**:
1. Periodically sync device list from Home Assistant to PostgreSQL
2. On each Housekeeper invocation, fetch device list and include in prompt
3. Remove need for model to call list_devices() - just match from context
- **Hypothesis**:
- Eliminates tool call step where errors occur
- Reduces context size by not returning full device list as tool output
- Makes entity matching a language task (in prompt) rather than tool result parsing
- **Trade-offs**:
- Stale data if sync is infrequent
- Prompt size increase (but less than tool call response)
- Need sync mechanism and storage
- **Results**: [TO BE RECORDED]
- **Success rate**: [TO BE RECORDED]
---
## Key Problem Identified (Solved)
The model struggled with:
1. **Parameter schema adherence** - uses `device_id` when schema requires `entity_id`
2. **Value preservation** - truncates values after validation errors (drops `light.` prefix)
3. **Semantic matching** - prefers substring matches ("studeerlamp" contains "studer") over exact matches (`light.study`)
4. **Following explicit instructions** - ignores step-by-step processes even when examples are provided
5. **Positional bias** - may not "see" items at the end of long lists
**Solution**: These issues were resolved by:
1. Using q5_1 quantization instead of default q4 (higher precision weights)
2. Sorting room groups to top of device list (address positional bias)
3. Explicit prompt guidance with negative constraints and examples
---
## Potential Next Experiments
### Experiment 5: Room Groups First (List Ordering)
- **Hypothesis**: Positional bias - model focuses on items earlier in list
- **Change**: Sort device list to put room groups (entities matching `light.<single_word>`) at the TOP
- **Effort**: Low - modify list_devices output formatting
- **Risk**: May affect other use cases where individual devices are needed
### Experiment 6: Simplified Device List Format
- **Hypothesis**: Markdown formatting adds noise that confuses the model
- **Change**: Return simple list: `light.study (Study - GROUP), light.study_main (Ceiling light), ...`
- **Effort**: Low - modify list_devices output
- **Risk**: Less human-readable responses
---
## Learnings to Apply Elsewhere
1. **Quantization matters** - q5_1 dramatically outperforms q4 for tool calling (100% vs 70%)
2. **Positional bias is real** - sort important items to top of lists
3. **Smaller models need simpler workflows** - fewer tool calls, more context injection
4. **Validation errors don't teach** - model often makes worse mistakes on retry
5. **Entity IDs are hard** - domain.name format confuses the model
6. **Consider pre-computation** - move matching logic to code, not LLM
7. **Use explicit negative constraints** - "NEVER do X" works better than "always do Y"
---
## Notes
- Librarian may need higher temperature for creative synthesis
- All "action" agents (Housekeeper, future agents) should use low temperature
- Consider testing with Gemma 2 9B for better function calling (Google, open weights)