fix(steward): route on the declared DELEGATE line, not on prose
The prompt tells the Steward to state its choice on a DELEGATE line and to explain itself on REASON, COMPLEXITY and CONTEXT lines. Extraction ignored that structure and substring-matched capability domains across the entire response, so ordinary English in the explanation selected agents: "description" contains the housekeeper domain "script", "discover" contains "cover", "acknowledge" contains "knowledge" and "know", "economy" contains the biographer domain "my". Every one of those was a real delegation. A spurious librarian is a multi-second web call on a query that asked for arithmetic. It also made prose length a routing input, which would have quietly corrupted the thinking benchmark this was found during: anything that shortened the Steward's output reduces accidental substring hits and so reads as improved routing. Resolution is now layered, most explicit first — a DELEGATE line opening with a capability name, then a capability named anywhere on that line, then a domain on that line. With no DELEGATE line at all the response is matched on capability names only, never domains, so the conversational path still answers with no capabilities. Matching is whole-word throughout. Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -19,35 +19,88 @@ from .schemas import ConversationContext, StewardRecommendation
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
_DELEGATE_LINE_RE = re.compile(r"^[ \t]*DELEGATE:[ \t]*(.+)$", re.IGNORECASE | re.MULTILINE)
|
||||
|
||||
|
||||
def _mentions(needle: str, haystack: str) -> bool:
|
||||
"""Whole-word containment. Substring matching is what made this go wrong."""
|
||||
return re.search(rf"(?<!\w){re.escape(needle)}(?!\w)", haystack) is not None
|
||||
|
||||
|
||||
def _extract_capabilities(text: str) -> list[str]:
|
||||
"""
|
||||
Extract capability names from Steward's text response.
|
||||
Extract capability names from the Steward's declared delegation.
|
||||
|
||||
Uses keyword matching to find mentioned capabilities.
|
||||
The prompt instructs the Steward to answer in a fixed shape::
|
||||
|
||||
DELEGATE: <capability> to <action> <task>
|
||||
REASON: ...
|
||||
COMPLEXITY: ...
|
||||
CONTEXT: ...
|
||||
|
||||
Only the DELEGATE line states intent; the rest is free prose. An earlier
|
||||
version substring-matched capability *domains* across the whole response,
|
||||
which routed on ordinary English: "description" contains "script" and
|
||||
"discover" contains "cover" (both housekeeper domains), "acknowledge"
|
||||
contains "knowledge" and "know" (librarian, biographer), and "economy"
|
||||
contains "my" (biographer). Any REASON line could therefore summon agents
|
||||
the Steward never asked for, and a spurious librarian is a real
|
||||
multi-second web call.
|
||||
|
||||
It also made prose length a routing input, so anything that shortened the
|
||||
Steward's output — such as disabling model thinking — would look like it had
|
||||
improved routing.
|
||||
|
||||
Resolution is layered, most explicit first:
|
||||
1. a DELEGATE line beginning with a capability name — the documented shape
|
||||
2. a capability named anywhere on a DELEGATE line
|
||||
3. a capability *domain* on a DELEGATE line, for a loosely worded answer
|
||||
4. no DELEGATE line: capability names only, never domains
|
||||
|
||||
Args:
|
||||
text: Steward's plain text analysis
|
||||
|
||||
Returns:
|
||||
List of capability names (e.g., ['tatlock_core'])
|
||||
List of capability names (e.g. ['tatlock_core']), de-duplicated.
|
||||
"""
|
||||
text_lower = text.lower()
|
||||
registry = get_household_registry()
|
||||
capabilities = registry.get_all_capabilities()
|
||||
delegate_lines = [line.strip().lower() for line in _DELEGATE_LINE_RE.findall(text or "")]
|
||||
|
||||
found_caps = []
|
||||
found_caps: list[str] = []
|
||||
|
||||
for cap in capabilities:
|
||||
# Check if capability name is mentioned
|
||||
if cap.name.lower() in text_lower:
|
||||
found_caps.append(cap.name)
|
||||
def _add(name: str) -> None:
|
||||
if name not in found_caps:
|
||||
found_caps.append(name)
|
||||
|
||||
if not delegate_lines:
|
||||
# Either the Steward judged no capability necessary — the prompt's
|
||||
# conversational path, whose correct answer is [] — or it ignored the
|
||||
# format. Names only: domain words are ordinary English and would fire
|
||||
# on any prose, which is the bug described above.
|
||||
haystack = (text or "").lower()
|
||||
for cap in capabilities:
|
||||
if _mentions(cap.name.lower(), haystack):
|
||||
_add(cap.name)
|
||||
return found_caps
|
||||
|
||||
for line in delegate_lines:
|
||||
leading = next((c for c in capabilities if line.startswith(c.name.lower())), None)
|
||||
if leading is not None:
|
||||
_add(leading.name)
|
||||
continue
|
||||
|
||||
# Check if any domains are mentioned
|
||||
for domain in cap.domains:
|
||||
if domain.lower() in text_lower:
|
||||
found_caps.append(cap.name)
|
||||
break
|
||||
named = [c for c in capabilities if _mentions(c.name.lower(), line)]
|
||||
if named:
|
||||
for cap in named:
|
||||
_add(cap.name)
|
||||
continue
|
||||
|
||||
# Last resort. Scoped to this line, so the REASON and CONTEXT prose that
|
||||
# caused the original misrouting can no longer reach it.
|
||||
for cap in capabilities:
|
||||
if any(_mentions(domain.lower(), line) for domain in cap.domains):
|
||||
_add(cap.name)
|
||||
|
||||
return found_caps
|
||||
|
||||
|
||||
Reference in New Issue
Block a user