jpmschweitzer/odysseus
1
0
Fork 0
mirror of https://github.com/pewdiepie-archdaemon/odysseus.git synced 2026-10-06 06:52:20 +02:00
Code Issues Packages Projects Releases Wiki Activity

Record latency decomposition and saved-evidence synthesis control

Browse Source
This commit is contained in:
pewdiepie-archdaemon
2026-09-17 21:24:29 +00:00
parent 40edb59864
commit 4d1abfb22b
1 changed files with 4 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/search-quality-audit-20260917.md
+4
View File
@@ -75,6 +75,10 @@ Further provider inspection found that the news-to-general fallback dropped the
### Additional informal/multi-part live checks
Timed `reports/clean-v3-search-quality-2026-09-17T21-22-21-304Z.json`: Firefox 31.0s/five rounds, tool execution 5.779s; news 69.8s/eight rounds, tool execution 1.672s. Remaining time includes inference, streaming and orchestration—not proven pure GPU time. The source-link retry still failed on Firefox. Main observed delay is outside tool execution, not search-provider time in these cached runs.
`4b6a9721` applies the explicit-query requirement to initial calls too: the weekly-news trace had copied a whole compound request into a missing query. Regression suites: 1,249 passed. Weekly-news replay launched. `reports/search-synthesis-probe-1789680251422.json` feeds the saved Firefox evidence to the same model without live recovery history/offered tools: canonical-system answer took 4.98s and concise research-system answer 4.66s; both supplied a link. This proves the model can emit the link in simplified context, not factual correctness—the linked support page was access-blocked, and some feature assertions still need grounding. Do not infer the system prompt alone or lack of tool schemas uniquely explains the difference.
`reports/clean-v3-search-quality-2026-09-17T21-20-16-154Z.json`: rejecting fabricated follow-up queries did not yield a latency win; typo-news used eight rounds/57.6 seconds and still lacked source URLs. Natural weekly-news request returned a failure in 27.0 seconds. Do not claim speed improvement. `7a3e1567` exposes measured execution time per tool (separate from total runtime) in live reports, and recognizes explicit imperative source requests such as “link the instructions” that the prior link-noun patterns missed. Related suites: 1,226 passed; timed news/privacy replay running. Additional completion retries are not a substitute for auditing the underlying answer generation.
`reports/clean-v3-search-quality-2026-09-17T21-18-22-350Z.json` remains unsatisfactory: Mozilla lookup 13.9s failed to locate documentation, multi-part privacy request 22.0s omitted requested links and details, Chrome follow-up 28.8s supplied generic homepages instead of comparison. Do not promote based on mechanics.