mirror of
https://github.com/pewdiepie-archdaemon/odysseus.git
synced 2026-09-14 04:02:22 +02:00
Joeseph GreyandRaresKeY
2c394704c6
* fix(personal): run directory indexing off the event loop (#5558) POST /api/personal/add_directory called rag.index_personal_documents inline from an async handler, so the whole indexing job (os.walk, file reads, per-chunk embedding, Chroma inserts) ran on the event loop and every other request queued behind it. Indexing a real directory froze the UI and API for 25+ minutes with no sign of life. Move the blocking section into the threadpool via run_in_threadpool. personal_docs_manager.add_directory stays inside it because its refresh_index() re-extracts text across tracked directories, which is also blocking work. A module-level lock serializes index jobs so the threadpool move does not introduce parallel jobs racing PersonalDocsManager's unsynchronized list mutations and file writes; they previously serialized on the blocked loop, so one-at-a-time is behavior parity. * fix(personal): serialize add/remove/reload on an async job lock The #5558 fix took the job lock INSIDE the threadpool worker and only on the add path, so (1) remove_directory and /reload mutated PersonalDocsManager's unsynchronized list/index concurrently with an in-flight add — the inconsistent state the PR claimed to prevent — and (2) a queued add blocked on the lock while holding an AnyIO threadpool token, starving the shared pool. Move the lock to an asyncio.Lock acquired in the async handler BEFORE offloading, and route add, remove and reload through it. A waiting request now parks on the event loop instead of pinning a worker, and all three mutators are serialized so the 'add/remove are serialized and cannot leave inconsistent state' guarantee holds. remove and reload also run their blocking work off the event loop. The lock is per-router so each app binds it to its own loop; single-process scope. Tests: add-vs-remove and add-vs-reload serialization regressions (async via ASGITransport, since asyncio.Lock deadlocks starlette TestClient's portal); the existing add-vs-add test converted to the same driver. * fix(personal): route upload and delete through the index job lock /api/personal/upload and DELETE /api/personal/file mutated the same vector and tracking state add/remove/reload serialize on, outside _index_job_lock and inline on the event loop. Both now stage async work on the loop, then run the complete transition (vector writes, disk change, personal_docs_manager update) in one offloaded critical section under the shared lock, acquired before the offload so queued requests park on the loop rather than pinning a threadpool worker. Adds add-vs-upload and add-vs-file ordering regressions. * fix(personal): bound multi-file upload memory --------- Co-authored-by: RaresKeY <158580472+RaresKeY@users.noreply.github.com>