Joeseph GreyandRaresKeY 2c394704c6 fix(personal): run directory indexing off the event loop (#5634)
* fix(personal): run directory indexing off the event loop (#5558)

POST /api/personal/add_directory called rag.index_personal_documents
inline from an async handler, so the whole indexing job (os.walk, file
reads, per-chunk embedding, Chroma inserts) ran on the event loop and
every other request queued behind it. Indexing a real directory froze
the UI and API for 25+ minutes with no sign of life.

Move the blocking section into the threadpool via run_in_threadpool.
personal_docs_manager.add_directory stays inside it because its
refresh_index() re-extracts text across tracked directories, which is
also blocking work. A module-level lock serializes index jobs so the
threadpool move does not introduce parallel jobs racing
PersonalDocsManager's unsynchronized list mutations and file writes;
they previously serialized on the blocked loop, so one-at-a-time is
behavior parity.

* fix(personal): serialize add/remove/reload on an async job lock

The #5558 fix took the job lock INSIDE the threadpool worker and only on the
add path, so (1) remove_directory and /reload mutated PersonalDocsManager's
unsynchronized list/index concurrently with an in-flight add — the inconsistent
state the PR claimed to prevent — and (2) a queued add blocked on the lock while
holding an AnyIO threadpool token, starving the shared pool.

Move the lock to an asyncio.Lock acquired in the async handler BEFORE offloading,
and route add, remove and reload through it. A waiting request now parks on the
event loop instead of pinning a worker, and all three mutators are serialized so
the 'add/remove are serialized and cannot leave inconsistent state' guarantee
holds. remove and reload also run their blocking work off the event loop. The
lock is per-router so each app binds it to its own loop; single-process scope.

Tests: add-vs-remove and add-vs-reload serialization regressions (async via
ASGITransport, since asyncio.Lock deadlocks starlette TestClient's portal); the
existing add-vs-add test converted to the same driver.

* fix(personal): route upload and delete through the index job lock

/api/personal/upload and DELETE /api/personal/file mutated the same
vector and tracking state add/remove/reload serialize on, outside
_index_job_lock and inline on the event loop.

Both now stage async work on the loop, then run the complete transition
(vector writes, disk change, personal_docs_manager update) in one
offloaded critical section under the shared lock, acquired before the
offload so queued requests park on the loop rather than pinning a
threadpool worker.

Adds add-vs-upload and add-vs-file ordering regressions.

* fix(personal): bound multi-file upload memory

---------

Co-authored-by: RaresKeY <158580472+RaresKeY@users.noreply.github.com>
2026-08-15 10:12:47 +01:00
2026-06-15 16:00:13 +09:00
2026-07-23 14:49:02 +00:00
2026-05-31 23:58:26 +09:00
2026-07-23 14:49:02 +00:00

Odysseus

A self-hosted AI workspace for chat, agents, research, documents, email, notes, calendar, and local model workflows.

Quick Start · Setup Guide · Contributing · Roadmap

Packaging status

Odysseus interface


Quick Start

dev is the default branch and gets the newest changes first. Use main if you want the more curated branch.

git clone https://github.com/odysseus-dev/odysseus.git
cd odysseus
cp .env.example .env
docker compose up -d --build

Open http://localhost:7000 when the containers are healthy. The first admin password is printed in docker compose logs odysseus.

Native installs, GPU notes, Windows/macOS instructions, HTTPS, and configuration live in the setup guide.

Features

  • Chat + Agents — local/API models, tools, MCP, files, shell, skills, and memory.
  • Cookbook — hardware-aware model recommendations, downloads, and serving.
  • Deep Research — multi-step web research with source reading and report generation.
  • Compare — blind side-by-side model testing and synthesis.
  • Documents — writing-first editor with AI edits, suggestions, Markdown, HTML, CSV, and syntax highlighting.
  • Email — IMAP/SMTP inbox with triage, tags, summaries, reminders, and reply drafts.
  • Notes, Tasks + Calendar — reminders, todos, scheduled agent tasks, and CalDAV sync.
  • Extras — gallery/image editor, themes, uploads, web search, presets, sessions, and 2FA.

Demo

A full hover-to-play tour lives on the landing page: docs/index.html.

Contributing

Help is welcome. The best entry points are fresh-install testing, provider setup bugs, mobile/editor polish, docs, and small focused refactors. See CONTRIBUTING.md and ROADMAP.md.

Security

Odysseus is a self-hosted workspace with powerful local tools. Keep auth enabled, keep private data out of Git, and do not expose raw model/service ports publicly. Deployment details are in the setup guide.

Star History

Star History Chart

License

AGPL-3.0-or-later -- see LICENSE and ACKNOWLEDGMENTS.md.

S
Description
No description provided
Readme AGPL-3.0
443 MiB
Languages
Python 55.3%
JavaScript 35.3%
CSS 7%
HTML 1.9%
Shell 0.4%