Revert "fix(chats): preserve saved conversations during tidy"

Merge pull request #6978 from odysseus-dev/test/remediation-false-green-oracles
test: reject vacuous success in email, bridge, auth and deletion tests
2026-10-30 17:52:26 +01:00 · 2026-10-30 15:34:21 +00:00 · 2026-10-30 15:25:10 +00:00 · 2026-10-30 15:09:09 +00:00 · 2026-10-30 15:00:08 +00:00 · 2026-10-30 14:13:15 +00:00
9 changed files with 1099 additions and 90 deletions
@@ -1030,6 +1030,166 @@ def _append_llama_cpp_linux_accel_build_lines(runner_lines: list[str]) -> None:
    runner_lines.append('  fi  # end _odysseus_have_prebuilt guard')


+ODYSSEUS_LLAMA_SHIM_MARKER = "# Auto-generated by Odysseus Cookbook: a `llama-server` lookalike"
+
+
+def _append_llama_server_resolution_definition_lines(runner_lines: list[str]) -> None:
+    """Define the helper shell function that discovers, reconciles and prioritizes llama-server."""
+    marker = f"'{_bash_squote(ODYSSEUS_LLAMA_SHIM_MARKER)}'"
+    runner_lines.append('_odysseus_resolve_llama_server() {')
+    runner_lines.append('  _odysseus_llama_bin=""')
+    runner_lines.append(f'  local _marker={marker}')
+    runner_lines.append('  local _saved_ifs="$IFS"')
+    runner_lines.append("  IFS=':'")
+    runner_lines.append('  local -a _parts')
+    runner_lines.append('  read -r -a _parts <<< "$PATH"')
+    runner_lines.append('  IFS="$_saved_ifs"')
+    runner_lines.append('  local _dir _cand')
+    runner_lines.append('  for _dir in "${_parts[@]}"; do')
+    runner_lines.append('    [ -n "$_dir" ] || continue')
+    runner_lines.append('    _cand="$_dir/llama-server"')
+    runner_lines.append('    [ -x "$_cand" ] || continue')
+    runner_lines.append('    if [ -f "$_cand" ] && [ ! -L "$_cand" ] && grep -Fxq "$_marker" "$_cand" 2>/dev/null; then')
+    runner_lines.append('      continue')
+    runner_lines.append('    fi')
+    runner_lines.append('    if "$_cand" --version >/dev/null 2>&1; then')
+    runner_lines.append('      _odysseus_llama_bin="$_cand"')
+    runner_lines.append('      break')
+    runner_lines.append('    fi')
+    runner_lines.append('  done')
+    runner_lines.append('  if [ -f "$HOME/bin/llama-server" ] && [ ! -L "$HOME/bin/llama-server" ] && grep -Fxq "$_marker" "$HOME/bin/llama-server" 2>/dev/null; then')
+    runner_lines.append('    if [ -n "$_odysseus_llama_bin" ]; then')
+    runner_lines.append('      echo "[odysseus] Native llama-server detected — retiring Odysseus compatibility shim in ~/bin"')
+    runner_lines.append('      rm -f "$HOME/bin/llama-server"')
+    runner_lines.append('      hash -r')
+    runner_lines.append('    elif python3 -c "import llama_cpp.server" 2>/dev/null; then')
+    runner_lines.append('      _odysseus_llama_bin="$HOME/bin/llama-server"')
+    runner_lines.append('    else')
+    runner_lines.append('      echo "[odysseus] Removing stale Odysseus llama-server shim (python3 -m llama_cpp.server unavailable)"')
+    runner_lines.append('      rm -f "$HOME/bin/llama-server"')
+    runner_lines.append('      hash -r')
+    runner_lines.append('    fi')
+    runner_lines.append('  fi')
+    runner_lines.append('  if [ -n "$_odysseus_llama_bin" ]; then')
+    runner_lines.append('    local _cand_dir')
+    runner_lines.append('    _cand_dir="$(dirname "$_odysseus_llama_bin")"')
+    runner_lines.append('    if [ "${PATH%%:*}" != "$_cand_dir" ]; then')
+    runner_lines.append('      export PATH="$_cand_dir:$PATH"')
+    runner_lines.append('    fi')
+    runner_lines.append('    hash -r')
+    runner_lines.append('    return 0')
+    runner_lines.append('  fi')
+    runner_lines.append('  return 1')
+    runner_lines.append('}')
+
+
+def _append_llama_server_shim_definition_lines(runner_lines: list[str]) -> None:
+    """Define the helper shell function that generates the Odysseus compatibility shim."""
+    _append_llama_server_resolution_definition_lines(runner_lines)
+    marker = f"'{_bash_squote(ODYSSEUS_LLAMA_SHIM_MARKER)}'"
+    runner_lines.append('_odysseus_create_llama_shim() {')
+    runner_lines.append('  local _target="$HOME/bin/llama-server"')
+    runner_lines.append(f'  local _marker={marker}')
+    runner_lines.append('  if [ -e "$_target" ] || [ -L "$_target" ]; then')
+    runner_lines.append('    if [ -f "$_target" ] && [ ! -L "$_target" ] && grep -Fxq "$_marker" "$_target" 2>/dev/null; then')
+    runner_lines.append('      : # existing Odysseus shim — safe to replace')
+    runner_lines.append('    else')
+    runner_lines.append('      echo "[odysseus] WARNING: $_target already exists and is not an Odysseus-managed shim; leaving untouched." >&2')
+    runner_lines.append('      return 1')
+    runner_lines.append('    fi')
+    runner_lines.append('  fi')
+    runner_lines.append('  mkdir -p "$HOME/bin" || return 1')
+    runner_lines.append('  local _tmp')
+    runner_lines.append('  _tmp="$(mktemp "$HOME/bin/.llama-server.odysseus.tmp.XXXXXX")" || {')
+    runner_lines.append('    echo "[odysseus] ERROR: unable to create secure temporary llama-server shim in $HOME/bin" >&2')
+    runner_lines.append('    return 1')
+    runner_lines.append('  }')
+    runner_lines.append("  if ! cat > \"$_tmp\" <<'_ODY_LLAMA_SHIM_EOF'")
+    runner_lines.append('#!/usr/bin/env bash')
+    runner_lines.append(ODYSSEUS_LLAMA_SHIM_MARKER)
+    runner_lines.append('# that translates the native CLI to `python -m llama_cpp.server`.')
+    runner_lines.append('# Lets cookbook-generated launch commands run unchanged on hosts')
+    runner_lines.append('# where only the pip llama-cpp-python package is installed.')
+    runner_lines.append('ARGS=()')
+    runner_lines.append('while [ $# -gt 0 ]; do')
+    runner_lines.append('  case "$1" in')
+    runner_lines.append('    -ngl|--gpu-layers|--n-gpu-layers) ARGS+=(--n_gpu_layers "$2"); shift 2 ;;')
+    runner_lines.append('    -c|--ctx-size) ARGS+=(--n_ctx "$2"); shift 2 ;;')
+    runner_lines.append('    -b|--batch-size) ARGS+=(--n_batch "$2"); shift 2 ;;')
+    runner_lines.append('    -ub|--ubatch-size) shift 2 ;;  # llama-cpp-python has no separate ubatch')
+    runner_lines.append('    --flash-attn) ARGS+=(--flash_attn true); shift 2 ;;')
+    runner_lines.append('    --cache-type-k) ARGS+=(--type_k "$2"); shift 2 ;;')
+    runner_lines.append('    --cache-type-v) ARGS+=(--type_v "$2"); shift 2 ;;')
+    runner_lines.append('    --n-cpu-moe) ARGS+=(--n_cpu_moe "$2"); shift 2 ;;')
+    runner_lines.append('    --mmproj) ARGS+=(--clip_model_path "$2"); shift 2 ;;')
+    runner_lines.append('    --image-max-tokens) shift 2 ;;  # native-only')
+    runner_lines.append('    --no-mmap) ARGS+=(--no_mmap true); shift ;;')
+    runner_lines.append('    --no-warmup) shift ;;  # native-only')
+    runner_lines.append('    --chat-template) ARGS+=(--chat_format "$2"); shift 2 ;;')
+    runner_lines.append('    --fit|--split-mode|--tensor-split|--main-gpu|--parallel) shift 2 ;;  # native-only')
+    runner_lines.append('    --mlock) ARGS+=(--use_mlock true); shift ;;')
+    runner_lines.append('    *) ARGS+=("$1"); shift ;;')
+    runner_lines.append('  esac')
+    runner_lines.append('done')
+    runner_lines.append('exec python3 -m llama_cpp.server "${ARGS[@]}"')
+    runner_lines.append('_ODY_LLAMA_SHIM_EOF')
+    runner_lines.append('  then')
+    runner_lines.append('    rm -f -- "$_tmp"')
+    runner_lines.append('    return 1')
+    runner_lines.append('  fi')
+    runner_lines.append('  chmod +x "$_tmp" || { rm -f -- "$_tmp"; return 1; }')
+    runner_lines.append('  mv -f -- "$_tmp" "$_target" || { rm -f -- "$_tmp"; return 1; }')
+    runner_lines.append('  hash -r')
+    runner_lines.append('  echo "[odysseus] Created llama-server shim → python -m llama_cpp.server (no native binary needed)"')
+    runner_lines.append('}')
+
+
+def _append_llama_server_shim_reconciliation_lines(runner_lines: list[str]) -> None:
+    """Reconcile an existing ~/bin/llama-server if it is owned by Odysseus Cookbook.
+
+    Invariants:
+    - Never touch an arbitrary user-owned executable or symlink.
+    - If ~/bin/llama-server is an Odysseus generated shim:
+        - If a native llama-server is present and healthy elsewhere on PATH, remove the shim
+          so the native binary is used.
+        - If python3 -m llama_cpp.server is unavailable, remove the stale shim
+          and hash -r so the fallback/build path is triggered.
+        - If no healthy native server exists and llama_cpp.server is working, preserve it.
+    - If a healthy native candidate exists on PATH (even later in PATH), prioritize it and
+      ensure the launched executable is the validated candidate.
+    """
+    runner_lines.append('[ -r ~/.config/odysseus-llama-cpp-env ] && . ~/.config/odysseus-llama-cpp-env')
+    runner_lines.append('if ! declare -F _odysseus_resolve_llama_server >/dev/null 2>&1; then')
+    _append_llama_server_resolution_definition_lines(runner_lines)
+    runner_lines.append('fi')
+    runner_lines.append('_odysseus_resolve_llama_server || true')
+
+
+def _append_llama_server_finalization_lines(runner_lines: list[str], platform_hint: str = "") -> None:
+    """Reconcile post-install launcher availability and enforce fail-closed invariant.
+
+    Guarantees that if python3 -m llama_cpp.server is usable but the `llama-server`
+    command is unresolvable, the compatibility launcher is created. If no usable
+    launcher exists after all build/install attempts, sets ODYSSEUS_PREFLIGHT_EXIT=127.
+    """
+    err_msg = (
+        f"ERROR: llama.cpp serving is not available on {platform_hint} after install attempts."
+        if platform_hint
+        else "ERROR: llama.cpp serving is not available after install/build attempts."
+    )
+    runner_lines.append('if ! declare -F _odysseus_resolve_llama_server >/dev/null 2>&1; then')
+    _append_llama_server_resolution_definition_lines(runner_lines)
+    runner_lines.append('fi')
+    runner_lines.append('if [ -z "$_odysseus_llama_bin" ] && python3 -c "import llama_cpp.server" 2>/dev/null; then')
+    runner_lines.append('  _odysseus_create_llama_shim')
+    runner_lines.append('  _odysseus_resolve_llama_server || true')
+    runner_lines.append('fi')
+    runner_lines.append('if [ -z "$_odysseus_llama_bin" ]; then')
+    runner_lines.append(f'  echo "{err_msg}"')
+    runner_lines.append('  ODYSSEUS_PREFLIGHT_EXIT=127')
+    runner_lines.append('fi')
+
+
 def _llama_cpp_rebuild_cmd(update_source: bool = False) -> str:
    """Shell command that clears the Cookbook-managed llama.cpp build.

@@ -57,9 +57,11 @@ from routes.cookbook_helpers import (
    _pip_install_no_cache, _user_shell_path_bootstrap, _venv_safe_local_pip_install_cmd,
    _diagnose_serve_output, run_ssh_command_async,
    _ollama_bind_from_cmd, _pip_install_fallback_chain, _pip_install_no_cache,
-    _user_shell_path_bootstrap, _venv_safe_local_pip_install_cmd,
    _append_pip_install_runner_lines, _pip_install_command_without_break_system_packages,
    _normalize_llama_cpp_python_cache_types,
+    _append_llama_server_shim_definition_lines,
+    _append_llama_server_shim_reconciliation_lines,
+    _append_llama_server_finalization_lines,
    ModelDownloadRequest, ServeRequest,
 )

@@ -2161,7 +2163,7 @@ def setup_cookbook_routes() -> APIRouter:
                ps_lines.append('}')
            elif "llama_cpp" in req.cmd or "llama-server" in req.cmd:
                ps_lines.append('# Auto-install llama-cpp-python if missing')
-                ps_lines.append('try { python -c "import llama_cpp" 2>$null } catch {}')
+                ps_lines.append('try { python -c "import llama_cpp.server" 2>$null } catch {}')
                ps_lines.append('if ($LASTEXITCODE -ne 0) {')
                ps_lines.append('  Write-Host "Installing llama-cpp-python..."')
                ps_lines.append('  python -m pip install llama-cpp-python[server]')
@@ -2247,14 +2249,17 @@ def setup_cookbook_routes() -> APIRouter:
                # ollama is found (otherwise macOS falls back to a slow source build).
                # /opt/homebrew = Apple Silicon, /usr/local = Intel; harmless on Linux.
                runner_lines.append('export PATH="$HOME/.local/bin:$HOME/bin:$HOME/llama.cpp/build/bin:/opt/homebrew/bin:/usr/local/bin:$PATH"')
+                _append_llama_server_shim_definition_lines(runner_lines)
+                _append_llama_server_shim_reconciliation_lines(runner_lines)
                runner_lines.append('if [ -d /data/data/com.termux ]; then')
                runner_lines.append('  # Termux: no native build — use the Python bindings (CPU).')
-                runner_lines.append('  if ! python3 -c "import llama_cpp" 2>/dev/null; then')
+                runner_lines.append('  if ! python3 -c "import llama_cpp.server" 2>/dev/null; then')
                runner_lines.append('    pkg install -y cmake 2>/dev/null')
                runner_lines.append('    pip install numpy diskcache jinja2 2>/dev/null')
                runner_lines.append('    CMAKE_ARGS="-DGGML_BLAS=OFF -DGGML_LLAMAFILE=OFF" pip install \'llama-cpp-python[server]\' --no-build-isolation --no-cache-dir 2>&1 || true')
                runner_lines.append('  fi')
-                runner_lines.append('elif ! command -v llama-server &>/dev/null; then')
+                _append_llama_server_finalization_lines(runner_lines, platform_hint="Termux")
+                runner_lines.append('elif [ -z "$_odysseus_llama_bin" ]; then')
                runner_lines.append('  echo "Native llama-server not found — building from source (one-time, may take a few minutes)..."')
                runner_lines.append('  mkdir -p ~/bin')
                runner_lines.append('  cd ~ && [ -d llama.cpp ] || git clone --depth 1 https://github.com/ggml-org/llama.cpp')
@@ -2271,8 +2276,7 @@ def setup_cookbook_routes() -> APIRouter:
                # would reuse the bad settings and fail again. CMAKE_BUILD_TYPE is
                # explicit so the binary is optimized (Metal auto-enables on macOS).
                runner_lines.append('    cd ~/llama.cpp && rm -rf build && cmake -B build -DCMAKE_BUILD_TYPE=Release \\')
-                runner_lines.append('      && cmake --build build -j"$NPROC" --target llama-server \\')
-                runner_lines.append('      && ln -sf ~/llama.cpp/build/bin/llama-server ~/bin/llama-server')
+                runner_lines.append('      && cmake --build build -j"$NPROC" --target llama-server')
                runner_lines.append('  else')
                _append_llama_cpp_linux_accel_build_lines(runner_lines)
                runner_lines.append('  fi')
@@ -2296,58 +2300,13 @@ def setup_cookbook_routes() -> APIRouter:
                runner_lines.append('      fi')
                runner_lines.append('    fi')
                runner_lines.append('  fi')
-                # SHORT-CIRCUIT before the build/pip fallback: if the
-                # native binary is missing but llama_cpp Python is already
-                # installed, drop a wrapper at ~/bin/llama-server that
-                # translates llama-server CLI args to llama_cpp.server's
-                # underscore-style flags. The user's serve command stays
-                # `llama-server ...` and "just works" — no build, no cmake,
-                # no second install. This is the path that unblocks every
-                # remote where pip-installed llama-cpp-python is already
-                # working but Cookbook used to insist on a native binary.
-                runner_lines.append('  if ! command -v llama-server >/dev/null 2>&1 && python3 -c "import llama_cpp" 2>/dev/null; then')
-                runner_lines.append('    mkdir -p ~/bin')
-                runner_lines.append('    cat > ~/bin/llama-server <<\'_ODY_LLAMA_SHIM_EOF\'')
-                runner_lines.append('#!/usr/bin/env bash')
-                runner_lines.append('# Auto-generated by Odysseus Cookbook: a `llama-server` lookalike')
-                runner_lines.append('# that translates the native CLI to `python -m llama_cpp.server`.')
-                runner_lines.append('# Lets cookbook-generated launch commands run unchanged on hosts')
-                runner_lines.append('# where only the pip llama-cpp-python package is installed.')
-                runner_lines.append('ARGS=()')
-                runner_lines.append('while [ $# -gt 0 ]; do')
-                runner_lines.append('  case "$1" in')
-                runner_lines.append('    -ngl|--gpu-layers|--n-gpu-layers) ARGS+=(--n_gpu_layers "$2"); shift 2 ;;')
-                runner_lines.append('    -c|--ctx-size) ARGS+=(--n_ctx "$2"); shift 2 ;;')
-                runner_lines.append('    -b|--batch-size) ARGS+=(--n_batch "$2"); shift 2 ;;')
-                runner_lines.append('    -ub|--ubatch-size) shift 2 ;;  # llama-cpp-python has no separate ubatch')
-                runner_lines.append('    --flash-attn) ARGS+=(--flash_attn true); shift 2 ;;')
-                runner_lines.append('    --cache-type-k) ARGS+=(--type_k "$2"); shift 2 ;;')
-                runner_lines.append('    --cache-type-v) ARGS+=(--type_v "$2"); shift 2 ;;')
-                runner_lines.append('    --n-cpu-moe) ARGS+=(--n_cpu_moe "$2"); shift 2 ;;')
-                runner_lines.append('    --mmproj) ARGS+=(--clip_model_path "$2"); shift 2 ;;')
-                runner_lines.append('    --image-max-tokens) shift 2 ;;  # native-only')
-                runner_lines.append('    --no-mmap) ARGS+=(--no_mmap true); shift ;;')
-                runner_lines.append('    --no-warmup) shift ;;  # native-only')
-                runner_lines.append('    --chat-template) ARGS+=(--chat_format "$2"); shift 2 ;;')
-                runner_lines.append('    --fit|--split-mode|--tensor-split|--main-gpu|--parallel) shift 2 ;;  # native-only')
-                runner_lines.append('    --mlock) ARGS+=(--use_mlock true); shift ;;')
-                runner_lines.append('    *) ARGS+=("$1"); shift ;;')
-                runner_lines.append('  esac')
-                runner_lines.append('done')
-                runner_lines.append('exec python3 -m llama_cpp.server "${ARGS[@]}"')
-                runner_lines.append('_ODY_LLAMA_SHIM_EOF')
-                runner_lines.append('    chmod +x ~/bin/llama-server')
-                runner_lines.append('    echo "[odysseus] Created llama-server shim → python -m llama_cpp.server (no native binary needed)"')
-                runner_lines.append('  fi')
-                runner_lines.append('  # If the native build failed, fall back to the Python bindings.')
-                runner_lines.append('  if ! command -v llama-server &>/dev/null && ! python3 -c "import llama_cpp" 2>/dev/null; then')
+                runner_lines.append('  _odysseus_resolve_llama_server || true')
+                # If the native build failed, fall back to the Python bindings.
+                runner_lines.append('  if [ -z "$_odysseus_llama_bin" ] && ! python3 -c "import llama_cpp.server" 2>/dev/null; then')
                runner_lines.append('    echo "llama-server build failed — installing Python bindings as fallback..."')
                runner_lines.append(f"    {_pip_install_fallback_chain('llama-cpp-python[server]', python_cmd='pip')} || true")
                runner_lines.append('  fi')
-                runner_lines.append('  if ! command -v llama-server &>/dev/null && ! python3 -c "import llama_cpp" 2>/dev/null; then')
-                runner_lines.append('    echo "ERROR: llama.cpp serving is not available after install/build attempts."')
-                runner_lines.append('    ODYSSEUS_PREFLIGHT_EXIT=127')
-                runner_lines.append('  fi')
+                _append_llama_server_finalization_lines(runner_lines)
                runner_lines.append('fi')
            elif re.search(r"\bollama\s+serve\b", req.cmd):
                handled_ollama_serve = True
@@ -10,6 +10,7 @@ import logging
 import os
 import re
 import asyncio 
+import contextlib
 from typing import Any, Dict, List, Optional, Set, Tuple
 from src.database import McpServer, SessionLocal

@@ -383,12 +384,21 @@ class McpManager:

            provider = build_provider(server_id, url, on_redirect=_on_redirect)
            stack = AsyncExitStack()
-            transport = await stack.enter_async_context(streamablehttp_client(url, auth=provider))
-            read_stream, write_stream, _get_session_id = transport
-            session = await stack.enter_async_context(ClientSession(read_stream, write_stream))
-            await session.initialize()
+            try:
+                transport = await stack.enter_async_context(streamablehttp_client(url, auth=provider))
+                read_stream, write_stream, _get_session_id = transport
+                session = await stack.enter_async_context(ClientSession(read_stream, write_stream))
+                await session.initialize()

-            tools_result = await session.list_tools()
+                tools_result = await session.list_tools()
+            except BaseException:
+                # Close the half-open transport here, in the task that entered
+                # it. Left for the garbage collector, its AnyIO cancel scope is
+                # exited from the wrong task and the event loop keeps spinning
+                # (#5518). Its own teardown error would only mask the real one.
+                with contextlib.suppress(Exception):
+                    await stack.aclose()
+                raise
            tools = []
            for tool in tools_result.tools:
                tools.append({
@@ -1929,23 +1929,60 @@ def test_place_layer_reports_false_on_queued_response(monkeypatch):
    assert placed is False


-def test_bridge_await_cancellation_handled_cleanly():
+def test_bridge_await_cancellation_handled_cleanly(monkeypatch):
    import asyncio
+    from contextlib import suppress
    from httpx import ASGITransport, AsyncClient
    app = _bridge_app()

    async def scenario():
+        entered_wait = asyncio.Event()
+        real_wait_for = asyncio.wait_for
+        waits = []
+
+        async def observe_wait(awaitable, timeout):
+            # Observe the real wait; do not replace its result or cancellation.
+            waits.append(awaitable)
+            entered_wait.set()
+            return await real_wait_for(awaitable, timeout)
+
+        monkeypatch.setattr(eaar.asyncio, "wait_for", observe_wait)
        async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as c:
            cmd_resp = await c.post("/api/editor/bridge/command", json={"session": "cancel-sess", "op": "document.info", "wait": False})
+            assert cmd_resp.status_code == 200
+            assert cmd_resp.json()["completed"] is False
            cid = cmd_resp.json()["queued"][0]

            await_task = asyncio.create_task(c.post("/api/editor/bridge/await", json={"id": cid, "timeout": 30}))
-            await asyncio.sleep(0.05)
-            await_task.cancel()
            try:
-                await await_task
-            except asyncio.CancelledError:
-                pass
+                await real_wait_for(entered_wait.wait(), 5)
+                assert len(waits) == 1
+                assert not waits[0].done()
+                assert not await_task.done(), "await returned without a queued result"
+                await_task.cancel()
+                response = await real_wait_for(await_task, 5)
+                assert response.status_code == 200
+                assert response.json() == {
+                    "ok": False, "error": "cancelled", "status": "cancelled",
+                    "completed": False, "id": cid,
+                }
+            finally:
+                if not await_task.done():
+                    await_task.cancel()
+                    with suppress(asyncio.CancelledError):
+                        await await_task
+
+            # Cancelling the waiter must not cancel the queued command itself.
+            payload = {"width": 137, "height": 251}
+            delivered = await c.post("/api/editor/bridge/result", json={"id": cid, "ok": True, "result": payload})
+            assert delivered.status_code == 200
+            completed = await c.post("/api/editor/bridge/await", json={"id": cid, "timeout": 1})
+            assert completed.status_code == 200
+            assert completed.json()["status"] == "completed"
+            assert completed.json()["completed"] is True
+            assert completed.json()["ok"] is True
+            assert completed.json()["result"] == payload
+            assert len(waits) == 2

    asyncio.run(scenario())

@@ -206,35 +206,50 @@ class TestDiskConsistency:
    @pytest.mark.slow
    def test_file_always_valid_json_during_concurrent_ops(self, tmp_path):
        mgr = _fresh_auth_manager(tmp_path)
-        mgr.create_user("admin", "adminpw", is_admin=True)
+        assert mgr.create_user("admin", "adminpw", is_admin=True)

        stop_event = threading.Event()
+        reader_ready = threading.Event()
        corruption_found = []
+        observations = {"reads": 0, "with_writes": 0}

        def reader():
            while not stop_event.is_set():
                try:
                    with open(mgr.auth_path, "r") as f:
                        content = f.read()
-                    json.loads(content)
+                    data = json.loads(content)
+                    observations["reads"] += 1
+                    if any(name.startswith("stress") for name in data["users"]):
+                        observations["with_writes"] += 1
+                    reader_ready.set()
                except json.JSONDecodeError as e:
                    corruption_found.append(str(e))
                    break
-                except FileNotFoundError:
-                    pass
                time.sleep(0.001)

        def writer():
+            assert reader_ready.wait(5), "reader never observed the initial file"
+            results = []
            for i in range(50):
-                mgr.create_user(f"stress{i}", f"pw{i}")
+                results.append(mgr.create_user(f"stress{i}", f"pw{i}"))
+            return results

-        reader_thread = threading.Thread(target=reader)
-        writer_thread = threading.Thread(target=writer)
-
-        reader_thread.start()
-        writer_thread.start()
-        writer_thread.join()
-        stop_event.set()
-        reader_thread.join()
+        # Futures propagate reader/writer exceptions to the test thread.
+        with ThreadPoolExecutor(max_workers=2) as pool:
+            reading = pool.submit(reader)
+            writing = pool.submit(writer)
+            try:
+                results = writing.result(timeout=120)
+            finally:
+                stop_event.set()
+            reading.result(timeout=5)

        assert not corruption_found, f"Corrupt JSON detected: {corruption_found[0]}"
+        assert len(results) == 50 and all(results), "writer operations did not succeed"
+        assert observations["reads"] > 0
+        assert observations["with_writes"] > 0, "reader saw no persisted writer activity"
+        expected = {"admin"} | {f"stress{i}" for i in range(50)}
+        with open(mgr.auth_path) as persisted:
+            assert set(json.load(persisted)["users"]) == expected
+        assert set(_fresh_auth_manager(tmp_path).users) == expected
@@ -1,4 +1,14 @@
+import os
+import subprocess
 from pathlib import Path
+import pytest
+from core.platform_compat import find_bash, git_bash_path
+from routes.cookbook_helpers import (
+    ODYSSEUS_LLAMA_SHIM_MARKER,
+    _append_llama_server_shim_definition_lines,
+    _append_llama_server_shim_reconciliation_lines,
+    _append_llama_server_finalization_lines,
+)

 from tests.helpers.stylesheets import app_css

@@ -615,3 +625,730 @@ def test_local_dependency_probe_refreshes_user_site_visibility():
    # deps (realesrgan) probe as not-installed until a restart. See #4810.
    assert "if user_site and os.path.isdir(user_site):" in source
    assert "site.addsitedir(user_site)" in source
+
+
+def test_llama_runner_path_preserves_standard_user_bin_order():
+    """The runner PATH must preserve standard user $HOME/bin precedence while
+    relying on ownership-aware reconciliation to retire Odysseus shims when
+    native builds exist."""
+    source = _read("routes/cookbook_routes.py")
+
+    assert (
+        'export PATH="$HOME/.local/bin:$HOME/bin:$HOME/llama.cpp/build/bin:/opt/homebrew/bin:/usr/local/bin:$PATH"'
+        in source
+    )
+
+
+def _run_shell(tmp_path, lines, cwd=None, path=None, extra_env=None):
+    from core.platform_compat import find_bash, git_bash_path
+    bash = find_bash() or "bash"
+    env = dict(os.environ, HOME=git_bash_path(tmp_path))
+    if path is not None:
+        env["PATH"] = path
+    if extra_env:
+        env.update(extra_env)
+    return subprocess.run(
+        [bash, "-c", "\n".join(lines)],
+        cwd=str(cwd) if cwd else None,
+        env=env,
+        capture_output=True,
+        text=True,
+    )
+
+
+def _write_odysseus_shim(path: Path) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    path.write_text(
+        f"#!/usr/bin/env bash\n{ODYSSEUS_LLAMA_SHIM_MARKER}\nexec python3 -m llama_cpp.server \"$@\"\n",
+        encoding="utf-8",
+    )
+    path.chmod(0o755)
+
+
+def _write_mock_native(path: Path, succeeds: bool = True) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    if succeeds:
+        content = '#!/usr/bin/env bash\nif [ "$1" = "--version" ]; then echo "version: 0.1.0-dev"; exit 0; fi\necho "RUN_NATIVE: $@"\nexit 0\n'
+    else:
+        content = '#!/usr/bin/env bash\necho "error: missing libllama.so" >&2\nexit 127\n'
+    path.write_text(content, encoding="utf-8")
+    path.chmod(0o755)
+
+
+def _write_mock_python3(path: Path, succeeds: bool = True) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    path.write_text(f"#!/usr/bin/env bash\nexit {0 if succeeds else 1}\n", encoding="utf-8")
+    path.chmod(0o755)
+
+
+def test_odysseus_create_llama_shim_never_overwrites_user_regular_file(tmp_path):
+    """Creation boundary: never overwrite a non-executable user-owned regular file."""
+    user_file = tmp_path / "bin/llama-server"
+    user_file.parent.mkdir(parents=True, exist_ok=True)
+    original_content = "user configuration or binary data without marker"
+    user_file.write_text(original_content, encoding="utf-8")
+
+    lines = []
+    _append_llama_server_shim_definition_lines(lines)
+    lines.append('_odysseus_create_llama_shim')
+
+    res = _run_shell(tmp_path, lines)
+    assert res.returncode != 0, "Creation should return non-zero when target is a user file"
+    assert user_file.read_text(encoding="utf-8") == original_content, "User file must remain byte-for-byte unchanged"
+
+
+def test_odysseus_create_llama_shim_never_overwrites_user_symlink(tmp_path):
+    """Creation boundary: never follow or overwrite a user-owned symlink."""
+    bin_dir = tmp_path / "bin"
+    bin_dir.mkdir(parents=True, exist_ok=True)
+    target_file = tmp_path / "user_target.txt"
+    target_file.write_text("precious user target content", encoding="utf-8")
+    user_symlink = bin_dir / "llama-server"
+    user_symlink.symlink_to(target_file)
+
+    lines = []
+    _append_llama_server_shim_definition_lines(lines)
+    lines.append('_odysseus_create_llama_shim')
+
+    res = _run_shell(tmp_path, lines)
+    assert res.returncode != 0, "Creation should return non-zero when target is a user symlink"
+    assert user_symlink.is_symlink(), "Symlink must remain intact"
+    assert target_file.read_text(encoding="utf-8") == "precious user target content", "Target file must not be modified"
+
+
+def test_odysseus_create_llama_shim_never_overwrites_dangling_symlink(tmp_path):
+    """Creation boundary: never overwrite a dangling user-owned symlink."""
+    bin_dir = tmp_path / "bin"
+    bin_dir.mkdir(parents=True, exist_ok=True)
+    dangling_symlink = bin_dir / "llama-server"
+    dangling_symlink.symlink_to(tmp_path / "nonexistent_target")
+
+    lines = []
+    _append_llama_server_shim_definition_lines(lines)
+    lines.append('_odysseus_create_llama_shim')
+
+    res = _run_shell(tmp_path, lines)
+    assert res.returncode != 0, "Creation should return non-zero when target is a dangling symlink"
+    assert dangling_symlink.is_symlink(), "Dangling symlink must remain intact"
+
+
+def test_odysseus_create_llama_shim_safely_replaces_existing_odysseus_shim(tmp_path):
+    """Creation boundary: safely replace an existing Odysseus-owned compatibility shim."""
+    shim_file = tmp_path / "bin/llama-server"
+    shim_file.parent.mkdir(parents=True, exist_ok=True)
+    shim_file.write_text(f"#!/bin/bash\n{ODYSSEUS_LLAMA_SHIM_MARKER}\n# old shim body\n", encoding="utf-8")
+
+    lines = []
+    _append_llama_server_shim_definition_lines(lines)
+    lines.append('_odysseus_create_llama_shim')
+
+    res = _run_shell(tmp_path, lines)
+    assert res.returncode == 0
+    new_content = shim_file.read_text(encoding="utf-8")
+    assert ODYSSEUS_LLAMA_SHIM_MARKER in new_content
+    assert "exec python3 -m llama_cpp.server" in new_content
+
+
+def test_odysseus_create_llama_shim_secure_temp_prevents_symlink_attack(tmp_path):
+    """Creation boundary: secure mktemp prevents predictable $$ temp symlink attacks.
+
+    Given:
+    - A victim file at ~/victim containing DO_NOT_TOUCH
+    - A pre-created symlink at ~/bin/.llama-server.odysseus.tmp.$$ pointing to ~/victim
+    - _odysseus_create_llama_shim executed within that exact same bash PID ($$)
+
+    Prove:
+    - ~/victim remains unmodified
+    - Pre-existing crafted symlink is NOT followed or overwritten
+    - Legitimate shim is successfully created at ~/bin/llama-server via secure temp path
+    """
+    lines = [
+        'victim="$HOME/victim"',
+        'printf "DO_NOT_TOUCH\\n" > "$victim"',
+        'mkdir -p "$HOME/bin"',
+        'old_tmp="$HOME/bin/.llama-server.odysseus.tmp.$$";',
+        'ln -s "$victim" "$old_tmp"',
+    ]
+    _append_llama_server_shim_definition_lines(lines)
+    lines.append('_odysseus_create_llama_shim')
+    lines.append('res=$?')
+    lines.append('echo "CREATION_RES=$res"')
+
+    res = _run_shell(tmp_path, lines)
+    assert res.returncode == 0
+    assert "CREATION_RES=0" in res.stdout
+
+    victim_file = tmp_path / "victim"
+    assert victim_file.exists()
+    assert victim_file.read_text(encoding="utf-8") == "DO_NOT_TOUCH\n", "Victim file must remain untouched"
+
+    target_shim = tmp_path / "bin/llama-server"
+    assert target_shim.exists()
+    assert ODYSSEUS_LLAMA_SHIM_MARKER in target_shim.read_text(encoding="utf-8")
+    assert target_shim.stat().st_mode & 0o111 != 0, "Shim must be executable"
+
+
+def test_stale_odysseus_shim_removed_when_python_server_unusable(tmp_path):
+    """Scenario 1: Stale Odysseus shim + missing Python server -> owned shim removed."""
+    shim_file = tmp_path / "bin/llama-server"
+    _write_odysseus_shim(shim_file)
+
+    lines = []
+    _append_llama_server_shim_reconciliation_lines(lines)
+
+    res = _run_shell(tmp_path, lines)
+    assert res.returncode == 0
+    assert not shim_file.exists(), "Stale Odysseus shim must be removed when python server is unavailable"
+
+
+def test_healthy_odysseus_shim_retained_when_no_native_server(tmp_path):
+    """Scenario 2: Healthy Odysseus shim + no native server -> shim preserved."""
+    shim_file = tmp_path / "bin/llama-server"
+    _write_odysseus_shim(shim_file)
+    mock_py_dir = tmp_path / "mock_bin"
+    _write_mock_python3(mock_py_dir / "python3", succeeds=True)
+
+    lines = []
+    _append_llama_server_shim_reconciliation_lines(lines)
+
+    res = _run_shell(tmp_path, lines, path=f"{git_bash_path(mock_py_dir)}:{os.environ.get('PATH', '')}")
+    assert res.returncode == 0
+    assert shim_file.exists(), "Healthy Odysseus shim must be preserved when no native server is present"
+
+
+def test_reconciliation_does_not_retire_healthy_shim_for_broken_native_binary(tmp_path):
+    """Reconciliation health check: a broken native candidate executable must NOT retire
+    a healthy Odysseus Python shim."""
+    shim_file = tmp_path / "bin/llama-server"
+    _write_odysseus_shim(shim_file)
+    _write_mock_native(tmp_path / "llama.cpp/build/bin/llama-server", succeeds=False)
+    mock_py_dir = tmp_path / "mock_bin"
+    _write_mock_python3(mock_py_dir / "python3", succeeds=True)
+
+    lines = [f'export PATH="$HOME/llama.cpp/build/bin:{git_bash_path(mock_py_dir)}:$PATH"']
+    _append_llama_server_shim_reconciliation_lines(lines)
+
+    res = _run_shell(tmp_path, lines)
+    assert res.returncode == 0
+    assert shim_file.exists(), "Healthy Odysseus shim must NOT be retired when native binary is broken"
+
+
+def test_odysseus_shim_retired_when_native_server_healthy(tmp_path):
+    """Scenario 3: Odysseus shim + healthy native server available -> native server wins."""
+    shim_file = tmp_path / "bin/llama-server"
+    _write_odysseus_shim(shim_file)
+    _write_mock_native(tmp_path / "llama.cpp/build/bin/llama-server", succeeds=True)
+
+    lines = ['export PATH="$HOME/bin:$HOME/llama.cpp/build/bin:$PATH"']
+    _append_llama_server_shim_reconciliation_lines(lines)
+
+    res = _run_shell(tmp_path, lines)
+    assert res.returncode == 0
+    assert not shim_file.exists(), "Odysseus shim must be retired when healthy native server is available"
+
+
+def test_user_owned_bin_llama_server_never_touched(tmp_path):
+    """Scenario 4: Arbitrary user ~/bin/llama-server -> untouched and not deprioritized."""
+    user_bin = tmp_path / "bin/llama-server"
+    user_bin.parent.mkdir(parents=True, exist_ok=True)
+    user_bin.write_text("#!/bin/bash\n# Custom user build script\nexec /opt/custom/llama-server\n", encoding="utf-8")
+    user_bin.chmod(0o755)
+
+    _write_mock_native(tmp_path / "llama.cpp/build/bin/llama-server", succeeds=True)
+
+    lines = ['export PATH="$HOME/bin:$HOME/llama.cpp/build/bin:$PATH"']
+    _append_llama_server_shim_reconciliation_lines(lines)
+
+    res = _run_shell(tmp_path, lines)
+    assert res.returncode == 0
+    assert user_bin.exists(), "User-owned ~/bin/llama-server must never be deleted"
+    assert "Custom user build script" in user_bin.read_text(encoding="utf-8")
+
+
+def test_reconciliation_ignores_native_binary_outside_effective_path(tmp_path):
+    """Native candidate outside effective PATH must NOT be considered."""
+    shim_file = tmp_path / "bin/llama-server"
+    _write_odysseus_shim(shim_file)
+    # Native binary placed in /usr_custom/bin which is NOT in PATH
+    _write_mock_native(tmp_path / "usr_custom/bin/llama-server", succeeds=True)
+    mock_py_dir = tmp_path / "mock_bin"
+    _write_mock_python3(mock_py_dir / "python3", succeeds=True)
+
+    lines = []
+    _append_llama_server_shim_reconciliation_lines(lines)
+
+    # PATH only contains ~/bin and mock_py_dir — usr_custom is omitted
+    path = f"{git_bash_path(tmp_path / 'bin')}:{git_bash_path(mock_py_dir)}"
+    res = _run_shell(tmp_path, lines, path=path)
+    assert res.returncode == 0
+    assert shim_file.exists(), "Shim must NOT be retired by native binary outside effective PATH"
+
+
+def test_reconciliation_skips_home_bin_with_or_without_trailing_slash(tmp_path):
+    """Reconciliation must exclude $HOME/bin and $HOME/bin/ from native candidate discovery."""
+    shim_file = tmp_path / "bin/llama-server"
+    _write_odysseus_shim(shim_file)
+    mock_py_dir = tmp_path / "mock_bin"
+    _write_mock_python3(mock_py_dir / "python3", succeeds=True)
+
+    lines = []
+    _append_llama_server_shim_reconciliation_lines(lines)
+
+    # PATH has ~/bin/ with trailing slash
+    path = f"{git_bash_path(tmp_path / 'bin')}/:{git_bash_path(mock_py_dir)}"
+    res = _run_shell(tmp_path, lines, path=path)
+    assert res.returncode == 0
+    assert shim_file.exists(), "Shim must NOT rediscover itself when PATH has trailing slash in $HOME/bin/"
+
+
+def test_reconciliation_does_not_discover_cwd_executable_when_path_omits_dot(tmp_path):
+    """Reconciliation must NOT synthesize an empty/trailing PATH element that searches cwd.
+
+    Given:
+    - Current working directory contains an executable `llama-server`
+    - Healthy Odysseus Python shim in ~/bin/llama-server
+    - PATH does NOT contain '.' or cwd
+    - No other healthy native binary on PATH
+
+    Prove:
+    - Reconciliation does not discover ./llama-server
+    - The healthy Odysseus shim in ~/bin/llama-server is preserved
+    """
+    shim_file = tmp_path / "bin/llama-server"
+    _write_odysseus_shim(shim_file)
+    mock_py_dir = tmp_path / "mock_bin"
+    _write_mock_python3(mock_py_dir / "python3", succeeds=True)
+
+    cwd_dir = tmp_path / "workdir"
+    _write_mock_native(cwd_dir / "llama-server", succeeds=True)
+
+    lines = []
+    _append_llama_server_shim_reconciliation_lines(lines)
+
+    path = f"{git_bash_path(tmp_path / 'bin')}:{git_bash_path(mock_py_dir)}:/usr/bin:/bin"
+    res = _run_shell(tmp_path, lines, cwd=cwd_dir, path=path)
+    assert res.returncode == 0
+    assert shim_file.exists(), "Healthy Odysseus shim must NOT be retired by finding ./llama-server in cwd"
+
+
+def test_reconciliation_discovers_healthy_native_on_explicit_other_path_entry(tmp_path):
+    """Reconciliation must discover healthy native binaries placed on explicit PATH entries."""
+    shim_file = tmp_path / "bin/llama-server"
+    _write_odysseus_shim(shim_file)
+    custom_opt_bin = tmp_path / "custom_opt/bin"
+    _write_mock_native(custom_opt_bin / "llama-server", succeeds=True)
+
+    lines = []
+    _append_llama_server_shim_reconciliation_lines(lines)
+
+    path = f"{git_bash_path(tmp_path / 'bin')}:{git_bash_path(custom_opt_bin)}:/usr/bin:/bin"
+    res = _run_shell(tmp_path, lines, path=path)
+    assert res.returncode == 0
+    assert not shim_file.exists(), "Odysseus shim must be retired when healthy native server is on explicit PATH entry"
+
+
+def test_reconciliation_safely_iterates_path_with_glob_metacharacters(tmp_path):
+    """Reconciliation must not perform pathname expansion / globbing on PATH components.
+
+    Given:
+    - A directory path containing shell glob characters: ~/custom[dir]/bin
+    - A native binary ~/custom[dir]/bin/llama-server
+    - A working directory containing matching glob entries
+    - An existing Odysseus shim in ~/bin/llama-server
+
+    Prove:
+    - Reconciliation correctly parses the literal path without expanding [dir]
+    - Discovers the healthy native server and retires the Odysseus shim
+    """
+    shim_file = tmp_path / "bin/llama-server"
+    _write_odysseus_shim(shim_file)
+
+    glob_dir = tmp_path / "custom[dir]/bin"
+    _write_mock_native(glob_dir / "llama-server", succeeds=True)
+
+    cwd_dir = tmp_path / "cwd"
+    cwd_dir.mkdir(parents=True, exist_ok=True)
+    (cwd_dir / "customd").mkdir(parents=True, exist_ok=True)
+    (cwd_dir / "customi").mkdir(parents=True, exist_ok=True)
+
+    lines = []
+    _append_llama_server_shim_reconciliation_lines(lines)
+
+    path = f"{git_bash_path(tmp_path / 'bin')}:{git_bash_path(glob_dir)}:/usr/bin:/bin"
+    res = _run_shell(tmp_path, lines, cwd=cwd_dir, path=path)
+    assert res.returncode == 0
+    assert not shim_file.exists(), "Reconciliation must discover literal bracketed PATH without globbing"
+
+
+def test_fallback_install_success_creates_usable_launcher(tmp_path):
+    """Scenario 5: No native + no Python server -> fallback install succeeds
+    -> usable compatibility launcher exists and resolves command -v llama-server."""
+    mock_py_dir = tmp_path / "mock_bin"
+    _write_mock_python3(mock_py_dir / "python3", succeeds=True)
+
+    lines = [f'export PATH="$HOME/bin:{git_bash_path(mock_py_dir)}:$PATH"']
+    _append_llama_server_shim_definition_lines(lines)
+    _append_llama_server_finalization_lines(lines)
+    lines.append('command -v llama-server')
+
+    res = _run_shell(tmp_path, lines)
+    assert res.returncode == 0
+    created_shim = tmp_path / "bin/llama-server"
+    assert created_shim.exists()
+    assert ODYSSEUS_LLAMA_SHIM_MARKER in created_shim.read_text(encoding="utf-8")
+    assert str(tmp_path / "bin/llama-server") in res.stdout
+
+
+def test_failed_fallback_sets_preflight_exit_127(tmp_path):
+    """Scenario 6: No native + failed fallback installation -> ODYSSEUS_PREFLIGHT_EXIT=127."""
+    lines = [
+        'ODYSSEUS_PREFLIGHT_EXIT=""',
+        'export PATH="$HOME/bin:$PATH"',
+    ]
+    _append_llama_server_shim_definition_lines(lines)
+    _append_llama_server_finalization_lines(lines)
+    lines.append('echo "EXIT=$ODYSSEUS_PREFLIGHT_EXIT"')
+
+    res = _run_shell(tmp_path, lines)
+    assert res.returncode == 0
+    assert "EXIT=127" in res.stdout
+
+
+def test_python_server_readiness_requires_llama_cpp_server_module():
+    """Scenario 7 & 8: Python server readiness requires `import llama_cpp.server`,
+    while GPU offload capability checks still use base `import llama_cpp`."""
+    routes_source = _read("routes/cookbook_routes.py")
+    helpers_source = _read("routes/cookbook_helpers.py")
+
+    # Windows fallback readiness check requires .server
+    assert 'try { python -c "import llama_cpp.server" 2>$null } catch {}' in routes_source
+    # Termux fallback readiness check requires .server
+    assert 'if ! python3 -c "import llama_cpp.server" 2>/dev/null; then' in routes_source
+    # Build-failure fallback pip check requires .server
+    assert 'if [ -z "$_odysseus_llama_bin" ] && ! python3 -c "import llama_cpp.server" 2>/dev/null; then' in routes_source
+    # Shim reconciliation and post-install finalization require .server
+    assert 'elif python3 -c "import llama_cpp.server" 2>/dev/null; then' in helpers_source
+    assert 'if [ -z "$_odysseus_llama_bin" ] && python3 -c "import llama_cpp.server" 2>/dev/null; then' in helpers_source
+
+    # GPU offload support checks still intentionally inspect base llama_cpp
+    assert 'llama_cpp.llama_supports_gpu_offload()' in routes_source
+
+
+def test_darwin_native_build_does_not_force_write_bin_symlink():
+    """Darwin source build must not overwrite ~/bin/llama-server with a forced symlink."""
+    source = _read("routes/cookbook_routes.py")
+
+    assert "&& ln -sf ~/llama.cpp/build/bin/llama-server ~/bin/llama-server" not in source
+    assert 'cmake --build build -j"$NPROC" --target llama-server' in source
+
+
+@pytest.mark.asyncio
+async def test_generated_runner_preflight_retires_stale_shim_before_backend_selection(monkeypatch, tmp_path):
+    """Integration regression: The actual generated Cookbook runner must execute
+    ownership reconciliation BEFORE evaluating the backend selection (`if Termux ... elif ! command -v llama-server`).
+
+    Given:
+    - Executable stale Odysseus shim in ~/bin/llama-server (with canonical marker)
+    - Healthy native llama-server in ~/llama.cpp/build/bin/llama-server
+    - ~/bin before ~/llama.cpp/build/bin on PATH
+
+    Prove:
+    1. Generated runner contains reconciliation shell lines strictly before the backend branching
+    2. Executing the generated runner script retires the stale ~/bin/llama-server shim
+    3. Resolves to the healthy native binary without attempting source build or pip fallback.
+    """
+    from unittest.mock import AsyncMock, MagicMock
+    from routes.cookbook_helpers import ServeRequest
+    import routes.cookbook_routes as cookbook_routes
+    from starlette.requests import Request
+
+    fake_home = tmp_path / "fake_home"
+    stale_shim = fake_home / "bin/llama-server"
+    _write_odysseus_shim(stale_shim)
+    native_bin = fake_home / "llama.cpp/build/bin/llama-server"
+    _write_mock_native(native_bin, succeeds=True)
+
+    # Mock environment to generate the real runner file with isolated DB
+    tmux_log_dir = tmp_path / "tmux_logs"
+    tmux_log_dir.mkdir(parents=True)
+    monkeypatch.setattr(cookbook_routes, "TMUX_LOG_DIR", tmux_log_dir)
+    monkeypatch.setattr(cookbook_routes, "require_admin", lambda request: None)
+    monkeypatch.setattr(cookbook_routes, "_binary_available", AsyncMock(return_value=True))
+    monkeypatch.setattr(cookbook_routes, "load_stored_hf_token", lambda **kwargs: "")
+    monkeypatch.setattr("core.database.SessionLocal", MagicMock())
+
+    class _Proc:
+        returncode = 0
+        stderr = None
+        async def wait(self):
+            return 0
+    monkeypatch.setattr(cookbook_routes.asyncio, "create_subprocess_shell", AsyncMock(return_value=_Proc()))
+
+    router = cookbook_routes.setup_cookbook_routes()
+    endpoint = None
+    for route in router.routes:
+        if route.path == "/api/model/serve" and "POST" in route.methods:
+            endpoint = route.endpoint
+            break
+    assert endpoint is not None
+
+    from types import SimpleNamespace
+    http_req = Request({
+        "type": "http",
+        "method": "POST",
+        "path": "/api/model/serve",
+        "headers": [],
+        "app": SimpleNamespace(state=SimpleNamespace(auth_manager=SimpleNamespace(
+            is_configured=True, is_admin=lambda user: user == "admin"))),
+    })
+    http_req.state.current_user = "admin"
+    http_req.state.local_model_authority = True
+
+    req = ServeRequest(
+        repo_id="test/repo",
+        cmd="llama-server -m /dummy/model.gguf --port 8085",
+        remote_host=None,
+    )
+
+    resp = await endpoint(http_req, req)
+    assert resp["ok"] is True
+    session_id = resp["session_id"]
+    runner_file = tmux_log_dir / f"{session_id}_run.sh"
+    assert runner_file.exists()
+    runner_script = runner_file.read_text(encoding="utf-8")
+
+    # Assert structural ordering in the generated script:
+    # Reconciliation must appear strictly BEFORE the Termux/backend check
+    recon_idx = runner_script.find("Native llama-server detected — retiring Odysseus compatibility shim in ~/bin")
+    termux_idx = runner_script.find("if [ -d /data/data/com.termux ]; then")
+    backend_idx = runner_script.find('elif [ -z "$_odysseus_llama_bin" ]; then')
+    assert recon_idx != -1, "Reconciliation block missing from generated runner"
+    assert termux_idx != -1, "Termux check missing from generated runner"
+    assert backend_idx != -1, "Backend selection missing from generated runner"
+    assert recon_idx < termux_idx < backend_idx, "Reconciliation must execute strictly before backend selection"
+
+    # Execute the generated runner preflight in a real subshell
+    res = _run_shell(fake_home, [str(runner_file)], path=f"{git_bash_path(fake_home / 'bin')}:{git_bash_path(fake_home / 'llama.cpp/build/bin')}:/usr/bin:/bin")
+    assert res.returncode == 0
+    assert not stale_shim.exists(), "Stale ~/bin/llama-server must have been retired by the generated preflight"
+    assert "RUN_NATIVE: -m /dummy/model.gguf --port 8085" in res.stdout
+    assert "Native llama-server not found — building from source" not in res.stdout
+
+
+def test_broken_first_native_candidate_healthy_second_candidate_executes_healthy(tmp_path):
+    """Regression finding P2: When PATH contains an earlier broken candidate followed by a
+    healthy candidate, reconciliation and launch must select and execute the healthy candidate."""
+    from core.platform_compat import git_bash_path
+    broken_dir = tmp_path / "broken_bin"
+    _write_mock_native(broken_dir / "llama-server", succeeds=False)
+
+    healthy_dir = tmp_path / "healthy_bin"
+    _write_mock_native(healthy_dir / "llama-server", succeeds=True)
+
+    shim_file = tmp_path / "bin/llama-server"
+    _write_odysseus_shim(shim_file)
+
+    lines = [
+        f'export PATH="$HOME/bin:{git_bash_path(broken_dir)}:{git_bash_path(healthy_dir)}:/usr/bin:/bin"',
+    ]
+    _append_llama_server_shim_reconciliation_lines(lines)
+    lines.append('echo "RESOLVED=$_odysseus_llama_bin"')
+    lines.append('command -v llama-server')
+    lines.append('llama-server -m /dummy/model.gguf --port 8080')
+
+    res = _run_shell(tmp_path, lines)
+    assert res.returncode == 0
+    assert not shim_file.exists(), "Stale Odysseus shim must be retired"
+    assert f"RESOLVED={git_bash_path(healthy_dir / 'llama-server')}" in res.stdout
+    assert str(healthy_dir / "llama-server") in res.stdout
+    assert "RUN_NATIVE: -m /dummy/model.gguf --port 8080" in res.stdout
+    assert "error: missing libllama.so" not in res.stdout
+
+
+def test_paths_with_spaces_broken_first_healthy_second_launch(tmp_path):
+    """Regression: Paths containing spaces must be safely resolved, prioritized and launched."""
+    from core.platform_compat import git_bash_path
+    broken_dir = tmp_path / "broken path with spaces/bin"
+    _write_mock_native(broken_dir / "llama-server", succeeds=False)
+
+    healthy_dir = tmp_path / "healthy path with spaces/bin"
+    _write_mock_native(healthy_dir / "llama-server", succeeds=True)
+
+    shim_file = tmp_path / "bin/llama-server"
+    _write_odysseus_shim(shim_file)
+
+    model_dir = tmp_path / "models with spaces"
+    model_dir.mkdir(parents=True, exist_ok=True)
+    model_file = model_dir / "my model.gguf"
+    model_file.write_text("dummy", encoding="utf-8")
+
+    lines = [
+        f'export PATH="$HOME/bin:{git_bash_path(broken_dir)}:{git_bash_path(healthy_dir)}:/usr/bin:/bin"',
+    ]
+    _append_llama_server_shim_reconciliation_lines(lines)
+    lines.append(f'llama-server -m "{git_bash_path(model_file)}" --port 8080')
+
+    res = _run_shell(tmp_path, lines)
+    assert res.returncode == 0
+    assert not shim_file.exists(), "Stale shim must be retired"
+    assert f"RUN_NATIVE: -m {git_bash_path(model_file)} --port 8080" in res.stdout
+    assert "error: missing libllama.so" not in res.stdout
+
+
+def test_command_hash_invalidation_switches_from_cached_broken_to_healthy(tmp_path):
+    """Regression: Command cache invalidation (`hash -r`) must prevent bash from reusing a
+    previously cached lookup to an earlier broken executable."""
+    from core.platform_compat import git_bash_path
+    broken_dir = tmp_path / "broken_bin"
+    _write_mock_native(broken_dir / "llama-server", succeeds=False)
+
+    healthy_dir = tmp_path / "healthy_bin"
+    _write_mock_native(healthy_dir / "llama-server", succeeds=True)
+
+    lines = [
+        # Populate bash hash cache with broken binary
+        f'export PATH="{git_bash_path(broken_dir)}:/usr/bin:/bin"',
+        'llama-server >/dev/null 2>&1 || true',
+        'hash',
+        # Now add healthy candidate later in PATH and run reconciliation
+        f'export PATH="$HOME/bin:{git_bash_path(broken_dir)}:{git_bash_path(healthy_dir)}:/usr/bin:/bin"',
+    ]
+    _append_llama_server_shim_reconciliation_lines(lines)
+    lines.append('llama-server -m /dummy/test.gguf')
+    lines.append('hash')
+
+    res = _run_shell(tmp_path, lines)
+    assert res.returncode == 0
+    assert "RUN_NATIVE: -m /dummy/test.gguf" in res.stdout
+    # Verify hash table ended with healthy binary, not broken
+    assert str(healthy_dir / "llama-server") in res.stdout
+
+
+def test_broken_native_candidate_falls_back_to_healthy_python_compatibility_shim(tmp_path):
+    """Regression: A broken native candidate must not shadow or retire a healthy Python shim."""
+    from core.platform_compat import git_bash_path
+    broken_dir = tmp_path / "broken_bin"
+    _write_mock_native(broken_dir / "llama-server", succeeds=False)
+
+    shim_file = tmp_path / "bin/llama-server"
+    _write_odysseus_shim(shim_file)
+
+    mock_py_dir = tmp_path / "mock_bin"
+    _write_mock_python3(mock_py_dir / "python3", succeeds=True)
+
+    lines = [
+        f'export PATH="$HOME/bin:{git_bash_path(broken_dir)}:{git_bash_path(mock_py_dir)}:/usr/bin:/bin"',
+    ]
+    _append_llama_server_shim_reconciliation_lines(lines)
+    lines.append('echo "RESOLVED=$_odysseus_llama_bin"')
+    lines.append('command -v llama-server')
+
+    res = _run_shell(tmp_path, lines)
+    assert res.returncode == 0
+    assert shim_file.exists(), "Healthy Python compatibility shim must be preserved"
+    assert f"RESOLVED={git_bash_path(shim_file)}" in res.stdout
+    assert str(shim_file) in res.stdout
+
+
+def test_broken_native_and_missing_python_backend_sets_preflight_127(tmp_path):
+    """Regression: When the only native candidates are broken and Python server is unavailable,
+    finalization must fail closed with ODYSSEUS_PREFLIGHT_EXIT=127 rather than claiming readiness."""
+    from core.platform_compat import git_bash_path
+    broken_dir = tmp_path / "broken_bin"
+    _write_mock_native(broken_dir / "llama-server", succeeds=False)
+
+    mock_py_dir = tmp_path / "mock_bin"
+    _write_mock_python3(mock_py_dir / "python3", succeeds=False)
+
+    lines = [
+        'ODYSSEUS_PREFLIGHT_EXIT=""',
+        f'export PATH="$HOME/bin:{git_bash_path(broken_dir)}:{git_bash_path(mock_py_dir)}:/usr/bin:/bin"',
+    ]
+    _append_llama_server_shim_definition_lines(lines)
+    _append_llama_server_shim_reconciliation_lines(lines)
+    _append_llama_server_finalization_lines(lines)
+    lines.append('echo "EXIT=$ODYSSEUS_PREFLIGHT_EXIT"')
+
+    res = _run_shell(tmp_path, lines)
+    assert res.returncode == 0
+    assert "EXIT=127" in res.stdout
+
+
+@pytest.mark.asyncio
+async def test_generated_runner_broken_first_healthy_second_e2e(monkeypatch, tmp_path):
+    """Integration regression: The full generated Cookbook runner script must launch the healthy
+    native candidate even when an earlier broken candidate exists on PATH."""
+    from unittest.mock import AsyncMock, MagicMock
+    from types import SimpleNamespace
+    from routes.cookbook_helpers import ServeRequest
+    import routes.cookbook_routes as cookbook_routes
+    from starlette.requests import Request
+    from core.platform_compat import git_bash_path
+
+    fake_home = tmp_path / "fake_home"
+    stale_shim = fake_home / "bin/llama-server"
+    _write_odysseus_shim(stale_shim)
+
+    broken_bin = fake_home / "broken/bin/llama-server"
+    _write_mock_native(broken_bin, succeeds=False)
+
+    healthy_bin = fake_home / "healthy/bin/llama-server"
+    _write_mock_native(healthy_bin, succeeds=True)
+
+    tmux_log_dir = tmp_path / "tmux_logs"
+    tmux_log_dir.mkdir(parents=True)
+    monkeypatch.setattr(cookbook_routes, "TMUX_LOG_DIR", tmux_log_dir)
+    monkeypatch.setattr(cookbook_routes, "require_admin", lambda request: None)
+    monkeypatch.setattr(cookbook_routes, "_binary_available", AsyncMock(return_value=True))
+    monkeypatch.setattr(cookbook_routes, "load_stored_hf_token", lambda **kwargs: "")
+    monkeypatch.setattr("core.database.SessionLocal", MagicMock())
+
+    class _Proc:
+        returncode = 0
+        stderr = None
+        async def wait(self):
+            return 0
+    monkeypatch.setattr(cookbook_routes.asyncio, "create_subprocess_shell", AsyncMock(return_value=_Proc()))
+
+    router = cookbook_routes.setup_cookbook_routes()
+    endpoint = None
+    for route in router.routes:
+        if route.path == "/api/model/serve" and "POST" in route.methods:
+            endpoint = route.endpoint
+            break
+    assert endpoint is not None
+
+    http_req = Request({
+        "type": "http",
+        "method": "POST",
+        "path": "/api/model/serve",
+        "headers": [],
+        "app": SimpleNamespace(state=SimpleNamespace(auth_manager=SimpleNamespace(
+            is_configured=True, is_admin=lambda user: user == "admin"))),
+    })
+    http_req.state.current_user = "admin"
+    http_req.state.local_model_authority = True
+
+    req = ServeRequest(
+        repo_id="test/repo",
+        cmd="llama-server -m /dummy/model.gguf --port 8090",
+        remote_host=None,
+    )
+
+    resp = await endpoint(http_req, req)
+    assert resp["ok"] is True
+    session_id = resp["session_id"]
+    runner_file = tmux_log_dir / f"{session_id}_run.sh"
+    assert runner_file.exists()
+
+    # Broken is earlier on PATH than healthy
+    res = _run_shell(
+        fake_home,
+        [str(runner_file)],
+        path=f"{git_bash_path(fake_home / 'bin')}:{git_bash_path(broken_bin.parent)}:{git_bash_path(healthy_bin.parent)}:/usr/bin:/bin",
+    )
+    assert res.returncode == 0
+    assert not stale_shim.exists(), "Stale shim must have been retired"
+    assert "RUN_NATIVE: -m /dummy/model.gguf --port 8090" in res.stdout
+    assert "error: missing libllama.so" not in res.stdout
+    assert "Native llama-server not found — building from source" not in res.stdout
@@ -11,6 +11,8 @@ cache from the DB, where the rows are already gone, and the token is rejected.
 """
 import asyncio
 import types
+import pytest
+from fastapi import HTTPException

 from routes.auth_routes import setup_auth_routes, DeleteUserRequest

@@ -60,12 +62,9 @@ def test_refused_delete_does_not_invalidate_cache():
    invalidations = []
    router = setup_auth_routes(_auth_manager(delete_result=False))
    handler = _handler(router)
-    try:
+    with pytest.raises(HTTPException) as refused:
        asyncio.run(handler(DeleteUserRequest(username="admin"), _fake_request(invalidations)))
-        raised = False
-    except Exception:
-        raised = True
-    assert raised, "a refused delete should raise (HTTP 400)"
+    assert refused.value.status_code == 400
    assert invalidations == [], "a refused delete must not touch the token cache"


@@ -56,14 +56,35 @@ def test_cold_start_with_no_snapshot_and_no_index_is_not_marked_stale(monkeypatc
    # falls through to the real (blocking) path, so the response is not a
    # stale-marked snapshot.
    endpoint = _list_endpoint()
+    calls = []
+
+    class EmptyMailbox:
+        def select(self, folder, readonly=False):
+            calls.append(("select", folder, readonly))
+            return "OK", [b"0"]
+
+        def uid(self, command, *args):
+            assert command == "SEARCH"
+            calls.append(("search", args))
+            return "OK", [b""]
+
+        def logout(self):
+            pass
+
+    def connect(*args, **kwargs):
+        calls.append(("connect", args, kwargs))
+        return EmptyMailbox()
+
+    monkeypatch.setattr(er, "_start_poller", lambda: None)
    monkeypatch.setattr(er, "_email_index_list", lambda *a, **k: ([], 0, None))
-    monkeypatch.setattr(er, "_imap_connect", lambda *a, **k: (_ for _ in ()).throw(
-        RuntimeError("offline test")))
-    try:
-        result = _call(endpoint)
-    except Exception:
-        return  # the cold path legitimately attempted real IMAP
+    monkeypatch.setattr(er, "_imap_connect", connect)
+    result = _call(endpoint)
+    assert [call[0] for call in calls] == ["connect", "select", "search"]
+    assert calls[1][2] is True
    assert isinstance(result, dict)
+    assert result["emails"] == []
+    assert result["total"] == 0
+    assert not result.get("error")
    assert not (result.get("sync") or {}).get("stale")


@@ -39,3 +39,74 @@ def test_http_transport_routes_to_start_http_connect():
        result = asyncio.run(mgr.connect_server("id1", "n", "http", url="https://x/mcp"))
    assert result == "ROUTED"
    m.assert_called_once()
+
+
+def _patch_http_transport(monkeypatch, initialize):
+    """Fake the Streamable HTTP transport and record, for each close, whether
+    it ran in the task that opened it (AnyIO requires that)."""
+    import contextlib
+    import mcp
+    import mcp.client.streamable_http as streamable_http
+    import src.mcp_oauth as mcp_oauth
+
+    closes = []
+
+    @contextlib.asynccontextmanager
+    async def fake_client(url, auth=None):
+        opener = asyncio.current_task()
+        try:
+            yield (object(), object(), lambda: None)
+        finally:
+            closes.append(asyncio.current_task() is opener)
+
+    class FakeSession:
+        def __init__(self, *a):
+            pass
+
+        async def __aenter__(self):
+            return self
+
+        async def __aexit__(self, *exc):
+            return False
+
+    FakeSession.initialize = initialize
+    monkeypatch.setattr(streamable_http, "streamablehttp_client", fake_client)
+    monkeypatch.setattr(mcp, "ClientSession", FakeSession)
+    monkeypatch.setattr(mcp_oauth, "build_provider", lambda *a, **k: None)
+    return closes
+
+
+def test_failed_http_connect_closes_transport_in_its_own_task(monkeypatch):
+    """#5518: a failed connect left its transport for the GC, which exited the
+    AnyIO cancel scope from the wrong task and kept the event loop spinning."""
+    async def initialize(self):
+        raise RuntimeError("Session terminated")
+
+    closes = _patch_http_transport(monkeypatch, initialize)
+    mgr = McpManager()
+
+    ok = asyncio.run(mgr._start_http_connect("s1", "remote", "http://x/mcp/", wait=2))
+
+    assert ok is False
+    assert mgr._connections["s1"]["status"] == "error"
+    assert "Session terminated" in mgr._connections["s1"]["error"]
+    assert closes == [True]
+    assert "s1" not in mgr._stacks
+
+
+def test_cancelled_http_connect_closes_transport(monkeypatch):
+    """Deleting a server mid-connect cancels the connect task; the transport
+    must still be closed rather than leaked."""
+    async def initialize(self):
+        await asyncio.sleep(3600)
+
+    closes = _patch_http_transport(monkeypatch, initialize)
+    mgr = McpManager()
+
+    async def run():
+        await mgr._start_http_connect("s1", "remote", "http://x/mcp/", wait=0.05)
+        await mgr.disconnect_server("s1")
+        await asyncio.sleep(0)
+
+    asyncio.run(run())
+    assert closes == [True]
Author	SHA1	Message	Date
Alexandre Teixeira	7937f87e54	Revert "fix(chats): preserve saved conversations during tidy"	2026-10-30 15:34:21 +00:00
Alexandre Teixeira	0e49c4ef40	Merge pull request #6978 from odysseus-dev/test/remediation-false-green-oracles test: reject vacuous success in email, bridge, auth and deletion tests	2026-10-30 15:25:10 +00:00
Alexandre Teixeira	1aff71836b	Merge pull request #6611 from robyroro/fix/chat-tidy-preserve-content fix(chats): preserve saved conversations during tidy	2026-10-30 15:09:09 +00:00
Alexandre Teixeira	b1c3125865	test: accept positional arguments in task endpoint mock for auto-sort	2026-10-30 15:00:08 +00:00
Alexandre Teixeira	dbee1155e6	Merge pull request #6080 from alteixeira20/fix/cookbook-stale-llama-server-shim fix(cookbook): recover from stale llama-server shims	2026-10-30 14:13:15 +00:00
Alexandre Teixeira	56c7cdf804	Merge pull request #6544 from harshit-ojha0324/fix/mcp-http-connect-leak fix(mcp): close the HTTP transport when a remote connect fails	2026-10-30 14:09:43 +00:00
Alexandre Teixeira	6037707430	fix(cookbook): launch validated llama-server candidate and handle PATH precedence Ensure the native candidate that passed the health check is prioritized and executed when an earlier broken candidate exists on PATH. Reconcile Odysseus-owned shims safely, invalidate command hash cache, and verify launchability rather than command existence before serve dispatch. Fixes #6079	2026-10-30 13:15:40 +00:00
Alexandre Teixeira	5c542d03ac	Merge remote-tracking branch 'origin/dev' into fix/pr-6080-llama-recovery	2026-10-30 13:02:47 +00:00
Alexandre Teixeira	6f5c28b345	test: require real cold reads, cancellation and auth writes	2026-10-29 17:12:20 +00:00
robyroro	ad3e309074	fix(chats): preserve saved conversations during tidy	2026-10-08 06:48:41 +03:00
harshit-ojha0324	50b05bf54f	fix(mcp): close the HTTP transport when a remote connect fails _connect_http entered streamablehttp_client and ClientSession on an AsyncExitStack but never closed it when initialize or list_tools failed. The half-open transport was left to the garbage collector, which exited its AnyIO cancel scope from the wrong task, and the event loop kept spinning: about 18% of a core per failed connect, growing with each one, until restart. Close the stack in the connecting task on any failure or cancellation, the same way _connect_sse already does. Fixes #5518	2026-10-06 19:01:10 -04:00
Alexandre Teixeira	a7c03f85bf	fix(cookbook): recover from stale llama-server shims Reconcile Odysseus-owned Python compatibility shims when llama_cpp.server is no longer available, allowing healthy native llama-server binaries in PATH to be discovered and executed without shadowing. Fixes #6079	2026-08-17 00:59:13 +01:00