Two small polish items in the Cookbook Serve panel.
Saved-config badge
The little count badge next to the Save button ("3 ▾" etc.) had a
generic "Saved launch configs" tooltip, so the number reads like a
notification dot. Make it spell out what it is and what clicking does:
"3 saved launch configs for <model> — click ▾ to load or delete"
(and "No saved launch configs for <model> yet — click Save to add
one" when empty). Tooltip stays in sync via _updateSavedToggleLabel
so save/delete updates both the count and the hint.
GPU chip on mixed-GPU boxes (#711)
The chip label was `${gpuCount}x ${gpu_name}`, where gpu_name is
just gpus[0].name — so a 4090 + 3060 reads as "2x RTX 4090". The
backend already emits gpu_groups (identical cards grouped, used by
the serve flow to pin CUDA_VISIBLE_DEVICES) and a per-card gpus[]
array, so use them:
- Label renders each homogeneous pool: "1× RTX 4090 + 1× RTX 3060".
Homogeneous setups keep the existing "2× RTX 4090" form.
- Tooltip lists each GPU with its index + VRAM, useful for picking
the right device when launching.
Refs #711.
The model-name detector treated every Qwen model as a Qwen3, falling
into the qwen3_xml parser:
if (n.includes('qwen3') && n.includes('coder')) return 'qwen3_coder';
if (n.includes('qwen')) return 'qwen3_xml'; // catches qwen2.5 too
qwen3_xml is the parser for Qwen3 reasoning/instruct models. Qwen2.5
(and Qwen2, Qwen1.5) ship with hermes-style tool calling, so the
qwen3_xml parser never recognises their tool calls — they leak through
as plain text in the assistant reply and the agent silently fails to
execute anything.
Reproduces with:
vllm serve Qwen/Qwen2.5-Coder-14B-Instruct-AWQ ... \
--enable-auto-tool-choice --tool-call-parser qwen3_xml
→ ask the agent to call any tool → JSON shows up in chat, no call runs.
Fix the ordering:
qwen3 + coder → qwen3_coder
qwen3 → qwen3_xml
qwen → hermes (Qwen2.5 / Qwen2 / Qwen1.5)
Verified against the model matrix:
Qwen2.5-Coder-14B-Instruct-AWQ → hermes
Qwen2.5-7B-Instruct → hermes
Qwen3-8B → qwen3_xml
Qwen3-32B → qwen3_xml
Qwen3-Coder-30B-A3B → qwen3_coder
Qwen2-72B-Instruct → hermes
Qwen1.5-7B-Chat → hermes
Two bugs hid the popup that opens on double-click (or right-click) of
a GPU button in the Serve panel:
1. z-index 240 vs the cookbook modal at 260 — popup rendered behind
the modal it was spawned from.
2. Horizontal position was just `button.left`, with no clamp against
the viewport. GPU buttons sit near the right edge of the modal, so
the popup got anchored at a left that pushed most of its body past
the viewport's right edge.
Switch the popup to position:fixed (escapes scrolling / transform
stacking contexts on any ancestor), bump z-index to 10010 (above the
themed-confirm / overlay layer that sits around 9000-10000), and
clamp left/top after measuring the rendered size — including flipping
above the button if there isn't room below. The popup is now fully
visible regardless of which GPU button it's anchored to or how
narrow the viewport is.
Typing / in the chat composer now shows a filtered popup listing all
available commands with their description. Arrow keys or Tab to select,
Enter/Tab to insert, Esc to close, click also works.
- New module: static/js/slashAutocomplete.js
Reads the existing COMMANDS registry (and LEGACY_ALIASES) from
slashCommands.js — no command logic added here, just discovery UI.
Excludes easter-egg commands (flip, roll, 8ball, fortune, odyssey,
ascii). Promotes short legacy aliases (/new, /clear, /web, /compact,
/research, etc.) as first-class rows so users don't have to know the
full /session new form.
- slashCommands.js: export COMMANDS and LEGACY_ALIASES so the new
module can read the registry.
- chat.js: lazy-import slashAutocomplete on init, wire to #message
textarea.
- style.css: popup + row styles using existing CSS variables.
The Cookbook fit scanner was reporting impossibly low VRAM requirements
for some pre-quantized models — e.g. cyankiwi/Qwen3-Coder-Next-REAM-AWQ-4bit
shown as 7.1 GB ('perfect' on a 12 GB card) when the real load is ~40 GB.
Root cause is in the catalog builder. When _entry_from_modelinfo falls
back to safetensors metadata for the parameter count, it stored
safetensors.total directly. For pre-quantized repos that figure reflects
*packed* element counts: AWQ/GPTQ-Int4 pack 8x 4-bit weights into one
I32, AWQ-8bit/GPTQ-Int8/FP8 pack 4x. The catalog therefore recorded
~1/8 of the real parameter count, and min_vram_gb = packed * bpp
double-applied the quantization.
Fix the safetensors fallback:
* prefer the per-dtype parameters dict when available and unpack only the
I32/I64 entries (the F16/BF16 scale/zero tensors and embeddings are
already at their real element counts)
* fall back to total * pack_factor when only total is exposed
Patch the catalog entries that were affected by the old fallback so the
fit ratings reflect reality without waiting for a full catalog rebuild:
* cyankiwi/Qwen3-Coder-Next-REAM-AWQ-4bit 11.4B -> 79.7B (40.8 GB VRAM)
* stelterlab/Qwen3-Coder-30B-A3B-Instruct-AWQ 4.6B -> 30.5B
* stelterlab/NVIDIA-Nemotron-3-Nano-30B-A3B-AWQ 5.1B -> 30.5B
* warshanks/Qwen3-8B-abliterated-AWQ 2.2B -> 8.2B
* QuantTrio/sarvam-30b-AWQ 7B -> 30B
* QuantTrio/sarvam-105b-AWQ 19B -> 105B
Closes#377.
POST /api/model-endpoints always inserted a new row, so Settings -> Add
Models -> Scan for Servers re-added any endpoint a user had already
registered manually — once under its model name (from the earlier
manual add) and again under its host:port (auto-generated when scan
posts without a name). The success toast then misreported the result
as "added N new".
Look up an existing endpoint with the same base_url accessible to the
caller (shared or owned by them) before inserting. If found, return it
with `existing: true` so the client can tell the difference between
an actual add and a dedupe hit. Toast now reads, e.g.,
"Found 1 server with 1 model — 1 already added".
Tested: POSTing the same base_url three times (incl. trailing-slash
variation) returns the same id each time; only one row exists.
Opt-in overlays under docker/ that pass the host GPU into the odysseus
container. Pick one in .env:
COMPOSE_FILE=docker-compose.yml:docker/gpu.nvidia.yml
COMPOSE_FILE=docker-compose.yml:docker/gpu.amd.yml
Non-GPU users are unaffected (no default merge). README now points at
the overlays instead of the old ad-hoc `gpus: all` suggestion.
Each overlay header notes that it only exposes the GPU devices — the
slim image still needs vLLM / llama-cpp-python / etc. installed via
Cookbook -> Dependencies before models can serve on GPU.
Tested on Arch + Docker 29.5.1 + RTX 4090:
docker compose exec odysseus nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 4090 (UUID: GPU-...)
Cookbook hardware scan reports the 24 GB GPU and recommends GPU-fit
models. `docker compose config` validates cleanly for all three
COMPOSE_FILE variants (base, +nvidia, +amd).
Builds on the structure proposed in #91 by @krllus with the path /
docs fixes from the review on that PR.
Closes#163.
Co-authored-by: krllus <krllus@users.noreply.github.com>
When Cookbook installs vllm via `pip install --user vllm`, pip pulls in
nvidia-cuda-* wheels under /app/.local but doesn't set CUDA_HOME or
create /usr/local/cuda. vllm 0.22+ then crashes during engine init:
RuntimeError: Could not find nvcc and default cuda_home='/usr/local/cuda' doesn't exist
After that, the mixed cuda-nvcc 13.3 / cuda-runtime 13.0 wheel combo
fails FlashInfer's JIT sampler with:
error: "CUDA compiler and CUDA toolkit headers are incompatible"
Detect the pip-installed nvcc on startup, point CUDA_HOME at it, and
default VLLM_USE_FLASHINFER_SAMPLER=0 (sampler only, no attention
impact) so the engine boots. No-op when vllm isn't installed.
Fixes#214.
Co-authored-by: sirs <sirs@local>
Sirsyorrz