mirror of
https://github.com/pewdiepie-archdaemon/odysseus.git
synced 2026-10-16 11:52:25 +02:00
37 lines
2.0 KiB
Markdown
37 lines
2.0 KiB
Markdown
# llama.cpp Provider Shape
|
|
|
|
Last updated: dev@86b886c | 2026-10-14
|
|
|
|
## Scope
|
|
|
|
Canonical provider ID `llamacpp`; OpenAI Chat/Responses and Anthropic Messages compatibility plus native server metadata; reader `src/model_capability_readers/llamacpp.py`.
|
|
|
|
## Metadata Shapes
|
|
|
|
`/v1/models` provides served identity and can include server model entries; native `/props` is authoritative for the running model/server combination:
|
|
|
|
- `model_alias`/`model_path`;
|
|
- `default_generation_settings.n_ctx` and sampling `params`;
|
|
- `total_slots` and optional `/slots[].n_ctx` fallback;
|
|
- `chat_template_caps` for tools/system role;
|
|
- `modalities.vision|audio`;
|
|
- current server/build state.
|
|
|
|
Capability depends on weights, projection/model assets, chat template, parser, and launch flags. It is endpoint evidence, not a checkpoint-name claim. `/props` and `/v1/models` can be merged only for the same served identity.
|
|
|
|
## Request And Response Shape
|
|
|
|
llama-server supports several OpenAI-compatible tasks and native extensions. Do not infer embeddings/rerank/chat solely from the OpenAI model card; use an explicit server model capability field or endpoint configuration. Tool and reasoning correctness can depend on selected chat template and parser.
|
|
|
|
## Fallback And Safety
|
|
|
|
The registry selects llama.cpp through an explicit vendor or endpoint kind; it does not auto-detect `/props` from payload shape. Port 8000 currently maps to the vLLM placeholder, while 8080 falls through to generic OpenAI-compatible. llama.cpp-only `session_id` and `cache_prompt` affinity fields must remain local endpoint behavior and never leak to strict cloud providers (#4640 and current affinity tests).
|
|
|
|
Runtime context lookup in `src/model_context.py` tries `/slots` first and falls back to `/props.default_generation_settings.n_ctx`, covering servers launched without `--slots`. This runtime path remains separate from canonical reader merging.
|
|
|
|
## Current Gaps
|
|
|
|
- Multi-model routing requires per-served-model `/props` association.
|
|
- Parser/template configuration is not yet fully represented in canonical
|
|
endpoint metadata.
|