Compare commits

..
Author SHA1 Message Date
RaresKeY 96c88c27c8 fix(personal): bound multi-file upload memory 2026-08-15 09:02:55 +00:00
Joeseph Grey 6edd771cc9 Merge branch 'dev' into fix/add-directory-event-loop 2026-08-12 10:39:22 -06:00
StressTestor 938251000b fix(personal): route upload and delete through the index job lock
/api/personal/upload and DELETE /api/personal/file mutated the same
vector and tracking state add/remove/reload serialize on, outside
_index_job_lock and inline on the event loop.

Both now stage async work on the loop, then run the complete transition
(vector writes, disk change, personal_docs_manager update) in one
offloaded critical section under the shared lock, acquired before the
offload so queued requests park on the loop rather than pinning a
threadpool worker.

Adds add-vs-upload and add-vs-file ordering regressions.
2026-08-12 10:17:36 -06:00
StressTestor e222e92153 fix(personal): serialize add/remove/reload on an async job lock
The #5558 fix took the job lock INSIDE the threadpool worker and only on the
add path, so (1) remove_directory and /reload mutated PersonalDocsManager's
unsynchronized list/index concurrently with an in-flight add — the inconsistent
state the PR claimed to prevent — and (2) a queued add blocked on the lock while
holding an AnyIO threadpool token, starving the shared pool.

Move the lock to an asyncio.Lock acquired in the async handler BEFORE offloading,
and route add, remove and reload through it. A waiting request now parks on the
event loop instead of pinning a worker, and all three mutators are serialized so
the 'add/remove are serialized and cannot leave inconsistent state' guarantee
holds. remove and reload also run their blocking work off the event loop. The
lock is per-router so each app binds it to its own loop; single-process scope.

Tests: add-vs-remove and add-vs-reload serialization regressions (async via
ASGITransport, since asyncio.Lock deadlocks starlette TestClient's portal); the
existing add-vs-add test converted to the same driver.
2026-07-27 20:43:04 +00:00
StressTestor b91f48f50a fix(personal): run directory indexing off the event loop (#5558)
POST /api/personal/add_directory called rag.index_personal_documents
inline from an async handler, so the whole indexing job (os.walk, file
reads, per-chunk embedding, Chroma inserts) ran on the event loop and
every other request queued behind it. Indexing a real directory froze
the UI and API for 25+ minutes with no sign of life.

Move the blocking section into the threadpool via run_in_threadpool.
personal_docs_manager.add_directory stays inside it because its
refresh_index() re-extracts text across tracked directories, which is
also blocking work. A module-level lock serializes index jobs so the
threadpool move does not introduce parallel jobs racing
PersonalDocsManager's unsynchronized list mutations and file writes;
they previously serialized on the blocked loop, so one-at-a-time is
behavior parity.
2026-07-27 20:43:04 +00:00
221 changed files with 3591 additions and 31442 deletions
+2 -29
View File
@@ -76,24 +76,12 @@ SEARXNG_INSTANCE=http://localhost:8080
# Change this if another local service already uses 7000 (macOS AirPlay often does).
# APP_PORT=7000
# Optional HTTP address advertised in companion/mobile pairing codes. Set this
# when Docker would otherwise advertise a container address or loopback. Use a
# LAN or Tailscale IPv4 address, a single-label hostname, or an mDNS *.local
# name that the phone can reach. HTTPS and public hostnames are not supported
# by the current companion client. Do not include credentials, a path, query,
# or fragment.
# COMPANION_BASE_URL=http://192.168.1.50:7000
# Development-only auth bypass for loopback requests.
# Keep false for Docker, LAN, reverse proxy, and any shared deployment.
# LOCALHOST_BYPASS=false
# Mark session cookies Secure. Left unset, this follows the request scheme:
# an HTTPS login gets a Secure cookie, a plain-HTTP one does not. Set true to
# force it on, or false to force it off while you still serve plain HTTP.
# Upgrading: this used to default to false. Drop a leftover SECURE_COOKIES=false
# from your .env unless you still need that escape hatch — it keeps HTTPS logins
# on a non-Secure cookie.
# Mark session cookies Secure. Set true when Odysseus is served through HTTPS
# by a trusted reverse proxy or private access gateway.
# SECURE_COOKIES=true
# Optional: pre-seed the first admin password during setup.
@@ -163,21 +151,6 @@ SEARXNG_INSTANCE=http://localhost:8080
# Local HTTP setups may use the callback URL inferred by the application.
# GOOGLE_OAUTH_REDIRECT_URI=https://your-domain.com/api/email/oauth/google/callback
# Origin the MCP OAuth callback is sent back to, for remote (Streamable HTTP)
# MCP servers that register it dynamically. Defaults to http://localhost:$APP_PORT,
# which is right only when you reach Odysseus directly on that port. Set it for
# HTTPS, reverse-proxy, hosted, and Docker installs — inside the container the
# app always listens on 7000 and cannot see the host port map, so the default is
# wrong there whenever APP_PORT is not 7000.
#
# Not for Google MCP servers. Those use Desktop App credentials, and Google only
# accepts loopback redirect URIs for that client type, so a public origin here is
# rejected with redirect_uri_mismatch. Leave it unset for a Google-only install:
# the loopback default is what Google wants, and remote users finish through the
# paste-back page, which never has to load the redirect.
# https://developers.google.com/identity/protocols/oauth2/native-app
# OAUTH_REDIRECT_BASE_URL=https://your-domain.com
# ============================================================
# Misc
# ============================================================
-7
View File
@@ -15,13 +15,6 @@ docker/entrypoint.sh text eol=lf
*.cmd text eol=crlf
*.bat text eol=crlf
# Vendored third-party bundles in static/lib/ are published minified artifacts
# and must stay byte-identical to what npm ships — stripping trailing whitespace
# to satisfy `git diff --check` would desync them from the upstream release. Turn
# the whitespace check off for that tree instead, and keep the bundles out of
# GitHub's language statistics.
static/lib/** -whitespace linguist-vendored
# Binary assets — never normalize.
*.png binary
*.jpg binary
-12
View File
@@ -26,18 +26,6 @@ body:
- label: I am running the latest code from the `dev` branch (the default branch you get on clone, where fixes land first) and the bug still reproduces there. Please `git pull` the latest `dev` before filing.
required: true
- type: input
id: revision
attributes:
label: Odysseus Revision
description: |
From the repository root (on the host when using Docker), run
`git show -s --abbrev=12 --format='%h (%cs)' HEAD`
and paste the output exactly.
placeholder: "1fef4929cf1d (2026-08-11)"
validations:
required: true
- type: dropdown
id: install-method
attributes:
-1
View File
@@ -28,7 +28,6 @@ Fixes #
- [ ] This PR targets `dev`
- [ ] My changes are limited to the scope described above — no unrelated refactors or whitespace changes mixed in.
- [ ] I actually ran the app (`docker compose up` or `uvicorn app:app`) and verified the change works end-to-end. Type-checks and unit tests are not enough.
- [ ] I did not run the app/runtime validation and stated that gap in **How to Test**. Leave this unchecked when the app-run box above is checked.
## How to Test
@@ -41,14 +41,6 @@ module.exports = async ({ github, context, core }) => {
break;
case 'bug': {
const revisionText = section('Odysseus Revision');
if (!/^[0-9a-f]{12} \(\d{4}-\d{2}-\d{2}\)$/i.test(revisionText)) {
failures.push(
'**Odysseus Revision** — paste the 12-character commit SHA and date, ' +
'for example `1fef4929cf1d (2026-08-11)`',
);
}
if (!section('Install Method')) {
failures.push('**Install Method** — select how you installed Odysseus');
}
+32 -142
View File
@@ -21,11 +21,11 @@ module.exports = async ({ github, context, core }) => {
return strip(m?.[0].replace(new RegExp(`#+\\s+${heading}`, 'i'), '') ?? '');
}
const descriptionProblems = [];
const problems = [];
// 1. Summary must be filled in.
if (section('Summary').length < 20) {
descriptionProblems.push('**Summary** is empty or too short — describe what changed and why.');
problems.push('**Summary** is empty or too short — describe what changed and why.');
}
// 2. Linked Issue must reference a real issue. Accept a bare #NNN, a closing
@@ -34,18 +34,18 @@ module.exports = async ({ github, context, core }) => {
const linkedSection = section('Linked Issue');
const hasIssueRef = /#\d+\b/.test(linkedSection) || /\/issues\/\d+/.test(linkedSection);
if (!linkedSection || !hasIssueRef) {
descriptionProblems.push('**Linked Issue** — add a reference like `Fixes #NNN`, a bare `#NNN`, or a link to the issue.');
problems.push('**Linked Issue** — add a reference like `Fixes #NNN`, a bare `#NNN`, or a link to the issue.');
}
// 3. At least one Type of Change box must be checked.
const typeBlock = body.match(/##\s+Type of Change[\s\S]*?(?=\n##\s|$)/i)?.[0] ?? '';
if (!/- \[x\]/i.test(typeBlock)) {
descriptionProblems.push('**Type of Change** — check at least one box.');
problems.push('**Type of Change** — check at least one box.');
}
// 4. Duplicate-search checklist item must be checked.
if (!/- \[x\] I searched/i.test(body)) {
descriptionProblems.push('**Checklist** — check the duplicate-search box to confirm you searched existing issues and PRs.');
problems.push('**Checklist** — check the duplicate-search box to confirm you searched existing issues and PRs.');
}
// 5. How to Test must contain enough real detail for a reviewer to act on.
@@ -53,83 +53,7 @@ module.exports = async ({ github, context, core }) => {
// code block — so we only require non-trivial content, not a specific shape.
const howTo = section('How to Test');
if (howTo.length < 30) {
descriptionProblems.push('**How to Test** — explain how a reviewer can verify this change. Numbered steps, the commands you ran, or a short code block all work — give a sentence or two of real detail (not just "tested locally").');
}
// Classify paths from GitHub's API. This workflow runs in the privileged base
// context, so it must never check out or execute code from the PR branch.
const changedFiles = await github.paginate(github.rest.pulls.listFiles, {
owner, repo, pull_number: prNum, per_page: 100,
});
const changedPaths = changedFiles.map(file => file.filename);
function isUiSensitivePath(filename) {
const path = filename.toLowerCase();
return path.startsWith('static/')
|| path.startsWith('templates/')
|| /\.(?:html?|css|svg)$/.test(path);
}
function isDocsOnlyPath(filename) {
const path = filename.toLowerCase();
return /\.(?:md|mdx|rst|adoc|txt)$/.test(path)
|| (path.startsWith('docs/') && !isUiSensitivePath(path));
}
function isRuntimeSensitivePath(filename) {
const path = filename.toLowerCase();
if (isUiSensitivePath(path)) return false;
if (path.startsWith('tests/') || path.startsWith('.github/')) return false;
return /^(?:app\.py|routes\/|services\/|src\/|core\/|mcp_servers\/|scripts\/|docker\/)/.test(path)
|| /^(?:dockerfile|docker-compose.*\.ya?ml|requirements(?:-optional)?\.txt|pyproject\.toml|setup\.py)$/.test(path)
|| /\.(?:py|sh|ps1|bat)$/.test(path);
}
let classification = 'tooling';
if (changedPaths.some(isUiSensitivePath)) {
classification = 'UI-sensitive';
} else if (changedPaths.some(isRuntimeSensitivePath)) {
classification = 'backend/runtime';
} else if (changedPaths.length > 0 && changedPaths.every(isDocsOnlyPath)) {
classification = 'docs-only';
}
const appRan = /- \[x\]\s+I actually ran the app\b/i.test(body);
const appNotRun = /- \[x\]\s+I did not run the app\/runtime validation\b/i.test(body);
// Anchor on the wording, not the template's emphasis: a ticked box the author
// retyped without the surrounding ** renders identically on the PR page, so
// treating it as unchecked is invisible from their side. Matches the two
// attestations above, which already ignore formatting.
const screenshotChecked = /- \[x\]\s+[*_]{0,2}Screenshot or short clip[*_]{0,2}/i.test(body);
const screenshotSection = section('Screenshots / clips');
const hasVisualEvidence = /!\[[^\]]*\]\([^)]+\)|<(?:img|video|source)\b[^>]*(?:src|href)=|https?:\/\/[^\s)]+/i.test(screenshotSection);
const evidenceGaps = [];
let needsRuntimeValidation = false;
let needsVisualEvidence = false;
if (classification === 'backend/runtime' || classification === 'UI-sensitive') {
if (appRan && appNotRun) {
needsRuntimeValidation = true;
evidenceGaps.push('The app-run and explicit not-run boxes are both checked. Select the one state that is true.');
} else if (!appRan) {
needsRuntimeValidation = true;
if (appNotRun) {
evidenceGaps.push('The author explicitly reports that app/runtime validation was not performed.');
} else {
evidenceGaps.push('App/runtime validation is not author-attested. Check the run box only after running it, or check the explicit not-run box and describe the gap.');
}
}
}
if (classification === 'UI-sensitive') {
if (!screenshotChecked) {
needsVisualEvidence = true;
evidenceGaps.push('The screenshot/clip checkbox is not checked for this UI-sensitive change.');
}
if (!hasVisualEvidence) {
needsVisualEvidence = true;
evidenceGaps.push('The Screenshots / clips section does not contain an actual attachment or link.');
}
problems.push('**How to Test** — explain how a reviewer can verify this change. Numbered steps, the commands you ran, or a short code block all work — give a sentence or two of real detail (not just "tested locally").');
}
// ── Comment ──────────────────────────────────────────────────────────────
@@ -138,43 +62,22 @@ module.exports = async ({ github, context, core }) => {
});
const existing = comments.find(c => (c.body ?? '').includes(MARKER));
if (descriptionProblems.length === 0 && evidenceGaps.length === 0) {
if (problems.length === 0) {
if (existing) {
await github.rest.issues.deleteComment({ owner, repo, comment_id: existing.id });
}
} else {
const commentLines = [MARKER];
if (descriptionProblems.length > 0) {
commentLines.push(
'⚠️ **PR description — action needed**',
'',
'The following required sections are missing or incomplete. Please update the PR description to address them:',
'',
descriptionProblems.map(problem => `- ${problem}`).join('\n'),
);
} else {
commentLines.push(
'⚠️ **PR description is complete; validation evidence is still outstanding**',
'',
`Changed-file classification: **${classification}**.`,
);
}
if (evidenceGaps.length > 0) {
commentLines.push(
'',
'**Author-reported runtime / visual state**',
'',
evidenceGaps.map(gap => `- ${gap}`).join('\n'),
'',
'Checkboxes are author attestations. GitHub Actions results remain the execution evidence for CI; this check does not prove that a local command ran.',
);
}
commentLines.push(
const commentBody = [
MARKER,
'⚠️ **PR description — action needed**',
'',
'The following required sections are missing or incomplete. Please update the PR description to address them:',
'',
problems.map(p => `- ${p}`).join('\n'),
'',
'---',
'_This comment updates automatically when the description or changed files change._',
);
const commentBody = commentLines.join('\n');
'_This comment is deleted automatically once all sections are complete._',
].join('\n');
if (existing) {
await github.rest.issues.updateComment({ owner, repo, comment_id: existing.id, body: commentBody });
@@ -194,47 +97,34 @@ module.exports = async ({ github, context, core }) => {
return true;
} catch (e) {
if (e.status === 404) return false;
if (e.status === 403) {
core.warning(`Could not inspect label "${name}" — token lacks label read access; skipping.`);
return false;
}
throw e;
}
}
async function setLabel(name, wanted) {
if (wanted && await labelExists(name)) {
async function swapLabel(num, add, remove) {
if (await labelExists(add)) {
try {
await github.rest.issues.addLabels({ owner, repo, issue_number: prNum, labels: [name] });
await github.rest.issues.addLabels({ owner, repo, issue_number: num, labels: [add] });
} catch (e) {
// Fail soft on a token that can't write labels so a label permission
// problem never masks the actual description verdict.
if (e.status !== 403 && e.status !== 404) throw e;
core.warning(`Could not add "${name}" — label is unavailable or the token lacks label write access; skipping.`);
if (e.status !== 403) throw e;
core.warning(`Could not add "${add}" — token lacks label write here; skipping.`);
}
} else if (wanted) {
core.warning(`Label "${name}" does not exist in the repo — skipping. Create it once to enable labelling.`);
} else {
try {
await github.rest.issues.removeLabel({ owner, repo, issue_number: prNum, name });
} catch (e) {
if (e.status !== 404 && e.status !== 410 && e.status !== 403) throw e;
}
core.warning(`Label "${add}" does not exist in the repo — skipping. Create it once to enable labelling.`);
}
try {
await github.rest.issues.removeLabel({ owner, repo, issue_number: num, name: remove });
} catch (e) {
if (e.status !== 404 && e.status !== 410 && e.status !== 403) throw e;
}
}
const descriptionComplete = descriptionProblems.length === 0;
const evidenceComplete = evidenceGaps.length === 0;
const isDraft = Boolean(context.payload.pull_request.draft);
await setLabel(
'ready for review',
descriptionComplete && evidenceComplete && !isDraft,
);
await setLabel('needs work', !descriptionComplete);
await setLabel('needs runtime validation', needsRuntimeValidation);
await setLabel('needs visual evidence', needsVisualEvidence);
if (!descriptionComplete) {
core.setFailed(`PR description has ${descriptionProblems.length} issue(s) — see bot comment for details.`);
if (problems.length === 0) {
await swapLabel(prNum, 'ready for review', 'needs work');
} else {
await swapLabel(prNum, 'needs work', 'ready for review');
core.setFailed(`PR description has ${problems.length} issue(s) — see bot comment for details.`);
}
};
+3 -9
View File
@@ -5,11 +5,7 @@ on:
# works on fork PRs. Safe here: the checkout pins to the base branch (no fork
# code runs) and the scripts only read context.payload and call the GitHub API.
pull_request_target: # zizmor: ignore[dangerous-triggers]
types: [opened, edited, synchronize, reopened, ready_for_review, converted_to_draft]
concurrency:
group: pr-description-${{ github.event.pull_request.number }}
cancel-in-progress: true
types: [opened, edited, synchronize, reopened, ready_for_review]
# Default-deny at the workflow level; each job opts into only the scopes it needs.
# Note: modifying a PR's labels/comments needs pull-requests:write even though the
@@ -63,14 +59,12 @@ jobs:
check-mergeable:
name: Flag unmergeable PRs
needs: check-description
runs-on: ubuntu-latest
permissions:
pull-requests: write
issues: write
# Run after description validation failures, but never from an obsolete
# workflow run canceled by a newer PR event.
if: ${{ !cancelled() && github.event.pull_request.user.type != 'Bot' }}
# Skip bots: they open PRs programmatically and have their own process.
if: github.event.pull_request.user.type != 'Bot'
steps:
- uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
+2 -10
View File
@@ -65,16 +65,6 @@ Vendored in `static/lib/` and served directly:
| [jsPDF](https://github.com/parallax/jsPDF) (bundled in html2pdf) | PDF generation | MIT |
| [html2canvas](https://github.com/niklasvh/html2canvas) (bundled in html2pdf) | DOM → canvas rasterization | MIT |
| [node-qrcode](https://github.com/soldair/node-qrcode) (`qrcode.min.js`) | QR-code rendering (2FA setup) | MIT |
| [KaTeX](https://github.com/KaTeX/KaTeX) v0.16.22 (`katex/katex.min.{js,css}` + `katex/fonts/*.woff2`) | Math typesetting | MIT ([`licenses/KaTeX-MIT-LICENSE.txt`](licenses/KaTeX-MIT-LICENSE.txt)) |
| [Mermaid](https://github.com/mermaid-js/mermaid) v11.16.1 (`mermaid.min.js`) | Diagrams from text | MIT ([`licenses/Mermaid-MIT-LICENSE.txt`](licenses/Mermaid-MIT-LICENSE.txt)) |
KaTeX and Mermaid are loaded on first use by `static/js/markdown.js` rather than
from `index.html`, so a session that renders no math and no diagram never fetches
either. Only the `.woff2` KaTeX fonts are shipped, matching `static/fonts/`; the
`.woff` and `.ttf` variants its stylesheet also lists are never requested by a
browser that supports `woff2`. The bundles are the published npm artifacts,
unmodified — `.gitattributes` turns the whitespace check off for `static/lib/`
so they can stay byte-identical to upstream.
## Front-end libraries loaded at runtime (CDN)
@@ -82,6 +72,8 @@ Referenced from `cdn.jsdelivr.net` / `cdnjs.cloudflare.com` at runtime — not v
| Library | Purpose | License |
|---|---|---|
| [KaTeX](https://github.com/KaTeX/KaTeX) 0.16.22 | Math typesetting | MIT |
| [Mermaid](https://github.com/mermaid-js/mermaid) 11 | Diagrams from text | MIT |
| [Pyodide](https://github.com/pyodide/pyodide) 0.27.5 | In-browser Python runtime | MPL-2.0 |
| [PDFObject](https://github.com/pipwerks/PDFObject) 2.1.1 | Inline PDF embedding | MIT |
+5 -10
View File
@@ -59,20 +59,15 @@ Help is welcome. The best entry points are fresh-install testing, provider setup
## Security
Odysseus is a self-hosted workspace with powerful local tools. Keep auth enabled, keep private data out of Git, and do not expose raw model/service ports publicly.
- Keep `AUTH_ENABLED=true` for any network-accessible deployment.
- Keep `LOCALHOST_BYPASS=false` outside local development.
Deployment details are in the [setup guide](docs/setup.md#security-notes).
Odysseus is a self-hosted workspace with powerful local tools. Keep auth enabled, keep private data out of Git, and do not expose raw model/service ports publicly. Deployment details are in the [setup guide](docs/setup.md#security-notes).
## Star History
<a href="https://star-history.dera.page/#odysseus-dev/odysseus&type=date&legend=top-left">
<a href="https://www.star-history.com/?repos=odysseus-dev%2Fodysseus&type=date&legend=top-left">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://star-history.dera.page/svg?repos=odysseus-dev/odysseus&type=date&theme=dark&legend=top-left" />
<source media="(prefers-color-scheme: light)" srcset="https://star-history.dera.page/svg?repos=odysseus-dev/odysseus&type=date&legend=top-left" />
<img alt="Star History Chart" src="https://star-history.dera.page/svg?repos=odysseus-dev/odysseus&type=date&legend=top-left" />
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=odysseus-dev/odysseus&type=date&theme=dark&legend=top-left" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=odysseus-dev/odysseus&type=date&legend=top-left" />
<img alt="Star History Chart" src="https://api.star-history.com/chart?repos=odysseus-dev/odysseus&type=date&legend=top-left" />
</picture>
</a>
+1 -1
View File
@@ -10,7 +10,7 @@ Security fixes are handled on the default branch until formal releases are cut.
- Keep `AUTH_ENABLED=true` for any network-accessible deployment.
- Keep `LOCALHOST_BYPASS=false` outside local development.
- Leave `SECURE_COOKIES` unset unless you need to override it: session cookies are marked `Secure` whenever the request arrives over HTTPS. Set `SECURE_COOKIES=true` to force it on (for a proxy Odysseus cannot see the scheme of), or `SECURE_COOKIES=false` to force it off while you still serve plain HTTP alongside HTTPS.
- Set `SECURE_COOKIES=true` when Odysseus is served through HTTPS by a trusted reverse proxy or private access gateway.
- Use HTTPS when exposing the app beyond localhost.
- Put the authenticated Odysseus web/API entrypoint behind a trusted reverse proxy or private access layer such as Cloudflare Access, Tailscale, or a VPN.
- Keep ChromaDB, SearXNG, ntfy, Ollama, vLLM, llama.cpp, databases, and raw model/provider APIs internal-only.
+1 -1
View File
@@ -37,7 +37,7 @@ Non-admin defaults are in `core/auth.py:DEFAULT_PRIVILEGES`. Tool enforcement is
- **Sessions:** bcrypt passwords, 7-day session tokens stored atomically in `data/sessions.json` via `core/atomic_io.py`.
- **2FA:** TOTP with 8 single-use backup codes. Verified after password check, before session issuance.
- **Reserved usernames:** request sentinels and the Default/Local storage owner cannot be registered or renamed into. Defined in `core/auth.py:RESERVED_USERNAMES`.
- **Reserved usernames:** `internal-tool`, `api`, `demo`, `system` cannot be registered or renamed into. Defined in `core/auth.py:RESERVED_USERNAMES`.
- `internal-tool` is security-critical: `core/middleware.py:require_admin` treats any request where `request.state.current_user == "internal-tool"` as the in-process tool loopback and grants admin unconditionally. A real account with that name would silently pass every `require_admin` check.
- **Orphan sessions:** `validate_token` re-checks that the user record still exists on every call. A deleted user's cookie is dropped on next request rather than continuing to authenticate.
+10 -34
View File
@@ -67,13 +67,7 @@ from core.constants import (
REQUEST_TIMEOUT, OPENAI_API_KEY, AUTH_FILE,
)
from core.database import SessionLocal, ApiToken
from core.middleware import (
SecurityHeadersMiddleware,
get_application_route_path,
is_cors_preflight,
path_is_route_or_child,
with_asgi_root_path,
)
from core.middleware import SecurityHeadersMiddleware, is_cors_preflight
from core.auth import AuthManager, normalize_known_username
from core.exceptions import (
SessionNotFoundError, InvalidFileUploadError,
@@ -84,7 +78,6 @@ import bcrypt as _bcrypt
from src.app_helpers import abs_join, serve_html_with_nonce
from src.generated_images import GENERATED_IMAGE_HEADERS, resolve_generated_image_path
from src.owner_identity import auth_disabled
from starlette.responses import RedirectResponse
# ========= LOGGING =========
@@ -255,7 +248,7 @@ from routes.auth_routes import setup_auth_routes, SESSION_COOKIE
auth_manager = AuthManager()
app.state.auth_manager = auth_manager
AUTH_ENABLED = not auth_disabled()
AUTH_ENABLED = os.getenv("AUTH_ENABLED", "true").lower() != "false"
LOCALHOST_BYPASS = os.getenv("LOCALHOST_BYPASS", "false").lower() == "true"
if LOCALHOST_BYPASS:
logger.warning("LOCALHOST_BYPASS is enabled, loopback requests bypass authentication. Do not expose this instance to a network.")
@@ -291,7 +284,7 @@ if AUTH_ENABLED:
def _is_auth_exempt(path: str) -> bool:
if path in AUTH_EXEMPT_EXACT:
return True
if any(path_is_route_or_child(path, p) for p in AUTH_EXEMPT_PREFIXES):
if any(path.startswith(p) for p in AUTH_EXEMPT_PREFIXES):
return True
return any(p.match(path) for p in AUTH_EXEMPT_PATTERNS)
@@ -362,7 +355,7 @@ if AUTH_ENABLED:
class AuthMiddleware(BaseHTTPMiddleware):
async def dispatch(self, request: Request, call_next):
path = get_application_route_path(request.scope)
path = request.url.path
# A genuine CORS preflight (OPTIONS + Access-Control-Request-Method)
# carries no credentials by design and must reach CORSMiddleware to be
# answered. AuthMiddleware is the outermost middleware, so gating the
@@ -406,10 +399,7 @@ if AUTH_ENABLED:
if not auth_manager.is_configured:
# No users yet — redirect to login for first-time setup
if not path.startswith("/api/"):
return RedirectResponse(
url=with_asgi_root_path(request.scope, "/login"),
status_code=302,
)
return RedirectResponse(url="/login", status_code=302)
return JSONResponse(status_code=401, content={"error": "Setup required"})
# --- Bearer token auth (API tokens for external integrations) ---
@@ -471,10 +461,7 @@ if AUTH_ENABLED:
if not auth_manager.validate_token(token):
if path.startswith("/api/"):
return JSONResponse(status_code=401, content={"error": "Not authenticated"})
return RedirectResponse(
url=with_asgi_root_path(request.scope, "/login"),
status_code=302,
)
return RedirectResponse(url="/login", status_code=302)
# Attach current username to request state for downstream routes
request.state.current_user = auth_manager.get_username_for_token(token)
@@ -643,24 +630,13 @@ app.include_router(auth_router)
@app.post("/api/activity/heartbeat")
async def activity_heartbeat():
from src.interactive_gate import (
mark_browser_activity,
maybe_stop_background_tasks_for_heartbeat,
)
from src.interactive_gate import mark_browser_activity
await mark_browser_activity()
async def _stop_background():
try:
await maybe_stop_background_tasks_for_heartbeat(
task_scheduler.stop_background_tasks_for_foreground
)
await task_scheduler.stop_background_tasks_for_foreground(reason="browser heartbeat")
except Exception:
logging.getLogger("app.foreground_gate").debug(
"heartbeat task stop failed",
exc_info=True,
)
logging.getLogger("app.foreground_gate").debug("heartbeat task stop failed", exc_info=True)
asyncio.create_task(_stop_background())
return {"ok": True}
@@ -784,7 +760,7 @@ from src.task_scheduler import TaskScheduler
task_scheduler = TaskScheduler(session_manager)
from src.event_bus import set_task_scheduler
set_task_scheduler(task_scheduler)
from routes.task.task_routes import setup_task_routes
from routes.task_routes import setup_task_routes
app.include_router(setup_task_routes(task_scheduler))
from routes.assistant_routes import setup_assistant_routes
-4
View File
@@ -73,10 +73,6 @@ cat > "$APP/Contents/MacOS/$APP_NAME.tmpl" <<'LAUNCHER'
INSTALL_DIR="__INSTALL_DIR__"
PORT="__PORT__"
URL="http://127.0.0.1:${PORT}"
# uvicorn is started with --port below, but APP_PORT is what the app itself
# reads when it needs to build a URL for this instance (internal_api_base(),
# companion pairing, the MCP OAuth callback), so export it as well.
export APP_PORT="$PORT"
export PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:$PATH"
UVICORN="$INSTALL_DIR/venv/bin/uvicorn"
-99
View File
@@ -6,14 +6,11 @@ units so the route layer stays thin and the logic is directly testable.
from __future__ import annotations
import ipaddress
import json
import os
import re
import secrets
import socket
import uuid
from urllib.parse import urlsplit
import bcrypt
@@ -23,102 +20,6 @@ PAIRING_VERSION = 1
COMPANION_SCOPE = "chat"
_COMPANION_IPV4_NETWORKS = tuple(
ipaddress.ip_network(cidr)
for cidr in (
"10.0.0.0/8",
"100.64.0.0/10",
"127.0.0.0/8",
"169.254.0.0/16",
"172.16.0.0/12",
"192.168.0.0/16",
)
)
_DNS_LABEL_RE = re.compile(r"[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?\Z")
def _valid_companion_client_host(host: str) -> bool:
"""Match the host forms supported by the current v1 Expo client."""
if not host or len(host) > 253 or not host.isascii() or "%" in host:
return False
try:
address = ipaddress.ip_address(host)
except ValueError:
labels = host.split(".")
if any(not _DNS_LABEL_RE.fullmatch(label) for label in labels):
return False
if any(label.startswith("xn--") for label in labels):
return False
# WHATWG URL parsers treat a decimal or ``0x`` single-label hostname
# as an IPv4 number even though Python's strict ``ipaddress`` parser
# rejects that spelling. The v1 client interpolates this host back
# into a URL, so accepting e.g. ``134744072`` would make the phone send
# its bearer token to public 8.8.8.8. Keep DNS labels unambiguous.
if len(labels) == 1 and (
labels[0].isdigit()
or re.fullmatch(r"0x[0-9a-f]*", labels[0]) is not None
):
return False
return len(labels) == 1 or (len(labels) >= 2 and labels[-1] == "local")
return isinstance(address, ipaddress.IPv4Address) and any(
address in network for network in _COMPANION_IPV4_NETWORKS
)
def parse_companion_base_url(value: str) -> tuple[str, int]:
"""Validate a v1 companion address and return its legacy (host, port).
The deployed client understands only HTTP plus a LAN-style host and port.
Reject anything outside that exact contract instead of advertising a URL
the client would reject, downgrade, or interpret differently.
"""
if not isinstance(value, str) or not value:
raise ValueError("COMPANION_BASE_URL must be a canonical HTTP LAN origin")
if not value.isascii():
raise ValueError("COMPANION_BASE_URL must contain only ASCII characters")
if any(
ord(char) <= 32 or ord(char) == 127 or char in {"\\", "%"}
for char in value
):
raise ValueError(
"COMPANION_BASE_URL contains a forbidden character"
)
try:
parsed = urlsplit(value)
port = parsed.port
except ValueError as exc:
raise ValueError("COMPANION_BASE_URL must be a valid HTTP LAN origin") from exc
host = parsed.hostname
if parsed.scheme.lower() != "http" or not parsed.netloc or not host:
raise ValueError("COMPANION_BASE_URL must be a canonical HTTP LAN origin")
if parsed.username is not None or parsed.password is not None:
raise ValueError("COMPANION_BASE_URL must not contain credentials")
if parsed.path or parsed.query or parsed.fragment:
raise ValueError("COMPANION_BASE_URL must not contain a path, query, or fragment")
if port is not None and not 1 <= port <= 65535:
raise ValueError("COMPANION_BASE_URL port must be between 1 and 65535")
if not _valid_companion_client_host(host):
raise ValueError("COMPANION_BASE_URL host is not supported by companion v1")
netloc = f"{host}:{port}" if port is not None else host
origin = f"http://{netloc}"
if value != origin:
raise ValueError("COMPANION_BASE_URL must be a canonical HTTP LAN origin")
return host, port or 80
def configured_companion_origin() -> tuple[str, int] | None:
"""Return the validated operator-configured v1 address, if any."""
value = os.environ.get("COMPANION_BASE_URL")
if value is None or value == "":
return None
return parse_companion_base_url(value)
def default_port() -> int:
"""Best guess at the port the server is reachable on. Callers that know the
real request port should pass it explicitly."""
+8 -23
View File
@@ -23,7 +23,7 @@ from fastapi import APIRouter, HTTPException, Request
from fastapi.responses import HTMLResponse
from core.middleware import require_admin
from src.auth_helpers import _auth_disabled, get_current_user
from src.auth_helpers import get_current_user
from companion import pairing as _pairing
@@ -113,9 +113,8 @@ def setup_companion_routes() -> APIRouter:
The stock /api/models route scopes to get_current_user, which for a
bearer token is the sandboxed pseudo-user "api" (owns nothing). Here we
scope to the token's real owner instead, plus legacy null-owner shared
rows -- the same rule as owner_filter. Explicit auth-disabled mode keeps
the stock route's single-user all-endpoints view. Read-only; never
returns api_key material.
rows -- the same rule as owner_filter. Read-only; never returns api_key
material.
"""
require_models_scope(request)
import json as _json
@@ -124,11 +123,6 @@ def setup_companion_routes() -> APIRouter:
from src.endpoint_resolver import build_chat_url
owner = token_owner(request)
single_user_mode = (
owner is None
and not getattr(request.state, "api_token", False)
and _auth_disabled()
)
out = []
db = SessionLocal()
try:
@@ -139,7 +133,7 @@ def setup_companion_routes() -> APIRouter:
if owner:
q = q.filter((ModelEndpoint.owner == owner) | (ModelEndpoint.owner == None)) # noqa: E711
for ep in q.all():
if not single_user_mode and not owner_can_see(ep.owner, owner):
if not owner_can_see(ep.owner, owner):
continue
try:
model_ids = _json.loads(ep.cached_models) if ep.cached_models else []
@@ -200,27 +194,19 @@ def setup_companion_routes() -> APIRouter:
the code works immediately, no restart. `?format=json` returns the
payload for an in-app pairing screen."""
require_admin(request)
try:
configured_origin = _pairing.configured_companion_origin()
except ValueError as exc:
raise HTTPException(500, str(exc)) from None
owner = get_current_user(request)
invalidate = getattr(request.app.state, "invalidate_token_cache", None)
token_id, raw_token = mint_pairing_token(owner, invalidate)
if configured_origin:
host, port = configured_origin
hosts = [host]
else:
hosts = _pairing.lan_ip_candidates()
host = hosts[0] if hosts else "127.0.0.1"
port = request.url.port or _pairing.default_port()
hosts = _pairing.lan_ip_candidates()
host = hosts[0] if hosts else "127.0.0.1"
port = request.url.port or _pairing.default_port()
payload = _pairing.pairing_payload(host, port, raw_token)
qr = _pairing.pairing_qr_png_data_uri(payload)
qr_ok = bool(qr and qr.startswith("data:image/png;base64,"))
if (request.query_params.get("format") or "").lower() == "json":
response = {
return {
"host": host,
"port": port,
"token": raw_token,
@@ -229,7 +215,6 @@ def setup_companion_routes() -> APIRouter:
"payload": payload,
"qr": qr if qr_ok else None,
}
return response
import json as _json
payload_json = _json.dumps(payload, separators=(",", ":"))
+16 -9
View File
@@ -20,6 +20,7 @@ logger = logging.getLogger(__name__)
from core.atomic_io import atomic_write_json as _atomic_write_json # noqa: E402
from core.middleware import INTERNAL_TOOL_USER # noqa: E402
DEFAULT_PRIVILEGES = {
"can_use_agent": True,
@@ -48,18 +49,24 @@ ADMIN_PRIVILEGES["allowed_models_restricted"] = False
ADMIN_PRIVILEGES["block_all_models"] = False
from src.constants import AUTH_FILE, PASSWORD_MIN_LENGTH
from src.owner_identity import RESERVED_AUTH_USERNAMES
DEFAULT_AUTH_PATH = AUTH_FILE
TOKEN_TTL = 60 * 60 * 24 * 7 # 7 days
# Usernames the auth + middleware layer reserves for request sentinels and
# internal storage owners; they must never belong to a real login account.
# "internal-tool" is the most dangerous because `core.middleware.require_admin`
# treats it as the in-process tool loopback. "api" collides with bearer-token
# attribution. "demo"/"system" are synthetic owners already special-cased by
# scheduler/assistant/research paths. The Default/Local owner is a storage
# bucket for explicit auth-disabled no-login mode, not a login username.
RESERVED_USERNAMES = frozenset(RESERVED_AUTH_USERNAMES)
# Usernames the auth + middleware layer reserve as internal "synthetic owner"
# sentinels; they must never belong to a real account. The most dangerous is
# "internal-tool": `core.middleware.require_admin` treats any request whose
# `current_user == "internal-tool"` as the in-process tool loopback and grants
# admin, and because the cookie auth path sets `current_user` to the raw
# username, an account literally named "internal-tool" would be silently
# treated as an admin by every `require_admin`-gated route. "api" collides with
# the bearer-token owner-attribution sentinel. "demo"/"system" round out the
# synthetic-owner set the rest of the codebase already special-cases (see
# `_SYNTHETIC_OWNERS` in routes/assistant_routes.py and the matching guards in
# src/task_scheduler.py / routes/research_routes.py) — a real account with one
# of those names would be denied an assistant and inconsistently owner-scoped.
# Refuse to create or rename into any of them so the sentinels can't be
# impersonated. (Keep this in sync with that synthetic-owner set.)
RESERVED_USERNAMES = frozenset({INTERNAL_TOOL_USER, "api", "demo", "system"})
def normalize_known_username(users: Dict[str, Any], username: str | None) -> Optional[str]:
+2 -19
View File
@@ -1491,25 +1491,8 @@ def _migrate_assign_legacy_owner():
with open(prefs_path, "r", encoding="utf-8") as f:
prefs = _json.load(f)
if "_users" not in prefs and prefs:
# Flat format → nest ordinary preferences under the admin
# user. Foreground fallback is an explicit per-owner opt-in,
# so auth-disabled consent must remain inert at the flat root
# rather than becoming consent for the first named owner.
foreground_keys = {
"foreground_fallback_enabled",
"foreground_model_fallbacks",
}
named_prefs = {
key: value
for key, value in prefs.items()
if key not in foreground_keys
}
new_prefs = {
key: prefs[key]
for key in foreground_keys
if key in prefs
}
new_prefs["_users"] = {admin_user: named_prefs}
# Flat format → nest under admin user
new_prefs = {"_users": {admin_user: prefs}}
with open(prefs_path, "w", encoding="utf-8") as f:
_json.dump(new_prefs, f, indent=2)
logger.info(f"Migrated user_prefs.json to per-user format under '{admin_user}'")
+3 -29
View File
@@ -3,14 +3,10 @@
import os
import secrets
from collections.abc import Mapping
from fastapi import HTTPException, Request
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.responses import Response
from starlette.routing import get_route_path
from src.owner_identity import INTERNAL_TOOL_USER, auth_disabled
# Per-process token that lets the in-app tool layer hit admin-gated
@@ -19,30 +15,8 @@ from src.owner_identity import INTERNAL_TOOL_USER, auth_disabled
# same value from this module. Never persisted or exposed externally.
INTERNAL_TOOL_TOKEN = os.environ.get("ODYSSEUS_INTERNAL_TOKEN") or secrets.token_hex(32)
INTERNAL_TOOL_HEADER = "X-Odysseus-Internal-Token"
def get_application_route_path(scope: Mapping[str, object]) -> str:
"""Return the application-relative path used by Starlette routing.
Uvicorn prefixes ``scope["path"]`` with a configured ASGI ``root_path``;
Starlette removes that prefix before matching routes. Middleware policy
must use the same path form or a deployment prefix can change which policy
applies to an otherwise unchanged application route.
"""
return get_route_path(scope)
def with_asgi_root_path(scope: Mapping[str, object], path: str) -> str:
"""Prefix an application path for a client-facing redirect target."""
root_path = scope.get("root_path", "")
if not isinstance(root_path, str) or not root_path:
return path
return f"{root_path.rstrip('/')}{path}"
def path_is_route_or_child(path: str, prefix: str) -> bool:
"""Return whether ``path`` is exactly ``prefix`` or below that route."""
return path == prefix or path.startswith(prefix + "/")
# Pseudo-username on in-process tool-loopback requests; require_admin trusts it and it is reserved.
INTERNAL_TOOL_USER = "internal-tool"
def is_cors_preflight(method: str, headers) -> bool:
@@ -73,7 +47,7 @@ def require_admin(request: Request):
pass
auth_mgr = getattr(request.app.state, "auth_manager", None)
if auth_disabled():
if os.getenv("AUTH_ENABLED", "true").lower() == "false":
return
if not auth_mgr or not auth_mgr.is_configured:
raise HTTPException(403, "Admin only")
+1 -17
View File
@@ -14,8 +14,6 @@ import logging
from datetime import datetime, timezone, timedelta
from typing import Dict, Optional
from sqlalchemy import func
from .database import Session as DbSession, ChatMessage as DbChatMessage, Document as DbDocument, SessionLocal, utcnow_naive
from .models import Session, ChatMessage
from src.attachment_refs import persistable_message_content
@@ -94,28 +92,14 @@ class SessionManager:
try:
db_sessions = db.query(DbSession).filter(
DbSession.archived == False,
DbSession.messages.any(),
DbSession.message_count > 0,
).order_by(DbSession.last_accessed.desc()).limit(100).all()
# message_count is derived metadata and can drift after interrupted
# or legacy writes. Count only the bounded discovery set so startup
# remains metadata-only while lazy hydration sees an authoritative
# positive count for every discovered non-empty session.
message_counts = {}
if db_sessions:
message_counts = dict(
db.query(DbChatMessage.session_id, func.count(DbChatMessage.id))
.filter(DbChatMessage.session_id.in_([row.id for row in db_sessions]))
.group_by(DbChatMessage.session_id)
.all()
)
loaded_count = 0
for db_session in db_sessions:
try:
session = self._db_to_session_meta(db_session)
if session is not None:
session.message_count = message_counts[db_session.id]
self.sessions[db_session.id] = session
loaded_count += 1
except Exception as e:
+1 -12
View File
@@ -46,11 +46,10 @@ services:
- DATABASE_URL=${DATABASE_URL:-sqlite:///./data/app.db}
- AUTH_ENABLED=${AUTH_ENABLED:-true}
- LOCALHOST_BYPASS=${LOCALHOST_BYPASS:-false}
- COMPANION_BASE_URL=${COMPANION_BASE_URL:-}
- ODYSSEUS_ADMIN_USER=${ODYSSEUS_ADMIN_USER:-admin}
- ODYSSEUS_ADMIN_PASSWORD=${ODYSSEUS_ADMIN_PASSWORD:-}
- ALLOWED_ORIGINS=${ALLOWED_ORIGINS:-http://localhost,http://127.0.0.1}
- SECURE_COOKIES=${SECURE_COOKIES:-}
- SECURE_COOKIES=${SECURE_COOKIES:-false}
- EMBEDDING_URL=${EMBEDDING_URL:-}
- EMBEDDING_MODEL=${EMBEDDING_MODEL:-}
- EMBEDDING_API_KEY=${EMBEDDING_API_KEY:-}
@@ -75,11 +74,6 @@ services:
- GOOGLE_OAUTH_CLIENT_ID=${GOOGLE_OAUTH_CLIENT_ID:-}
- GOOGLE_OAUTH_CLIENT_SECRET=${GOOGLE_OAUTH_CLIENT_SECRET:-}
- GOOGLE_OAUTH_REDIRECT_URI=${GOOGLE_OAUTH_REDIRECT_URI:-}
# Externally reachable origin for MCP OAuth callbacks. The container
# always listens on 7000 and cannot see the host port map above, so
# remote MCP OAuth needs this set whenever the browser reaches
# Odysseus on anything other than http://localhost:7000.
- OAUTH_REDIRECT_BASE_URL=${OAUTH_REDIRECT_BASE_URL:-}
- TAVILY_API_KEY=${TAVILY_API_KEY:-}
- SERPER_API_KEY=${SERPER_API_KEY:-}
# PUID / PGID — the user/group the container drops to before
@@ -135,17 +129,12 @@ services:
fi
sed "s|__SEARXNG_SECRET__|$$secret|g" /tmp/searxng-settings.yml.template > /etc/searxng/settings.yml
fi
# Advisory: a settings file the migration cannot parse or rewrite must
# not be what stops searxng from booting. It explains itself on stderr
# and we carry on, letting searxng report anything genuinely wrong.
/usr/local/searxng/.venv/bin/python /tmp/migrate-searxng-settings.py /etc/searxng/settings.yml || true
exec /usr/local/searxng/entrypoint.sh
ports:
- "127.0.0.1:8080:8080"
volumes:
- searxng-data:/etc/searxng
- ./config/searxng/settings.yml:/tmp/searxng-settings.yml.template:ro,z
- ./scripts/migrate_searxng_settings.py:/tmp/migrate-searxng-settings.py:ro,z
environment:
- SEARXNG_BASE_URL=http://localhost:8080/
- SEARXNG_SECRET=${SEARXNG_SECRET:-}
+1 -12
View File
@@ -45,11 +45,10 @@ services:
- DATABASE_URL=${DATABASE_URL:-sqlite:///./data/app.db}
- AUTH_ENABLED=${AUTH_ENABLED:-true}
- LOCALHOST_BYPASS=${LOCALHOST_BYPASS:-false}
- COMPANION_BASE_URL=${COMPANION_BASE_URL:-}
- ODYSSEUS_ADMIN_USER=${ODYSSEUS_ADMIN_USER:-admin}
- ODYSSEUS_ADMIN_PASSWORD=${ODYSSEUS_ADMIN_PASSWORD:-}
- ALLOWED_ORIGINS=${ALLOWED_ORIGINS:-http://localhost,http://127.0.0.1}
- SECURE_COOKIES=${SECURE_COOKIES:-}
- SECURE_COOKIES=${SECURE_COOKIES:-false}
- EMBEDDING_URL=${EMBEDDING_URL:-}
- EMBEDDING_MODEL=${EMBEDDING_MODEL:-}
- EMBEDDING_API_KEY=${EMBEDDING_API_KEY:-}
@@ -74,11 +73,6 @@ services:
- GOOGLE_OAUTH_CLIENT_ID=${GOOGLE_OAUTH_CLIENT_ID:-}
- GOOGLE_OAUTH_CLIENT_SECRET=${GOOGLE_OAUTH_CLIENT_SECRET:-}
- GOOGLE_OAUTH_REDIRECT_URI=${GOOGLE_OAUTH_REDIRECT_URI:-}
# Externally reachable origin for MCP OAuth callbacks. The container
# always listens on 7000 and cannot see the host port map above, so
# remote MCP OAuth needs this set whenever the browser reaches
# Odysseus on anything other than http://localhost:7000.
- OAUTH_REDIRECT_BASE_URL=${OAUTH_REDIRECT_BASE_URL:-}
- TAVILY_API_KEY=${TAVILY_API_KEY:-}
- SERPER_API_KEY=${SERPER_API_KEY:-}
# PUID / PGID — the user/group the container drops to before
@@ -138,17 +132,12 @@ services:
fi
sed "s|__SEARXNG_SECRET__|$$secret|g" /tmp/searxng-settings.yml.template > /etc/searxng/settings.yml
fi
# Advisory: a settings file the migration cannot parse or rewrite must
# not be what stops searxng from booting. It explains itself on stderr
# and we carry on, letting searxng report anything genuinely wrong.
/usr/local/searxng/.venv/bin/python /tmp/migrate-searxng-settings.py /etc/searxng/settings.yml || true
exec /usr/local/searxng/entrypoint.sh
ports:
- "127.0.0.1:8080:8080"
volumes:
- searxng-data:/etc/searxng
- ./config/searxng/settings.yml:/tmp/searxng-settings.yml.template:ro,z
- ./scripts/migrate_searxng_settings.py:/tmp/migrate-searxng-settings.py:ro,z
environment:
- SEARXNG_BASE_URL=http://localhost:8080/
- SEARXNG_SECRET=${SEARXNG_SECRET:-}
+1 -12
View File
@@ -34,11 +34,10 @@ services:
- DATABASE_URL=${DATABASE_URL:-sqlite:///./data/app.db}
- AUTH_ENABLED=${AUTH_ENABLED:-true}
- LOCALHOST_BYPASS=${LOCALHOST_BYPASS:-false}
- COMPANION_BASE_URL=${COMPANION_BASE_URL:-}
- ODYSSEUS_ADMIN_USER=${ODYSSEUS_ADMIN_USER:-admin}
- ODYSSEUS_ADMIN_PASSWORD=${ODYSSEUS_ADMIN_PASSWORD:-}
- ALLOWED_ORIGINS=${ALLOWED_ORIGINS:-http://localhost,http://127.0.0.1}
- SECURE_COOKIES=${SECURE_COOKIES:-}
- SECURE_COOKIES=${SECURE_COOKIES:-false}
- EMBEDDING_URL=${EMBEDDING_URL:-}
- EMBEDDING_MODEL=${EMBEDDING_MODEL:-}
- EMBEDDING_API_KEY=${EMBEDDING_API_KEY:-}
@@ -63,11 +62,6 @@ services:
- GOOGLE_OAUTH_CLIENT_ID=${GOOGLE_OAUTH_CLIENT_ID:-}
- GOOGLE_OAUTH_CLIENT_SECRET=${GOOGLE_OAUTH_CLIENT_SECRET:-}
- GOOGLE_OAUTH_REDIRECT_URI=${GOOGLE_OAUTH_REDIRECT_URI:-}
# Externally reachable origin for MCP OAuth callbacks. The container
# always listens on 7000 and cannot see the host port map above, so
# remote MCP OAuth needs this set whenever the browser reaches
# Odysseus on anything other than http://localhost:7000.
- OAUTH_REDIRECT_BASE_URL=${OAUTH_REDIRECT_BASE_URL:-}
- TAVILY_API_KEY=${TAVILY_API_KEY:-}
- SERPER_API_KEY=${SERPER_API_KEY:-}
# PUID / PGID — the user/group the container drops to before
@@ -116,17 +110,12 @@ services:
fi
sed "s|__SEARXNG_SECRET__|$$secret|g" /tmp/searxng-settings.yml.template > /etc/searxng/settings.yml
fi
# Advisory: a settings file the migration cannot parse or rewrite must
# not be what stops searxng from booting. It explains itself on stderr
# and we carry on, letting searxng report anything genuinely wrong.
/usr/local/searxng/.venv/bin/python /tmp/migrate-searxng-settings.py /etc/searxng/settings.yml || true
exec /usr/local/searxng/entrypoint.sh
ports:
- "127.0.0.1:8080:8080"
volumes:
- searxng-data:/etc/searxng
- ./config/searxng/settings.yml:/tmp/searxng-settings.yml.template:ro,z
- ./scripts/migrate_searxng_settings.py:/tmp/migrate-searxng-settings.py:ro,z
environment:
- SEARXNG_BASE_URL=http://localhost:8080/
- SEARXNG_SECRET=${SEARXNG_SECRET:-}
+3 -173
View File
@@ -441,19 +441,10 @@ A grab-bag of small gotchas that otherwise turn into long debugging sessions.
| Package | Feature unlocked |
|---------|-----------------|
| `faster-whisper` | Local speech-to-text (microphone -> text) via the "local" STT provider. |
| `kokoro`, `soundfile` | Local Kokoro-82M text-to-speech on a CUDA GPU. The pinned Kokoro release supports Odysseus installs on Python 3.11-3.12; these packages are intentionally skipped on Python 3.13+ (including the Python 3.14 container image). |
| `ddgs` | DuckDuckGo as a search provider option. |
| `PyMuPDF` | PDF page rendering in the side viewer panel and form-filling. (Note: AGPL-3.0) |
| `markitdown` | Office/EPUB document text extraction (converts .docx/.xlsx/.pptx/.xls/.epub to Markdown). |
Install the optional set only when you need these features:
```bash
pip install -r requirements-optional.txt
```
The default Docker image currently uses Python 3.14, while Kokoro 0.9.4 declares Python `>=3.10,<3.13`. Odysseus itself continues to support Python 3.11+, but this pinned optional local-TTS feature requires a native Python 3.11 or 3.12 environment. Kokoro declares `torch`, but the local provider only activates when that torch build has CUDA and a GPU is visible; install the CUDA build appropriate for your host. Browser and configured endpoint TTS remain available on Python 3.13+ and in the container image.
### Faster, reproducible installs with uv (optional)
[uv](https://docs.astral.sh/uv/) works as a drop-in replacement for the
venv + pip steps in the native install guides, no project changes are needed but this change results in faster installs along with a lockfile for reproducible environments. After [installing `uv`](https://docs.astral.sh/uv/getting-started/installation/), use:
@@ -484,7 +475,7 @@ Odysseus is a self-hosted workspace with powerful local tools: shell access, fil
- Keep `AUTH_ENABLED=true` for any network-accessible deployment.
- Keep `LOCALHOST_BYPASS=false` outside local development.
- Leave `SECURE_COOKIES` unset unless you need to override it: session cookies are marked `Secure` whenever the request arrives over HTTPS. Use `SECURE_COOKIES=true` to force it on for a proxy whose scheme Odysseus cannot see, or `SECURE_COOKIES=false` to force it off while you still serve plain HTTP alongside HTTPS.
- Use `SECURE_COOKIES=true` when Odysseus is served through HTTPS by a trusted reverse proxy or private access gateway.
- Do not expose it directly to the public internet without HTTPS and a trusted reverse proxy or private access layer.
- Keep `.env`, `data/`, `logs/`, databases, uploads, generated media, backups, auth/session files, API keys, and model/provider tokens out of Git and private shares. They are ignored by default.
- Review `data/auth.json` after first boot: disable open signup unless you intentionally want it, make only your own account admin, and keep demo/test accounts non-admin.
@@ -495,14 +486,6 @@ Odysseus is a self-hosted workspace with powerful local tools: shell access, fil
- Keep ChromaDB, SearXNG, ntfy, Ollama, vLLM, llama.cpp, databases, and raw model/provider APIs internal-only. Expose only the authenticated Odysseus web/API entrypoint through your trusted proxy or private access layer.
- Before publishing a fork, run `git status --short` and confirm no private files from `.env`, `data/`, `logs/`, uploads, backups, or local databases are staged.
> **Upgrading an existing install:** `SECURE_COOKIES` used to default to
> `false`, so an install set up before scheme derivation may still carry
> `SECURE_COOKIES=false` in its own `.env`. That explicit value stays
> authoritative, so HTTPS logins keep getting a non-`Secure` session cookie.
> Pulling this change updates the tracked Compose files, but nothing rewrites
> your `.env` — drop the line from it unless you deliberately serve plain HTTP
> alongside HTTPS and want the escape hatch.
### Private or proxied deployments
Odysseus serves plain HTTP on its app port. Docker Compose binds Odysseus and the bundled services to `127.0.0.1` by default, so a typical production/private setup is:
@@ -511,162 +494,9 @@ Odysseus serves plain HTTP on its app port. Docker Compose binds Odysseus and th
3. Put the authenticated Odysseus web/API entrypoint behind that layer.
4. Keep raw service and model ports internal-only.
Cloudflare Access, Tailscale, Caddy, nginx, and Traefik can all fit this pattern; none are required by Odysseus. If your access layer reaches Odysseus on the same host, proxy to `http://127.0.0.1:7000` and keep `AUTH_ENABLED=true` and `LOCALHOST_BYPASS=false`. Any proxy that forwards `X-Forwarded-Proto: https` gets `Secure` session cookies without configuration, so `SECURE_COOKIES` only needs setting when you want to override that — force it on for a proxy that forwards no scheme at all, or off while you still serve plain HTTP.
Cloudflare Access, Tailscale, Caddy, nginx, and Traefik can all fit this pattern; none are required by Odysseus. If your access layer reaches Odysseus on the same host, proxy to `http://127.0.0.1:7000` and keep `AUTH_ENABLED=true`, `LOCALHOST_BYPASS=false`, and `SECURE_COOKIES=true`.
`ALLOWED_ORIGINS` lists exact permitted origins for cross-origin browser/API clients; ordinary same-origin reverse-proxy access usually does not need a special CORS entry.
#### Faster over the network: HTTP/2
The frontend is raw ES modules with no bundler, so a page load is a few hundred
small same-origin requests. Over HTTP/1.1 browsers typically allow only a small
number of concurrent connections per host (commonly around six), so many of
those requests are serialized across multiple round trips. On localhost that
costs almost nothing. Over a LAN, VPN, or remote link it can become a major
part of load time, especially as latency increases.
HTTP/2 multiplexes them onto one connection and the serialisation disappears.
Odysseus needs no changes for this — uvicorn keeps speaking HTTP/1.1 on
loopback and the proxy speaks HTTP/2 to the browser. Mainstream browsers
negotiate HTTP/2 for normal web pages over TLS; they do not use the cleartext
h2c mode here, so browser-facing HTTP/2 requires a certificate. The
`--ssl-certfile` route in *HTTPS + LAN/Tailscale exposure* above gives you
HTTPS but not HTTP/2 — uvicorn does not speak it.
**1. Install Caddy.** See the [install docs](https://caddyserver.com/docs/install)
for your platform; on macOS, `brew install caddy`.
**2. Write a `Caddyfile`.** Pick the block that matches how you reach the
machine. Replace `7000` if Odysseus listens elsewhere — the macOS start script
uses `7860`.
Public domain, Caddy obtains and renews the certificate itself:
```
odysseus.example.com {
reverse_proxy 127.0.0.1:7000
}
```
Tailscale, no public DNS needed — `tailscale cert` issues a browser-trusted
certificate for a tailnet name and writes `<domain>.crt` and `<domain>.key`:
```bash
tailscale cert myhost.tailnet-name.ts.net
```
```
myhost.tailnet-name.ts.net {
tls /path/to/myhost.tailnet-name.ts.net.crt /path/to/myhost.tailnet-name.ts.net.key
reverse_proxy 127.0.0.1:7000
}
```
LAN with your own certificate — same shape, your own files:
```
odysseus.lan {
tls /path/to/cert.pem /path/to/key.pem
reverse_proxy 127.0.0.1:7000
}
```
Give `tls` absolute paths: a service starts in a working directory you did not
choose. If port 443 is already taken, append a port to the site address
(`odysseus.example.com:8443`) and use it in the URL. That alone does not free
port 80 — Caddy still binds it for the HTTP-to-HTTPS redirect, and fails to
start with `listen tcp :80: bind: address already in use` if something else
holds it. Turn the redirect off with a global block at the top of the file:
```
{
auto_https disable_redirects
}
```
**3. Run it in the foreground first:**
```bash
caddy run --config ./Caddyfile
```
Once that works, run it as a service:
```bash
brew services start caddy # macOS — reads $(brew --prefix)/etc/Caddyfile, not ./Caddyfile
sudo systemctl enable --now caddy # Linux, if your package installed the unit
```
Odysseus's own service is unchanged; the proxy runs alongside it. Under Docker,
run the proxy as another container, or on the host pointing at the published
port.
**4. Point Odysseus at the new origin** in `.env`, then restart it.
A proxy that exposes the HTTPS request scheme to Odysseus needs no `SECURE_COOKIES` setting. Only force it on when the proxy cannot expose that scheme:
```bash
# only if the proxy cannot expose the external HTTPS scheme to Odysseus:
SECURE_COOKIES=true
# only if you use remote MCP servers with OAuth:
OAUTH_REDIRECT_BASE_URL=https://odysseus.example.com
```
Gmail OAuth needs nothing here when the proxy runs on the same host: the
redirect URI is built from the incoming request, and uvicorn rewrites the
scheme from `X-Forwarded-Proto` for proxies it trusts — by default only
`127.0.0.1`. A proxy in a separate container or on another machine is not
trusted, so pin the URI there:
```bash
GOOGLE_OAUTH_REDIRECT_URI=https://odysseus.example.com/api/email/oauth/google/callback
```
(uvicorn's own `FORWARDED_ALLOW_IPS` widens that trust, but it has to be in the
environment uvicorn starts with — `.env` is read by the app afterwards, too
late for it to take effect.)
**5. Confirm HTTP/2 is really on:**
```bash
curl -s -o /dev/null -w '%{http_version}\n' https://odysseus.example.com/
# 2
```
The status code is not the thing to check here — a logged-out request redirects
to the login page, so `curl -I` shows `HTTP/2 302`, and the `HTTP/2` prefix is
the part that matters. The browser reports the same in the Network panel's
Protocol column (`h2`); in Chrome and Firefox that column is hidden until you
enable it by right-clicking the column headers.
Three things bite when moving an existing install behind TLS:
- Leave `SECURE_COOKIES` unset when Odysseus can see the external HTTPS scheme;
the cookie then follows the request automatically. If your proxy cannot expose
that scheme, set `SECURE_COOKIES=true` **at the same time** you stop serving
plain HTTP, not before. An explicit `true` applies to every login, so while an
HTTP entrypoint is still reachable the browser will reject the `Secure` cookie
there and login will appear to loop.
- `OAUTH_REDIRECT_BASE_URL` defaults to `http://localhost:7000`. Unlike the
Gmail redirect URI it cannot be derived from a request — it is registered
with each MCP authorization server up front — so set it to the external
origin if you use remote MCP servers over OAuth.
- Odysseus sends `Strict-Transport-Security` once it sees `X-Forwarded-Proto:
https`. HSTS applies to the whole hostname and ignores the port, so any other
plain-HTTP service on that same hostname becomes unreachable in browsers that
have visited Odysseus. Give Odysseus its own hostname, or strip the header at
the proxy (`header_down -Strict-Transport-Security` in Caddy).
Server-sent events are not buffered by this configuration, so chat streaming
arrives token by token; add `flush_interval -1` inside the `reverse_proxy`
block if you want that pinned explicitly. nginx needs `proxy_buffering off;`
for the same reason.
Changing the external origin also affects state scoped to it. Service workers
and their caches are origin-scoped, so moving to a different origin starts with
a cold load. Cookies follow their own domain/path/security rules rather than
being port-scoped: changing the hostname normally requires a new login, while
changing only the scheme or port does not by itself guarantee that existing
cookies disappear.
Common internal-only ports from the default docs/compose setup:
| Port | Service |
@@ -697,7 +527,7 @@ Key settings:
| `AUTH_ENABLED` | `true` | Enable/disable login |
| `LOCALHOST_BYPASS` | `false` | Development-only auth bypass for loopback requests. Keep false for shared/network deployments. |
| `ALLOWED_ORIGINS` | `http://localhost,http://127.0.0.1` | Comma-separated exact permitted origins for cross-origin browser/API clients. |
| `SECURE_COOKIES` | derived from the request scheme | Marks session cookies `Secure` on HTTPS requests. Set true to force it on, false to force it off. |
| `SECURE_COOKIES` | `false` | Set true when serving Odysseus through HTTPS at a trusted proxy or private access gateway. |
| `DATABASE_URL` | `sqlite:///./data/app.db` | Database connection string |
| `CHROMADB_HOST` | `localhost` | ChromaDB host for vector memory. Docker overrides this to `chromadb`. |
| `CHROMADB_PORT` | `8100` | ChromaDB port for manual host runs. Docker overrides this to `8000`. |
-4
View File
@@ -163,10 +163,6 @@ if (Test-Path $cudaBase) {
}
# 7. Start the server (use `python -m uvicorn` - bare `uvicorn` may not be on PATH)
# -Port only reaches uvicorn as a flag. Everything that builds a URL for this
# instance - internal_api_base(), companion pairing, the MCP OAuth callback -
# reads APP_PORT, so set it too or they all assume 7000.
$env:APP_PORT = $Port
Write-Step ("Starting Odysseus at http://{0}:{1}" -f $BindHost, $Port)
Write-Host "Press Ctrl+C to stop."
Write-Host ""
-21
View File
@@ -1,21 +0,0 @@
The MIT License (MIT)
Copyright (c) 2013-2020 Khan Academy and other contributors
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
-21
View File
@@ -1,21 +0,0 @@
The MIT License (MIT)
Copyright (c) 2014 - 2022 Knut Sveidqvist
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+8
View File
@@ -1802,6 +1802,7 @@ async def _ai_draft_reply_to_email(uid, folder="INBOX", reply_all=False, account
from src.endpoint_resolver import (
resolve_endpoint,
resolve_utility_fallback_candidates,
resolve_chat_fallback_candidates,
)
from src.llm_core import llm_call_async_with_fallback
except Exception as exc:
@@ -1842,6 +1843,13 @@ async def _ai_draft_reply_to_email(uid, folder="INBOX", reply_all=False, account
utility_fallbacks = resolve_utility_fallback_candidates() or []
for cand in utility_fallbacks:
_add(*cand)
try:
chat_fallbacks = resolve_chat_fallback_candidates(owner=None) or []
except TypeError:
chat_fallbacks = resolve_chat_fallback_candidates() or []
for cand in chat_fallbacks:
_add(*cand)
if not candidates:
return {"error": "No LLM endpoint configured for AI reply"}
-10
View File
@@ -12,16 +12,6 @@
# GPU-accelerated transcription — it's auto-detected, CPU is used otherwise.
faster-whisper
# Local text-to-speech via Kokoro-82M for the "local" TTS provider.
# Kokoro 0.9.4 declares Python >=3.10,<3.13; Odysseus itself requires 3.11+,
# so pip installs these extras on 3.11-3.12 and deliberately skips them on
# Python 3.13+ (including the Python 3.14 container image). Kokoro declares
# torch; the local provider still
# requires a CUDA-enabled torch build and GPU at runtime. SoundFile is separate
# in Kokoro's official install instructions and is not a transitive dependency.
kokoro==0.9.4; python_version >= "3.11" and python_version < "3.13"
soundfile; python_version >= "3.11" and python_version < "3.13"
# DuckDuckGo as a search provider option.
# Install if you want DDG in the search-provider dropdown.
# Alternatives: SearXNG, Brave, Tavily, Serper, Google PSE.
+3 -4
View File
@@ -16,7 +16,7 @@ from pydantic import BaseModel
from core.database import SessionLocal, CrewMember, ScheduledTask
from src.auth_helpers import get_current_user
from src.owner_identity import REQUEST_SENTINEL_OWNERS
from core.auth import RESERVED_USERNAMES
from src.task_scheduler import compute_next_run
@@ -90,12 +90,11 @@ def setup_assistant_routes(task_scheduler) -> APIRouter:
# check-in tasks seeded. Hitting any /assistant route under one of these
# used to seed a full CrewMember + Morning/Midday/Evening tasks under that
# owner, which then double-fired alongside the real user's check-ins.
# REQUEST_SENTINEL_OWNERS covers request-only identities; Default/Local is a
# reserved login name but remains a valid storage owner.
# RESERVED_USERNAMES covers the same set; the `not owner` guard handles "".
async def _get_or_create(owner: str) -> CrewMember:
"""Return the per-owner assistant CrewMember, creating it on demand."""
if not owner or owner in REQUEST_SENTINEL_OWNERS:
if not owner or owner in RESERVED_USERNAMES:
raise HTTPException(status_code=400, detail=f"Cannot seed assistant for {owner!r}")
db = SessionLocal()
try:
+3 -34
View File
@@ -22,8 +22,6 @@ from src.settings import (
load_features as _load_features,
save_features as _save_features,
DEFAULT_SETTINGS,
RETIRED_SETTING_KEYS,
without_retired_settings,
)
from src.integrations import (
load_integrations,
@@ -86,33 +84,6 @@ class SetOpenRegistrationRequest(BaseModel):
SESSION_COOKIE = "odysseus_session"
def _secure_cookie(request: Request) -> bool:
"""Decide the ``Secure`` attribute of the session cookie.
``SECURE_COOKIES`` stays authoritative when it holds an explicit value:
``true`` always marks the cookie Secure (the documented knob for a TLS
proxy), ``false`` never does, which is the escape hatch for an install
that still answers on plain HTTP alongside HTTPS. Anything else
unset, or the present-but-empty value docker-compose injects for a
variable the host has not defined derives it from the request, so an
HTTPS login gets a Secure cookie without any configuration.
Either the connection scheme or ``X-Forwarded-Proto`` saying https is
enough, which is the same test ``core/middleware.py`` applies before it
sends HSTS. Uvicorn's proxy-headers middleware already folds that header
into the scheme for the proxies it trusts, so reading it here only adds
the case of a terminator that is not on a trusted address; the cost is
that a client talking to the app directly can set the header and lock
its own session out over plain HTTP.
"""
configured = os.getenv("SECURE_COOKIES", "").strip().lower()
if configured in ("true", "false"):
return configured == "true"
# A chained proxy sends a list — the client-facing hop comes first.
forwarded_proto = request.headers.get("x-forwarded-proto", "").split(",")[0]
return request.url.scheme == "https" or forwarded_proto.strip().lower() == "https"
def setup_auth_routes(auth_manager: AuthManager) -> APIRouter:
router = APIRouter(prefix="/api/auth", tags=["auth"])
@@ -186,7 +157,7 @@ def setup_auth_routes(auth_manager: AuthManager) -> APIRouter:
value=token,
httponly=True,
samesite="lax",
secure=_secure_cookie(request),
secure=os.getenv("SECURE_COOKIES", "false").lower() == "true",
path="/",
)
if body.remember:
@@ -718,7 +689,7 @@ def setup_auth_routes(auth_manager: AuthManager) -> APIRouter:
a scrubbed copy with secret keys blanked. The frontend uses this
for keybinds + TTS prefs, so it stays callable without admin."""
user = _get_current_user(request)
settings = without_retired_settings(_load_settings())
settings = _load_settings()
if user and auth_manager.is_admin(user):
return settings
return scrub_settings(settings)
@@ -738,8 +709,6 @@ def setup_auth_routes(auth_manager: AuthManager) -> APIRouter:
"agent_max_tool_calls": (0, 1000), # 0 = unlimited
}
for key in DEFAULT_SETTINGS:
if key in RETIRED_SETTING_KEYS:
continue
if key not in body:
continue
val = body[key]
@@ -752,7 +721,7 @@ def setup_auth_routes(auth_manager: AuthManager) -> APIRouter:
val = max(lo, min(val, hi))
current[key] = val
_save_settings(current)
return without_retired_settings(current)
return current
# ---- Integrations CRUD ----
+100 -47
View File
@@ -15,7 +15,7 @@ from core.database import Session as DBSession, ModelEndpoint
from src.llm_core import normalize_model_id
from src.endpoint_resolver import normalize_base
from src.context_compactor import maybe_compact, trim_for_context
from src.model_context import estimate_tokens, get_context_length
from src.model_context import estimate_tokens
from src.auth_helpers import effective_user
from src.prompt_security import untrusted_context_message
from src.attachment_refs import attachment_ref
@@ -152,38 +152,10 @@ class ChatContext:
# Uploads attached to this user turn, resolved and owner-checked for the
# agent's private context. This is not emitted to the browser.
uploaded_files: list = field(default_factory=list)
# Route-neutral prompt before any model-window compaction/trimming. This is
# retained only when explicit foreground fallbacks are enabled so each
# concrete candidate can apply its own context budget independently.
route_messages: list = field(default_factory=list)
# ── Helpers ────────────────────────────────────────────────────────────── #
def _allowed_models_from_privileges(privs: dict) -> Optional[frozenset[str]]:
if privs.get("block_all_models"):
return frozenset()
allowed_raw = privs.get("allowed_models")
allowed = allowed_raw if isinstance(allowed_raw, list) else []
restricted = bool(privs.get("allowed_models_restricted")) or bool(allowed)
return frozenset(model for model in allowed if isinstance(model, str)) if restricted else None
def _allowed_models_for_request(request) -> Optional[frozenset[str]]:
"""Return the caller's model allowlist, or ``None`` when unrestricted."""
try:
user = effective_user(request)
except Exception:
user = None
if not user:
return None
auth_manager = getattr(getattr(request.app, "state", None), "auth_manager", None)
if not auth_manager:
return None
privs = auth_manager.get_privileges(user) or {}
return _allowed_models_from_privileges(privs)
def _enforce_chat_privileges(request, sess) -> None:
"""Apply the per-user privilege gates (allowed_models + max_messages_per_day)
that both /api/chat and /api/chat_stream must enforce BEFORE any LLM work.
@@ -213,8 +185,10 @@ def _enforce_chat_privileges(request, sess) -> None:
if privs.get("block_all_models"):
raise HTTPException(403, f"Your account is not allowed to use model '{sess.model}'.")
allowed_models = _allowed_models_from_privileges(privs)
if allowed_models is not None and sess.model and sess.model not in allowed_models:
allowed_raw = privs.get("allowed_models")
allowed = allowed_raw if isinstance(allowed_raw, list) else []
restricted = bool(privs.get("allowed_models_restricted")) or bool(allowed)
if restricted and sess.model and sess.model not in allowed:
raise HTTPException(403, f"Your account is not allowed to use model '{sess.model}'.")
cap = int(privs.get("max_messages_per_day") or 0)
@@ -313,6 +287,96 @@ async def auto_name_session(session_manager, sess):
logger.error(f"Auto-name failed for {sess.id}: {e}\n{traceback.format_exc()}")
def try_fallback_endpoint(sess, session_id: str) -> dict | None:
"""Find an alternative working endpoint when the current one fails.
Returns {"model": ..., "endpoint_url": ..., "endpoint_name": ...} or None.
"""
import requests as _req
from src.endpoint_resolver import (
build_chat_url,
build_headers,
build_models_url,
normalize_base,
resolve_endpoint_runtime,
)
from src.chatgpt_subscription import is_chatgpt_subscription_base
current_url = sess.endpoint_url or ""
owner = getattr(sess, "owner", None)
db = SessionLocal()
try:
q = db.query(ModelEndpoint).filter(
ModelEndpoint.is_enabled == True
)
if owner:
from src.auth_helpers import owner_filter
q = owner_filter(q, ModelEndpoint, owner)
endpoints = q.all()
finally:
db.close()
for ep in endpoints:
base = normalize_base(ep.base_url)
# Skip current endpoint
if current_url and base in current_url:
continue
try:
base, api_key = resolve_endpoint_runtime(ep, owner=owner)
except Exception:
continue
ping_url = build_models_url(base)
headers = build_headers(api_key, base)
try:
if ping_url:
r = _req.get(ping_url, headers=headers, timeout=5)
r.raise_for_status()
data = r.json()
models = [m.get("id") for m in (data.get("data") or []) if m.get("id")]
if not models:
models = [
m.get("name") or m.get("model")
for m in (data.get("models") or [])
if m.get("name") or m.get("model")
]
else:
models = json.loads(ep.cached_models or "[]")
if not models:
continue
# Found a working endpoint — update session
new_model = models[0]
chat_url = build_chat_url(base)
new_headers = build_headers(api_key, base)
persisted_headers = {} if is_chatgpt_subscription_base(base) else new_headers
sess.model = new_model
sess.endpoint_url = chat_url
sess.headers = new_headers
# Persist
_db = SessionLocal()
try:
_db.query(DBSession).filter(DBSession.id == session_id).update({
"model": new_model,
"endpoint_url": chat_url,
"headers": persisted_headers,
})
_db.commit()
finally:
_db.close()
logger.info(f"Fallback: switched session {session_id} from {current_url} to {ep.name} ({new_model})")
return {
"model": new_model,
"endpoint_url": chat_url,
"endpoint_name": ep.name,
}
except Exception:
continue
return None
def extract_preset(chat_handler, preset_id) -> PresetInfo:
"""Extract preset parameters via chat_handler."""
temperature, max_tokens, system_prompt, char_name = (
@@ -623,7 +687,6 @@ async def build_chat_context(
use_enhanced_message: bool = False,
agent_mode: bool = False,
allow_tool_preprocessing: bool = True,
defer_context_shaping: bool = False,
) -> ChatContext:
"""Build the full context (preface + messages) for an LLM call.
@@ -767,22 +830,13 @@ async def build_chat_context(
except Exception:
logger.debug("Failed to add current date/time context", exc_info=True)
route_messages = list(messages)
# Explicit fallback routing must shape from the same route-neutral prompt
# for every candidate. Running selected-model compaction here would mutate
# session history before we know which route can answer and would make a
# later larger-context candidate unable to recover discarded history.
if defer_context_shaping:
context_length = get_context_length(sess.endpoint_url, sess.model)
was_compacted = False
else:
messages, context_length, was_compacted = await maybe_compact(
sess, sess.endpoint_url, sess.model, messages, sess.headers, owner=user,
)
# Auto-compact
messages, context_length, was_compacted = await maybe_compact(
sess, sess.endpoint_url, sess.model, messages, sess.headers, owner=user,
)
_before_trim_messages = len(messages)
_before_trim_tokens = estimate_tokens(messages)
if not defer_context_shaping:
messages = trim_for_context(messages, context_length)
messages = trim_for_context(messages, context_length)
_after_trim_messages = len(messages)
_after_trim_tokens = estimate_tokens(messages)
_context_trimmed = _after_trim_messages < _before_trim_messages or _after_trim_tokens < _before_trim_tokens
@@ -806,7 +860,6 @@ async def build_chat_context(
context_tokens_after_trim=_after_trim_tokens,
auto_opened_docs=auto_opened_docs,
uploaded_files=uploaded_files,
route_messages=route_messages,
)
+40 -664
View File
File diff suppressed because it is too large Load Diff
-35
View File
@@ -1204,41 +1204,6 @@ def _safe_env_prefix(ep: str | None) -> str | None:
return f'[ -f "{path}" ] && source "{path}" || true'
def _local_windows_bash_env_prefix(ep: str | None) -> str | None:
"""Convert a frontend PowerShell venv prefix for the local Git Bash runner."""
if not ep:
return ep
prefix = ep.strip()
if not prefix.startswith("&"):
return ep
raw_path = prefix[1:].lstrip()
if not raw_path:
return ep
if raw_path.startswith("'"):
if len(raw_path) < 2 or not raw_path.endswith("'"):
return ep
quoted_path = raw_path[1:-1]
if "'" in quoted_path.replace("''", ""):
return ep
path = quoted_path.replace("''", "'")
else:
path = raw_path.rstrip()
if "'" in path or '"' in path:
return ep
if any(c in path for c in "\r\n;&|`$<>"):
return ep
if not path.replace("\\", "/").casefold().endswith("/scripts/activate.ps1"):
return ep
bash_path = _git_bash_path(path)
if "\\" in bash_path:
return ep
bash_path = bash_path[: -len("Activate.ps1")] + "activate"
return "source " + shlex.quote(bash_path)
def _ssh_ps(host, script_path, port=None):
"""Build SSH command to run a PowerShell script on a Windows remote."""
pf = f"-p {port} " if port and port != "22" else ""
+3 -3
View File
@@ -50,7 +50,7 @@ from routes.cookbook_helpers import (
_SESSION_ID_RE, _validate_repo_id, _validate_serve_model_id, _validate_include, _validate_token,
_validate_local_dir, _validate_gpus, _shell_path,
_ps_squote, _bash_squote, _validate_serve_cmd, _parse_serve_phase, OLLAMA_MISSING_HINT,
_safe_env_prefix, _local_windows_bash_env_prefix, _local_tooling_path_export, _append_serve_preflight_exit_lines,
_safe_env_prefix, _local_tooling_path_export, _append_serve_preflight_exit_lines,
_append_serve_exit_code_lines, _append_llama_cpp_linux_accel_build_lines, _cached_model_scan_script,
load_stored_hf_token,
_append_vllm_linux_preflight_lines, _ollama_bind_from_cmd, _pip_install_fallback_chain,
@@ -1336,7 +1336,7 @@ def setup_cookbook_routes() -> APIRouter:
# Local: run hf download in the background (tmux on POSIX, a detached
# process + logfile on Windows where tmux doesn't exist).
if req.env_prefix:
lines.append(_safe_env_prefix(_local_windows_bash_env_prefix(req.env_prefix) if local_windows else req.env_prefix))
lines.append(_safe_env_prefix(req.env_prefix))
else:
lines.append("deactivate 2>/dev/null; hash -r")
# Show whether the HF token reached this run (masked) — tells a gated
@@ -2166,7 +2166,7 @@ def setup_cookbook_routes() -> APIRouter:
if req.gpus:
runner_lines.append(f"export CUDA_VISIBLE_DEVICES='{req.gpus}'")
if req.env_prefix:
runner_lines.append(_safe_env_prefix(_local_windows_bash_env_prefix(req.env_prefix) if local_windows else req.env_prefix))
runner_lines.append(_safe_env_prefix(req.env_prefix))
else:
runner_lines.append("deactivate 2>/dev/null; hash -r")
_append_venv_nvidia_library_path_lines(runner_lines, cmd=req.cmd)
+10 -3
View File
@@ -5004,6 +5004,7 @@ def setup_email_routes():
from src.endpoint_resolver import (
resolve_endpoint,
resolve_utility_fallback_candidates,
resolve_chat_fallback_candidates,
)
from src.llm_core import llm_call_async_with_fallback
@@ -5065,6 +5066,8 @@ def setup_email_routes():
pass
for cand in resolve_utility_fallback_candidates(owner=owner) or []:
_add(*cand)
for cand in resolve_chat_fallback_candidates(owner=owner) or []:
_add(*cand)
if not candidates:
return {"success": False, "error": "No LLM endpoint configured"}
@@ -5324,11 +5327,13 @@ def setup_email_routes():
# Build a candidate chain so a stale session-stored API key
# (the most common cause of "authentication failed" here)
# doesn't kill AI Reply outright — fall through to the
# user's Utility / Default endpoints and active Utility fallback
# chain. Dedupe by url+model so we don't retry the same endpoint.
# user's Utility / Default endpoints and the active Utility
# fallback chain. The retired default-fallback hook stays empty.
# Dedupe by url+model so we don't retry the same broken endpoint.
from src.llm_core import llm_call_async_with_fallback
from src.endpoint_resolver import (
resolve_utility_fallback_candidates,
resolve_chat_fallback_candidates,
)
_seen = set()
_candidates = []
@@ -5353,9 +5358,11 @@ def setup_email_routes():
_add(_d_url, _d_model, _d_headers)
except Exception:
pass
# Active Utility fallbacks last.
# Active Utility fallbacks, then the retired default hook.
for cand in resolve_utility_fallback_candidates(owner=owner) or []:
_add(*cand)
for cand in resolve_chat_fallback_candidates(owner=owner) or []:
_add(*cand)
_messages = [
{"role": "system", "content": system_prompt},
{"role": "user", "content": user_msg},
+7 -13
View File
@@ -475,7 +475,7 @@ def setup_mcp_routes(mcp_manager: McpManager):
return RedirectResponse(auth_url)
else:
# Remote device — show paste-back page
return HTMLResponse(_oauth_authorize_page(auth_url, server_id, redirect_uri))
return HTMLResponse(_oauth_authorize_page(auth_url, server_id, host, redirect_uri))
finally:
db.close()
@@ -612,13 +612,15 @@ def setup_mcp_routes(mcp_manager: McpManager):
def _oauth_authorize_page(
auth_url: str,
server_id: str,
redirect_uri: str,
host: str,
redirect_uri: str = "http://localhost:7000/api/mcp/oauth/callback",
) -> str:
"""Page with Google sign-in link and URL paste-back form for remote access."""
# Escape values interpolated into the page: `server_id` comes from the OAuth
# state and is not trusted.
# Escape values interpolated into the page: `host` comes from the request
# Host header and `server_id` from the OAuth state — neither is trusted.
auth_url = html.escape(auth_url, quote=True)
server_id = html.escape(server_id, quote=True)
host = html.escape(host, quote=True)
redirect_uri = html.escape(redirect_uri, quote=True)
return f"""<!DOCTYPE html>
<html><head>
@@ -662,15 +664,7 @@ def _oauth_authorize_page(
</div>
<a class="auth-link" href="{auth_url}" target="_blank" rel="noopener">Sign in with Google</a>
<div class="divider"></div>
<!-- Relative action: the browser resolves it against the origin this page was
served from, so the form follows the user through any proxy without the
app having to know the scheme or the host. An absolute http:// action is
blocked as mixed content on exactly the HTTPS deployments that need
paste-back, and request.url.scheme cannot be trusted to spot them
uvicorn only honours X-Forwarded-Proto from a peer in
--forwarded-allow-ips, which defaults to 127.0.0.1 and excludes a proxy
arriving over the Docker bridge. -->
<form method="POST" action="/api/mcp/oauth/exchange/{server_id}">
<form method="POST" action="http://{host}/api/mcp/oauth/exchange/{server_id}">
<p>Paste the URL from your browser after signing in:</p>
<input type="text" name="callback_url" placeholder="{redirect_uri}?code=..." required>
<br><button type="submit">Connect</button>
+8 -12
View File
@@ -46,7 +46,6 @@ _ENDPOINT_SETTING_FIELDS = {
}
_ENDPOINT_FALLBACK_FIELDS = {
"foreground_model_fallbacks": "Foreground Model Fallbacks",
"utility_model_fallbacks": "Utility Model Fallbacks",
"vision_model_fallbacks": "Vision Model Fallbacks",
}
@@ -181,12 +180,7 @@ def _clear_user_pref_endpoint_refs(all_prefs: dict, ep_id: str) -> int:
if not isinstance(all_prefs, dict):
return 0
users = all_prefs.get("_users")
# A mixed store can contain auth-disabled foreground policy at the root
# alongside named-owner preferences. Both are active namespaces; legacy
# `default_model_fallbacks` remains untouched by the field allowlist.
pref_sets = [all_prefs]
if isinstance(users, dict):
pref_sets.extend(users.values())
pref_sets = users.values() if isinstance(users, dict) else [all_prefs]
cleared_users = 0
for prefs in pref_sets:
if isinstance(prefs, dict) and _clear_endpoint_settings_for_endpoint(prefs, ep_id):
@@ -1351,14 +1345,14 @@ def _legacy_visible_api_models(ep) -> List[str]:
def _picker_models_for_endpoint(ep, base_url: str, kind: str):
"""Return model IDs that should appear in the picker for an endpoint.
API providers expose remote inventory from /v1/models. Default to that
visible inventory until an explicit pinned-model allow-list is saved.
Local/self-hosted endpoints keep the older hide-list behavior.
API providers expose remote inventory from /v1/models. Treat that cache as
inventory, not approval: only manually pinned API models should appear in
the picker. Local/self-hosted endpoints keep the older hide-list behavior.
"""
pinned = _normalize_model_ids(getattr(ep, "pinned_models", None))
if _picker_requires_pinning(base_url, kind):
if not _has_explicit_pinned_models(ep):
pinned = _legacy_visible_api_models(ep)
pinned = _legacy_visible_api_models(ep) if _hidden_model_ids(ep) else []
return pinned, pinned
return _visible_models(
_cached_model_ids(ep),
@@ -2342,7 +2336,9 @@ def setup_model_routes(model_discovery):
else:
response.headers["X-Model-Refresh-Status"] = "failed"
response.headers["X-Model-Refresh-Warning"] = "Model refresh failed or returned no models; kept cached models."
_, pinned = _picker_models_for_endpoint(ep, base, kind)
pinned = _normalize_model_ids(getattr(ep, "pinned_models", None))
if picker_requires_pinning and not _has_explicit_pinned_models(ep):
pinned = _legacy_visible_api_models(ep)
pinned_set = set(pinned)
return [
{
+11 -51
View File
@@ -7,10 +7,6 @@ from src.auth_helpers import get_current_user
from src.constants import USER_PREFS_FILE
PREFS_FILE = USER_PREFS_FILE
_FOREGROUND_POLICY_KEYS = (
"foreground_fallback_enabled",
"foreground_model_fallbacks",
)
def _load():
@@ -30,27 +26,14 @@ def _save(prefs):
def _load_for_user(user: Optional[str] = None) -> dict:
"""Load preferences for a specific user."""
all_prefs = _load()
users = all_prefs.get("_users")
if isinstance(users, dict):
if "_users" in all_prefs:
if user is None:
# Auth disabled — return first user's prefs for backward compat
prefs = dict(next(iter(users.values()), {}))
# Foreground fallback consent is never borrowed from a named
# owner. Auth-disabled operation has a separate flat/root opt-in
# that remains inert when authentication is enabled again.
for key in _FOREGROUND_POLICY_KEYS:
prefs.pop(key, None)
if key in all_prefs:
prefs[key] = all_prefs[key]
return prefs
prefs = users.get(user, {})
return dict(prefs) if isinstance(prefs, dict) else {}
# A legacy flat store belongs only to auth-disabled single-user mode.
# Copying it into the first named user's new `_users` record during an
# auth transition would silently transfer another user's preferences and,
# critically, foreground fallback consent. Named owners therefore start
# with an empty record and must write their own preferences explicitly.
return dict(all_prefs) if user is None else {}
users = all_prefs["_users"]
return dict(next(iter(users.values()), {}))
return dict(all_prefs["_users"].get(user, {}))
# Legacy flat format — return as-is
return dict(all_prefs)
def _save_for_user(user: Optional[str], prefs: dict):
@@ -62,40 +45,17 @@ def _save_for_user(user: Optional[str], prefs: dict):
# `prefs` flat would overwrite the whole `_users` map and destroy every
# other user's preferences. Instead write back into the same (first)
# slot _load_for_user(None) reads from, preserving the others.
users = all_prefs.get("_users")
if isinstance(users, dict):
if "_users" in all_prefs:
users = all_prefs["_users"]
first_key = next(iter(users), None)
if first_key is not None:
existing_named = users.get(first_key)
existing_named = (
dict(existing_named)
if isinstance(existing_named, dict)
else {}
)
named_foreground = {
key: existing_named[key]
for key in _FOREGROUND_POLICY_KEYS
if key in existing_named
}
users[first_key] = {
key: value
for key, value in prefs.items()
if key not in _FOREGROUND_POLICY_KEYS
}
users[first_key].update(named_foreground)
for key in _FOREGROUND_POLICY_KEYS:
if key in prefs:
all_prefs[key] = prefs[key]
users[first_key] = prefs
_save(all_prefs)
return
_save(prefs)
return
if not isinstance(all_prefs.get("_users"), dict):
# Preserve the flat single-user object as inert legacy data while
# creating the first named-owner namespace. In particular, historical
# fallback values must not be deleted or copied into the new owner.
all_prefs = dict(all_prefs)
all_prefs["_users"] = {}
if "_users" not in all_prefs:
all_prefs = {"_users": {}}
all_prefs["_users"][user] = prefs
_save(all_prefs)
+2 -2
View File
@@ -15,7 +15,7 @@ from pydantic import BaseModel, Field
from core.middleware import INTERNAL_TOOL_USER
from src.endpoint_resolver import resolve_endpoint
from src.auth_helpers import _auth_disabled, get_current_user
from src.owner_identity import REQUEST_SENTINEL_OWNERS
from core.auth import RESERVED_USERNAMES
from src.constants import DEEP_RESEARCH_DIR
_SESSION_ID_RE = re.compile(r"^[a-zA-Z0-9-]{1,128}$")
@@ -496,7 +496,7 @@ def setup_research_routes(research_handler, session_manager=None) -> APIRouter:
user = require_privilege(request, "can_use_research")
if user == INTERNAL_TOOL_USER:
tool_owner = (request.headers.get("X-Odysseus-Owner") or "").strip()
if tool_owner and tool_owner not in REQUEST_SENTINEL_OWNERS:
if tool_owner and tool_owner not in RESERVED_USERNAMES:
auth_mgr = getattr(request.app.state, "auth_manager", None)
if auth_mgr is not None and getattr(auth_mgr, "is_configured", False):
try:
+17 -251
View File
@@ -18,7 +18,6 @@ from pydantic import BaseModel, Field
from services.memory.skills import SkillsManager
from src.auth_helpers import get_current_user
from src.prompt_security import untrusted_context_message
from core.middleware import require_admin
logger = logging.getLogger(__name__)
@@ -108,23 +107,6 @@ def _skill_test_task(skill: dict) -> str:
)
def _skill_test_messages(md: str, task: str) -> list[dict]:
"""Keep user-editable skill text out of the trusted system role."""
return [
{
"role": "system",
"content": (
"You are TESTING a skill. Follow the supplied reusable procedure "
"to complete the user's task for real, using available tools step "
"by step. If the skill is wrong, unclear, or references tools that "
"do not exist, do your best; the problems will be reviewed afterward."
),
},
untrusted_context_message("skill under test", md),
{"role": "user", "content": task},
]
async def _eval_skill_run(skill_md: str, task: str, transcript: str,
url: str, model: str, headers: Optional[dict]) -> dict:
"""LLM-as-judge: grade a skill test run from its transcript. Advisory only.
@@ -429,21 +411,7 @@ async def _eval_skill_retrieval_precision(skill_md: str, others: list,
_skill_test_jobs: dict = {}
async def _run_skill_test_job(
key,
name,
md,
task,
url,
model,
headers,
owner,
skills_manager=None,
*,
messages=None,
transcript=None,
exact_approval=None,
):
async def _run_skill_test_job(key, name, md, task, url, model, headers, owner, skills_manager=None):
"""Background coroutine: run the skill in an agent loop, capture a condensed
log + transcript, then have the judge grade it. Writes into _skill_test_jobs."""
import json as _json
@@ -453,7 +421,7 @@ async def _run_skill_test_job(
if job is None:
return
log = job["log"]
transcript = transcript if isinstance(transcript, list) else []
transcript = []
say_buf = []
def _flush_say():
@@ -461,12 +429,18 @@ async def _run_skill_test_job(
log.append({"type": "say", "text": "".join(say_buf)})
say_buf.clear()
messages = list(messages) if isinstance(messages, list) else _skill_test_messages(md, task)
messages = [
{"role": "system", "content":
"You are TESTING a skill. Below is a reusable skill (a procedure). Follow it "
"to complete the user's task for real, using your available tools, step by "
"step. If the skill is wrong, unclear, or references tools that don't exist, "
"do your best — the problems will be reviewed afterward.\n\n=== SKILL ===\n" + md},
{"role": "user", "content": task},
]
try:
async for chunk in stream_agent_loop(
url, model, messages, headers=headers,
temperature=0.3, max_tokens=0, max_rounds=8, owner=owner,
exact_approval=exact_approval,
):
if not chunk.startswith("data: ") or chunk.strip() == "data: [DONE]":
continue
@@ -484,25 +458,8 @@ async def _run_skill_test_job(
elif d.get("type") == "tool_output":
_flush_say()
out = str(d.get("output") or "")[:600]
tool_log = {"type": "tool_output", "output": out}
approval = d.get("ask_user")
if isinstance(approval, dict):
tool_log["ask_user"] = approval
log.append(tool_log)
log.append({"type": "tool_output", "output": out})
transcript.append(f"[output] {out}\n")
if (
isinstance(approval, dict)
and approval.get("kind") == "tool_approval"
and approval.get("approval_id")
):
# Manual skill tests have their own polling UI instead of a
# chat session. Pause the run and retain only server-side
# continuation state until the same owner approves/denies
# this exact sealed action.
job["status"] = "awaiting_approval"
job["approval"] = approval
job["_transcript"] = transcript
return
elif d.get("type") == "agent_step":
_flush_say()
log.append({"type": "agent_step", "round": d.get("round")})
@@ -514,9 +471,6 @@ async def _run_skill_test_job(
_flush_say()
log.append({"type": "error", "error": str(e)})
job.pop("approval", None)
job.pop("_transcript", None)
job.pop("_run", None)
log.append({"type": "evaluating"})
try:
job["verdict"] = await _eval_skill_run(md, task, "".join(transcript), url, model, headers)
@@ -740,8 +694,12 @@ async def _run_skill_test_once(md: str, task: str, url, model, headers, owner) -
import json as _json
from src.agent_loop import stream_agent_loop
transcript = []
approval_required = None
messages = _skill_test_messages(md, task)
messages = [
{"role": "system", "content":
"You are TESTING a skill. Follow this skill's procedure to complete the task "
"for real, using your tools, step by step.\n\n=== SKILL ===\n" + md},
{"role": "user", "content": task},
]
try:
# max_tokens explicitly set: passing 0 lets some upstreams (Ollama,
# OpenAI-compat) generate an empty completion, which manifested as
@@ -761,44 +719,11 @@ async def _run_skill_test_once(md: str, task: str, url, model, headers, owner) -
transcript.append(f"\n[tool {d.get('tool')}] {str(d.get('command') or d.get('args') or '')[:300]}\n")
elif d.get("type") == "tool_output":
transcript.append(f"[output] {str(d.get('output') or '')[:600]}\n")
approval = d.get("ask_user")
if (
isinstance(approval, dict)
and approval.get("kind") == "tool_approval"
):
approval_required = approval
break
elif d.get("type") == "agent_step":
transcript.append(f"\n--- round {d.get('round')} ---\n")
except Exception as e:
transcript.append(f"\n[run error] {e}\n")
text = "".join(transcript)
if approval_required is not None:
# Unattended audits have no authority to approve and no UI that could
# resume this record. Destructively deny it now instead of leaving a
# reusable opaque grant pending until TTL/cap eviction.
try:
from src.tool_approvals import tool_approval_store
tool_approval_store.consume(
approval_required.get("approval_id"),
decision="deny",
owner=owner,
session_id=None,
)
except Exception:
logger.debug("Could not retire unattended skill approval", exc_info=True)
return text, {
"verdict": "inconclusive",
"confidence": 1.0,
"summary": (
"This automated audit reached an exact action that requires "
"a human approval; no action was executed."
),
"issues": [
"Run this skill's manual test and review the sealed action."
],
"approval_required": True,
}
verdict = await _eval_skill_run(md, task, text, url, model, headers)
return text, verdict
@@ -938,26 +863,6 @@ async def _audit_one_skill(skills_manager, skill, url, model, headers,
transcript, verdict = await _run_skill_test_once(md, task, url, model, headers, owner)
v = verdict.get("verdict")
log(f"{name}: verdict = {v} ({verdict.get('summary', '')[:80]})")
if verdict.get("approval_required"):
# An unattended audit is not authority for an action influenced by the
# skill under test. Preserve the skill's current publication/confidence
# state and route the exact action to the manual test UI instead of
# letting a safety pause demote, rewrite, or auto-publish the skill.
skills_manager.set_audit(
name,
"inconclusive",
by_teacher=False,
worker_model=model,
owner=owner,
)
status = skill.get("status") or "draft"
log(f"{name}: {status} unchanged — exact action needs manual approval")
return {
"skill": name,
"result": "approval_required",
"verdict": verdict,
"status": status,
}
if v == "pass":
# Procedure works. If the reviewer still flagged metadata (tags/category/
# when_to_use/description), do ONE fixer pass to correct the frontmatter
@@ -1526,19 +1431,6 @@ def setup_skills_routes(skills_manager: SkillsManager) -> APIRouter:
logger.warning(f"Skill-test model resolve failed: {_e}")
key = (user or "", name)
previous_job = _skill_test_jobs.get(key) or {}
previous_approval = previous_job.get("approval") or {}
if previous_approval.get("approval_id"):
try:
from src.tool_approvals import tool_approval_store
tool_approval_store.consume(
previous_approval["approval_id"],
decision="deny",
owner=user,
session_id=None,
)
except Exception:
logger.debug("Could not retire replaced skill approval", exc_info=True)
_skill_test_jobs[key] = {
"status": "running",
"task": task,
@@ -1547,135 +1439,10 @@ def setup_skills_routes(skills_manager: SkillsManager) -> APIRouter:
"started": _time.time(),
"log": [{"type": "skill_test_start", "task": task, "skill": name, "model": model}],
"verdict": None,
"_run": {
"md": md,
"url": url,
"model": model,
"headers": headers,
"owner": user,
},
}
_asyncio.create_task(_run_skill_test_job(key, name, md, task, url, model, headers, user, skills_manager))
return {"ok": True, "status": "running", "skill": name, "model": model}
@router.post("/{skill_id}/test-approval")
async def approve_skill_test_action(request: Request, skill_id: str):
"""Resume a manual skill test with one exact server-sealed action."""
import asyncio as _asyncio
from src.tool_approvals import tool_approval_store
user = _owner(request)
skills = skills_manager.load(owner=user)
match = next(
(s for s in skills if s.get("name") == skill_id or s.get("id") == skill_id),
None,
)
if not match:
raise HTTPException(404, "Skill not found")
_verify_owner(match, user)
name = match.get("name")
key = (user or "", name)
job = _skill_test_jobs.get(key)
if not job or job.get("status") != "awaiting_approval":
raise HTTPException(409, "This skill test is not awaiting an approval.")
body = await request.json()
if not isinstance(body, dict):
raise HTTPException(400, "Tool approval body must be a JSON object.")
approval_id = str(body.get("approval_id") or "")
decision = str(body.get("decision") or "").strip().lower()
expected = job.get("approval") or {}
if approval_id != str(expected.get("approval_id") or ""):
raise HTTPException(409, "This approval does not match the pending skill test action.")
if decision not in {"approve", "deny"}:
raise HTTPException(400, "Invalid tool approval decision.")
pending = tool_approval_store.peek(approval_id)
normalized_owner = str(user or "").strip().casefold()
if (
pending is None
or pending.owner != normalized_owner
or pending.session_id != ""
):
raise HTTPException(409, "This tool approval is invalid or expired.")
exact_approval = tool_approval_store.consume(
approval_id,
decision=decision,
owner=user,
session_id=None,
)
if decision == "approve" and exact_approval is None:
raise HTTPException(409, "This tool approval could not be consumed.")
job.pop("approval", None)
if decision == "deny":
job.pop("_transcript", None)
job.pop("_run", None)
job["log"].append({
"type": "approval_denied",
"text": "Exact action denied; the skill test stopped without executing it.",
})
job["verdict"] = {
"verdict": "inconclusive",
"confidence": 1.0,
"summary": "The test stopped because its exact action was denied.",
"issues": [],
}
job["status"] = "done"
return {"ok": True, "status": "done", "decision": "deny"}
run = job.get("_run") or {}
transcript = job.pop("_transcript", [])
# stream_agent_loop owns its per-round message list internally. Rebuild
# continuation context from the original untrusted skill plus the
# accumulated transcript so repeated approvals do not lose earlier
# approved results, while keeping every transcript byte tainted.
messages = _skill_test_messages(
run.get("md", ""),
job.get("task", ""),
)
if transcript:
messages.append(untrusted_context_message(
"skill test transcript",
"".join(str(item) for item in transcript),
))
messages.extend([
{
"role": "assistant",
"content": str(expected.get("question") or "Allow this exact action once?"),
},
{
"role": "user",
"content": (
f"Approved the exact {exact_approval.pending.tool_name} "
"action shown above once."
),
},
])
job["status"] = "running"
job["log"].append({
"type": "approval_granted",
"text": (
f"Approved exact {exact_approval.pending.tool_name} action once; "
"resuming test."
),
})
_asyncio.create_task(_run_skill_test_job(
key,
name,
run.get("md", ""),
job.get("task", ""),
run.get("url"),
run.get("model"),
run.get("headers"),
run.get("owner"),
skills_manager,
messages=messages,
transcript=transcript,
exact_approval=exact_approval,
))
return {"ok": True, "status": "running", "decision": "approve"}
@router.get("/{skill_id}/test-status")
async def test_skill_status(request: Request, skill_id: str):
"""Current background-test state for a skill (status / log / verdict)."""
@@ -1692,7 +1459,6 @@ def setup_skills_routes(skills_manager: SkillsManager) -> APIRouter:
"model": job.get("model"),
"log": job.get("log", []),
"verdict": job.get("verdict"),
"approval": job.get("approval"),
}
@router.post("/audit-all")
-5
View File
@@ -1,5 +0,0 @@
"""Task route domain package (slice 2p, #4082/#4071).
Contains task_routes.py, migrated from the flat routes/ directory.
Backward-compat shim at routes/task_routes.py re-exports from here.
"""
File diff suppressed because it is too large Load Diff
+1177 -14
View File
File diff suppressed because it is too large Load Diff
-166
View File
@@ -1,166 +0,0 @@
#!/usr/bin/env python3
"""Make retained SearXNG settings inherit defaults without replacing them."""
from __future__ import annotations
import os
import stat
import sys
import tempfile
from pathlib import Path
import yaml
from yaml.nodes import MappingNode
from yaml.tokens import BlockMappingStartToken, FlowMappingStartToken
_UTF8_BOM = b"\xef\xbb\xbf"
def _parse_root_mapping(text: str) -> tuple[MappingNode | None, dict]:
"""Parse settings with the same safe YAML semantics SearXNG uses."""
try:
loaded = yaml.safe_load(text)
node = yaml.compose(text, Loader=yaml.SafeLoader)
except yaml.YAMLError:
raise ValueError("settings file is not valid single-document YAML") from None
if loaded is None and node is None:
return None, {}
if not isinstance(loaded, dict) or not isinstance(node, MappingNode):
raise ValueError("settings root is not a mapping")
return node, loaded
def _flow_mapping_start(text: str) -> int:
"""Return the root flow mapping's opening-brace character offset."""
try:
for token in yaml.scan(text, Loader=yaml.SafeLoader):
if isinstance(token, FlowMappingStartToken):
return token.start_mark.index
except yaml.YAMLError:
pass
raise ValueError("flow-style settings mapping has no opening brace")
def _newline_for(contents: bytes) -> bytes:
first_lf = contents.find(b"\n")
if first_lf > 0 and contents[first_lf - 1 : first_lf + 1] == b"\r\n":
return b"\r\n"
return b"\n"
def _block_mapping_position(text: str, root: MappingNode | None) -> tuple[int, int]:
"""Return a safe character offset and indent for a root block mapping key."""
if root is None:
return len(text), 0
try:
for token in yaml.scan(text, Loader=yaml.SafeLoader):
if not isinstance(token, BlockMappingStartToken):
continue
line_start = token.start_mark.index - token.start_mark.column
if not text[line_start : token.start_mark.index].strip():
return line_start, token.start_mark.column
return root.end_mark.index, token.start_mark.column
except yaml.YAMLError:
pass
return root.end_mark.index, root.start_mark.column
def _add_block_default_inheritance(
contents: bytes, text: str, root: MappingNode | None
) -> bytes:
newline = _newline_for(contents)
character_offset, indent_width = _block_mapping_position(text, root)
bom_length = len(_UTF8_BOM) if contents.startswith(_UTF8_BOM) else 0
offset = bom_length + len(text[:character_offset].encode("utf-8"))
separator = b""
if offset not in (0, bom_length) and not contents[:offset].endswith((b"\n", b"\r")):
separator = newline
addition = (
separator
+ b" " * indent_width
+ b"use_default_settings: true"
+ newline
)
return contents[:offset] + addition + contents[offset:]
def migrate_settings(path: Path) -> bool:
"""Add the missing inheritance key atomically; return whether the file changed."""
source_stat = path.lstat()
if not stat.S_ISREG(source_stat.st_mode):
raise ValueError(f"settings path is not a regular file: {path}")
contents = path.read_bytes()
if not contents:
return False
text = contents.decode("utf-8-sig")
root, loaded = _parse_root_mapping(text)
if "use_default_settings" in loaded:
return False
if root is not None and root.flow_style:
start = _flow_mapping_start(text)
bom_length = len(_UTF8_BOM) if contents.startswith(_UTF8_BOM) else 0
offset = bom_length + len(text[: start + 1].encode("utf-8"))
separator = b", " if root.value else b""
updated = (
contents[:offset]
+ b"use_default_settings: true"
+ separator
+ contents[offset:]
)
else:
updated = _add_block_default_inheritance(contents, text, root)
fd, temporary_name = tempfile.mkstemp(
prefix=f".{path.name}.odysseus-", dir=path.parent
)
temporary = Path(temporary_name)
try:
# chmod before chown: the Compose cap set is `cap_drop: ALL` plus
# CHOWN/SETGID/SETUID/DAC_OVERRIDE, with no FOWNER. Once the temporary
# file belongs to searxng:searxng — which every retained settings file
# does, because searxng's entrypoint chowns /etc/searxng — root can no
# longer chmod it and the migration dies with EPERM.
os.fchmod(fd, stat.S_IMODE(source_stat.st_mode))
os.fchown(fd, source_stat.st_uid, source_stat.st_gid)
with os.fdopen(fd, "wb") as handle:
fd = -1
handle.write(updated)
handle.flush()
os.fsync(handle.fileno())
os.replace(temporary, path)
directory_fd = os.open(path.parent, os.O_RDONLY | os.O_DIRECTORY)
try:
os.fsync(directory_fd)
finally:
os.close(directory_fd)
finally:
if fd >= 0:
os.close(fd)
temporary.unlink(missing_ok=True)
return True
def main(argv: list[str]) -> int:
if len(argv) > 2:
print(f"usage: {Path(argv[0]).name} [settings.yml]", file=sys.stderr)
return 2
path = Path(argv[1]) if len(argv) == 2 else Path("/etc/searxng/settings.yml")
try:
changed = migrate_settings(path)
except (OSError, UnicodeError, ValueError) as exc:
print(f"SearXNG settings migration failed: {exc}", file=sys.stderr)
return 1
if changed:
print("Added use_default_settings inheritance to retained SearXNG settings")
return 0
if __name__ == "__main__":
raise SystemExit(main(sys.argv))
+3 -11
View File
@@ -2,7 +2,7 @@
"""odysseus-webhook — shell wrapper for scheduled-task webhook tokens.
Tasks in the scheduled-task system can carry a `webhook_token`. Any
HTTP POST to `/api/tasks/<task-id>/webhook/<token>` fires the task. This CLI lists,
HTTP POST to `/api/webhook/<token>` fires the task. This CLI lists,
rotates, and revokes those tokens.
odysseus-webhook list # tasks that have a token
@@ -21,7 +21,6 @@ quiet_logs()
import argparse, json, logging, os, secrets, sys
from pathlib import Path
from urllib.parse import quote
try:
from core.database import SessionLocal, ScheduledTask
@@ -54,14 +53,6 @@ def _summary(t: "ScheduledTask", reveal: bool = False) -> dict:
}
def _task_webhook_url(base: str, task_id: str, token: str) -> str:
"""Build the live task-route URL without leaking ids into path syntax."""
root = (base or "http://localhost:7000").rstrip("/")
task_part = quote(str(task_id), safe="")
token_part = quote(str(token), safe="")
return f"{root}/api/tasks/{task_part}/webhook/{token_part}"
def cmd_list(args):
db = SessionLocal()
try:
@@ -118,7 +109,8 @@ def cmd_url(args):
fail(f"no task with id {args.id!r}")
if not t.webhook_token:
fail(f"task {args.id!r} has no webhook token (rotate one first)")
url = _task_webhook_url(args.base, t.id, t.webhook_token)
base = (args.base or "http://localhost:7000").rstrip("/")
url = f"{base}/api/webhook/{t.webhook_token}"
emit({
"task_id": t.id,
"name": t.name,
+11 -41
View File
@@ -50,46 +50,16 @@ class DocsService:
List of DocChunk objects
"""
results = self.rag.search(query, k=top_k)
chunks = []
for result in results:
if not isinstance(result, dict):
continue
metadata = result.get("metadata")
if not isinstance(metadata, dict):
metadata = {}
text = result.get("document")
if text is None:
text = result.get("text")
if text is None:
text = result.get("content")
if text is None:
text = ""
source = result.get("source")
if source is None:
source = metadata.get("source")
if source is None:
source = "unknown"
score = result.get("similarity")
if score is None:
score = result.get("score")
if score is None:
score = 0.0
chunks.append(
DocChunk(
text=text,
source=source,
score=score,
metadata=metadata,
)
return [
DocChunk(
text=r.get("text", r.get("content", "")),
source=r.get("source", r.get("metadata", {}).get("source", "unknown")),
score=r.get("score", 0.0),
metadata=r.get("metadata"),
)
return chunks
for r in results
if isinstance(r, dict)
]
async def index(self, directory: str) -> IndexResult:
"""
@@ -103,8 +73,8 @@ class DocsService:
"""
result = self.rag.index_personal_documents(directory)
return IndexResult(
indexed=result.get("indexed_count", result.get("indexed", 0)),
failed=result.get("failed_count", result.get("failed", 0)),
indexed=result.get("indexed", 0),
failed=result.get("failed", 0),
errors=result.get("errors", []),
)
+43 -213
View File
@@ -1,18 +1,16 @@
"""Import SKILL.md bundles from public GitHub (or skills.sh → GitHub) URLs."""
from __future__ import annotations
import ipaddress
import logging
import os
import time
import re
from dataclasses import dataclass
from typing import Dict, Iterable, List, Optional, Tuple, cast
from typing import Dict, List, Optional, Tuple
from urllib.parse import quote, urljoin, urlparse
import httpcore
import httpx
from src.url_safety import _default_resolver, check_outbound_url
from src.url_safety import check_outbound_url
logger = logging.getLogger(__name__)
@@ -27,7 +25,6 @@ TEXT_NAMES = {"skill.md", "license", "license.md", "readme.md"}
_GITHUB_HOSTS = frozenset({
"github.com", "www.github.com", "api.github.com", "raw.githubusercontent.com",
})
_SKILLS_SH_HOSTS = frozenset({"skills.sh", "www.skills.sh"})
def _github_host(url: str) -> str:
@@ -75,158 +72,18 @@ def _is_text_file(name: str) -> bool:
_MAX_FETCH_REDIRECTS = 5
def _validated_ips(raw_ips: List[str]) -> List[ipaddress._BaseAddress]:
"""Parse and de-duplicate one resolver snapshot in resolver order."""
ips: List[ipaddress._BaseAddress] = []
seen = set()
for raw in raw_ips:
if not isinstance(raw, str):
continue
try:
ip = ipaddress.ip_address(raw.split("%", 1)[0])
except ValueError:
continue
if ip in seen:
continue
seen.add(ip)
ips.append(ip)
return ips
def _check_fetch_url(url: str) -> None:
"""SSRF guard for skill-import fetches (defense-in-depth).
def _resolve_and_check_url(url: str) -> List[ipaddress._BaseAddress]:
"""Return the exact address snapshot approved for one fetch hop."""
resolved_ips: List[str] = []
def _recording_resolver(host: str) -> List[str]:
answers = list(_default_resolver(host))
resolved_ips[:] = answers
return answers
ok, reason = check_outbound_url(
url,
block_private=True,
resolver=_recording_resolver,
)
Skill bundles only ever come from public GitHub, never an internal
address, so block private/loopback/link-local targets on every hop
matching the hardened web-fetch path in
``services/search/content.py:_get_public_url`` rather than the lenient
default used for admin-configured model endpoints.
"""
ok, reason = check_outbound_url(url, block_private=True)
if not ok:
raise SkillImportError(f"outbound URL blocked: {reason}")
pinned_ips = _validated_ips(resolved_ips)
if not pinned_ips:
raise SkillImportError("outbound URL blocked: host did not resolve to a usable address")
return pinned_ips
# Backward compatibility alias for tests importing _check_fetch_url directly
_check_fetch_url = _resolve_and_check_url
class _PinnedBackend(httpcore.NetworkBackend):
"""Connect only to addresses from one validated DNS snapshot."""
def __init__(self, ips: List[ipaddress._BaseAddress]):
self._ips = [str(ip) for ip in ips]
self._real = httpcore.SyncBackend()
def connect_tcp(
self,
host: str,
port: int,
timeout: float | None = None,
local_address: str | None = None,
socket_options=None,
):
deadline = None if timeout is None else time.monotonic() + timeout
last_exc: Optional[Exception] = None
for ip in self._ips:
remaining = None if deadline is None else max(0.0, deadline - time.monotonic())
try:
return self._real.connect_tcp(
ip,
port,
remaining,
local_address,
socket_options,
)
except (httpcore.ConnectError, httpcore.ConnectTimeout) as exc:
last_exc = exc
if deadline is not None and time.monotonic() >= deadline:
break
if last_exc is not None:
raise last_exc
raise httpcore.ConnectError("no validated address available")
def connect_unix_socket(self, path, timeout=None, socket_options=None):
return self._real.connect_unix_socket(path, timeout, socket_options)
def sleep(self, seconds: float) -> None:
return self._real.sleep(seconds)
_HTTPCORE_TO_HTTPX_EXC = {
httpcore.ConnectError: httpx.ConnectError,
httpcore.ConnectTimeout: httpx.ConnectTimeout,
httpcore.LocalProtocolError: httpx.LocalProtocolError,
httpcore.NetworkError: httpx.NetworkError,
httpcore.PoolTimeout: httpx.PoolTimeout,
httpcore.ProtocolError: httpx.ProtocolError,
httpcore.ProxyError: httpx.ProxyError,
httpcore.ReadError: httpx.ReadError,
httpcore.ReadTimeout: httpx.ReadTimeout,
httpcore.RemoteProtocolError: httpx.RemoteProtocolError,
httpcore.TimeoutException: httpx.TimeoutException,
httpcore.UnsupportedProtocol: httpx.UnsupportedProtocol,
httpcore.WriteError: httpx.WriteError,
httpcore.WriteTimeout: httpx.WriteTimeout,
}
class _PinnedTransport(httpx.BaseTransport):
"""Pin socket connects while preserving URL authority, Host, and TLS SNI."""
def __init__(self, ips: List[ipaddress._BaseAddress]):
self._pinned_ips = list(ips)
self._pool = httpcore.ConnectionPool(
ssl_context=httpx.create_ssl_context(),
http1=True,
http2=False,
network_backend=_PinnedBackend(ips),
)
def handle_request(self, request: httpx.Request) -> httpx.Response:
core_request = httpcore.Request(
method=request.method,
url=httpcore.URL(
scheme=request.url.raw_scheme,
host=request.url.raw_host,
port=request.url.port,
target=request.url.raw_path,
),
headers=request.headers.raw,
content=request.stream,
extensions=request.extensions,
)
core_response = None
try:
core_response = self._pool.handle_request(core_request)
content = b"".join(cast(Iterable[bytes], core_response.stream))
except Exception as exc:
mapped = _HTTPCORE_TO_HTTPX_EXC.get(type(exc))
if mapped is not None:
raise mapped(str(exc)) from exc
raise
finally:
if core_response is not None:
core_response.close()
return httpx.Response(
status_code=core_response.status,
headers=core_response.headers,
content=content,
extensions=core_response.extensions,
)
def close(self) -> None:
self._pool.close()
raise SkillImportError(reason)
def _get_checked(
@@ -243,76 +100,49 @@ def _get_checked(
hand lets us re-validate every hop, closing that blind-SSRF gap.
"""
current = url
for _ in range(_MAX_FETCH_REDIRECTS + 1):
pinned_ips = _resolve_and_check_url(current)
with httpx.Client(
transport=_PinnedTransport(pinned_ips),
follow_redirects=False,
timeout=timeout,
) as client:
with httpx.Client(follow_redirects=False, timeout=timeout) as client:
for _ in range(_MAX_FETCH_REDIRECTS + 1):
_check_fetch_url(current)
r = client.get(current, headers=headers)
if r.status_code in (301, 302, 303, 307, 308):
location = r.headers.get("location")
if not location:
return r
current = urljoin(str(r.url), location)
continue
return r
if r.status_code in (301, 302, 303, 307, 308):
location = r.headers.get("location")
if not location:
return r
current = urljoin(str(r.url), location)
continue
return r
raise SkillImportError("too many redirects while fetching skill bundle")
def parse_skill_source(url: str) -> ResolvedSource:
"""Normalize skills.sh / GitHub web URLs into owner/repo/ref/path."""
url = (url or "").strip()
if not url:
raw = (url or "").strip()
if not raw:
raise SkillImportError("URL is required")
# ``urlparse`` only reports an unambiguous scheme when the URL carries the
# ``scheme://`` form. Opaque schemes (``mailto:``, ``javascript:``) and a
# schemeless ``host:port`` both parse a "scheme" that is not one, so they
# fall through to the host check below and are rejected on the host instead.
scheme = urlparse(url).scheme.lower()
if scheme not in ("http", "https"):
if scheme and url.lower().startswith(f"{scheme}://"):
raise SkillImportError(f"unsupported URL scheme: {scheme}")
# Schemeless "github.com/owner/repo" — accept only a supported host.
rough_host = (urlparse("//" + url).hostname or "").lower()
if rough_host not in _GITHUB_HOSTS and rough_host not in _SKILLS_SH_HOSTS:
raise SkillImportError("Only GitHub or skills.sh URLs are supported")
url = "https://" + url
parsed = urlparse(url)
hostname = (parsed.hostname or "").lower()
if hostname not in _GITHUB_HOSTS and hostname not in _SKILLS_SH_HOSTS:
raise SkillImportError("Only GitHub or skills.sh URLs are supported")
# A skills.sh link is only usable if it redirects to an exact supported
# GitHub host. Scraping the page body for a github.com link cannot work:
# skill pages only ever link the repository root, never the skill's
# subdirectory, so the scrape resolves every skill in a repo to the same
# (wrong) bundle. Fail with an actionable message instead.
if hostname in _SKILLS_SH_HOSTS:
r = _get_checked(url, timeout=20.0)
# skills.sh often links to GitHub; try to unwrap ?url= or redirect target later.
if "skills.sh" in raw and "github.com" not in raw:
r = _get_checked(raw, timeout=20.0)
if r.status_code >= 400:
raise _github_response_error(r)
final = str(r.url)
if _github_host(final) not in _GITHUB_HOSTS:
raise SkillImportError(
"skills.sh did not redirect to GitHub — open the skill's "
"repository on GitHub, navigate to the exact skill folder or "
"SKILL.md file, and paste that URL; the repository-root link "
"alone is not sufficient"
)
url = final
_assert_github_url(final, context="redirect target")
# Page may embed a github link; prefer final URL if redirected.
if "github.com" in final:
raw = final
else:
m = re.search(r"https?://github\.com/[^\s\"')]+", r.text or "")
if m:
raw = m.group(0).rstrip(".,)")
# Update parsed and hostname to reflect the new GitHub URL
parsed = urlparse(url)
hostname = (parsed.hostname or "").lower()
parsed = urlparse(raw)
host = _github_host(raw)
if host not in _GITHUB_HOSTS:
raise SkillImportError(
"Only GitHub URLs are supported (https://github.com/... or raw.githubusercontent.com/...)"
)
_assert_github_url(url)
if hostname == "raw.githubusercontent.com":
if host == "raw.githubusercontent.com":
# /owner/repo/ref/path/to/file
bits = [p for p in parsed.path.split("/") if p]
if len(bits) < 4:
+331 -31
View File
@@ -2,18 +2,22 @@
import copy
import io
import ipaddress
import json
import os
import re
import logging
import socket
import ssl
from datetime import datetime, timedelta
from typing import List
from typing import Iterable, List, cast
from urllib.parse import urljoin, urlparse
import httpx
import httpcore
from bs4 import BeautifulSoup
from src.constants import WEB_FETCH_SOFT_MAX_BYTES, WEB_FETCH_HARD_MAX_BYTES, WEB_FETCH_USER_AGENT
from src import outbound_fetch as _outbound_fetch
from .analytics import RateLimitError, error_logger
from .cache import (
@@ -25,40 +29,336 @@ from .cache import (
logger = logging.getLogger(__name__)
def _is_private_address(addr):
return _outbound_fetch._is_private_address(addr)
_PRIVATE_NETWORKS = (
ipaddress.ip_network("0.0.0.0/8"),
ipaddress.ip_network("10.0.0.0/8"),
ipaddress.ip_network("127.0.0.0/8"),
ipaddress.ip_network("169.254.0.0/16"),
ipaddress.ip_network("172.16.0.0/12"),
ipaddress.ip_network("192.168.0.0/16"),
ipaddress.ip_network("::1/128"),
ipaddress.ip_network("fc00::/7"),
ipaddress.ip_network("fe80::/10"),
)
def _resolve_hostname_ips(hostname):
return _outbound_fetch._resolve_hostname_ips(hostname)
def _public_http_url(url):
return _outbound_fetch._public_http_url(url, resolver=_resolve_hostname_ips)
def _resolve_public_ips(url):
return _outbound_fetch._resolve_public_ips(url, resolver=_resolve_hostname_ips)
_PinnedBackend = _outbound_fetch._PinnedBackend
_PinnedTransport = _outbound_fetch._PinnedTransport
BodyTooLargeError = _outbound_fetch.BodyTooLargeError
_CappedFetch = _outbound_fetch._CappedFetch
def _get_public_url(url, headers, timeout, max_redirects=5, max_bytes=None):
return _outbound_fetch._get_public_url(
url,
headers=headers,
timeout=timeout,
max_redirects=max_redirects,
max_bytes=max_bytes,
resolve_public_ips=_resolve_public_ips,
transport_factory=_PinnedTransport,
def _is_private_address(addr: ipaddress._BaseAddress) -> bool:
if isinstance(addr, ipaddress.IPv6Address) and addr.ipv4_mapped is not None:
addr = addr.ipv4_mapped
return (
addr.is_private
or addr.is_loopback
or addr.is_link_local
or addr.is_reserved
or addr.is_multicast
or addr.is_unspecified
or any(addr in net for net in _PRIVATE_NETWORKS)
)
def _resolve_hostname_ips(hostname: str) -> list[ipaddress._BaseAddress]:
try:
infos = socket.getaddrinfo(hostname, None)
except Exception:
return []
out = []
for info in infos:
try:
out.append(ipaddress.ip_address(info[4][0]))
except Exception:
continue
return out
def _public_http_url(url: str) -> bool:
try:
parsed = urlparse(url)
if parsed.scheme not in ("http", "https"):
return False
host = (parsed.hostname or "").strip()
if not host:
return False
lower = host.lower()
if lower in ("localhost", "metadata", "metadata.google.internal"):
return False
if lower.endswith((".local", ".localhost", ".internal", ".lan", ".intranet")):
return False
try:
return not _is_private_address(ipaddress.ip_address(host))
except ValueError:
pass
addrs = _resolve_hostname_ips(host)
return bool(addrs) and not any(_is_private_address(a) for a in addrs)
except Exception:
return False
def _resolve_public_ips(url: str) -> list[ipaddress._BaseAddress]:
parsed = urlparse(url)
if parsed.scheme not in ("http", "https") or not parsed.hostname:
raise httpx.RequestError(f"Blocked non-public URL: {url}")
host = (parsed.hostname or "").strip().lower()
if host in ("localhost", "metadata", "metadata.google.internal"):
raise httpx.RequestError(f"Blocked non-public hostname: {host}")
try:
ip = ipaddress.ip_address(host)
if _is_private_address(ip):
raise httpx.RequestError(f"Blocked non-public IP literal: {host}")
return [ip]
except httpx.RequestError:
raise
except ValueError:
pass
addrs = _resolve_hostname_ips(host)
if not addrs or any(_is_private_address(a) for a in addrs):
raise httpx.RequestError(f"Blocked non-public URL: {url}")
return addrs
class _PinnedBackend(httpcore.NetworkBackend):
"""Network backend that connects to a pre-resolved IP.
httpcore derives the TLS SNI and the ``Host`` header from the URL's
origin, not from the host argument passed to ``connect_tcp``. So
routing the TCP connect to a resolved IP while leaving the URL
untouched keeps SNI / vhost behaviour correct and closes the
DNS-rebinding TOCTOU between the SSRF check and the connect.
"""
def __init__(self, ip: ipaddress._BaseAddress):
self._ip = str(ip)
self._real = httpcore.SyncBackend()
def connect_tcp(
self,
host: str,
port: int,
timeout: float | None = None,
local_address: str | None = None,
socket_options=None,
):
return self._real.connect_tcp(
self._ip, port, timeout, local_address, socket_options
)
def connect_unix_socket(self, path, timeout=None, socket_options=None):
return self._real.connect_unix_socket(path, timeout, socket_options)
def sleep(self, seconds: float) -> None:
return self._real.sleep(seconds)
# Map httpcore exception classes to their httpx equivalents. Built
# once at import time from the public exception classes; avoids any
# import of httpx's private transport machinery. httpcore's
# ``ConnectionNotAvailable`` is a pool-internal signal (the pool will
# close and retry on its own) — we never expect to see it surface to
# a transport caller, so it has no httpx counterpart here.
_HTTPCORE_TO_HTTPX_EXC = {
httpcore.ConnectError: httpx.ConnectError,
httpcore.ConnectTimeout: httpx.ConnectTimeout,
httpcore.LocalProtocolError: httpx.LocalProtocolError,
httpcore.NetworkError: httpx.NetworkError,
httpcore.PoolTimeout: httpx.PoolTimeout,
httpcore.ProtocolError: httpx.ProtocolError,
httpcore.ProxyError: httpx.ProxyError,
httpcore.ReadError: httpx.ReadError,
httpcore.ReadTimeout: httpx.ReadTimeout,
httpcore.RemoteProtocolError: httpx.RemoteProtocolError,
httpcore.TimeoutException: httpx.TimeoutException,
httpcore.UnsupportedProtocol: httpx.UnsupportedProtocol,
httpcore.WriteError: httpx.WriteError,
httpcore.WriteTimeout: httpx.WriteTimeout,
}
class _PinnedTransport(httpx.BaseTransport):
"""Transport that pins every TCP connect to a pre-resolved IP.
Uses only the public ``httpcore`` and ``httpx`` APIs no
subclassing of ``httpx.HTTPTransport``, no reads of private
``httpcore.ConnectionPool`` attributes, no imports from
``httpx private transport internals``. The URL is passed through unchanged so SNI
/ vhost work as if httpx had been given the hostname directly;
only the TCP destination is pinned, closing the DNS-rebinding
TOCTOU between the SSRF check and the connect.
"""
def __init__(self, ip: ipaddress._BaseAddress, *, http2: bool = False):
self._pool = httpcore.ConnectionPool(
ssl_context=ssl.create_default_context(),
http1=True,
http2=http2,
network_backend=_PinnedBackend(ip),
)
def __enter__(self):
self._pool.__enter__()
return self
def __exit__(self, exc_type=None, exc_value=None, traceback=None) -> None:
self._pool.__exit__(exc_type, exc_value, traceback)
def handle_request(self, request: httpx.Request) -> httpx.Response:
httpcore_req = httpcore.Request(
method=request.method,
url=httpcore.URL(
scheme=request.url.raw_scheme,
host=request.url.raw_host,
port=request.url.port,
target=request.url.raw_path,
),
headers=request.headers.raw,
content=request.stream,
extensions=request.extensions,
)
try:
httpcore_resp = self._pool.handle_request(httpcore_req)
# Eager materialisation matches the original
# ``response.text`` usage in fetch_webpage_content. The
# sync pool's stream is a plain Iterable[bytes] despite
# the httpcore type hint unioning the async variant.
content = b"".join(cast(Iterable[bytes], httpcore_resp.stream))
except Exception as exc:
mapped = _HTTPCORE_TO_HTTPX_EXC.get(type(exc))
if mapped is not None:
raise mapped(str(exc)) from exc
raise
return httpx.Response(
status_code=httpcore_resp.status,
headers=httpcore_resp.headers,
content=content,
extensions=httpcore_resp.extensions,
)
def close(self) -> None:
self._pool.close()
class BodyTooLargeError(Exception):
"""The server declared a body larger than the hard fetch ceiling."""
def __init__(self, url: str, declared_bytes: int):
self.url = url
self.declared_bytes = declared_bytes
super().__init__(
f"response body is {declared_bytes:,} bytes, over the "
f"{WEB_FETCH_HARD_MAX_BYTES:,}-byte hard cap"
)
class _CappedFetch:
"""Result of a size-capped streaming GET.
Carries just what fetch_webpage_content needs from an httpx.Response,
plus the cap bookkeeping: the (possibly truncated) body, whether the
cap cut it short, and the size the server declared via Content-Length
(wire bytes; None when absent).
"""
__slots__ = ("status_code", "headers", "content", "truncated",
"declared_bytes", "encoding", "url")
def __init__(self, status_code, headers, content, truncated,
declared_bytes, encoding, url):
self.status_code = status_code
self.headers = headers
self.content = content
self.truncated = truncated
self.declared_bytes = declared_bytes
self.encoding = encoding
self.url = url
@property
def text(self) -> str:
return self.content.decode(self.encoding or "utf-8", errors="replace")
def raise_for_status(self):
if self.status_code >= 400:
request = httpx.Request("GET", self.url)
raise httpx.HTTPStatusError(
f"HTTP {self.status_code} for {self.url}",
request=request,
response=httpx.Response(self.status_code, request=request),
)
def _get_public_url(url: str, headers: dict, timeout: int, max_redirects: int = 5,
max_bytes: int = None) -> "_CappedFetch":
"""Capped streaming GET with SSRF-guarded, DNS-pinned manual redirects.
Each hop is resolved once, validated as public, and then the actual TCP
connection is pinned to that resolved IP. The request URL is left unchanged
so Host and TLS SNI keep the original hostname.
"""
cap = min(max_bytes or WEB_FETCH_SOFT_MAX_BYTES, WEB_FETCH_HARD_MAX_BYTES)
current = url
for _ in range(max_redirects + 1):
ips = _resolve_public_ips(current)
# Force identity transfer-encoding. With gzip/deflate the wire bytes
# and Content-Length can be a small fraction of the decoded body, so a
# tiny compressed response could pass the hard-cap preflight and then
# expand past the ceiling in one decoded chunk before the streamed cap
# below can slice it.
req_headers = dict(headers or {})
req_headers["Accept-Encoding"] = "identity"
with httpx.Client(
headers=req_headers,
timeout=timeout,
follow_redirects=False,
transport=_PinnedTransport(ips[0]),
) as client:
with client.stream("GET", current) as response:
if response.status_code in (301, 302, 303, 307, 308):
location = response.headers.get("location")
if not location:
return _CappedFetch(response.status_code, response.headers, b"",
False, None, response.encoding, str(response.url))
current = urljoin(str(response.url), location)
continue
# A server can ignore the identity request and still return a
# compressed body; httpx.iter_bytes would then decode it, and a
# tiny gzip can balloon into one decoded chunk far past the cap.
# Refuse compressed Content-Encoding so the streamed cap stays
# a real memory bound.
enc = (response.headers.get("content-encoding") or "").strip().lower()
if enc and enc != "identity":
raise httpx.RequestError(
f"Refusing compressed response (Content-Encoding: {enc}) after "
"requesting identity: cannot bound decoded body size",
request=httpx.Request("GET", current),
)
declared = None
raw_len = response.headers.get("content-length")
if raw_len and raw_len.isdigit():
declared = int(raw_len)
if declared is not None and declared > WEB_FETCH_HARD_MAX_BYTES:
raise BodyTooLargeError(current, declared)
chunks = []
read = 0
truncated = False
for chunk in response.iter_bytes():
read += len(chunk)
if read > cap:
keep = cap - (read - len(chunk))
if keep > 0:
chunks.append(chunk[:keep])
truncated = True
break
chunks.append(chunk)
return _CappedFetch(response.status_code, response.headers,
b"".join(chunks), truncated, declared,
response.encoding, str(response.url))
raise httpx.RequestError("Too many redirects", request=httpx.Request("GET", current))
# PDF extraction (optional dependency)
try:
from pdfminer.high_level import extract_text as pdf_extract_text
+467 -1646
View File
File diff suppressed because it is too large Load Diff
+26 -84
View File
@@ -17,14 +17,13 @@ close / navigation / refresh). It does NOT survive a server restart.
import asyncio
import json
import logging
import uuid
from typing import AsyncGenerator, Dict, Optional
logger = logging.getLogger(__name__)
class _Run:
__slots__ = ("buffer", "subscribers", "status", "task", "evict_task", "run_id")
__slots__ = ("buffer", "subscribers", "status", "task", "evict_task")
def __init__(self) -> None:
self.buffer: list = [] # ordered SSE event strings (replay log)
@@ -32,9 +31,6 @@ class _Run:
self.status: str = "running" # running | done | error | stopped
self.task: Optional[asyncio.Task] = None
self.evict_task: Optional[asyncio.Task] = None
# Stable across every subscription/replay of this exact detached run.
# The browser uses it to make local cost accounting replay-idempotent.
self.run_id: str = uuid.uuid4().hex
_RUNS: Dict[str, _Run] = {}
@@ -57,24 +53,13 @@ def _publish(run: _Run, ev: str) -> None:
pass
def _wake_run_subscribers(run: _Run) -> None:
"""Close subscribers even when the drain task never reached its body."""
for q in list(run.subscribers):
try:
q.put_nowait((None, None))
except Exception:
pass
def _schedule_evict(session_id: str, expected_run: Optional[_Run] = None) -> None:
def _schedule_evict(session_id: str) -> None:
"""(Re)arm a grace-period eviction for a terminal run with no subscribers.
Identity-checked so a run that gets replaced/reused is never evicted by a
stale timer."""
run = _RUNS.get(session_id)
if run is None:
return
if expected_run is not None and run is not expected_run:
return
if run.evict_task and not run.evict_task.done():
run.evict_task.cancel()
@@ -100,38 +85,25 @@ def get_status(session_id: str) -> Optional[str]:
return r.status if r else None
def get_run_id(session_id: str) -> Optional[str]:
"""Return the opaque identity of the current detached run, if present."""
r = _RUNS.get(session_id)
return r.run_id if r else None
def get_active_run(session_id: str) -> Optional[_Run]:
"""Return the exact active run currently registered for a session."""
r = _RUNS.get(session_id)
return r if r and r.status == "running" else None
async def _drain(session_id: str, run: _Run, agen: AsyncGenerator[str, None],
async def _drain(session_id: str, agen: AsyncGenerator[str, None],
prev_task: Optional[asyncio.Task] = None) -> None:
"""Pull every event from the wrapped generator into the run buffer, fanning
each out to live subscribers. Runs to completion regardless of subscribers."""
subscribers_woken = False
def _wake_subscribers() -> None:
nonlocal subscribers_woken
if subscribers_woken:
return
subscribers_woken = True
_wake_run_subscribers(run)
run = _RUNS.get(session_id)
if run is None:
return
# If this run replaced an in-flight one (rapid double-send), wait for that
# one to fully finish first. Its CancelledError handler calls aclose(), which
# persists its partial response — letting it complete before we start writing
# keeps the two runs' session saves sequential instead of interleaved.
try:
if prev_task is not None and not prev_task.done():
if prev_task is not None and not prev_task.done():
try:
await asyncio.wait({prev_task})
except asyncio.CancelledError:
raise # our own cancellation — propagate
except Exception:
pass
try:
async for ev in agen:
_publish(run, ev)
if run.status == "running":
@@ -144,16 +116,6 @@ async def _drain(session_id: str, run: _Run, agen: AsyncGenerator[str, None],
await agen.aclose()
except Exception:
pass
# A rapid third replacement can cancel this task while it is still
# waiting for its predecessor. Close this run's subscribers promptly,
# but keep the task alive until the predecessor finishes so the next
# run still observes the transitive session-save ordering barrier.
_wake_subscribers()
if prev_task is not None and not prev_task.done():
try:
await asyncio.shield(prev_task)
except (asyncio.CancelledError, Exception):
pass
except Exception as e:
logger.error("[agent-run] %s failed: %s", session_id, e, exc_info=True)
run.status = "error"
@@ -165,11 +127,15 @@ async def _drain(session_id: str, run: _Run, agen: AsyncGenerator[str, None],
_publish(run, "data: [DONE]\n\n")
finally:
# Wake every subscriber with the end sentinel so their SSE closes.
_wake_subscribers()
for q in list(run.subscribers):
try:
q.put_nowait((None, None))
except Exception:
pass
# Run is terminal — arm the grace timer so it (and its buffer) is
# eventually freed even if nobody ever reconnects. subscribe() cancels
# this on connect and re-arms on disconnect.
_schedule_evict(session_id, run)
_schedule_evict(session_id)
def start(session_id: str, agen: AsyncGenerator[str, None]) -> _Run:
@@ -179,37 +145,20 @@ def start(session_id: str, agen: AsyncGenerator[str, None]) -> _Run:
prev_task: Optional[asyncio.Task] = None
if prev:
if prev.task and not prev.task.done():
# A task cancelled before its first instruction never enters
# _drain(), so its except/finally blocks cannot update status or
# wake a response already bound to this exact run. Terminalize it
# synchronously before cancelling; _drain's cleanup is idempotent
# when the task had already started.
if prev.status == "running":
prev.status = "stopped"
_wake_run_subscribers(prev)
prev.task.cancel()
prev_task = prev.task # new run awaits this before it starts writing
if prev.evict_task and not prev.evict_task.done():
prev.evict_task.cancel()
run = _Run()
_RUNS[session_id] = run
run.task = asyncio.create_task(_drain(session_id, run, agen, prev_task))
run.task = asyncio.create_task(_drain(session_id, agen, prev_task))
return run
async def subscribe(
session_id: str,
expected_run: Optional[_Run] = None,
) -> AsyncGenerator[str, None]:
async def subscribe(session_id: str) -> AsyncGenerator[str, None]:
"""Replay the run's buffer from the start, then stream live until it ends.
Safe to call repeatedly (reconnect) and from multiple clients at once.
``expected_run`` binds a lazy StreamingResponse body to the same run whose
identity was put in its response headers. Without that binding, a rapid
replacement between response construction and body iteration could replay
the replacement run under the prior run's identity.
"""
run = expected_run or _RUNS.get(session_id)
Safe to call repeatedly (reconnect) and from multiple clients at once."""
run = _RUNS.get(session_id)
if run is None:
return
q: asyncio.Queue = asyncio.Queue()
@@ -252,19 +201,12 @@ async def subscribe(
# Last subscriber gone on a finished run — (re)arm eviction so the
# buffer doesn't linger indefinitely.
if not run.subscribers and run.status != "running":
_schedule_evict(session_id, run)
_schedule_evict(session_id)
def stop(session_id: str, expected_run_id: Optional[str] = None) -> bool:
"""Cancel the matching in-flight run (which saves its partial output).
A stale browser may issue Stop after another tab has replaced the session's
run. Once the caller knows its opaque run identity, fail closed rather than
cancelling that newer run.
"""
def stop(session_id: str) -> bool:
"""Cancel an in-flight run (the wrapped generator saves its partial)."""
run = _RUNS.get(session_id)
if not expected_run_id or run is None or run.run_id != expected_run_id:
return False
if run and run.task and not run.task.done():
run.task.cancel()
return True
+6 -18
View File
@@ -510,12 +510,7 @@ async def do_manage_settings(content: str, owner: Optional[str] = None) -> Dict:
# set/get/list/delete operate on the REAL app settings (the same store
# the Settings panel writes), so changing a model / voice / search
# engine / reminder channel from chat actually takes effect.
from src.settings import (
DEFAULT_SETTINGS,
RETIRED_SETTING_KEYS,
load_settings,
save_settings,
)
from src.settings import load_settings, save_settings, DEFAULT_SETTINGS
# Secrets/credentials the agent must NOT write: kept read-only (masked)
# so API keys never flow through chat. User sets these in the panel.
@@ -567,9 +562,6 @@ async def do_manage_settings(content: str, owner: Optional[str] = None) -> Dict:
return k2
return _ALIASES_SET.get(k2, (k or "").strip())
def _is_managed_key(key):
return key in DEFAULT_SETTINGS and key not in RETIRED_SETTING_KEYS
_ENUMS = {
"image_quality": ["low", "medium", "high"],
"reminder_channel": ["browser", "email", "ntfy", "webhook"],
@@ -632,18 +624,14 @@ async def do_manage_settings(content: str, owner: Optional[str] = None) -> Dict:
if action == "list":
s = load_settings()
shown = {
k: _mask(k, v)
for k, v in s.items()
if _is_managed_key(k) and not isinstance(v, dict)
}
shown = {k: _mask(k, v) for k, v in s.items() if k in DEFAULT_SETTINGS and not isinstance(v, dict)}
return {"response": f"{len(shown)} settings (use get/set with a key)", "settings": shown, "exit_code": 0}
elif action == "get":
key = _resolve(args.get("key", ""))
if not key:
return {"error": "key is required", "exit_code": 1}
if not _is_managed_key(key):
if key not in DEFAULT_SETTINGS:
return {"error": f"Unknown setting '{args.get('key')}'. Use action='list' to see them.", "exit_code": 1}
val = load_settings().get(key, DEFAULT_SETTINGS.get(key))
return {"response": f"{key} = {_mask(key, val)}", "value": _mask(key, val), "exit_code": 0}
@@ -654,11 +642,11 @@ async def do_manage_settings(content: str, owner: Optional[str] = None) -> Dict:
if not raw:
return {"error": "key is required", "exit_code": 1}
key = _resolve(raw)
if not _is_managed_key(key):
if key not in DEFAULT_SETTINGS:
return {"error": f"Unknown setting '{raw}'. Use action='list' to see available settings.", "exit_code": 1}
if _is_secret(key):
return {"response": f"'{key}' is a credential/secret. For security I can't set it from chat. Open Settings and set it there.", "exit_code": 0}
# Structured settings (dicts/lists like keybinds or vision fallbacks)
# Structured settings (dicts/lists like keybinds, default_model_fallbacks)
# have no safe scalar coercion; _coerce would pass a bare string
# straight through and clobber the structure. Refuse them here; they're
# edited in their dedicated panels. (reset/delete still restore the
@@ -687,7 +675,7 @@ async def do_manage_settings(content: str, owner: Optional[str] = None) -> Dict:
elif action == "delete" or action == "reset":
key = _resolve(args.get("key", ""))
if not _is_managed_key(key):
if key not in DEFAULT_SETTINGS:
return {"error": f"Unknown setting '{args.get('key')}'.", "exit_code": 1}
if _is_secret(key):
return {"response": f"'{key}' is a credential. Reset it in the panel.", "exit_code": 0}
-59
View File
@@ -2,7 +2,6 @@ from typing import Any, Dict, List, Optional
import logging
import re
from src.constants import MAX_READ_CHARS
from src.tool_approvals import document_content_digest
from src.tool_utils import _parse_tool_args, get_upload_handler
from src.upload_handler import reserve_upload_references
@@ -81,40 +80,6 @@ def _most_recent_owned_document(db, Document, owner: Optional[str], active_only:
return q.order_by(Document.updated_at.desc()).first()
def _approved_document_version_error(doc: Any, ctx: dict) -> Optional[Dict]:
"""Reject a sealed document action when its target changed meanwhile."""
expected_version = ctx.get("expected_document_version")
expected_digest = (
str(ctx.get("expected_document_digest") or "").strip().lower()
)
if expected_version is None and not expected_digest:
return None
try:
version_unchanged = (
expected_version is None
or int(getattr(doc, "version_count", -1)) == int(expected_version)
)
except (TypeError, ValueError):
version_unchanged = False
content_unchanged = True
if expected_digest:
content_unchanged = (
doc is not None
and document_content_digest(getattr(doc, "current_content", ""))
== expected_digest
)
if version_unchanged and content_unchanged:
return None
return {
"error": (
"The target document changed after this action was proposed. "
"Review the latest version and request the edit again."
),
"exit_code": 1,
"document_changed": True,
}
# ---------------------------------------------------------------------------
# Document tools — create/update/edit/suggest living documents
# ---------------------------------------------------------------------------
@@ -489,12 +454,6 @@ class UpdateDocumentTool:
doc = None
if target_id:
doc = _get_owned_document(db, Document, target_id, owner)
if (
not doc
and target_id
and ctx.get("expected_document_version") is not None
):
return _approved_document_version_error(None, ctx)
if not doc:
doc = _most_recent_owned_document(db, Document, owner)
if doc:
@@ -504,10 +463,6 @@ class UpdateDocumentTool:
if not doc:
return {"error": "No documents exist to update"}
version_error = _approved_document_version_error(doc, ctx)
if version_error:
return version_error
is_email_doc = doc.language == "email" or _looks_like_email_document(doc.current_content or "", doc.title or "")
new_content = _coerce_email_document_content(doc.current_content or "", content) if is_email_doc else content.strip()
if is_email_doc:
@@ -575,12 +530,6 @@ class EditDocumentTool:
doc = None
if target_id:
doc = _get_owned_document(db, Document, target_id, owner)
if (
not doc
and target_id
and ctx.get("expected_document_version") is not None
):
return _approved_document_version_error(None, ctx)
if not doc:
# Fallback: most recently updated document. Avoids "no active doc" errors
# after server restart or when the agent loses track of which doc to edit.
@@ -592,10 +541,6 @@ class EditDocumentTool:
if not doc:
return {"error": "No documents exist to edit"}
version_error = _approved_document_version_error(doc, ctx)
if version_error:
return version_error
is_email_doc = doc.language == "email" or _looks_like_email_document(doc.current_content or "", doc.title or "")
blank_find_edits = [e for e in edits if not (e.get("find") or "").strip()]
if blank_find_edits:
@@ -732,10 +677,6 @@ class SuggestDocumentTool:
if not doc:
return {"error": f"Document {target_id} not found"}
version_error = _approved_document_version_error(doc, ctx)
if version_error:
return version_error
# Validate that FIND text exists in document
valid = []
for s in suggestions:
+2 -8
View File
@@ -64,10 +64,7 @@ async def chat_with_model(content: str, session_id: Optional[str] = None, owner:
return {"model": model, "response": response}
except Exception as e:
logger.error(f"chat_with_model failed: {e}")
return {
"error": f"Failed to get response from {model_spec}: {e}",
"untrusted_content": True,
}
return {"error": f"Failed to get response from {model_spec}: {e}"}
async def ask_teacher(content: str, session_id: Optional[str] = None, owner: Optional[str] = None) -> Dict:
@@ -113,10 +110,7 @@ async def ask_teacher(content: str, session_id: Optional[str] = None, owner: Opt
return {"model": model, "response": response, "teacher": True}
except Exception as e:
logger.error(f"ask_teacher failed: {e}")
return {
"error": f"Teacher call failed ({model_spec}): {e}",
"untrusted_content": True,
}
return {"error": f"Teacher call failed ({model_spec}): {e}"}
async def list_models(content: str, session_id: Optional[str] = None, owner: Optional[str] = None) -> Dict:
+1 -4
View File
@@ -240,10 +240,7 @@ async def send_to_session(content: str, session_id: Optional[str] = None, owner:
}
except Exception as e:
logger.error(f"send_to_session failed: {e}")
return {
"error": f"Failed to send to session: {e}",
"untrusted_content": True,
}
return {"error": f"Failed to send to session: {e}"}
async def manage_session(content: str, session_id: Optional[str] = None, owner: Optional[str] = None) -> Dict:
"""Manage sessions: rename, archive, delete, important, truncate, fork.
+1 -6
View File
@@ -66,7 +66,6 @@ class WebSearchTool:
return {
"error": f"web_search failed: {type(e).__name__}: {str(e) or 'no details'}",
"exit_code": 1,
"untrusted_content": True,
}
if progress_cb:
await progress_cb({
@@ -137,11 +136,7 @@ class WebFetchTool:
if not text:
if err:
return {
"error": f"web_fetch: {url}: {err}",
"exit_code": 1,
"untrusted_content": True,
}
return {"error": f"web_fetch: {url}: {err}", "exit_code": 1}
return {"error": f"web_fetch: {url}: no readable text content (not HTML, or the page needs JS/login)", "exit_code": 1}
# Tell the model when the download budget cut the body short and how
+6 -24
View File
@@ -324,10 +324,7 @@ async def do_pipeline(content: str, session_id: Optional[str] = None, owner: Opt
}
except Exception as e:
logger.error(f"pipeline failed at step {len(step_outputs) + 1}: {e}")
return {
"error": f"Pipeline failed at step {len(step_outputs) + 1}: {e}",
"untrusted_content": True,
}
return {"error": f"Pipeline failed at step {len(step_outputs) + 1}: {e}"}
# ---------------------------------------------------------------------------
@@ -1092,10 +1089,7 @@ async def do_generate_image(content: str, session_id: Optional[str] = None, owne
error_text = err_json.get("error", {}).get("message", error_text) if isinstance(err_json.get("error"), dict) else str(err_json.get("error", error_text))
except Exception:
pass
return {
"error": f"Image generation failed ({resp.status_code}): {error_text}",
"untrusted_content": True,
}
return {"error": f"Image generation failed ({resp.status_code}): {error_text}"}
data = resp.json()
images = data.get("data", [])
@@ -1179,10 +1173,7 @@ async def do_generate_image(content: str, session_id: Optional[str] = None, owne
except httpx.TimeoutException:
return {"error": "Image generation timed out (300s). The model may be overloaded — try again or use quality=low."}
except Exception as e:
return {
"error": f"Image generation error: {str(e)}",
"untrusted_content": True,
}
return {"error": f"Image generation error: {str(e)}"}
async def do_edit_image(
@@ -1319,10 +1310,7 @@ async def do_edit_image(
error_text = err_json.get("detail") or err_json.get("error") or error_text
except Exception:
pass
return {
"error": f"Image edit fallback failed ({fallback_resp.status_code}): {error_text}",
"untrusted_content": True,
}
return {"error": f"Image edit fallback failed ({fallback_resp.status_code}): {error_text}"}
fallback_data = fallback_resp.json()
image_b64 = fallback_data.get("image")
if not image_b64:
@@ -1406,10 +1394,7 @@ async def do_edit_image(
"model for attached-image prompts."
)
}
return {
"error": f"Image edit failed ({resp.status_code}): {error_text}",
"untrusted_content": True,
}
return {"error": f"Image edit failed ({resp.status_code}): {error_text}"}
data = resp.json()
images = data.get("data", [])
@@ -1449,10 +1434,7 @@ async def do_edit_image(
except httpx.TimeoutException:
return {"error": "Image edit timed out. The model may still be loading or overloaded."}
except Exception as e:
return {
"error": f"Image edit error: {str(e)}",
"untrusted_content": True,
}
return {"error": f"Image edit error: {str(e)}"}
# ---------------------------------------------------------------------------
+1 -13
View File
@@ -4,8 +4,6 @@ import os
from typing import Optional
from fastapi import Request, HTTPException
from src.owner_identity import auth_disabled, effective_storage_owner
def get_current_user(request: Request) -> Optional[str]:
"""Get current username from request state (set by auth middleware)."""
@@ -58,17 +56,7 @@ def _auth_disabled() -> bool:
"""True when the operator has explicitly turned off auth via .env.
Mirrors the AUTH_ENABLED parse in app.py / core/middleware.py so the
three call sites agree on what "off" means."""
return auth_disabled()
def storage_owner_for_request(request: Request) -> Optional[str]:
"""Resolve the storage owner for code paths that need an owner bucket.
This does not replace route authentication. It only gives auth-disabled
no-login mode a stable storage identity instead of writing new data as
legacy NULL/ownerless state.
"""
return effective_storage_owner(effective_user(request))
return os.getenv("AUTH_ENABLED", "true").lower() == "false"
def require_user(request: Request) -> str:
+9 -20
View File
@@ -15,7 +15,6 @@ import json
import logging
from src import bg_jobs
from src.prompt_security import untrusted_context_message
logger = logging.getLogger(__name__)
@@ -26,16 +25,6 @@ POLL_INTERVAL_S = 5
_FOLLOWUP_MAX_ROUNDS = 12
def _background_result_message(rec):
inject = (
f"[Background job {rec['id']} finished]\n\n"
f"{bg_jobs.result_text(rec)}\n\n"
"Continue the task using this output. Don't repeat work that's already done. "
"If the task is now complete, give the user the final result."
)
return untrusted_context_message("background job output", inject)
async def _drain_agent(sess, messages):
"""Run the agent loop headless against a session. Returns
(final_prose, tool_events) tool_events in the same shape the live chat
@@ -73,19 +62,13 @@ async def _drain_agent(sess, messages):
round_num = d.get("round", round_num)
elif d.get("type") == "tool_output":
# Mirror the live chat's tool_event shape (chat_routes / chatRenderer).
tool_event = {
tool_events.append({
"round": round_num,
"tool": d.get("tool"),
"command": d.get("command"),
"output": d.get("output"),
"exit_code": d.get("exit_code"),
}
if isinstance(d.get("ask_user"), dict):
# Preserve exact-approval cards from a tainted background-job
# continuation so the user can authorize the sealed action on
# the next foreground turn instead of losing it headlessly.
tool_event["ask_user"] = d["ask_user"]
tool_events.append(tool_event)
})
return full, tool_events
@@ -118,8 +101,14 @@ async def _run_followup(rec: dict) -> bool:
except Exception:
pass
inject = (
f"[Background job {rec['id']} finished]\n\n"
f"{bg_jobs.result_text(rec)}\n\n"
"Continue the task using this output. Don't repeat work that's already done. "
"If the task is now complete, give the user the final result."
)
context = sess.get_context_messages()
context.append(_background_result_message(rec))
context.append({"role": "user", "content": inject})
full, tool_events = await _drain_agent(sess, context)
+1 -15
View File
@@ -810,27 +810,13 @@ async def action_tidy_research(owner: str, **kwargs) -> Tuple[str, bool]:
Research history lives entirely in data/deep_research/<id>.json and is NOT
backed by chat-session rows so a file must never be deleted just because
no chat session matches its id. Only prune files that fail to load.
A broken file has no readable owner stamp, so it cannot be matched against
`owner`. Clearing one is privileged: admins and the single-user operator
(AUTH_ENABLED=false) may, a regular user may not, and neither may anyone
during the pre-setup window before an admin exists.
"""
no chat session matches its id. Only prune files that fail to load."""
try:
from pathlib import Path
import json as _json
from src.tool_security import owner_is_admin_or_single_user
research_dir = Path(DEEP_RESEARCH_DIR)
if not research_dir.exists():
raise TaskNoop("no research directory")
if not owner_is_admin_or_single_user(owner):
# Return before the glob rather than filtering inside the loop: the
# loop reports "none broken" off an empty `removed`, which reaches
# Activity as a false report to a user whose files it skipped, and a
# regular user need not read every owner's file to learn it may
# delete none of them.
raise TaskNoop("not permitted to remove unattributable research files")
files = list(research_dir.glob("*.json"))
removed = []
for p in files:
+3 -35
View File
@@ -381,10 +381,7 @@ class ChatProcessor:
)
if len(rag_content) > 10000:
rag_content = rag_content[:10000] + "\n[Truncated]"
preface.append(untrusted_context_message(
"retrieved documents",
rag_content,
))
preface.append(untrusted_context_message("retrieved documents", rag_content))
except Exception as e:
logger.warning(f"RAG retrieval failed: {e}")
@@ -462,38 +459,12 @@ class ChatProcessor:
skip_url_fetch = len(message) > 2000 or len(non_yt_urls) > 3
if not skip_url_fetch:
for url in non_yt_urls:
try:
result = fetch_webpage_content(url)
except Exception:
# The URL and exception can both contain signed-query
# credentials or response-controlled text. Keep the log
# diagnostic stable as well as the model-facing context.
logger.warning("Automatic URL fetch failed while building context")
result = {"success": False, "error": ""}
result = fetch_webpage_content(url)
if result.get('success'):
content = result.get('content', '')[:10000]
preface.append(untrusted_context_message(
f"web page: {url}",
f"Content from {url}:\n\n{content}",
provenance_origin="external",
))
else:
# A failed automatic URL fetch is context too. Never pass
# exception text or response-controlled diagnostics back to
# the model: reduce the result to a small transport-owned
# status and explicitly state that the page was not read.
error = str(result.get("error") or "")
status = "the page was unavailable"
status_match = re.match(r"^HTTP\s+(\d{3})\b", error)
if status_match:
status = f"the server returned HTTP {status_match.group(1)}"
elif error.startswith("TooLarge:"):
status = "the response exceeded the fetch size limit"
elif error.startswith("Rate limit"):
status = "the request was rate limited"
preface.append(untrusted_context_message(
"web page fetch failure",
f"A linked page was not read: {status}.",
))
# Skills index — progressive disclosure. Only injected when the
@@ -517,9 +488,6 @@ class ChatProcessor:
for s in sorted(by_cat[cat], key=lambda x: x["name"]):
desc = s.get("description") or ""
lines.append(f" - {s['name']}: {desc}" if desc else f" - {s['name']}")
preface.append(untrusted_context_message(
"available skills index",
"\n".join(lines),
))
preface.append(untrusted_context_message("available skills index", "\n".join(lines)))
return preface, rag_sources, web_sources
+2 -62
View File
@@ -282,9 +282,7 @@ def trim_for_context(messages: List[Dict], context_length: int, reserve_tokens:
if essential_system:
sys_text = essential_system[0].get("content", "")
if len(sys_text) > 2000:
truncated_system = dict(essential_system[0])
truncated_system["content"] = sys_text[:2000] + "\n[System prompt truncated for context limits]"
essential_system[0] = truncated_system
essential_system[0] = {"role": "system", "content": sys_text[:2000] + "\n[System prompt truncated for context limits]"}
trimmed = essential_system + convo_msgs
if estimate_tokens(trimmed) <= budget:
return _sanitize_tool_messages(essential_system + protected_msgs + convo_msgs)
@@ -327,9 +325,6 @@ async def maybe_compact(
messages: List[Dict],
headers: Optional[Dict] = None,
owner: Optional[str] = None,
*,
persist: bool = True,
compaction_state: Optional[Dict[str, Any]] = None,
) -> tuple:
"""Check context usage and compact if above threshold.
@@ -421,17 +416,7 @@ async def maybe_compact(
# offset — session.history INCLUDES the system messages, but
# split_point is indexed against convo_msgs which does NOT. Without
# this, the slice drops the leading system message(s).
if compaction_state is not None:
compaction_state.update({
"split_point": split_point,
"summary": summary,
"system_msg_count": len(system_msgs),
"applied": False,
})
if persist:
_update_session_history(session, split_point, summary, system_msg_count=len(system_msgs))
if compaction_state is not None:
compaction_state["applied"] = True
_update_session_history(session, split_point, summary, system_msg_count=len(system_msgs))
new_used = estimate_tokens(compacted)
logger.info(
@@ -442,51 +427,6 @@ async def maybe_compact(
return compacted, context_length, True
def apply_compaction_state(session, compaction_state: Optional[Dict[str, Any]]) -> bool:
"""Persist a route-specific compaction after that route commits output.
Candidate prompts may be compacted speculatively while an explicit
foreground fallback chain is being tried. Persisting at construction time
would let an unavailable route rewrite history before another route answers,
so callers hold this small plan and apply only the winning route's plan.
"""
state = compaction_state if isinstance(compaction_state, dict) else None
if not state or state.get("applied"):
return False
summary = state.get("summary")
split_point = state.get("split_point")
system_msg_count = state.get("system_msg_count", 0)
if not isinstance(summary, str) or not isinstance(split_point, int):
return False
_update_session_history(
session,
split_point,
summary,
system_msg_count=system_msg_count if isinstance(system_msg_count, int) else 0,
)
state["applied"] = True
return True
def apply_compaction_state_for_session(
session_id: Optional[str],
compaction_state: Optional[Dict[str, Any]],
) -> bool:
"""Resolve an in-memory session and apply a deferred compaction plan."""
if not session_id:
return False
try:
from core.models import get_session_manager_instance
manager = get_session_manager_instance()
session = manager.get_session(session_id) if manager else None
except Exception:
session = None
return apply_compaction_state(session, compaction_state) if session else False
def _update_session_history(session, split_point: int, summary: str,
system_msg_count: int = 0):
"""Update the in-memory session history after compaction.
+20 -216
View File
@@ -5,7 +5,6 @@ Consolidates the 4+ copies of normalize_base / resolve_endpoint logic into one p
"""
import json
import ipaddress
import logging
import socket
import subprocess
@@ -28,43 +27,6 @@ _NON_CHAT_MODEL = (
)
def endpoint_cost_tracked(url: str, endpoint_kind: Optional[str] = None) -> bool:
"""Return whether token cost should be tracked for a concrete route.
This is intentionally a non-secret route classification. It mirrors the
frontend's local/subscription exclusions without exposing endpoint URLs to
message metadata.
"""
try:
parsed = urlparse(url or "")
host = (parsed.hostname or "").lower().rstrip(".")
path = (parsed.path or "").rstrip("/")
except Exception:
return False
if not host:
return False
if host == "chatgpt.com" and (
path == "/backend-api/codex" or path.startswith("/backend-api/codex/")
):
return False
kind = str(endpoint_kind or "auto").strip().lower()
if kind == "local":
return False
if kind in {"api", "proxy"}:
return True
if host in {"localhost", "0.0.0.0", "host.docker.internal"} or host.endswith(".local"):
return False
try:
ip = ipaddress.ip_address(host)
return ip.is_global
except ValueError:
pass
if "." not in host:
return False
return True
def _first_chat_model(models) -> Optional[str]:
"""First model that isn't an embedding/tts/etc.; falls back to models[0]."""
for m in (models or []):
@@ -434,14 +396,10 @@ def resolve_endpoint(
db.close()
def _resolve_endpoint_by_id_with_descriptor(
ep_id: str,
model: Optional[str] = None,
owner: Optional[str] = None,
*,
require_exact_model: bool = False,
) -> Optional[Tuple[Tuple[str, str, Dict], dict]]:
"""Resolve a concrete endpoint/model plus its non-secret descriptor.
def resolve_endpoint_by_id(
ep_id: str, model: Optional[str] = None, owner: Optional[str] = None
) -> Optional[Tuple[str, str, Dict]]:
"""Resolve a specific endpoint id (+ optional model) to (chat_url, model, headers).
Returns None if the endpoint doesn't exist or is disabled. Used to turn
a configured fallback entry ({endpoint_id, model}) into a dispatch target.
@@ -468,34 +426,15 @@ def _resolve_endpoint_by_id_with_descriptor(
chat_url = build_chat_url(base)
headers = build_headers(api_key, base)
m = (model or "").strip()
enabled_models = _endpoint_enabled_models(ep)
if require_exact_model:
# Explicit foreground fallback entries are concrete choices. A
# hidden or known-missing model must disable the entry instead of
# silently substituting another model from the endpoint.
if not m or m in _endpoint_hidden_models(ep):
return None
if enabled_models and m not in enabled_models:
return None
else:
# Legacy Utility/Vision chains retain their model-repair behavior.
if m and m in _endpoint_hidden_models(ep):
m = ""
if not m:
m = _first_chat_model(enabled_models) or ""
# Drop a model the user disabled on the endpoint, then pick the first
# enabled chat model rather than a hidden one.
if m and m in _endpoint_hidden_models(ep):
m = ""
if not m:
m = _first_chat_model(_endpoint_enabled_models(ep)) or ""
if not m:
return None
return (
(chat_url, m, headers),
{
"endpoint_id": ep.id,
"endpoint_label": getattr(ep, "name", None) or ep.id,
"endpoint_cost_tracked": endpoint_cost_tracked(
chat_url,
getattr(ep, "endpoint_kind", None),
),
},
)
return chat_url, m, headers
except Exception as e:
logger.debug(f"Could not resolve endpoint {ep_id}: {e}")
return None
@@ -503,101 +442,11 @@ def _resolve_endpoint_by_id_with_descriptor(
db.close()
def resolve_endpoint_by_id(
ep_id: str,
model: Optional[str] = None,
owner: Optional[str] = None,
*,
require_exact_model: bool = False,
) -> Optional[Tuple[str, str, Dict]]:
"""Resolve a specific endpoint id (+ optional model) to its runtime route."""
def resolve_chat_fallback_candidates(owner: Optional[str] = None) -> list:
"""Compatibility shim for the retired default-chat fallback chain."""
resolved = _resolve_endpoint_by_id_with_descriptor(
ep_id,
model,
owner=owner,
require_exact_model=require_exact_model,
)
return resolved[0] if resolved else None
def resolve_route_descriptor(
endpoint_url: str,
model: str,
headers: Optional[Dict] = None,
owner: Optional[str] = None,
) -> dict:
"""Return the visible endpoint identity for an already-resolved route.
Headers are compared only inside the process so two endpoints using the
same provider URL/model but different credentials remain distinguishable.
No credential material is returned or logged.
"""
if not endpoint_url or not model:
return {
"endpoint_id": None,
"endpoint_label": "Selected route",
"endpoint_cost_tracked": endpoint_cost_tracked(endpoint_url),
}
db = SessionLocal()
try:
q = db.query(ModelEndpoint).filter(ModelEndpoint.is_enabled == True)
if owner:
from src.auth_helpers import owner_filter
q = owner_filter(q, ModelEndpoint, owner)
expected = (endpoint_url.rstrip("/"), model, headers or {})
for ep in q.all():
resolved = _resolve_endpoint_by_id_with_descriptor(
ep.id,
model,
owner=owner,
require_exact_model=True,
)
if not resolved:
continue
candidate, descriptor = resolved
actual = (candidate[0].rstrip("/"), candidate[1], candidate[2] or {})
if actual == expected:
return descriptor
except Exception as e:
logger.debug("Could not identify selected endpoint route: %s", e)
finally:
db.close()
return {
"endpoint_id": None,
"endpoint_label": "Selected route",
"endpoint_cost_tracked": endpoint_cost_tracked(endpoint_url),
}
def resolve_route_descriptor_by_id(
endpoint_id: str,
endpoint_url: str,
model: str,
headers: Optional[Dict] = None,
owner: Optional[str] = None,
) -> Optional[dict]:
"""Resolve a selected route's identity without relying on row order.
The explicit endpoint id is still verified against the resolved runtime
route. This prevents stale or mismatched request metadata from being used
for attribution while disambiguating endpoints whose routes are otherwise
identical.
"""
resolved = _resolve_endpoint_by_id_with_descriptor(
endpoint_id,
model,
owner=owner,
require_exact_model=True,
)
if not resolved:
return None
candidate, descriptor = resolved
expected = ((endpoint_url or "").rstrip("/"), model, headers or {})
actual = (candidate[0].rstrip("/"), candidate[1], candidate[2] or {})
return descriptor if actual == expected else None
del owner
return []
def resolve_utility_fallback_candidates(owner: Optional[str] = None) -> list:
@@ -611,62 +460,17 @@ def resolve_vision_fallback_candidates(owner: Optional[str] = None) -> list:
def _resolve_fallback_candidates(setting_key: str, owner: Optional[str] = None) -> list:
out = []
try:
from src.settings import get_user_setting, load_settings
settings = load_settings()
chain = get_user_setting(setting_key, owner or "", settings.get(setting_key) or []) or []
except Exception:
return []
return resolve_fallback_entries(chain, owner=owner)
def resolve_fallback_entries(
entries,
owner: Optional[str] = None,
*,
require_exact_model: bool = False,
) -> list:
"""Resolve ordered endpoint/model entries within the caller's owner scope."""
out = []
for entry in entries or []:
return out
for entry in chain:
if not isinstance(entry, dict):
continue
resolved = resolve_endpoint_by_id(
entry.get("endpoint_id", ""),
entry.get("model", ""),
owner=owner,
require_exact_model=require_exact_model,
)
if resolved and resolved not in out:
resolved = resolve_endpoint_by_id(entry.get("endpoint_id", ""), entry.get("model", ""), owner=owner)
if resolved:
out.append(resolved)
return out
def resolve_fallback_entries_with_descriptors(
entries,
owner: Optional[str] = None,
*,
require_exact_model: bool = False,
) -> list:
"""Resolve ordered entries while retaining safe endpoint provenance."""
out = []
seen = []
for entry in entries or []:
if not isinstance(entry, dict):
continue
resolved = _resolve_endpoint_by_id_with_descriptor(
entry.get("endpoint_id", ""),
entry.get("model", ""),
owner=owner,
require_exact_model=require_exact_model,
)
if not resolved:
continue
candidate, descriptor = resolved
if any(candidate == prior for prior in seen):
continue
seen.append(candidate)
out.append((candidate, descriptor))
return out
+14 -189
View File
@@ -1,155 +1,22 @@
"""Explicit foreground Chat and Agent model-routing policy."""
"""Foreground Chat and Agent model-routing policy.
from dataclasses import dataclass
from typing import Any, Collection, Dict, FrozenSet, Optional, Tuple
The selected session model is strict by default. Historical
``default_model_fallbacks`` values remain stored for compatibility, but this
policy intentionally does not read or migrate them.
"""
from src.endpoint_resolver import (
endpoint_cost_tracked,
resolve_fallback_entries,
resolve_fallback_entries_with_descriptors,
resolve_route_descriptor,
resolve_route_descriptor_by_id,
)
_DEFAULT_FALLBACK_ENTRY_RESOLVER = resolve_fallback_entries
FOREGROUND_FALLBACK_ENABLED_KEY = "foreground_fallback_enabled"
FOREGROUND_FALLBACK_LIST_KEY = "foreground_model_fallbacks"
FOREGROUND_AVAILABILITY_STATUSES: FrozenSet[int] = frozenset({
408, 425, 429, 500, 502, 503, 504, 507, 508, 529,
})
MAX_FOREGROUND_FALLBACKS = 10
@dataclass(frozen=True)
class ForegroundModelPolicy:
"""Resolved per-user foreground fallback policy."""
enabled: bool = False
fallback_candidates: Tuple[tuple, ...] = ()
fallback_descriptors: Tuple[dict, ...] = ()
eligible_statuses: FrozenSet[int] = FOREGROUND_AVAILABILITY_STATUSES
fallback_on_empty: bool = False
def _load_policy_preferences(owner: Optional[str]) -> dict:
"""Load only preferences that explicitly belong to ``owner``.
The generic preferences loader intentionally treats a legacy flat store as
the single-user preferences object. That compatibility must not cross an
authentication transition: once a named owner is present, foreground
fallback consent exists only in an actual ``_users[owner]`` dictionary.
"""
from routes import prefs_routes
if owner is None:
prefs = prefs_routes._load_for_user(None)
return dict(prefs) if isinstance(prefs, dict) else {}
raw = prefs_routes._load()
users = raw.get("_users") if isinstance(raw, dict) else None
if not isinstance(users, dict):
return {}
prefs = users.get(owner)
return dict(prefs) if isinstance(prefs, dict) else {}
def resolve_foreground_model_policy(
owner: Optional[str] = None,
allowed_models: Optional[Collection[str]] = None,
) -> ForegroundModelPolicy:
"""Resolve an explicit owner-scoped policy, failing closed to strict mode.
The policy is stored in user preferences even when authentication is
disabled. Historical ``default_model_fallbacks`` values are deliberately
unrelated and are never read or migrated.
"""
try:
prefs = _load_policy_preferences(owner)
except Exception:
return ForegroundModelPolicy()
if prefs.get(FOREGROUND_FALLBACK_ENABLED_KEY) is not True:
return ForegroundModelPolicy()
entries = prefs.get(FOREGROUND_FALLBACK_LIST_KEY)
if not isinstance(entries, list) or not entries:
return ForegroundModelPolicy()
if allowed_models is not None:
allowed = frozenset(allowed_models)
entries = [
entry for entry in entries
if (
isinstance(entry, dict)
and isinstance(entry.get("model"), str)
and entry.get("model") in allowed
)
]
if not entries:
return ForegroundModelPolicy()
entries = entries[:MAX_FOREGROUND_FALLBACKS]
if resolve_fallback_entries is not _DEFAULT_FALLBACK_ENTRY_RESOLVER:
# Preserve the long-standing resolver seam used by downstream tests and
# integrations. Production uses the descriptor-aware resolver below.
compatibility_candidates = resolve_fallback_entries(
entries,
owner=owner,
require_exact_model=True,
)
# Known limitation of this test-only seam: alignment matches on model
# alone, so when two entries share a model and the resolver skips the
# first, the surviving candidate inherits the skipped entry's
# endpoint_id. Production uses the descriptor-aware branch below,
# which is unaffected.
resolved_routes = []
remaining_entries = list(entries)
for candidate in compatibility_candidates:
matching_index = next(
(
index for index, entry in enumerate(remaining_entries)
if isinstance(entry, dict)
and entry.get("model") == candidate[1]
),
None,
)
matching_entry = (
remaining_entries.pop(matching_index)
if matching_index is not None
else {}
)
descriptor = {
"endpoint_id": matching_entry.get("endpoint_id"),
"endpoint_label": matching_entry.get("endpoint_id") or "Fallback route",
"endpoint_cost_tracked": endpoint_cost_tracked(candidate[0]),
}
resolved_routes.append((candidate, descriptor))
else:
resolved_routes = resolve_fallback_entries_with_descriptors(
entries,
owner=owner,
require_exact_model=True,
)
candidates = [candidate for candidate, _descriptor in resolved_routes]
if not candidates:
return ForegroundModelPolicy()
return ForegroundModelPolicy(
enabled=True,
fallback_candidates=tuple(candidates),
fallback_descriptors=tuple(
dict(descriptor) for _candidate, descriptor in resolved_routes
),
)
from typing import Any, Dict, Optional
def resolve_foreground_fallback_candidates(owner: Optional[str] = None) -> list:
"""Return only candidates explicitly enabled by the current user."""
"""Return fallback candidates for a foreground Chat or Agent request.
return list(resolve_foreground_model_policy(owner).fallback_candidates)
Foreground routing is strict, so no alternate endpoint/model is eligible.
``owner`` is accepted to keep this policy boundary owner-aware.
"""
del owner
return []
def build_foreground_model_candidates(
@@ -157,50 +24,8 @@ def build_foreground_model_candidates(
model: str,
headers: Optional[Dict[str, Any]] = None,
owner: Optional[str] = None,
policy: Optional[ForegroundModelPolicy] = None,
) -> list:
"""Build the ordered candidate list for a foreground request."""
policy = policy or resolve_foreground_model_policy(owner)
primary = (endpoint_url, model, headers or {})
candidates = [primary]
for candidate in policy.fallback_candidates:
if candidate not in candidates:
candidates.append(candidate)
return candidates
def build_foreground_route_descriptors(
endpoint_url: str,
model: str,
headers: Optional[Dict[str, Any]] = None,
owner: Optional[str] = None,
policy: Optional[ForegroundModelPolicy] = None,
selected_endpoint_id: Optional[str] = None,
) -> list:
"""Build safe route metadata parallel to foreground candidates."""
policy = policy or resolve_foreground_model_policy(owner)
selected = None
if selected_endpoint_id:
selected = resolve_route_descriptor_by_id(
selected_endpoint_id,
endpoint_url,
model,
headers or {},
owner=owner,
)
if selected is None:
selected = resolve_route_descriptor(endpoint_url, model, headers or {}, owner=owner)
primary = (endpoint_url, model, headers or {})
candidates = [primary]
descriptors = [selected]
for candidate, descriptor in zip(
policy.fallback_candidates,
policy.fallback_descriptors,
):
if candidate in candidates:
continue
candidates.append(candidate)
descriptors.append(dict(descriptor))
return descriptors
return [primary] + resolve_foreground_fallback_candidates(owner=owner)
+1 -8
View File
@@ -719,14 +719,7 @@ async def execute_api_call(
output = f"HTTP {status}\n{formatted}"
if status >= 400:
return {
"error": output,
"exit_code": 1,
# The error string includes the remote response body. Preserve
# it for diagnostics, but make its provenance explicit so the
# agent gate does not treat HTTP failure as content-free.
"untrusted_content": True,
}
return {"error": output, "exit_code": 1}
return {"output": output, "exit_code": 0}
-16
View File
@@ -63,11 +63,8 @@ _PASSIVE_EXACT_PATHS = {
"/api/activity/heartbeat",
"/api/client-perf",
"/api/tasks/notifications",
"/api/tasks/runs/recent",
"/api/research/active",
"/api/email/urgency-state",
# UI idle poll sibling of urgency-state; must not pre-empt background tasks.
"/api/email/unread-state",
}
_PASSIVE_PREFIXES = (
@@ -77,19 +74,6 @@ _PASSIVE_PREFIXES = (
)
async def maybe_stop_background_tasks_for_heartbeat(stop_background) -> bool:
"""Stop background work for browser activity only when the gate is enabled.
``stop_background`` is injected by the application boundary so this policy
remains independently testable without importing the full FastAPI app.
"""
if not _enabled():
return False
await stop_background(reason="browser heartbeat")
return True
def should_track_interactive_request(path: str, method: str = "GET") -> bool:
if not _enabled():
return False
+126 -885
View File
File diff suppressed because it is too large Load Diff
-2
View File
@@ -530,8 +530,6 @@ class McpManager:
"stderr": output if is_error else "",
"exit_code": 1 if is_error else 0,
}
if is_error and output:
result_dict["untrusted_content"] = True
if images:
result_dict["images"] = images
return result_dict
+10 -24
View File
@@ -15,32 +15,18 @@ from urllib.parse import urlparse, parse_qs
logger = logging.getLogger(__name__)
def _resolve_redirect_base() -> str:
"""Origin the browser is sent back to after authorizing.
Falls back to the port the app binds natively (APP_PORT, read the same way
by app.py and launcher.py) rather than a fixed 7000: the macOS launcher
defaults to 7860, and a callback on the wrong port reaches nothing. The
hostname stays `localhost` rather than internal_api_base()'s 127.0.0.1 —
this URI is registered with the authorization server (via DCR, or by hand
for Google clients), so changing the host invalidates registrations that
already exist.
"""
return (
os.environ.get("OAUTH_REDIRECT_BASE_URL")
or os.environ.get("APP_PUBLIC_URL")
or f"http://localhost:{os.environ.get('APP_PORT', '7000')}"
).rstrip("/")
# OAuth redirect URI registered with every authorization server via DCR. Loopback
# is allowed for native/desktop clients (RFC 8252); remote users finish via the
# paste-back flow. Deployments whose externally reachable origin differs from the
# port Odysseus binds — reverse proxy, public domain, or Docker, whose host port
# map is invisible inside the container — must set OAUTH_REDIRECT_BASE_URL (or
# APP_PUBLIC_URL), otherwise the redirect never lands back on Odysseus.
_REDIRECT_BASE = _resolve_redirect_base()
# paste-back flow. Deployments not reachable at http://localhost:7000 (custom
# port, reverse proxy, or public domain) must set OAUTH_REDIRECT_BASE_URL (or
# APP_PUBLIC_URL) to their externally reachable origin so the redirect lands back
# on Odysseus. APP_PORT is intentionally not used: it is only the Docker host
# port-map; the app always listens on 7000 inside the container.
_REDIRECT_BASE = (
os.environ.get("OAUTH_REDIRECT_BASE_URL")
or os.environ.get("APP_PUBLIC_URL")
or "http://localhost:7000"
).rstrip("/")
REDIRECT_URI = f"{_REDIRECT_BASE}/api/mcp/oauth/callback"
# How long the background connect waits for the user to authorize before giving up.
+6 -11
View File
@@ -290,22 +290,17 @@ def detect_vendor(base_url: Any = "", endpoint_kind: Any = "") -> str:
return kind_map[kind]
parsed = urlparse(compact_str(base_url))
host = (parsed.hostname or "").lower().rstrip(".")
host = (parsed.hostname or "").lower()
port = parsed.port
def host_matches(domain: str) -> bool:
domain = domain.lower().rstrip(".")
return host == domain or host.endswith(f".{domain}")
if host_matches("openrouter.ai"):
if host.endswith("openrouter.ai"):
return VENDOR_OPENROUTER
if host_matches("openai.com"):
if host.endswith("openai.com"):
return VENDOR_OPENAI
if host_matches("anthropic.com"):
if host.endswith("anthropic.com"):
return VENDOR_ANTHROPIC
if host_matches("googleapis.com"):
if host.endswith("googleapis.com"):
return VENDOR_GOOGLE
if host_matches("ollama.com") or port == 11434:
if host.endswith("ollama.com") or port == 11434:
return VENDOR_OLLAMA
if port == 1234:
return VENDOR_LMSTUDIO
-354
View File
@@ -1,354 +0,0 @@
"""SSRF-guarded synchronous HTTP fetching primitives.
This module owns outbound URL classification, one-resolution-per-hop DNS
pinning, redirects, and response-body budgets. It deliberately has no search
or content-extraction dependencies so callers outside search can reuse the
same transport boundary.
"""
from __future__ import annotations
import ipaddress
import socket
import ssl
from typing import Callable, Iterable, cast
from urllib.parse import urljoin, urlparse
import httpcore
import httpx
from src.constants import WEB_FETCH_HARD_MAX_BYTES, WEB_FETCH_SOFT_MAX_BYTES
_PRIVATE_NETWORKS = (
ipaddress.ip_network("0.0.0.0/8"),
ipaddress.ip_network("10.0.0.0/8"),
ipaddress.ip_network("127.0.0.0/8"),
ipaddress.ip_network("169.254.0.0/16"),
ipaddress.ip_network("172.16.0.0/12"),
ipaddress.ip_network("192.168.0.0/16"),
ipaddress.ip_network("::1/128"),
ipaddress.ip_network("fc00::/7"),
ipaddress.ip_network("fe80::/10"),
)
def _is_private_address(addr: ipaddress._BaseAddress) -> bool:
if isinstance(addr, ipaddress.IPv6Address) and addr.ipv4_mapped is not None:
addr = addr.ipv4_mapped
return (
addr.is_private
or addr.is_loopback
or addr.is_link_local
or addr.is_reserved
or addr.is_multicast
or addr.is_unspecified
or any(addr in net for net in _PRIVATE_NETWORKS)
)
def _resolve_hostname_ips(hostname: str) -> list[ipaddress._BaseAddress]:
try:
infos = socket.getaddrinfo(hostname, None)
except Exception:
return []
out = []
for info in infos:
try:
out.append(ipaddress.ip_address(info[4][0]))
except Exception:
continue
return out
def _public_http_url(
url: str,
*,
resolver: Callable[[str], list[ipaddress._BaseAddress]] | None = None,
) -> bool:
resolver = resolver or _resolve_hostname_ips
try:
parsed = urlparse(url)
if parsed.scheme not in ("http", "https"):
return False
host = (parsed.hostname or "").strip()
if not host:
return False
lower = host.lower()
if lower in ("localhost", "metadata", "metadata.google.internal"):
return False
if lower.endswith((".local", ".localhost", ".internal", ".lan", ".intranet")):
return False
try:
return not _is_private_address(ipaddress.ip_address(host))
except ValueError:
pass
addrs = resolver(host)
return bool(addrs) and not any(_is_private_address(a) for a in addrs)
except Exception:
return False
def _resolve_public_ips(
url: str,
*,
resolver: Callable[[str], list[ipaddress._BaseAddress]] | None = None,
) -> list[ipaddress._BaseAddress]:
resolver = resolver or _resolve_hostname_ips
parsed = urlparse(url)
if parsed.scheme not in ("http", "https") or not parsed.hostname:
raise httpx.RequestError(f"Blocked non-public URL: {url}")
host = (parsed.hostname or "").strip().lower()
if host in ("localhost", "metadata", "metadata.google.internal"):
raise httpx.RequestError(f"Blocked non-public hostname: {host}")
try:
ip = ipaddress.ip_address(host)
if _is_private_address(ip):
raise httpx.RequestError(f"Blocked non-public IP literal: {host}")
return [ip]
except httpx.RequestError:
raise
except ValueError:
pass
addrs = resolver(host)
if not addrs or any(_is_private_address(a) for a in addrs):
raise httpx.RequestError(f"Blocked non-public URL: {url}")
return addrs
class _PinnedBackend(httpcore.NetworkBackend):
"""Network backend that connects to a pre-resolved IP."""
def __init__(self, ip: ipaddress._BaseAddress):
self._ip = str(ip)
self._real = httpcore.SyncBackend()
def connect_tcp(
self,
host: str,
port: int,
timeout: float | None = None,
local_address: str | None = None,
socket_options=None,
):
return self._real.connect_tcp(
self._ip, port, timeout, local_address, socket_options
)
def connect_unix_socket(self, path, timeout=None, socket_options=None):
return self._real.connect_unix_socket(path, timeout, socket_options)
def sleep(self, seconds: float) -> None:
return self._real.sleep(seconds)
_HTTPCORE_TO_HTTPX_EXC = {
httpcore.ConnectError: httpx.ConnectError,
httpcore.ConnectTimeout: httpx.ConnectTimeout,
httpcore.LocalProtocolError: httpx.LocalProtocolError,
httpcore.NetworkError: httpx.NetworkError,
httpcore.PoolTimeout: httpx.PoolTimeout,
httpcore.ProtocolError: httpx.ProtocolError,
httpcore.ProxyError: httpx.ProxyError,
httpcore.ReadError: httpx.ReadError,
httpcore.ReadTimeout: httpx.ReadTimeout,
httpcore.RemoteProtocolError: httpx.RemoteProtocolError,
httpcore.TimeoutException: httpx.TimeoutException,
httpcore.UnsupportedProtocol: httpx.UnsupportedProtocol,
httpcore.WriteError: httpx.WriteError,
httpcore.WriteTimeout: httpx.WriteTimeout,
}
class _PinnedTransport(httpx.BaseTransport):
"""Transport that pins every TCP connect to a pre-resolved IP."""
def __init__(self, ip: ipaddress._BaseAddress, *, http2: bool = False):
self._pool = httpcore.ConnectionPool(
ssl_context=ssl.create_default_context(),
http1=True,
http2=http2,
network_backend=_PinnedBackend(ip),
)
def __enter__(self):
self._pool.__enter__()
return self
def __exit__(self, exc_type=None, exc_value=None, traceback=None) -> None:
self._pool.__exit__(exc_type, exc_value, traceback)
def handle_request(self, request: httpx.Request) -> httpx.Response:
httpcore_req = httpcore.Request(
method=request.method,
url=httpcore.URL(
scheme=request.url.raw_scheme,
host=request.url.raw_host,
port=request.url.port,
target=request.url.raw_path,
),
headers=request.headers.raw,
content=request.stream,
extensions=request.extensions,
)
try:
httpcore_resp = self._pool.handle_request(httpcore_req)
content = b"".join(cast(Iterable[bytes], httpcore_resp.stream))
except Exception as exc:
mapped = _HTTPCORE_TO_HTTPX_EXC.get(type(exc))
if mapped is not None:
raise mapped(str(exc)) from exc
raise
return httpx.Response(
status_code=httpcore_resp.status,
headers=httpcore_resp.headers,
content=content,
extensions=httpcore_resp.extensions,
)
def close(self) -> None:
self._pool.close()
class BodyTooLargeError(Exception):
"""The server declared a body larger than the hard fetch ceiling."""
def __init__(self, url: str, declared_bytes: int):
self.url = url
self.declared_bytes = declared_bytes
super().__init__(
f"response body is {declared_bytes:,} bytes, over the "
f"{WEB_FETCH_HARD_MAX_BYTES:,}-byte hard cap"
)
class _CappedFetch:
"""Result of a size-capped streaming GET."""
__slots__ = (
"status_code",
"headers",
"content",
"truncated",
"declared_bytes",
"encoding",
"url",
)
def __init__(
self,
status_code,
headers,
content,
truncated,
declared_bytes,
encoding,
url,
):
self.status_code = status_code
self.headers = headers
self.content = content
self.truncated = truncated
self.declared_bytes = declared_bytes
self.encoding = encoding
self.url = url
@property
def text(self) -> str:
return self.content.decode(self.encoding or "utf-8", errors="replace")
def raise_for_status(self):
if self.status_code >= 400:
request = httpx.Request("GET", self.url)
raise httpx.HTTPStatusError(
f"HTTP {self.status_code} for {self.url}",
request=request,
response=httpx.Response(self.status_code, request=request),
)
def _get_public_url(
url: str,
headers: dict,
timeout: int,
max_redirects: int = 5,
max_bytes: int | None = None,
*,
resolve_public_ips: Callable[[str], list[ipaddress._BaseAddress]] | None = None,
transport_factory: Callable[[ipaddress._BaseAddress], httpx.BaseTransport] | None = None,
) -> _CappedFetch:
"""Capped streaming GET with SSRF-guarded, DNS-pinned redirects."""
resolve_public_ips = resolve_public_ips or _resolve_public_ips
transport_factory = transport_factory or _PinnedTransport
cap = min(max_bytes or WEB_FETCH_SOFT_MAX_BYTES, WEB_FETCH_HARD_MAX_BYTES)
current = url
for _ in range(max_redirects + 1):
ips = resolve_public_ips(current)
req_headers = dict(headers or {})
req_headers["Accept-Encoding"] = "identity"
with httpx.Client(
headers=req_headers,
timeout=timeout,
follow_redirects=False,
transport=transport_factory(ips[0]),
) as client:
with client.stream("GET", current) as response:
if response.status_code in (301, 302, 303, 307, 308):
location = response.headers.get("location")
if not location:
return _CappedFetch(
response.status_code,
response.headers,
b"",
False,
None,
response.encoding,
str(response.url),
)
current = urljoin(str(response.url), location)
continue
enc = (response.headers.get("content-encoding") or "").strip().lower()
if enc and enc != "identity":
raise httpx.RequestError(
f"Refusing compressed response (Content-Encoding: {enc}) after "
"requesting identity: cannot bound decoded body size",
request=httpx.Request("GET", current),
)
declared = None
raw_len = response.headers.get("content-length")
if raw_len and raw_len.isdigit():
declared = int(raw_len)
if declared is not None and declared > WEB_FETCH_HARD_MAX_BYTES:
raise BodyTooLargeError(current, declared)
chunks = []
read = 0
truncated = False
for chunk in response.iter_bytes():
read += len(chunk)
if read > cap:
keep = cap - (read - len(chunk))
if keep > 0:
chunks.append(chunk[:keep])
truncated = True
break
chunks.append(chunk)
return _CappedFetch(
response.status_code,
response.headers,
b"".join(chunks),
truncated,
declared,
response.encoding,
str(response.url),
)
raise httpx.RequestError(
"Too many redirects", request=httpx.Request("GET", current)
)
-56
View File
@@ -1,56 +0,0 @@
"""Shared owner identity constants and helpers."""
from __future__ import annotations
import os
from typing import Optional
DEFAULT_LOCAL_OWNER = "__odysseus_local__"
DEFAULT_LOCAL_OWNER_LABEL = "Local"
INTERNAL_TOOL_USER = "internal-tool"
REQUEST_SENTINEL_OWNERS = frozenset({INTERNAL_TOOL_USER, "api", "demo", "system"})
RESERVED_AUTH_USERNAMES = REQUEST_SENTINEL_OWNERS | {DEFAULT_LOCAL_OWNER}
def auth_disabled() -> bool:
"""Return True only when auth is explicitly disabled by configuration."""
return os.getenv("AUTH_ENABLED", "true").strip().lower() == "false"
def normalize_owner(owner: str | None) -> Optional[str]:
"""Normalize an owner-like value without inventing a fallback identity."""
value = str(owner or "").strip()
return value or None
def owner_key(owner: str | None) -> Optional[str]:
normalized = normalize_owner(owner)
return normalized.lower() if normalized else None
def is_request_sentinel_owner(owner: str | None) -> bool:
return owner_key(owner) in REQUEST_SENTINEL_OWNERS
def effective_storage_owner(owner: str | None, *, auth_is_disabled: bool | None = None) -> Optional[str]:
"""Resolve the owner used for storage writes that need a real bucket.
``None`` still means no authenticated owner when auth is enabled. In the
explicit no-login mode, it resolves to the reserved local owner instead of
conflating local-operator writes with legacy NULL/ownerless rows.
"""
normalized = normalize_owner(owner)
if normalized:
if is_request_sentinel_owner(normalized):
return None
return normalized
disabled = auth_disabled() if auth_is_disabled is None else auth_is_disabled
if disabled:
return DEFAULT_LOCAL_OWNER
return None
def is_default_local_owner(owner: str | None) -> bool:
return owner_key(owner) == DEFAULT_LOCAL_OWNER
+2 -15
View File
@@ -61,13 +61,7 @@ def _sanitize_label(label: str) -> str:
return label
def untrusted_context_message(
label: str,
content: Any,
*,
provenance_origin: str | None = None,
arm_tool_gate: bool = True,
) -> Dict[str, Any]:
def untrusted_context_message(label: str, content: Any) -> Dict[str, Any]:
"""Return an LLM message that keeps retrieved/source text out of system role.
The template is structured so that *only* the hardcoded
@@ -79,13 +73,6 @@ def untrusted_context_message(
safe_label = _sanitize_label(label)
text = "" if content is None else str(content)
text = _escape_guard_markers(text)
metadata: Dict[str, Any] = {
"trusted": False,
"source": label,
"tool_gate_untrusted": bool(arm_tool_gate),
}
if provenance_origin:
metadata["provenance_origin"] = provenance_origin
return {
"role": "user",
"content": (
@@ -95,5 +82,5 @@ def untrusted_context_message(
f"{text}\n"
f"{GUARD_CLOSE}"
),
"metadata": metadata,
"metadata": {"trusted": False, "source": label},
}
-1
View File
@@ -12,7 +12,6 @@ class ChatRequest(BaseModel):
use_research: Optional[bool] = Field(default=False, description="Enable deep research")
time_filter: Optional[str] = Field(default=None, description="Time filter for search")
preset_id: Optional[str] = Field(default=None, description="Preset identifier")
selected_endpoint_id: Optional[str] = Field(default=None, description="Selected model endpoint ID")
@field_validator('message')
@classmethod
+1 -19
View File
@@ -14,13 +14,6 @@ from src.constants import SETTINGS_FILE, FEATURES_FILE
logger = logging.getLogger(__name__)
# Keys retained in the raw settings store for compatibility and rollback, but
# deliberately unavailable through generic settings APIs or agent tools. They
# must stay in ``DEFAULT_SETTINGS`` so old files continue to load without data
# loss; callers that present or mutate settings should use this set as a
# tombstone boundary.
RETIRED_SETTING_KEYS = frozenset({"default_model_fallbacks"})
# Tiny TTL cache for settings/features. get_setting() is called on hot paths
# (every chat, every preprocess); without this it re-parses the JSON each call.
# Picks up edits within _CACHE_TTL seconds, which is fine for human-edited config.
@@ -204,17 +197,6 @@ DEFAULT_SETTINGS = {
},
}
def without_retired_settings(settings: dict) -> dict:
"""Return a shallow copy suitable for generic settings interfaces."""
if not isinstance(settings, dict):
return {}
return {
key: value
for key, value in settings.items()
if key not in RETIRED_SETTING_KEYS
}
DEFAULT_FEATURES = {
"web_search": True,
"web_fetch": True,
@@ -287,7 +269,7 @@ _PER_USER_KEYS = {
# Default chat endpoint / model — without per-user resolution every new
# account inherited whatever the most-recent admin picked, which then
# got injected into the chat composer on first open.
"default_endpoint_id", "default_model",
"default_endpoint_id", "default_model", "default_model_fallbacks",
"utility_endpoint_id", "utility_model", "utility_model_fallbacks",
"research_endpoint_id", "research_model",
}
+5
View File
@@ -1,6 +1,7 @@
"""Shared resolver for background-task AI endpoints."""
from src.endpoint_resolver import (
resolve_chat_fallback_candidates,
resolve_endpoint,
resolve_utility_fallback_candidates,
)
@@ -31,6 +32,7 @@ def resolve_task_candidates(
2. Utility endpoint/model
3. Default endpoint/model
4. Utility fallback chain
5. Retired default-fallback compatibility hook (currently empty)
"""
candidates = []
@@ -47,6 +49,9 @@ def resolve_task_candidates(
_append(*resolve_endpoint("default", owner=owner))
for url, model, headers in resolve_utility_fallback_candidates(owner=owner):
_append(url, model, headers)
for url, model, headers in resolve_chat_fallback_candidates(owner=owner):
_append(url, model, headers)
return candidates
+1 -38
View File
@@ -10,7 +10,6 @@ from datetime import datetime, timedelta, timezone
from typing import Any, Awaitable, Callable, Dict, Tuple
from core.auth import RESERVED_USERNAMES
from src.owner_identity import REQUEST_SENTINEL_OWNERS
from src.task_action_policy import (
is_admin_only_task_action,
owner_has_admin_task_privileges,
@@ -1884,7 +1883,6 @@ class TaskScheduler:
pass
full_text = ""
tool_results = []
approval_pause = None
# Honor per-task max_steps (defense against runaway agent loops).
# Falls back to 20 if not set — the historical default.
@@ -1931,44 +1929,9 @@ class TaskScheduler:
tool_summary = data.get("stdout") or data.get("output") or data.get("result") or ""
if isinstance(tool_summary, str) and tool_summary.strip():
tool_results.append(f"[{data.get('tool', '?')}] {tool_summary[:500]}")
approval = data.get("ask_user")
if (
isinstance(approval, dict)
and approval.get("kind") == "tool_approval"
):
approval_pause = {
"tool": data.get("tool") or "tool",
"approval_id": approval.get("approval_id"),
}
# Scheduled tasks have no interactive surface that
# can safely resume a one-use grant. Retire the
# record immediately instead of leaving it pending
# and report an explicit manual-action boundary.
try:
from src.tool_approvals import tool_approval_store
tool_approval_store.consume(
approval_pause["approval_id"],
decision="deny",
owner=task.owner,
session_id=session_id,
)
except Exception:
logger.debug(
"Could not retire scheduled-task approval",
exc_info=True,
)
break
except (json.JSONDecodeError, KeyError):
pass
if approval_pause is not None:
return (
"Scheduled task paused safely: "
f"{approval_pause['tool']} requested an exact action after "
"untrusted context. That action was not executed. Run this task "
"interactively to inspect and approve the action."
)
# Grace summarization — if the model exhausted rounds on tool calls
# without producing a final text response, do one last LLM call
# asking it to summarize what it did. Guarantees output.
@@ -2521,7 +2484,7 @@ class TaskScheduler:
# check-ins seeded, which then double-fire alongside the human user's
# check-ins. This was the root cause of the duplicate 'Morning check-in'
# rows we had to manually clean up.
if not owner or owner in REQUEST_SENTINEL_OWNERS:
if not owner or owner in RESERVED_USERNAMES:
logger.info(f"ensure_assistant_defaults: skip synthetic owner {owner!r}")
return
from core.database import SessionLocal, CrewMember, ScheduledTask
+74 -110
View File
@@ -439,11 +439,56 @@ async def escalate_and_learn(
failure_reason: str,
owner: Optional[str] = None,
) -> Optional[str]:
"""Retire legacy background learning when no approval UI is available."""
logger.info(
"background teacher learning skipped: generated skills require an "
"interactive exact approval"
"""Call the teacher, evaluate ITS attempt, save a skill on success.
Returns the saved skill name (or None if the teacher couldn't
write one). Logs but doesn't raise — escalation is best-effort.
"""
from src.settings import get_setting
teacher_spec = (get_setting("teacher_model", "") or "").strip()
if not teacher_spec:
return None
prompt = _TEACHER_ESCALATION_PROMPT.format(
user_request=user_request or "(no user request captured)",
failure_reason=failure_reason or "(failure reason not captured)",
untrusted_trace_guard=_UNTRUSTED_TRACE_GUARD,
trace=_format_trace(tool_results, agent_reply),
)
response = await _call_teacher(teacher_spec, prompt, owner=owner)
if not response:
return None
skill = _extract_skill_json(response)
if not skill:
# Teacher chose not to write a skill — see prompt contract.
logger.info("teacher declined to write a skill for this failure")
return None
# Same regex eval applied to the teacher's response — if the
# teacher itself sounded uncertain ("I don't have a tool"), drop
# the skill rather than persist a sketchy one.
status, reason = evaluate_turn_regex([], response)
if status == "failure":
logger.info(f"teacher response failed eval, skipping skill save: {reason}")
return None
# Tag the skill with the escalation source for auditability.
skill.setdefault("source", "teacher-escalation")
skill.setdefault("teacher_model", teacher_spec)
# Force action=add regardless of what the teacher wrote.
skill["action"] = "add"
import json
from src.tool_implementations import do_manage_skills
try:
result = await do_manage_skills(json.dumps(skill), owner=owner)
if isinstance(result, dict) and not result.get("error"):
logger.info(f"teacher wrote skill: {skill.get('name')}")
return skill.get("name")
logger.warning(f"skill save failed: {result}")
except Exception as e:
logger.warning(f"skill save raised: {e}")
return None
@@ -518,12 +563,6 @@ async def run_teacher_inline(
student_tool_events: List[Dict[str, Any]],
student_reply: str,
owner: Optional[str] = None,
session_id: Optional[str] = None,
workspace: Optional[str] = None,
disabled_tools: Optional[set[str]] = None,
tool_policy: Any = None,
active_document: Any = None,
active_email: Optional[Dict[str, str]] = None,
):
"""Async generator. Yields SSE event strings.
@@ -622,7 +661,6 @@ async def run_teacher_inline(
from src.agent_loop import stream_agent_loop
captured_tool_events: List[Dict[str, Any]] = []
captured_text_parts: List[str] = []
captured_metrics: Dict[str, Any] = {}
async for evt_str in stream_agent_loop(
endpoint_url=teacher_url,
@@ -630,12 +668,6 @@ async def run_teacher_inline(
messages=teacher_messages,
headers=teacher_headers,
owner=owner,
session_id=session_id,
workspace=workspace,
disabled_tools=disabled_tools,
tool_policy=tool_policy,
active_document=active_document,
active_email=active_email,
_is_teacher_run=True,
):
# Swallow teacher's own [DONE] — outer loop emits the real one
@@ -650,21 +682,13 @@ async def run_teacher_inline(
if isinstance(payload, dict):
payload["teacher"] = True
typ = payload.get("type")
if typ == "metrics" and isinstance(payload.get("data"), dict):
# The outer chat route persists only the last metrics
# payload. Keep a copy so any approval produced after the
# recursive teacher run's metrics remains reloadable.
captured_metrics = dict(payload["data"])
if typ == "tool_output":
captured_tool_event = {
captured_tool_events.append({
"tool": payload.get("tool"),
"command": payload.get("command"),
"output": payload.get("output"),
"exit_code": payload.get("exit_code"),
}
if isinstance(payload.get("ask_user"), dict):
captured_tool_event["ask_user"] = payload["ask_user"]
captured_tool_events.append(captured_tool_event)
})
if "delta" in payload and isinstance(payload["delta"], str):
if payload.get("thinking"):
continue
@@ -673,12 +697,6 @@ async def run_teacher_inline(
continue
yield evt_str
# A takeover that paused for a question or exact action has not completed
# yet. Its server-owned approval card is already in the live/persisted tool
# events; do not evaluate the partial trace or distill it into a skill.
if any(event.get("ask_user") for event in captured_tool_events):
return
teacher_text = "".join(captured_text_parts).strip()
t_status, t_reason = evaluate_turn_regex(captured_tool_events, teacher_text)
if t_status == "failure":
@@ -722,85 +740,31 @@ async def run_teacher_inline(
skill.setdefault("source", "teacher-escalation")
skill.setdefault("teacher_model", teacher_spec)
if not session_id:
import json as _json
from src.tool_implementations import do_manage_skills
try:
result = await do_manage_skills(_json.dumps(skill), owner=owner)
if isinstance(result, dict) and not result.get("error"):
logger.info(f"teacher succeeded; saved skill: {skill.get('name')}")
yield (
'data: ' + json.dumps({
"type": "skill_saved",
"name": skill.get("name"),
"category": skill.get("category", "general"),
}) + '\n\n'
)
else:
yield (
'data: ' + json.dumps({
"type": "skill_save_failed",
"reason": str(result),
}) + '\n\n'
)
except Exception as e:
logger.warning(f"skill save raised: {e}")
yield (
'data: ' + json.dumps({
"type": "skill_save_failed",
"reason": (
"Teacher-generated skills require an interactive exact "
"approval before they can be saved."
),
"reason": str(e),
}) + '\n\n'
)
return
import json as _json
import uuid as _uuid
from src.tool_approvals import tool_approval_store
from src.tool_capabilities import capabilities_for_action
skill_content = _json.dumps(skill, ensure_ascii=False)
pending = tool_approval_store.create(
owner=owner,
session_id=session_id,
origin_run_id=f"teacher-skill-{_uuid.uuid4().hex}",
tool_name="manage_skills",
content=skill_content,
workspace=workspace,
external_untrusted_context_seen=True,
capabilities=capabilities_for_action("manage_skills", skill_content),
)
approval = pending.public_payload(
reason=(
"The teacher generated this reusable skill. Review and approve "
"the complete skill definition before it is saved."
),
)
persisted_metrics = dict(captured_metrics)
persisted_tool_events = list(persisted_metrics.get("tool_events") or [])
persisted_round_texts = list(persisted_metrics.get("round_texts") or [])
prior_rounds = [
event.get("round")
for event in persisted_tool_events
if isinstance(event, dict) and isinstance(event.get("round"), int)
]
approval_round = max([len(persisted_round_texts), *prior_rounds, 0]) + 1
approval_tool_event = {
"round": approval_round,
"model": teacher_model,
"tool": "manage_skills",
"command": str(skill.get("name") or "teacher-generated skill"),
"output": "Waiting for an exact user approval.",
"exit_code": None,
"ask_user": approval,
}
persisted_tool_events.append(approval_tool_event)
persisted_metrics["tool_events"] = persisted_tool_events
persisted_metrics.setdefault("model", teacher_model)
yield (
"data: "
+ json.dumps({"delta": "Review the teacher-generated skill before saving it."})
+ "\n\n"
)
yield (
"data: "
+ json.dumps({
"type": "tool_output",
**approval_tool_event,
"teacher": True,
})
+ "\n\n"
)
yield (
"data: "
+ json.dumps({"type": "ask_user", "data": approval, "teacher": True})
+ "\n\n"
)
# This must be the final metrics event: chat_routes saves only last_metrics
# when the outer stream reaches [DONE]. Without it, the live approval card
# disappears after a reload even though the server grant remains pending.
yield (
"data: "
+ json.dumps({"type": "metrics", "data": persisted_metrics, "teacher": True})
+ "\n\n"
)
-392
View File
@@ -1,392 +0,0 @@
"""Opaque, exact, one-use approvals for tainted model-requested actions.
The model may propose an action after untrusted context, but only the server
stores and later executes the exact approved tool input. Browser-visible
fields are display copies, never authority.
"""
from __future__ import annotations
import hashlib
import json
import os
import secrets
import threading
import time
from dataclasses import dataclass, field
from typing import Any
from src.tool_capabilities import ToolCapabilities, capabilities_for_action
DEFAULT_APPROVAL_TTL_SECONDS = 10 * 60
DEFAULT_MAX_PENDING_APPROVALS = 2048
def _normalized_owner(owner: Any) -> str:
return str(owner or "").strip().casefold()
def _normalized_workspace(workspace: Any) -> str:
if not isinstance(workspace, str) or not workspace.strip():
return ""
return os.path.realpath(os.path.expanduser(workspace))
def _canonical_digest(payload: dict[str, Any]) -> str:
encoded = json.dumps(
payload,
sort_keys=True,
separators=(",", ":"),
ensure_ascii=False,
).encode("utf-8")
return hashlib.sha256(encoded).hexdigest()
def document_content_digest(content: Any) -> str:
"""Return the stable server-side fingerprint used to seal a document."""
return hashlib.sha256(str(content or "").encode("utf-8")).hexdigest()
def _binding_payload(
*,
owner: Any,
session_id: Any,
origin_run_id: Any,
tool_name: Any,
content: Any,
workspace: Any,
document_id: Any,
document_version: Any,
document_digest: Any,
external_untrusted_context_seen: bool,
effects: tuple[str, ...],
result_integrity: str,
) -> dict[str, Any]:
return {
"owner": _normalized_owner(owner),
"session_id": str(session_id or ""),
"origin_run_id": str(origin_run_id or ""),
"tool_name": str(tool_name or ""),
"content": str(content or ""),
"workspace": _normalized_workspace(workspace),
"document_id": str(document_id or ""),
"document_version": (
int(document_version) if document_version is not None else None
),
"document_digest": str(document_digest or "").strip().lower(),
"external_untrusted_context_seen": bool(external_untrusted_context_seen),
"effects": list(effects),
"result_integrity": str(result_integrity),
}
@dataclass(frozen=True)
class PendingToolApproval:
approval_id: str
owner: str
session_id: str
origin_run_id: str
tool_name: str
content: str
workspace: str
document_id: str
document_version: int | None
document_digest: str
external_untrusted_context_seen: bool
effects: tuple[str, ...]
result_integrity: str
digest: str
created_at: float
expires_at: float
def public_payload(self, *, reason: str | None = None) -> dict[str, Any]:
return {
"kind": "tool_approval",
"approval_id": self.approval_id,
"question": "Allow this exact action once?",
"description": reason or (
"Untrusted context influenced this run, so this action needs "
"your explicit approval."
),
"options": [
{
"label": "Allow once",
"value": "approve",
"description": "Execute only the sealed action shown here.",
},
{
"label": "Deny",
"value": "deny",
"description": "Do not execute it.",
},
],
"action": {
"tool": self.tool_name,
# Show the complete sealed input so approval never hides
# trailing lines. This is not read back as authority.
"content": self.content,
"digest": self.digest[:16],
"effects": list(self.effects),
"workspace": self.workspace or None,
"document_id": self.document_id or None,
"document_version": self.document_version,
},
}
@dataclass
class ExactToolApproval:
"""A consumed grant that the dispatcher can claim exactly once."""
pending: PendingToolApproval
_claimed: bool = field(default=False, init=False, repr=False)
_lock: threading.Lock = field(default_factory=threading.Lock, init=False, repr=False)
def _matches_unlocked(
self,
*,
owner: Any,
session_id: Any,
tool_name: Any,
content: Any,
workspace: Any,
) -> bool:
if self._claimed:
return False
capabilities = capabilities_for_action(tool_name, content)
effects = tuple(sorted(effect.value for effect in capabilities.effects))
result_integrity = capabilities.result_integrity.value
if (
effects != self.pending.effects
or result_integrity != self.pending.result_integrity
):
return False
expected = _binding_payload(
owner=owner,
session_id=session_id,
origin_run_id=self.pending.origin_run_id,
tool_name=tool_name,
content=content,
workspace=workspace,
document_id=self.pending.document_id,
document_version=self.pending.document_version,
document_digest=self.pending.document_digest,
external_untrusted_context_seen=(
self.pending.external_untrusted_context_seen
),
effects=effects,
result_integrity=result_integrity,
)
return _canonical_digest(expected) == self.pending.digest
def matches(
self,
*,
owner: Any,
session_id: Any,
tool_name: Any,
content: Any,
workspace: Any,
) -> bool:
with self._lock:
return self._matches_unlocked(
owner=owner,
session_id=session_id,
tool_name=tool_name,
content=content,
workspace=workspace,
)
def claim(
self,
*,
owner: Any,
session_id: Any,
tool_name: Any,
content: Any,
workspace: Any,
) -> bool:
with self._lock:
if not self._matches_unlocked(
owner=owner,
session_id=session_id,
tool_name=tool_name,
content=content,
workspace=workspace,
):
return False
self._claimed = True
return True
class ToolApprovalStore:
"""Thread-safe pending approval registry with destructive consumption."""
def __init__(
self,
*,
ttl_seconds: int = DEFAULT_APPROVAL_TTL_SECONDS,
max_pending: int = DEFAULT_MAX_PENDING_APPROVALS,
):
self._ttl_seconds = max(1, int(ttl_seconds))
self._max_pending = max(1, int(max_pending))
self._pending: dict[str, PendingToolApproval] = {}
self._lock = threading.Lock()
def _purge_expired_locked(self, now: float) -> None:
expired = [
approval_id
for approval_id, pending in self._pending.items()
if pending.expires_at <= now
]
for approval_id in expired:
self._pending.pop(approval_id, None)
def create(
self,
*,
owner: Any,
session_id: Any,
origin_run_id: Any,
tool_name: Any,
content: Any,
workspace: Any,
document_id: Any = None,
document_version: Any = None,
document_digest: Any = None,
external_untrusted_context_seen: bool,
capabilities: ToolCapabilities,
) -> PendingToolApproval:
now = time.time()
effects = tuple(sorted(effect.value for effect in capabilities.effects))
result_integrity = capabilities.result_integrity.value
payload = _binding_payload(
owner=owner,
session_id=session_id,
origin_run_id=origin_run_id,
tool_name=tool_name,
content=content,
workspace=workspace,
document_id=document_id,
document_version=document_version,
document_digest=document_digest,
external_untrusted_context_seen=external_untrusted_context_seen,
effects=effects,
result_integrity=result_integrity,
)
pending = PendingToolApproval(
approval_id=secrets.token_urlsafe(32),
owner=payload["owner"],
session_id=payload["session_id"],
origin_run_id=payload["origin_run_id"],
tool_name=payload["tool_name"],
content=payload["content"],
workspace=payload["workspace"],
document_id=payload["document_id"],
document_version=payload["document_version"],
document_digest=payload["document_digest"],
external_untrusted_context_seen=payload[
"external_untrusted_context_seen"
],
effects=effects,
result_integrity=result_integrity,
digest=_canonical_digest(payload),
created_at=now,
expires_at=now + self._ttl_seconds,
)
with self._lock:
self._purge_expired_locked(now)
# The chat UI exposes one pending card per session, so supersede an
# older action there. Headless/manual-test callers use an empty
# session id; keep independent origin runs separate so two skill
# tests owned by the same user cannot invalidate each other.
superseded = [
approval_id
for approval_id, existing in self._pending.items()
if (
existing.owner == pending.owner
and existing.session_id == pending.session_id
and (
bool(pending.session_id)
or existing.origin_run_id == pending.origin_run_id
)
)
]
for approval_id in superseded:
self._pending.pop(approval_id, None)
while len(self._pending) >= self._max_pending:
oldest_id = min(
self._pending,
key=lambda approval_id: self._pending[approval_id].created_at,
)
self._pending.pop(oldest_id, None)
self._pending[pending.approval_id] = pending
return pending
def consume(
self,
approval_id: Any,
*,
decision: Any,
owner: Any,
session_id: Any,
) -> ExactToolApproval | None:
now = time.time()
with self._lock:
self._purge_expired_locked(now)
approval_key = str(approval_id or "")
pending = self._pending.get(approval_key)
if pending is None:
return None
if (
pending.owner != _normalized_owner(owner)
or pending.session_id != str(session_id or "")
):
# Authentication is checked before destructive consumption so
# a leaked/guessed opaque id cannot be used to invalidate
# another owner's pending action.
return None
self._pending.pop(approval_key, None)
if str(decision or "").strip().lower() != "approve":
return None
return ExactToolApproval(pending)
def peek(self, approval_id: Any) -> PendingToolApproval | None:
now = time.time()
with self._lock:
self._purge_expired_locked(now)
return self._pending.get(str(approval_id or ""))
def retire_for_session(self, *, owner: Any, session_id: Any) -> bool:
"""Discard pending actions superseded by an ordinary user turn.
Returns whether any retired action carried external provenance, so the
caller can preserve that security state without treating the new user
message as an approval continuation.
"""
now = time.time()
normalized_owner = _normalized_owner(owner)
normalized_session = str(session_id or "")
if not normalized_session:
return False
with self._lock:
self._purge_expired_locked(now)
retired_ids = [
approval_id
for approval_id, pending in self._pending.items()
if (
pending.owner == normalized_owner
and pending.session_id == normalized_session
)
]
carried_taint = any(
self._pending[approval_id].external_untrusted_context_seen
for approval_id in retired_ids
)
for approval_id in retired_ids:
self._pending.pop(approval_id, None)
return carried_taint
tool_approval_store = ToolApprovalStore()
-668
View File
@@ -1,668 +0,0 @@
"""Deterministic capability metadata for agent tools.
Model output requests an action; it never supplies the authority for that
action. This module classifies the effects of each built-in tool and applies
run-local integrity gates before dispatch.
"""
from __future__ import annotations
import json
import uuid
from dataclasses import dataclass, field
from enum import Enum
from types import MappingProxyType
from typing import Any, Iterable, Mapping
from src.tool_security import BUILTIN_EMAIL_TOOLS
class ToolEffect(str, Enum):
READ_PUBLIC = "read_public"
READ_WORKSPACE = "read_workspace"
READ_PRIVATE = "read_private"
WRITE_WORKSPACE = "write_workspace"
WRITE_PRIVATE = "write_private"
EXECUTE_CODE = "execute_code"
BROKERED_NETWORK_READ = "brokered_network_read"
NETWORK_EGRESS = "network_egress"
EXTERNAL_SIDE_EFFECT = "external_side_effect"
UI_SIDE_EFFECT = "ui_side_effect"
ADMIN_CHANGE = "admin_change"
DESTRUCTIVE = "destructive"
USER_INTERACTION = "user_interaction"
class ResultIntegrity(str, Enum):
SYSTEM = "system"
WORKSPACE_UNTRUSTED = "workspace_untrusted"
EXTERNAL_UNTRUSTED = "external_untrusted"
@dataclass(frozen=True)
class ToolCapabilities:
effects: frozenset[ToolEffect]
result_integrity: ResultIntegrity = ResultIntegrity.SYSTEM
known: bool = True
def _capabilities(
*effects: ToolEffect,
result_integrity: ResultIntegrity = ResultIntegrity.SYSTEM,
) -> ToolCapabilities:
return ToolCapabilities(frozenset(effects), result_integrity)
_REGISTRY: dict[str, ToolCapabilities] = {}
def _register(
names: Iterable[str],
*effects: ToolEffect,
result_integrity: ResultIntegrity = ResultIntegrity.SYSTEM,
) -> None:
capabilities = _capabilities(*effects, result_integrity=result_integrity)
for name in names:
if name in _REGISTRY:
raise RuntimeError(f"Duplicate tool capability classification: {name}")
_REGISTRY[name] = capabilities
_register(
{"ask_user", "update_plan"},
ToolEffect.USER_INTERACTION,
)
_register(
{
"list_cached_models",
"list_cookbook_servers",
"list_downloads",
"list_models",
"list_serve_presets",
"list_served_models",
},
ToolEffect.READ_PRIVATE,
# These readers return provider-controlled model identifiers or durable
# user/admin-authored Cookbook and process state. Local brokering does not
# make the returned text server-authored.
result_integrity=ResultIntegrity.EXTERNAL_UNTRUSTED,
)
_register(
{"search_hf_models"},
ToolEffect.BROKERED_NETWORK_READ,
result_integrity=ResultIntegrity.EXTERNAL_UNTRUSTED,
)
_register(
{"get_workspace", "glob", "grep", "ls", "read_file"},
ToolEffect.READ_WORKSPACE,
result_integrity=ResultIntegrity.WORKSPACE_UNTRUSTED,
)
_register(
{"web_search"},
ToolEffect.BROKERED_NETWORK_READ,
result_integrity=ResultIntegrity.EXTERNAL_UNTRUSTED,
)
_register(
{"web_fetch"},
ToolEffect.BROKERED_NETWORK_READ,
ToolEffect.NETWORK_EGRESS,
result_integrity=ResultIntegrity.EXTERNAL_UNTRUSTED,
)
_register(
{
"list_email_accounts",
"list_emails",
"read_email",
"resolve_contact",
"scan_email_unsubscribes",
"search_chats",
"search_emails",
"list_sessions",
"tail_serve_output",
"vault_get",
"vault_search",
},
ToolEffect.READ_PRIVATE,
result_integrity=ResultIntegrity.EXTERNAL_UNTRUSTED,
)
_register(
{"bash", "manage_bg_jobs", "python"},
ToolEffect.EXECUTE_CODE,
result_integrity=ResultIntegrity.WORKSPACE_UNTRUSTED,
)
_register(
{"apply_patch", "edit_file", "write_file"},
ToolEffect.WRITE_WORKSPACE,
# Successful writes include unified diffs that can echo arbitrary existing
# workspace content back into the next model round.
result_integrity=ResultIntegrity.WORKSPACE_UNTRUSTED,
)
_register(
{
"create_document",
"manage_calendar",
"manage_contact",
"manage_documents",
"manage_memory",
"manage_notes",
"manage_research",
"manage_session",
"manage_skills",
"manage_tasks",
"suggest_document",
"todowrite",
},
ToolEffect.WRITE_PRIVATE,
)
_register(
{
"ai_draft_email_reply",
"create_session",
"draft_email",
"draft_email_reply",
},
ToolEffect.WRITE_PRIVATE,
# These tools resolve user-configured endpoints/accounts or read stored
# email content before returning model-visible status text.
result_integrity=ResultIntegrity.EXTERNAL_UNTRUSTED,
)
_register(
{"edit_document", "update_document"},
ToolEffect.WRITE_PRIVATE,
# These tools can echo stored document content that was not present in
# their arguments. edit_document returns the complete edited document;
# update_document also preserves stored email headers/thread history.
result_integrity=ResultIntegrity.EXTERNAL_UNTRUSTED,
)
_register(
{"pipeline"},
ToolEffect.NETWORK_EGRESS,
result_integrity=ResultIntegrity.EXTERNAL_UNTRUSTED,
)
_register(
{"send_to_session"},
ToolEffect.NETWORK_EGRESS,
ToolEffect.WRITE_PRIVATE,
result_integrity=ResultIntegrity.EXTERNAL_UNTRUSTED,
)
_register(
{"chat_with_model", "ask_teacher"},
ToolEffect.NETWORK_EGRESS,
result_integrity=ResultIntegrity.EXTERNAL_UNTRUSTED,
)
_register(
{"download_attachment"},
ToolEffect.READ_PRIVATE,
ToolEffect.WRITE_WORKSPACE,
result_integrity=ResultIntegrity.EXTERNAL_UNTRUSTED,
)
_register(
{"edit_image", "generate_image", "trigger_research"},
ToolEffect.NETWORK_EGRESS,
ToolEffect.WRITE_PRIVATE,
result_integrity=ResultIntegrity.EXTERNAL_UNTRUSTED,
)
_register(
{
"archive_email",
"bulk_email",
"mark_email_read",
"reply_to_email",
"send_email",
"unsubscribe_email",
},
ToolEffect.EXTERNAL_SIDE_EFFECT,
# Email action results can include stored headers/account labels or remote
# SMTP/IMAP responses, even when the action itself succeeded.
result_integrity=ResultIntegrity.EXTERNAL_UNTRUSTED,
)
_register(
{"delete_email"},
ToolEffect.EXTERNAL_SIDE_EFFECT,
ToolEffect.DESTRUCTIVE,
result_integrity=ResultIntegrity.EXTERNAL_UNTRUSTED,
)
_register(
{"ui_control"},
ToolEffect.UI_SIDE_EFFECT,
# Model switches and custom-theme validation read mutable user settings.
result_integrity=ResultIntegrity.EXTERNAL_UNTRUSTED,
)
_register(
{
"adopt_served_model",
"cancel_download",
"download_model",
"serve_model",
"serve_preset",
"stop_served_model",
"vault_unlock",
},
ToolEffect.ADMIN_CHANGE,
# Cookbook/process operations can return stored presets, provider data,
# remote shell output, and command errors.
result_integrity=ResultIntegrity.EXTERNAL_UNTRUSTED,
)
_register(
{
"api_call",
"app_api",
"manage_endpoints",
"manage_mcp",
"manage_settings",
"manage_tokens",
"manage_webhooks",
},
ToolEffect.ADMIN_CHANGE,
# api_call/app_api return remote or stored application data, and the
# admin managers can echo user-controlled configuration. Conservatively
# retain the action effect while treating every successful result as data.
result_integrity=ResultIntegrity.EXTERNAL_UNTRUSTED,
)
TOOL_CAPABILITIES: Mapping[str, ToolCapabilities] = MappingProxyType(dict(_REGISTRY))
KNOWN_CAPABILITY_TOOLS = frozenset(TOOL_CAPABILITIES)
_UNKNOWN_CAPABILITIES = _capabilities(
ToolEffect.READ_PRIVATE,
ToolEffect.WRITE_WORKSPACE,
ToolEffect.WRITE_PRIVATE,
ToolEffect.EXECUTE_CODE,
ToolEffect.NETWORK_EGRESS,
ToolEffect.EXTERNAL_SIDE_EFFECT,
ToolEffect.ADMIN_CHANGE,
ToolEffect.DESTRUCTIVE,
result_integrity=ResultIntegrity.EXTERNAL_UNTRUSTED,
)
_UNKNOWN_CAPABILITIES = ToolCapabilities(
_UNKNOWN_CAPABILITIES.effects,
_UNKNOWN_CAPABILITIES.result_integrity,
known=False,
)
_BROWSER_MCP_READ_CAPABILITIES = _capabilities(
ToolEffect.BROKERED_NETWORK_READ,
result_integrity=ResultIntegrity.EXTERNAL_UNTRUSTED,
)
_BROWSER_MCP_READ_TOOLS = frozenset(
{
"mcp__builtin_browser__browser_console_messages",
"mcp__builtin_browser__browser_network_requests",
"mcp__builtin_browser__browser_snapshot",
"mcp__builtin_browser__browser_take_screenshot",
}
)
def capabilities_for_tool(tool_name: Any) -> ToolCapabilities:
"""Return deterministic capabilities; malformed and unknown tools fail high."""
if not isinstance(tool_name, str) or not tool_name:
return _UNKNOWN_CAPABILITIES
capabilities = TOOL_CAPABILITIES.get(tool_name)
if capabilities is not None:
return capabilities
if tool_name.startswith("mcp__email__"):
bare_name = tool_name[len("mcp__email__"):]
capabilities = TOOL_CAPABILITIES.get(bare_name)
if bare_name in BUILTIN_EMAIL_TOOLS and capabilities is not None:
return capabilities
if tool_name in _BROWSER_MCP_READ_TOOLS:
return _BROWSER_MCP_READ_CAPABILITIES
return _UNKNOWN_CAPABILITIES
_PRIVATE_ACTION_READS: Mapping[str, frozenset[str]] = MappingProxyType(
{
"manage_calendar": frozenset({"list_calendars", "list_events"}),
"manage_contact": frozenset({"list"}),
"manage_documents": frozenset({"list", "read", "view", "open", "get"}),
"manage_memory": frozenset({"list", "search"}),
"manage_notes": frozenset({"list", "search", "find", "view"}),
"manage_research": frozenset({"list", "read", "open", "view", "get"}),
"manage_session": frozenset({"list", "switch", "open", "select", "view"}),
"manage_skills": frozenset({"list", "index", "view", "view_ref", "search"}),
"manage_tasks": frozenset({"list"}),
}
)
_PRIVATE_ACTION_WRITES: Mapping[str, frozenset[str]] = MappingProxyType(
{
"manage_calendar": frozenset(
{"create_event", "update_event", "delete_event"}
),
"manage_contact": frozenset({"add", "update", "edit", "delete"}),
"manage_documents": frozenset({"delete", "tidy"}),
"manage_memory": frozenset({"add", "edit", "delete"}),
"manage_notes": frozenset({"add", "update", "delete", "toggle_item"}),
"manage_research": frozenset({"delete"}),
"manage_session": frozenset(
{
"rename",
"archive",
"unarchive",
"delete",
"important",
"unimportant",
"truncate",
"fork",
}
),
"manage_skills": frozenset({"add", "edit", "patch", "publish", "delete"}),
"manage_tasks": frozenset({"create", "edit", "delete", "pause", "resume", "run"}),
}
)
_ACTION_DESTRUCTIVE: Mapping[str, frozenset[str]] = MappingProxyType(
{
"manage_calendar": frozenset({"delete_event"}),
"manage_contact": frozenset({"delete"}),
"manage_documents": frozenset({"delete", "tidy"}),
"manage_endpoints": frozenset({"delete"}),
"manage_bg_jobs": frozenset({"kill", "stop", "cancel", "terminate"}),
"manage_memory": frozenset({"delete"}),
"manage_mcp": frozenset({"delete"}),
"manage_notes": frozenset({"delete"}),
"manage_research": frozenset({"delete"}),
"manage_session": frozenset({"delete", "truncate"}),
"manage_settings": frozenset({"delete", "reset"}),
"manage_skills": frozenset({"delete"}),
"manage_tasks": frozenset({"delete"}),
"manage_tokens": frozenset({"delete"}),
"manage_webhooks": frozenset({"delete"}),
}
)
_ACTION_DEFAULTS: Mapping[str, str] = MappingProxyType(
{
"manage_calendar": "list_events",
"manage_documents": "list",
"manage_research": "list",
"manage_tasks": "list",
}
)
_ACTION_ALIASES: Mapping[str, Mapping[str, str]] = MappingProxyType(
{
"manage_calendar": MappingProxyType(
{
"create": "create_event",
"update": "update_event",
"delete": "delete_event",
"list": "list_events",
}
),
"manage_notes": MappingProxyType(
{
"create": "add",
"new": "add",
"save": "add",
"remind": "add",
"reminder": "add",
"remove": "delete",
"remove_item": "toggle_item",
}
),
}
)
_LINE_ACTION_TOOLS = frozenset({"manage_memory", "manage_session"})
def _action_from_content(tool_name: str, content: Any) -> str | None:
"""Extract the action discriminator using the same accepted input shapes."""
if isinstance(content, Mapping):
payload: Any = dict(content)
elif isinstance(content, str):
raw = content.strip()
if tool_name in _LINE_ACTION_TOOLS and raw and not raw.startswith("{"):
return raw.splitlines()[0].strip().replace("-", "_").casefold() or None
try:
payload = json.loads(raw) if raw else {}
except (TypeError, ValueError):
return None
else:
payload = {}
if not isinstance(payload, dict):
return None
if (
len(payload) == 1
and isinstance(payload.get("body"), dict)
and "action" in payload["body"]
):
payload = payload["body"]
action = payload.get("action")
if (
not action
and tool_name == "manage_calendar"
and isinstance(payload.get("events"), list)
):
action = "create_event"
if not action and tool_name == "manage_tasks" and any(
payload.get(key) is not None
for key in ("task", "description", "schedule", "time", "day_of_week")
):
action = "create"
if not isinstance(action, str) or not action.strip():
action = _ACTION_DEFAULTS.get(tool_name)
if not action:
return None
normalized = action.strip().replace("-", "_").casefold()
return _ACTION_ALIASES.get(tool_name, {}).get(normalized, normalized)
def capabilities_for_action(tool_name: Any, content: Any) -> ToolCapabilities:
"""Classify a sealed multiplexed action; ambiguous actions fail high."""
base = capabilities_for_tool(tool_name)
if not isinstance(tool_name, str):
return base
action = _action_from_content(tool_name, content)
destructive = action in _ACTION_DESTRUCTIVE.get(tool_name, ())
if tool_name not in _PRIVATE_ACTION_READS:
if not destructive:
return base
return ToolCapabilities(
frozenset(set(base.effects) | {ToolEffect.DESTRUCTIVE}),
base.result_integrity,
known=base.known,
)
if action in _PRIVATE_ACTION_READS[tool_name]:
return _capabilities(
ToolEffect.READ_PRIVATE,
result_integrity=ResultIntegrity.EXTERNAL_UNTRUSTED,
)
if action in _PRIVATE_ACTION_WRITES[tool_name]:
effects = set(base.effects)
if destructive:
effects.add(ToolEffect.DESTRUCTIVE)
return ToolCapabilities(
frozenset(effects),
ResultIntegrity.EXTERNAL_UNTRUSTED,
known=base.known,
)
return _capabilities(
ToolEffect.READ_PRIVATE,
ToolEffect.WRITE_PRIVATE,
result_integrity=ResultIntegrity.EXTERNAL_UNTRUSTED,
)
def tool_result_is_successful(result: Any) -> bool:
"""Return whether a result actually introduced successful tool output."""
return bool(
isinstance(result, dict)
and not result.get("blocked")
and not result.get("approval_required")
and not result.get("error")
and result.get("exit_code") in (None, 0)
and result.get("success") is not False
)
def tool_result_should_arm_gate(
tool_name: Any,
result: Any,
content: Any = None,
) -> bool:
"""Return whether a result introduced non-system content to the model.
A blocked/approval placeholder and a genuinely content-free failure do not
change authority. Once a non-system tool returns text or structured data
that will be folded into model context, however, failure status cannot make
that payload trusted: MCP ``isError`` text, provider exception messages,
and HTTP error bodies are all attacker-controlled input surfaces.
"""
if not isinstance(result, dict):
return False
if result.get("blocked") or result.get("approval_required"):
return False
# A producer that knows a particular response body came from a remote or
# stored source overrides a coarse static SYSTEM default.
if result.get("untrusted_content") is True:
return True
capabilities = capabilities_for_action(tool_name, content)
if capabilities.result_integrity is ResultIntegrity.SYSTEM:
return False
if tool_result_is_successful(result):
return True
# ``format_tool_result`` serializes every additional structured field, so
# a fixed allowlist here would inevitably miss model-visible payloads such
# as ``details``, ``events``, or provider-specific response keys. Exclude
# only status/policy controls that carry no producer content; any other
# non-empty field crosses the same integrity boundary even on failure.
non_content_keys = frozenset(
{
"approval_required",
"blocked",
"exit_code",
"policy",
"success",
"untrusted_content",
}
)
return any(
key not in non_content_keys and value not in (None, "", [], {}, ())
for key, value in result.items()
)
POST_EXTERNAL_BLOCKED_EFFECTS = frozenset(
{
ToolEffect.READ_PRIVATE,
ToolEffect.WRITE_WORKSPACE,
ToolEffect.WRITE_PRIVATE,
ToolEffect.EXECUTE_CODE,
ToolEffect.NETWORK_EGRESS,
ToolEffect.EXTERNAL_SIDE_EFFECT,
ToolEffect.UI_SIDE_EFFECT,
ToolEffect.ADMIN_CHANGE,
ToolEffect.DESTRUCTIVE,
}
)
@dataclass(frozen=True)
class ToolGateDecision:
allowed: bool
reason: str | None = None
_EXTERNAL_MESSAGE_SOURCES = frozenset(
{
"injected research context",
"prefetched search context",
"research context",
"web search results",
"youtube transcript",
}
)
_EXTERNAL_MESSAGE_SOURCE_PREFIXES = ("web page:",)
def messages_contain_external_untrusted_context(messages: Iterable[dict]) -> bool:
"""Detect explicitly labelled external context already present in a run."""
for message in messages or ():
if not isinstance(message, dict):
continue
metadata = message.get("metadata")
if not isinstance(metadata, dict) or metadata.get("trusted") is not False:
continue
gate_marker = metadata.get("tool_gate_untrusted")
if gate_marker is True:
return True
if gate_marker is False:
# Explicit current-format opt-outs are authoritative. The source
# label heuristics below exist only for older saved wrappers that
# predate the marker.
continue
if metadata.get("provenance_origin") == "external":
return True
source = metadata.get("source")
if not isinstance(source, str):
continue
normalized_source = source.strip().casefold()
if normalized_source in _EXTERNAL_MESSAGE_SOURCES:
return True
if normalized_source.startswith(_EXTERNAL_MESSAGE_SOURCE_PREFIXES):
return True
return False
@dataclass
class ToolRunSecurityContext:
"""Server-owned integrity state for one agent run."""
external_untrusted_context_seen: bool = False
external_sources: list[str] = field(default_factory=list)
run_id: str = field(default_factory=lambda: uuid.uuid4().hex)
def observe_messages(self, messages: Iterable[dict]) -> None:
"""Promote any server-labelled untrusted prompt context into the gate."""
if messages_contain_external_untrusted_context(messages):
self.external_untrusted_context_seen = True
def decision_for(self, tool_name: Any, content: Any = None) -> ToolGateDecision:
if not self.external_untrusted_context_seen:
return ToolGateDecision(True)
capabilities = capabilities_for_action(tool_name, content)
blocked_effects = capabilities.effects & POST_EXTERNAL_BLOCKED_EFFECTS
if capabilities.known and not blocked_effects:
return ToolGateDecision(True)
effects = ", ".join(sorted(effect.value for effect in blocked_effects))
if not capabilities.known:
effects = "unknown/high-impact"
return ToolGateDecision(
False,
(
"External untrusted context has already influenced this run. "
f"Tool '{tool_name}' requires a separate user-authorized action "
f"because it can cause {effects}."
),
)
def observe_tool_result(
self,
tool_name: Any,
result: Any,
content: Any = None,
) -> None:
if not tool_result_should_arm_gate(tool_name, result, content):
return
self.external_untrusted_context_seen = True
if isinstance(tool_name, str) and tool_name not in self.external_sources:
self.external_sources.append(tool_name)
def blocked_tool_result(tool_name: Any, reason: str) -> tuple[str, dict]:
return (
f"{tool_name}: BLOCKED",
{
"error": reason,
"exit_code": 1,
"blocked": True,
"policy": "external_untrusted_context",
},
)
+2 -161
View File
@@ -27,24 +27,10 @@ from src.tool_security import (
is_public_blocked_tool,
owner_is_admin_or_single_user,
)
from src.tool_capabilities import ToolRunSecurityContext, blocked_tool_result
from src.tool_approvals import ExactToolApproval
from src.tool_policy import ToolPolicy
from src.constants import MAX_OUTPUT_CHARS, MAX_READ_CHARS, MAX_DIFF_LINES, DATA_DIR
from src.tool_utils import _truncate, get_mcp_manager
class _MissingToolSecurityContext:
pass
class _NoToolSecurityContext:
"""Explicit sentinel for non-agent callers that have no run provenance."""
_MISSING_TOOL_SECURITY_CONTEXT = _MissingToolSecurityContext()
NO_TOOL_SECURITY_CONTEXT = _NoToolSecurityContext()
# Persistent working directory for agent subprocesses.
# Resolves to <repo_root>/data, which is the bind-mounted volume in Docker
# (/app/data) and the local data directory for manual installs.
@@ -568,19 +554,10 @@ async def _document_tool_dispatch(
content: str,
session_id: Optional[str] = None,
owner: Optional[str] = None,
document_id: Optional[str] = None,
document_version: Optional[int] = None,
document_digest: Optional[str] = None,
) -> Optional[Dict]:
"""Route a document tool through TOOL_HANDLERS with the right ctx shape."""
from src.agent_tools import TOOL_HANDLERS
ctx = {
"session_id": session_id,
"owner": owner,
"doc_id": document_id,
"expected_document_version": document_version,
"expected_document_digest": document_digest,
}
ctx = {"session_id": session_id, "owner": owner}
if tool in TOOL_HANDLERS:
return await TOOL_HANDLERS[tool](content, ctx)
return None
@@ -598,12 +575,6 @@ async def execute_tool_block(
progress_cb: Optional[Callable[[Dict], Awaitable[None]]] = None,
workspace: Optional[str] = None,
tool_policy: Optional[Any] = None,
security_context: (
ToolRunSecurityContext
| _NoToolSecurityContext
| _MissingToolSecurityContext
) = _MISSING_TOOL_SECURITY_CONTEXT,
exact_approval: Optional[ExactToolApproval] = None,
) -> Tuple[str, Dict]:
"""Execute a single tool block. Returns (description, result_dict).
@@ -611,104 +582,6 @@ async def execute_tool_block(
cwd confine to it) for the duration of this call, then delegate. Reset on the
way out so the binding never leaks to the next tool call.
"""
if security_context is _MISSING_TOOL_SECURITY_CONTEXT:
raise TypeError(
"execute_tool_block requires security_context; pass a "
"ToolRunSecurityContext or NO_TOOL_SECURITY_CONTEXT explicitly"
)
if (
not isinstance(security_context, ToolRunSecurityContext)
and security_context is not NO_TOOL_SECURITY_CONTEXT
):
raise TypeError(
"security_context must be a ToolRunSecurityContext or "
"NO_TOOL_SECURITY_CONTEXT"
)
approval_claimed = False
if exact_approval is not None:
if (
not isinstance(security_context, ToolRunSecurityContext)
or not security_context.external_untrusted_context_seen
or not exact_approval.pending.external_untrusted_context_seen
):
return (
f"{getattr(block, 'tool_type', None)}: BLOCKED",
{
"error": "Exact-action approval requires an armed run security context.",
"exit_code": 1,
"blocked": True,
"policy": "exact_tool_approval",
},
)
if (
exact_approval.pending.tool_name
in {"edit_document", "suggest_document", "update_document"}
and (
not exact_approval.pending.document_id
or exact_approval.pending.document_version is None
or not exact_approval.pending.document_digest
)
):
return (
f"{getattr(block, 'tool_type', None)}: BLOCKED",
{
"error": (
"The approved document action has no sealed target and "
"cannot be executed."
),
"exit_code": 1,
"blocked": True,
"policy": "exact_tool_approval",
},
)
sealed_workspace = exact_approval.pending.workspace
if sealed_workspace and vet_workspace(sealed_workspace) != sealed_workspace:
return (
f"{getattr(block, 'tool_type', None)}: BLOCKED",
{
"error": (
"The approved workspace is no longer a valid safe "
"directory. Review the action again."
),
"exit_code": 1,
"blocked": True,
"policy": "exact_tool_approval",
},
)
approval_claimed = exact_approval.claim(
owner=owner,
session_id=session_id,
tool_name=getattr(block, "tool_type", None),
content=getattr(block, "content", None),
workspace=workspace,
)
if not approval_claimed:
return (
f"{getattr(block, 'tool_type', None)}: BLOCKED",
{
"error": "The exact-action approval did not match this tool request.",
"exit_code": 1,
"blocked": True,
"policy": "exact_tool_approval",
},
)
if isinstance(security_context, ToolRunSecurityContext) and not approval_claimed:
decision = security_context.decision_for(
getattr(block, "tool_type", None),
getattr(block, "content", None),
)
if not decision.allowed:
logger.warning(
"External-context policy blocked tool=%r",
getattr(block, "tool_type", None),
)
return blocked_tool_result(
getattr(block, "tool_type", None),
decision.reason or "Tool blocked by external-context policy.",
)
token = _active_workspace.set(workspace or None)
try:
output = await _execute_tool_block_impl(
@@ -718,28 +591,7 @@ async def execute_tool_block(
owner=owner,
progress_cb=progress_cb,
tool_policy=tool_policy,
approved_document_id=(
exact_approval.pending.document_id
if approval_claimed
else None
),
approved_document_version=(
exact_approval.pending.document_version
if approval_claimed
else None
),
approved_document_digest=(
exact_approval.pending.document_digest
if approval_claimed
else None
),
)
if isinstance(security_context, ToolRunSecurityContext):
security_context.observe_tool_result(
getattr(block, "tool_type", None),
output[1],
getattr(block, "content", None),
)
return output
finally:
_active_workspace.reset(token)
@@ -752,9 +604,6 @@ async def _execute_tool_block_impl(
owner: Optional[str] = None,
progress_cb: Optional[Callable[[Dict], Awaitable[None]]] = None,
tool_policy: Optional[Any] = None,
approved_document_id: Optional[str] = None,
approved_document_version: Optional[int] = None,
approved_document_digest: Optional[str] = None,
) -> Tuple[str, Dict]:
"""Execute a single tool block. Returns (description, result_dict).
@@ -916,15 +765,7 @@ async def _execute_tool_block_impl(
elif tool in ("create_document", "update_document", "edit_document",
"suggest_document", "manage_documents"):
desc = f"{tool}: {content.split(chr(10))[0][:80]}"
result = await _document_tool_dispatch(
tool,
content,
session_id,
owner,
document_id=approved_document_id,
document_version=approved_document_version,
document_digest=approved_document_digest,
) \
result = await _document_tool_dispatch(tool, content, session_id, owner) \
or {"error": f"{tool}: execution failed", "exit_code": 1}
if tool in ("edit_document", "suggest_document") and "title" in (result or {}):
desc = f"{tool}: {result.get('title', '')}"
+2 -10
View File
@@ -954,11 +954,7 @@ async def _cookbook_kill_session(session_id: str, *, remote_host: str = "",
resp = await client.post(f"{_INTERNAL_BASE}/api/shell/exec",
json={"command": cmd}, headers=headers)
if resp.status_code >= 400:
return {
"error": f"shell/exec returned HTTP {resp.status_code}: {resp.text[:200]}",
"exit_code": 1,
"untrusted_content": True,
}
return {"error": f"shell/exec returned HTTP {resp.status_code}: {resp.text[:200]}", "exit_code": 1}
try:
data = resp.json()
except Exception:
@@ -1087,11 +1083,7 @@ async def do_tail_serve_output(content: str, owner: Optional[str] = None) -> Dic
resp = await client.post(f"{_INTERNAL_BASE}/api/shell/exec",
json={"command": cmd}, headers=headers)
if resp.status_code >= 400:
return {
"error": f"shell/exec returned HTTP {resp.status_code}: {resp.text[:200]}",
"exit_code": 1,
"untrusted_content": True,
}
return {"error": f"shell/exec returned HTTP {resp.status_code}: {resp.text[:200]}", "exit_code": 1}
data = resp.json() if resp.content else {}
output_text = (data.get("stdout") or "").strip()
stderr_text = (data.get("stderr") or "").strip()
+1 -5
View File
@@ -123,11 +123,7 @@ async def do_trigger_research(content: str, owner: Optional[str] = None) -> Dict
resp = await client.post(f"{_INTERNAL_BASE}/api/research/start",
json=payload, headers=_internal_headers(owner))
if resp.status_code >= 400:
return {
"error": f"research/start returned HTTP {resp.status_code}: {resp.text[:200]}",
"exit_code": 1,
"untrusted_content": True,
}
return {"error": f"research/start returned HTTP {resp.status_code}: {resp.text[:200]}", "exit_code": 1}
data = resp.json()
sid = data.get("session_id", "?")
return {
-1
View File
@@ -725,7 +725,6 @@ async def do_app_api(content: str, owner: Optional[str] = None) -> Dict:
"status_code": resp.status_code,
"body": preview,
"exit_code": 1,
"untrusted_content": True,
}
return {
"output": f"{method} {path} -> {resp.status_code}\n{preview}",
-4
View File
@@ -34,10 +34,6 @@ fi
# values (APP_PORT / APP_BIND), then built-in defaults.
PORT="${ODYSSEUS_PORT:-${APP_PORT:-7860}}" # 7860, not 7000 — macOS AirPlay Receiver holds 7000.
HOST="${ODYSSEUS_HOST:-${APP_BIND:-127.0.0.1}}" # Set APP_BIND=0.0.0.0 in .env for LAN/Tailscale access.
# The port only reaches uvicorn as a flag, so export it too: everything that
# builds a URL for this instance — internal_api_base(), the companion pairing
# code, the MCP OAuth callback — reads APP_PORT and would otherwise assume 7000.
export APP_PORT="$PORT"
PROBE_HOST="$HOST"
if [ "$PROBE_HOST" = "0.0.0.0" ] || [ "$PROBE_HOST" = "::" ]; then
PROBE_HOST="127.0.0.1"
+12 -11
View File
@@ -10,9 +10,9 @@ import modelsModule from './js/models.js?v=20260715startupcalm2';
import ragModule from './js/rag.js';
import presetsModule from './js/presets.js';
import searchModule from './js/search.js';
import chatModule from './js/chat.js?v=20260815toolapproval4';
import chatModule from './js/chat.js?v=20260801fix1';
import compareModule from './js/compare/index.js?v=20260723compareicon2';
import documentModule from './js/document.js?v=20260815approvalsave1';
import documentModule from './js/document.js?v=20260722emailfastindex1';
import searchChatModule from './js/search-chat.js';
import { makeWindowDraggable } from './js/windowDrag.js';
import {
@@ -22,7 +22,7 @@ import {
settleSessionHydration
} from './js/startupShell.js';
import markdownModule from './js/markdown.js';
import chatRenderer from './js/chatRenderer.js?v=20260815toolapproval4';
import chatRenderer from './js/chatRenderer.js?v=20260722emailfastindex1';
import sessionModule from './js/sessions.js';
import memoryModule from './js/memory.js?v=20260722memoryloading1';
import voiceRecorderModule from './js/voiceRecorder.js';
@@ -33,7 +33,7 @@ import tasksModule from './js/tasks.js?v=20260723tasksbulkfeedback1';
import calendarModule from './js/calendar.js';
import notesModule from './js/notes.js';
import adminModule from './js/admin.js?v=20260716openrouter3';
import settingsModule from './js/settings.js?v=20260815approvalsave1';
import settingsModule from './js/settings.js?v=20260722emailfastindex1';
// Eagerly bind unified minimize/restore behavior across all tool modals.
import './js/modalManager.js?v=20260723compareicon2';
// Desktop window tiling — drag a modal near an edge/corner to snap.
@@ -50,7 +50,6 @@ import * as researchPanelModule from './js/research/panel.js?v=20260630researcht
import ttsModule from './js/tts-ai.js';
import spinnerModule from './js/spinner.js';
import { initKeyboardShortcuts } from './js/keyboard-shortcuts.js';
import { getSettings } from './js/appConfig.js';
import { initSidebarLayout, syncRailSide } from './js/sidebar-layout.js?v=20260715startupclean';
import { initSectionCollapse, initSectionDrag } from './js/section-management.js';
@@ -1519,11 +1518,13 @@ function initializeEventListeners() {
})
.catch(() => {});
// Hide Gallery when image generation is disabled in settings.
// getSettings() consumes the login prefetch itself, so every other module
// that asks for settings this load gets the same snapshot without a request.
window._initSettingsReady = getSettings()
.then(settings => {
// Hide Gallery when image generation is disabled in settings
const _prefetchedSettings = sessionStorage.getItem('ody-prefetch-settings');
sessionStorage.removeItem('ody-prefetch-settings');
window._initSettingsReady = (_prefetchedSettings
? Promise.resolve(JSON.parse(_prefetchedSettings))
: fetch(`${API_BASE}/api/auth/settings`, { credentials: 'same-origin' }).then(r => r.json())
).then(settings => {
// NOTE: image_gen_enabled only governs *generating* images in chat — the
// tool is blocked server-side (chat_routes / agent_loop). The Gallery
// holds uploads and past images too, so it stays visible regardless;
@@ -3704,7 +3705,7 @@ function startOdysseusApp() {
modelsModule.init(API_BASE);
ragModule.init(API_BASE);
presetsModule.init(API_BASE);
searchModule.init();
searchModule.init(API_BASE);
chatModule.init(API_BASE);
chatModule.initListeners();
groupModule.init(API_BASE);
+33 -64
View File
@@ -231,11 +231,23 @@
}
}
</style>
<!-- KaTeX and Mermaid are vendored in /static/lib and pulled in by
static/js/markdown.js the first time a page actually renders math or a
```mermaid fence. They used to load here from cdn.jsdelivr.net on every
page load, which cost ~985 KB on the wire, broke offline installs, and
announced every session to a third party. -->
<!-- KaTeX CSS is loaded with media="print" so it doesn't block render,
then flipped to "all" via JS after load. Mermaid init runs once the
library finishes loading. Both hooks are wired via addEventListener
below (inline onload= attrs are blocked by CSP script-src-attr). -->
<link id="katex-css" rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.16.22/dist/katex.min.css" media="print">
<script async src="https://cdn.jsdelivr.net/npm/katex@0.16.22/dist/katex.min.js"></script>
<script id="mermaid-script" async src="https://cdn.jsdelivr.net/npm/mermaid@11/dist/mermaid.min.js"></script>
<script nonce="{{CSP_NONCE}}">
(function(){
var k = document.getElementById('katex-css');
if (k) k.addEventListener('load', function(){ k.media = 'all'; }, { once: true });
var m = document.getElementById('mermaid-script');
if (m) m.addEventListener('load', function(){
if (window.odysseusInitMermaid) window.odysseusInitMermaid();
}, { once: true });
})();
</script>
<!-- Preload the two faces first paint actually uses: Fira Code 400 and 600,
the app font and the weight the sidebar and header text render at. They
are declared in style.css, so without a hint they are only discovered
@@ -246,8 +258,8 @@
<link rel="preload" as="font" type="font/woff2" crossorigin href="/static/fonts/FiraCode-Regular.woff2">
<link rel="preload" as="font" type="font/woff2" crossorigin href="/static/fonts/FiraCode-SemiBold.woff2">
<link rel="stylesheet" href="/static/style.css?v=20260808startupshell1">
<link rel="modulepreload" href="/static/app.js?v=20260815toolapproval4">
<link rel="modulepreload" href="/static/js/chat.js?v=20260815toolapproval4">
<link rel="modulepreload" href="/static/app.js?v=20260808startupshell1">
<link rel="modulepreload" href="/static/js/chat.js?v=20260801fix1">
<link rel="modulepreload" href="/static/js/ui.js">
<link rel="modulepreload" href="/static/js/sessions.js">
<link rel="modulepreload" href="/static/js/markdown.js">
@@ -1409,55 +1421,6 @@
</div>
<div class="settings-layout">
<div class="settings-sidebar">
<button
type="button"
class="settings-sidebar-toggle"
id="settings-sidebar-toggle"
aria-label="Collapse settings navigation"
title="Collapse settings navigation"
>
<svg class="settings-sidebar-toggle-collapse" width="12" height="12" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.3" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true">
<polyline points="15 18 9 12 15 6"></polyline>
</svg>
<svg class="settings-sidebar-toggle-expand" width="12" height="12" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.3" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true">
<polyline points="9 18 15 12 9 6"></polyline>
</svg>
</button>
<div
class="settings-sidebar-resize-handle"
id="settings-sidebar-resize-handle"
role="separator"
aria-orientation="vertical"
aria-label="Resize settings navigation"
aria-valuemin="150"
aria-valuemax="340"
aria-valuenow="220"
tabindex="0"
></div>
<div class="settings-sidebar-content">
<div class="settings-nav-search-wrap">
<svg class="settings-nav-search-icon" width="13" height="13" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" aria-hidden="true">
<circle cx="11" cy="11" r="7"></circle>
<path d="M20 20l-4-4"></path>
</svg>
<input
type="search"
id="settings-nav-search"
class="settings-nav-search"
placeholder="Find settings…"
autocomplete="off"
aria-label="Find settings"
aria-controls="settings-nav-search-results"
/>
<div
id="settings-nav-search-results"
class="settings-nav-search-results hidden"
role="listbox"
aria-label="Settings search results"
></div>
</div>
<!-- Section 1: AI plumbing (Add Models → AI Defaults → Search) -->
<button class="settings-nav-item active" data-settings-tab="services">
<svg width="15" height="15" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.6" stroke-linecap="round" stroke-linejoin="round"><path d="M12 5v14"/><path d="M5 12h14"/></svg>
@@ -1518,10 +1481,9 @@
<span>Users</span>
</button>
<button class="settings-nav-item admin-only" data-settings-tab="system">
<svg width="15" height="15" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="3"/><path d="M19.4 15a1.65 1.65 0 0 0 .33 1.82l.06.06a2 2 0 0 1-2.83 2.83l-.06-.06a1.65 1.65 0 0 0-1.82-.33 1.65 1.65 0 0 0-1 1.51V21a2 2 0 0 1-4 0v-.09A1.65 1.65 0 0 0 9 19.4a1.65 1.65 0 0 0-1.82.33l-.06.06a2 2 0 0 1-2.83-2.83l.06-.06A1.65 1.65 0 0 0 4.68 15a1.65 1.65 0 0 0-1.51-1H3a2 2 0 0 1 0-4h.09A1.65 1.65 0 0 0 4.6 9a1.65 1.65 0 0 0-.33-1.82l-.06-.06a2 2 0 0 1 2.83-2.83l.06.06A1.65 1.65 0 0 0 9 4.68a1.65 1.65 0 0 0 1-1.51V3a2 2 0 0 1 4 0v-.09a1.65 1.65 0 0 0 1 1.51 1.65 1.65 0 0 0 1.82-.33l.06-.06a2 2 0 0 1 2.83 2.83l-.06.06A1.65 1.65 0 0 0 19.4 9a1.65 1.65 0 0 0 1.51 1H21a2 2 0 0 1 0 4h-.09A1.65 1.65 0 0 0 19.4 15z"/></svg>
<svg width="15" height="15" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="3"/><path d="M19.4 15a1.65 1.65 0 0 0 .33 1.82l.06.06a2 2 0 0 1-2.83 2.83l-.06-.06a1.65 1.65 0 0 0-1.82-.33 1.65 1.65 0 0 0-1 1.51V21a2 2 0 0 1-4 0v-.09A1.65 1.65 0 0 0 9 19.4a1.65 1.65 0 0 0-1.82.33l-.06.06a2 2 0 0 1-2.83-2.83l.06-.06A1.65 1.65 0 0 0 4.68 15a1.65 1.65 0 0 0-1.51-1H3a2 2 0 0 1 0-4h.09A1.65 1.65 0 0 0 4.6 9a1.65 1.65 0 0 0-.33-1.82l-.06-.06a2 2 0 0 1 2.83-2.83l.06.06A1.65 1.65 0 0 0 9 4.68a1.65 1.65 0 0 0 1-1.51V3a2 2 0 0 1 4 0v.09a1.65 1.65 0 0 0 1 1.51 1.65 1.65 0 0 0 1.82-.33l.06-.06a2 2 0 0 1 2.83 2.83l-.06.06A1.65 1.65 0 0 0 19.4 9a1.65 1.65 0 0 0 1.51 1H21a2 2 0 0 1 0 4h-.09a1.65 1.65 0 0 0-1.51 1z"/></svg>
<span>System</span>
</button>
</div>
</div>
<div class="settings-panels">
@@ -1542,6 +1504,13 @@
<span class="adm-model-logo" id="set-defaultModelSelect-logo" style="display:inline-flex;align-items:center;justify-content:center;width:18px;height:18px;flex-shrink:0;opacity:0.9;color:var(--fg);"></span>
<select id="set-defaultModelSelect" class="settings-select"></select>
</div>
<div class="settings-row" style="align-items:flex-start;" hidden>
<label class="settings-label" style="margin-top:6px;">Fallbacks</label>
<div style="flex:1;display:flex;flex-direction:column;gap:6px;">
<div id="set-defaultFallbacks" class="settings-fallbacks"></div>
<button type="button" class="settings-fallback-add" id="set-defaultAddFallback" title="Add a model to try if the one above fails">+ Add fallback</button>
</div>
</div>
<div id="set-defaultChatMsg" style="font-size:11px;color:color-mix(in srgb, var(--fg) 45%, transparent);"></div>
</div>
</div>
@@ -2570,20 +2539,20 @@
<script type="module" src="/static/js/search.js"></script>
<script type="module" src="/static/js/spinner.js"></script>
<script type="module" src="/static/js/tts-ai.js"></script>
<script type="module" src="/static/js/document.js?v=20260815approvalsave1"></script>
<script type="module" src="/static/js/document.js?v=20260722emailfastindex1"></script>
<script type="module" src="/static/js/gallery.js?v=20260708match1"></script>
<script type="module" src="/static/js/chatRenderer.js?v=20260815toolapproval4"></script>
<script type="module" src="/static/js/chatRenderer.js?v=20260722emailfastindex1"></script>
<script type="module" src="/static/js/codeRunner.js"></script>
<script type="module" src="/static/js/chatStream.js?v=20260815approvalsave1"></script>
<script type="module" src="/static/js/chat.js?v=20260815toolapproval4"></script>
<script type="module" src="/static/js/chatStream.js?v=20260722emailfastindex1"></script>
<script type="module" src="/static/js/chat.js?v=20260801fix1"></script>
<script type="module" src="/static/js/cookbook.js"></script>
<script src="/static/js/cookbookSchedule.js"></script>
<script type="module" src="/static/js/search-chat.js"></script>
<script type="module" src="/static/js/theme.js"></script>
<script type="module" src="/static/js/censor.js"></script>
<script type="module" src="/static/js/settings.js?v=20260815approvalsave1"></script>
<script type="module" src="/static/js/settings.js?v=20260723compareicon1"></script>
<script type="module" src="/static/js/assistant.js"></script>
<script type="module" src="/static/app.js?v=20260815toolapproval4"></script> <!-- app.js must be LAST -->
<script type="module" src="/static/app.js?v=20260808startupshell1"></script> <!-- app.js must be LAST -->
<script type="module" src="/static/js/init.js?v=20260715freshroot3"></script>
<script type="module" src="/static/js/a11y.js"></script>
<script nonce="{{CSP_NONCE}}">if('serviceWorker' in navigator){navigator.serviceWorker.register('/static/sw.js').catch(()=>{});}</script>
+17 -65
View File
@@ -6,7 +6,6 @@ import settingsModule from './settings.js';
import { providerLogo, providerLogoFromUrl } from './providers.js';
import { sortModelObjects } from './modelSort.js';
import { PROVIDER_DEVICE_FLOWS, formatDeviceFlowError, runProviderDeviceFlow } from './providerDeviceFlow.js';
import { getSettings, getTools, invalidateSettings, invalidateTools } from './appConfig.js';
let initialized = false;
let modalEl = null;
@@ -346,7 +345,8 @@ function initSignupToggle() {
function initShareDefaultsToggle() {
const toggle = el('adm-shareDefaultsToggle');
getSettings()
fetch('/api/auth/settings', { credentials: 'same-origin' })
.then(r => r.json())
.then(d => { toggle.checked = !!d.share_defaults_with_users; })
.catch(e => console.warn('Settings fetch failed:', e));
toggle.addEventListener('change', async () => {
@@ -361,9 +361,6 @@ function initShareDefaultsToggle() {
toggle.checked = !!data.share_defaults_with_users;
} catch (e) {
toggle.checked = !toggle.checked;
} finally {
// Drop the shared snapshot: it still says what this toggle used to be.
invalidateSettings();
}
});
}
@@ -1896,16 +1893,8 @@ async function loadBuiltinTools() {
const list = el('adm-builtin-tools-list');
if (!list) return;
try {
// This panel is an editor, and its save posts the whole disabled list
// rebuilt from the checkboxes below. So it has to render authoritative
// state: a snapshot that went stale out of band (the manage_settings tool,
// another tab) would be re-posted wholesale on the next unrelated toggle
// and would silently undo the newer state. refreshAll() calls this on every
// panel open, so drop the shared entry and refill it. The startup read that
// chatRenderer.js shares is unaffected; this panel just never edits a cache,
// which is the same rule the settings panel follows by reading directly.
invalidateTools();
const data = await getTools();
const res = await fetch('/api/tools', { credentials: 'same-origin' });
const data = await res.json();
const tools = data.tools || [];
if (!tools.length) { list.innerHTML = '<div class="admin-empty">No tools found</div>'; return; }
@@ -1979,50 +1968,17 @@ async function loadBuiltinTools() {
});
});
// Merge only the user's intended changes onto authoritative server state.
// /api/tools replaces the full disabled list, so rebuilding it from this
// panel's DOM can undo a change made by another tab or manage_settings
// after the panel was opened.
async function _saveToolState(changes) {
invalidateTools();
const latest = await getTools();
const state = new Map(
(latest.tools || []).map(t => [t.id, !!t.enabled])
);
for (const change of changes) {
if (state.has(change.id)) {
state.set(change.id, !!change.enabled);
}
}
const disabled = Array.from(state.entries())
.filter(([, enabled]) => !enabled)
.map(([id]) => id);
try {
const res = await fetch('/api/tools', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ disabled }),
credentials: 'same-origin',
});
if (!res.ok) throw new Error(`Failed to update tools (${res.status})`);
// Bring the still-open editor forward to the same merged snapshot so an
// out-of-band change is visible instead of leaving stale checkboxes.
list.querySelectorAll('input[data-tool-id]').forEach(c => {
if (state.has(c.dataset.toolId)) {
c.checked = state.get(c.dataset.toolId);
}
});
list.querySelectorAll('.admin-tool-category').forEach(_updateCatCounter);
} finally {
// This route persists disabled_tools into the settings store
// (routes/model_routes.py), so both snapshots are now stale.
invalidateTools();
invalidateSettings();
}
// Helper: save disabled tools + update counters
async function _saveToolState() {
const allChecks = list.querySelectorAll('input[data-tool-id]');
const disabled = [];
allChecks.forEach(c => { if (!c.checked) disabled.push(c.dataset.toolId); });
await fetch('/api/tools', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ disabled }),
credentials: 'same-origin',
});
}
function _updateCatCounter(catEl) {
if (!catEl) return;
@@ -2037,9 +1993,7 @@ async function loadBuiltinTools() {
// Wire individual tool toggles
list.querySelectorAll('input[data-tool-id]').forEach(chk => {
chk.addEventListener('change', async () => {
await _saveToolState([
{ id: chk.dataset.toolId, enabled: chk.checked },
]);
await _saveToolState();
_updateCatCounter(chk.closest('.admin-tool-category'));
});
});
@@ -2050,10 +2004,8 @@ async function loadBuiltinTools() {
const catEl = chk.closest('.admin-tool-category');
if (!catEl) return;
const checked = chk.checked;
const changes = Array.from(catEl.querySelectorAll('input[data-tool-id]'))
.map(c => ({ id: c.dataset.toolId, enabled: checked }));
catEl.querySelectorAll('input[data-tool-id]').forEach(c => { c.checked = checked; });
await _saveToolState(changes);
await _saveToolState();
_updateCatCounter(catEl);
});
});
-86
View File
@@ -1,86 +0,0 @@
// static/js/appConfig.js
//
// One shared, invalidatable cache for the two config endpoints that every
// module wants at startup.
//
// Before this, /api/auth/settings was fetched independently by six modules and
// /api/tools by three, none of them aware of the others — 4 and 3 requests on a
// single cold load. Worse than the requests: each caller could observe a
// different snapshot of the same object, and chatRenderer.js is imported under
// three different ?v= query strings, so it is three separate module instances
// each issuing its own /api/tools fetch. Caching here fixes both, because the
// cache lives in one module every instance imports by the same specifier.
//
// URLs are bare paths on purpose. The callers that used `${API_BASE}/api/...`
// resolved to the identical URL — API_BASE is `window.location.origin`
// (app.js) — so nothing about the request changes for them.
//
// WRITERS MUST INVALIDATE. Anything that POSTs /api/auth/settings calls
// invalidateSettings(); anything that POSTs /api/tools calls invalidateTools()
// *and* invalidateSettings(), because that route persists `disabled_tools`
// into the same settings store (routes/model_routes.py). Miss one and the UI
// serves a stale settings object for the rest of the session, which is worse
// than the duplicate fetches this replaces.
//
// The resolved object is shared by reference, so treat it as read-only: copy
// before mutating (`{ ...await getSettings() }`).
// Written by login.html immediately before it redirects to '/', so the first
// load after a login can skip the request entirely. Consumed once per page
// load, by whichever module asks for settings first.
const PREFETCH_KEY = 'ody-prefetch-settings';
const _URLS = { settings: '/api/auth/settings', tools: '/api/tools' };
const _cache = { settings: null, tools: null };
function _readPrefetchedSettings() {
try {
const raw = sessionStorage.getItem(PREFETCH_KEY);
if (!raw) return null;
sessionStorage.removeItem(PREFETCH_KEY);
return JSON.parse(raw);
} catch (_) {
return null;
}
}
// A rejected promise must not stay in the slot. Plain `??=` memoisation would
// keep it, so one transient blip during boot would leave keybinds, TTS and the
// search provider on their defaults for the whole session with no retry. Clear
// the slot on failure — unless a later invalidate/refetch already replaced it —
// and rethrow, so every caller's existing .catch() still runs exactly as before.
function _get(key) {
if (_cache[key]) return _cache[key];
const pending = fetch(_URLS[key], { credentials: 'same-origin' })
.then(r => r.json())
.catch(err => {
if (_cache[key] === pending) _cache[key] = null;
throw err;
});
_cache[key] = pending;
return pending;
}
/** GET /api/auth/settings, once per page load (or once per invalidation). */
export function getSettings() {
if (!_cache.settings) {
const prefetched = _readPrefetchedSettings();
if (prefetched) _cache.settings = Promise.resolve(prefetched);
}
return _get('settings');
}
/** GET /api/tools, once per page load (or once per invalidation). */
export function getTools() {
return _get('tools');
}
/** Call after any write that can change settings. */
export function invalidateSettings() {
_cache.settings = null;
}
/** Call after any write that can change the tool enable/disable state. */
export function invalidateTools() {
_cache.tools = null;
}
+197 -642
View File
File diff suppressed because it is too large Load Diff
-104
View File
@@ -1,104 +0,0 @@
/** Select and update the response holder for a route-provenance event. */
export function applyModelRouteEventState(event, holder, roundHolder, defaultModel = '') {
const target = event && event.round && roundHolder ? roundHolder : holder;
if (!target) return null;
target._requestedModel = (
event.requested_model
|| event.selected_model
|| target._requestedModel
|| defaultModel
);
target._actualModel = (
event.model
|| event.answered_by
|| target._actualModel
|| target._requestedModel
);
const hasEndpointRoute = Boolean(
event.requested_endpoint_id
|| event.selected_endpoint_id
|| event.endpoint_id
|| event.answered_by_endpoint_id
|| event.requested_endpoint_label
|| event.selected_endpoint_label
|| event.endpoint_label
|| event.answered_by_endpoint_label
|| target._requestedEndpointLabel
);
if (hasEndpointRoute) {
target._requestedEndpointId = (
event.requested_endpoint_id
|| event.selected_endpoint_id
|| target._requestedEndpointId
|| null
);
target._requestedEndpointLabel = (
event.requested_endpoint_label
|| event.selected_endpoint_label
|| target._requestedEndpointLabel
|| 'Selected route'
);
target._actualEndpointId = (
event.endpoint_id
|| event.answered_by_endpoint_id
|| target._actualEndpointId
|| target._requestedEndpointId
|| null
);
target._actualEndpointLabel = (
event.endpoint_label
|| event.answered_by_endpoint_label
|| target._actualEndpointLabel
|| target._requestedEndpointLabel
);
}
return target;
}
/** Copy the active route into the bubble created for the next Agent round. */
export function inheritModelRouteState(holder, roundHolder, target, defaultModel = '') {
if (!target) return null;
const source = roundHolder || holder;
target._requestedModel = source?._requestedModel || defaultModel;
target._actualModel = source?._actualModel || target._requestedModel;
if (source?._requestedEndpointLabel || source?._actualEndpointLabel) {
target._requestedEndpointId = source?._requestedEndpointId || null;
target._requestedEndpointLabel = source?._requestedEndpointLabel || 'Selected route';
target._actualEndpointId = source?._actualEndpointId || target._requestedEndpointId;
target._actualEndpointLabel = source?._actualEndpointLabel || target._requestedEndpointLabel;
}
return target;
}
/** Apply final/metrics provenance to the active round, not the first bubble. */
export function applyModelMetricsState(metrics, holder, roundHolder, defaultModel = '') {
const target = roundHolder || holder;
if (!target || !metrics) return target || null;
const roundModels = Array.isArray(metrics.round_models) ? metrics.round_models : [];
const roundModel = roundHolder && roundModels.length
? roundModels[roundModels.length - 1]
: null;
target._requestedModel = metrics.requested_model || target._requestedModel || defaultModel;
target._actualModel = roundModel || metrics.model || target._actualModel || target._requestedModel;
const roundEndpointIds = Array.isArray(metrics.round_endpoint_ids) ? metrics.round_endpoint_ids : [];
const roundEndpointLabels = Array.isArray(metrics.round_endpoint_labels) ? metrics.round_endpoint_labels : [];
if (
metrics.requested_endpoint_label
|| metrics.endpoint_label
|| roundEndpointLabels.length
|| target._requestedEndpointLabel
) {
target._requestedEndpointId = metrics.requested_endpoint_id || target._requestedEndpointId || null;
target._requestedEndpointLabel = metrics.requested_endpoint_label || target._requestedEndpointLabel || 'Selected route';
const hasRoundEndpointId = Boolean(roundHolder && roundEndpointIds.length);
const hasRoundEndpointLabel = Boolean(roundHolder && roundEndpointLabels.length);
target._actualEndpointId = hasRoundEndpointId
? roundEndpointIds[roundEndpointIds.length - 1]
: (metrics.endpoint_id || target._actualEndpointId || target._requestedEndpointId);
target._actualEndpointLabel = hasRoundEndpointLabel
? roundEndpointLabels[roundEndpointLabels.length - 1]
: (metrics.endpoint_label || target._actualEndpointLabel || target._requestedEndpointLabel);
}
return target;
}
+57 -312
View File
@@ -9,9 +9,7 @@ import { providerLogo, providerLabel } from './providers.js';
import settingsModule from './settings.js';
import spinnerModule from './spinner.js';
import { bindMenuDismiss } from './escMenuStack.js';
import { loadPanel } from './panels.js';
import { matchModelKey } from './model/matchKey.js';
import { getTools } from './appConfig.js';
const SEARCH_ICON = '<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2"><circle cx="11" cy="11" r="8"/><path d="M21 21l-4.35-4.35"/></svg>';
const REPORT_ICON = '<svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M14 2H6a2 2 0 0 0-2 2v16a2 2 0 0 0 2 2h12a2 2 0 0 0 2-2V8z"/><polyline points="14 2 14 8 20 8"/><line x1="16" y1="13" x2="8" y2="13"/><line x1="16" y1="17" x2="8" y2="17"/><line x1="10" y1="9" x2="8" y2="9"/></svg>';
@@ -447,12 +445,8 @@ function stripExecutedFence(match, tag, inline, body) {
async function loadExecFenceRegex() {
try {
// Shared with admin.js, and — more to the point — with the other copies of
// this module: chatRenderer.js is imported under three different ?v= query
// strings, so it is instantiated three times per load and used to issue
// three identical /api/tools requests. appConfig.js is imported by one
// specifier from all of them, so they now share a single fetch.
const data = await getTools();
const res = await fetch('/api/tools', { credentials: 'same-origin' });
const data = await res.json();
const tags = (data.tools || [])
.map((t) => t.id)
.filter((id) => id && !EXEC_FENCE_NON_TOOL.has(id));
@@ -621,36 +615,10 @@ export function sameModelName(left, right) {
|| shortModel(a).toLowerCase() === shortModel(b).toLowerCase();
}
function shortEndpointLabel(label) {
const value = modelValue(label);
if (!value) return '';
return value.length > 18 ? value.slice(0, 17) + '…' : value;
}
export function modelRouteLabel(
requestedModel,
actualModel,
requestedEndpointLabel = '',
actualEndpointLabel = '',
requestedEndpointId = '',
actualEndpointId = '',
) {
export function modelRouteLabel(requestedModel, actualModel) {
const requested = modelValue(requestedModel);
const actual = modelValue(actualModel) || requested;
const requestedRoute = modelValue(requestedEndpointId || requestedEndpointLabel);
const actualRoute = modelValue(actualEndpointId || actualEndpointLabel);
const routeChanged = Boolean(
actualRoute
&& requestedRoute
&& actualRoute !== requestedRoute
);
if (!requested || sameModelName(requested, actual)) {
const model = shortModel(actual || requested);
if (!routeChanged) return model;
const from = shortEndpointLabel(requestedEndpointLabel || 'Selected route');
const to = shortEndpointLabel(actualEndpointLabel || actualEndpointId);
return model + ' (' + from + ' -> ' + to + ')';
}
if (!requested || sameModelName(requested, actual)) return shortModel(actual || requested);
return shortModel(requested) + ' -> ' + shortModel(actual);
}
@@ -661,24 +629,10 @@ export function replyModelPair(modelName, metadata) {
if (actualFromMeta || requestedFromMeta) {
const actual = actualFromMeta || requestedFromMeta || modelValue(modelName);
const requested = requestedFromMeta || actual;
return {
requestedModel: requested,
actualModel: actual,
requestedEndpointId: meta.requested_endpoint_id || null,
requestedEndpointLabel: meta.requested_endpoint_label || 'Selected route',
actualEndpointId: meta.endpoint_id || null,
actualEndpointLabel: meta.endpoint_label || meta.requested_endpoint_label || 'Selected route',
};
return { requestedModel: requested, actualModel: actual };
}
const fallback = modelValue(modelName);
return {
requestedModel: fallback,
actualModel: fallback,
requestedEndpointId: null,
requestedEndpointLabel: 'Selected route',
actualEndpointId: null,
actualEndpointLabel: 'Selected route',
};
return { requestedModel: fallback, actualModel: fallback };
}
/**
@@ -870,50 +824,12 @@ export function isCostTrackedEndpoint(url) {
}
/** Cost for the current turn, returning null for non-billable endpoints. */
function _billableCost(model, inputTokens, outputTokens, endpointCostTracked, selectedEndpointUrl) {
// Foreground fallback can answer on a different endpoint than the session's
// selected route. Prefer the backend's non-secret actual-route
// classification; retain the selected-endpoint check for older history.
if (endpointCostTracked === false) return null;
const selectedUrl = selectedEndpointUrl === undefined
? _currentEndpointUrl()
: selectedEndpointUrl;
if (endpointCostTracked !== true && !isCostTrackedEndpoint(selectedUrl)) {
return null;
}
function _billableCost(model, inputTokens, outputTokens) {
const url = _currentEndpointUrl();
if (!isCostTrackedEndpoint(url)) return null;
return getModelCost(model, inputTokens, outputTokens);
}
/** Sum cost using the route/model that produced each Agent round. */
function _metricsBillableCost(metrics, model, inputTokens, outputTokens, selectedEndpointUrl) {
const buckets = Array.isArray(metrics.usage_buckets) ? metrics.usage_buckets : [];
if (!buckets.length) {
return _billableCost(
model,
inputTokens,
outputTokens,
metrics.endpoint_cost_tracked,
selectedEndpointUrl,
);
}
let total = 0;
let hasPricedUsage = false;
for (const bucket of buckets) {
if (!bucket || typeof bucket !== 'object') continue;
const bucketCost = _billableCost(
bucket.model || model,
Number(bucket.input_tokens) || 0,
Number(bucket.output_tokens) || 0,
bucket.endpoint_cost_tracked,
selectedEndpointUrl,
);
if (bucketCost === null) continue;
total += bucketCost;
hasPricedUsage = true;
}
return hasPricedUsage ? total : null;
}
export function getImageCost(model, quality, size) {
if (!model) return null;
const m = model.toLowerCase();
@@ -928,9 +844,6 @@ export function getImageCost(model, quality, size) {
/* ── Session cost helpers ─────────────────────────────────────────── */
const _COST_KEY = 'ody-session-cost';
const _COST_RUNS_KEY = 'ody-session-cost-runs';
const _MAX_COST_RUNS_PER_SESSION = 256;
const _COST_LEDGER_LOCK = 'odysseus-session-cost-ledger';
/** Return the accumulated cost for the current (or given) session. */
export function getSessionCost(sessionId) {
@@ -938,14 +851,7 @@ export function getSessionCost(sessionId) {
if (!sid) return 0;
try {
const costs = JSON.parse(localStorage.getItem(_COST_KEY) || '{}');
const runCosts = JSON.parse(localStorage.getItem(_COST_RUNS_KEY) || '{}');
const recordedRuns = runCosts[sid] && typeof runCosts[sid] === 'object'
? Object.values(runCosts[sid])
: [];
return (costs[sid] || 0) + recordedRuns.reduce(
(total, value) => total + (Number(value) || 0),
0,
);
return costs[sid] || 0;
} catch (_e) { return 0; }
}
@@ -957,9 +863,6 @@ export function resetSessionCost(sessionId) {
const costs = JSON.parse(localStorage.getItem(_COST_KEY) || '{}');
delete costs[sid];
localStorage.setItem(_COST_KEY, JSON.stringify(costs));
const runCosts = JSON.parse(localStorage.getItem(_COST_RUNS_KEY) || '{}');
delete runCosts[sid];
localStorage.setItem(_COST_RUNS_KEY, JSON.stringify(runCosts));
} catch (_e) { /* ignore */ }
updateSessionCostUI();
}
@@ -968,8 +871,21 @@ export function resetSessionCost(sessionId) {
export function updateSessionCostUI() {
const el = document.getElementById('session-cost-display');
if (!el) return;
// The ledger records billable work already performed in this session. A
// selected local endpoint does not erase cost from a paid fallback route.
// Non-billable endpoint? Hide the badge and clear stale cost that a previous
// cloud-rate calculation may have left in localStorage for this session.
const _url = _currentEndpointUrl();
if (!isCostTrackedEndpoint(_url)) {
const sid = window.sessionModule && window.sessionModule.getCurrentSessionId();
if (sid && getSessionCost(sid) > 0) {
try {
const costs = JSON.parse(localStorage.getItem(_COST_KEY) || '{}');
delete costs[sid];
localStorage.setItem(_COST_KEY, JSON.stringify(costs));
} catch (_e) { /* ignore */ }
}
el.style.display = 'none';
return;
}
const cost = getSessionCost();
if (cost > 0) {
el.textContent = '$' + (cost < 0.01 ? cost.toFixed(4) : cost < 1 ? cost.toFixed(3) : cost.toFixed(2));
@@ -979,94 +895,6 @@ export function updateSessionCostUI() {
}
}
/** Record one metrics payload in a session ledger at most once. */
export function recordSessionMetricsCost(metrics, sessionId, selectedEndpointUrl) {
if (!metrics || typeof metrics !== 'object') return null;
const cost = _metricsBillableCost(
metrics,
metrics.model || 'Unknown',
metrics.input_tokens || 0,
metrics.output_tokens || 0,
selectedEndpointUrl,
);
if (metrics._fromHistory) return cost;
const sid = sessionId || (
window.sessionModule && window.sessionModule.getCurrentSessionId()
);
if (!sid || cost === null) return cost;
const runId = typeof metrics._costRecordId === 'string'
? metrics._costRecordId.trim()
: '';
if ((metrics._costRecorded || metrics._costRecordPending) && !runId) return cost;
// Recorded is only set once the write actually runs; pending covers the
// window while the write waits on the cross-tab lock, so a replay in that
// window cannot double-add and a tab closed mid-queue never claims recorded.
metrics._costRecordPending = true;
const writeCost = () => {
if (runId) {
try {
const runCosts = JSON.parse(localStorage.getItem(_COST_RUNS_KEY) || '{}');
const sessionRuns = runCosts[sid] && typeof runCosts[sid] === 'object'
? runCosts[sid]
: {};
// Assigning by detached-run identity is replay-idempotent even when a
// refresh produces a fresh metrics object. The Web Lock around this
// read/modify/write also keeps distinct runs from two tabs from
// overwriting one another's stale snapshot.
sessionRuns[runId] = cost;
const entries = Object.entries(sessionRuns);
if (entries.length > _MAX_COST_RUNS_PER_SESSION) {
const overflow = entries.slice(0, entries.length - _MAX_COST_RUNS_PER_SESSION);
const costs = JSON.parse(localStorage.getItem(_COST_KEY) || '{}');
costs[sid] = (costs[sid] || 0) + overflow.reduce(
(total, entry) => total + (Number(entry[1]) || 0),
0,
);
overflow.forEach(([oldRunId]) => delete sessionRuns[oldRunId]);
localStorage.setItem(_COST_KEY, JSON.stringify(costs));
}
runCosts[sid] = sessionRuns;
localStorage.setItem(_COST_RUNS_KEY, JSON.stringify(runCosts));
} catch (_e) { /* ignore */ }
} else {
try {
const costs = JSON.parse(localStorage.getItem(_COST_KEY) || '{}');
costs[sid] = (costs[sid] || 0) + cost;
localStorage.setItem(_COST_KEY, JSON.stringify(costs));
} catch (_e) { /* ignore */ }
}
metrics._costRecorded = true;
metrics._costRecordPending = false;
const currentSid = window.sessionModule && window.sessionModule.getCurrentSessionId();
if (currentSid === sid) updateSessionCostUI();
};
let writeStarted = false;
const guardedWrite = () => {
writeStarted = true;
writeCost();
};
try {
if (
typeof navigator !== 'undefined'
&& navigator.locks
&& typeof navigator.locks.request === 'function'
) {
const pendingWrite = navigator.locks.request(_COST_LEDGER_LOCK, guardedWrite);
if (pendingWrite && typeof pendingWrite.catch === 'function') {
pendingWrite.catch(() => {
if (!writeStarted) guardedWrite();
});
}
} else {
guardedWrite();
}
} catch (_e) {
if (!writeStarted) guardedWrite();
}
return cost;
}
/** Create a timestamp span for role labels.
* Pass an ISO string / Date / epoch-ms to render the message's own time
* (used when replaying history). Falls back to "now" when no value is given. */
@@ -1373,7 +1201,7 @@ document.addEventListener('click', function(e) {
} catch {}
});
} else if (kind === 'document') {
import('./document.js?v=20260815approvalsave1').then(mod => {
import('./document.js?v=20260722emailfastindex1').then(mod => {
const open = mod.loadDocument
|| mod.openDocument
|| (mod.default && (mod.default.loadDocument || mod.default.openDocument));
@@ -1395,7 +1223,7 @@ document.addEventListener('click', function(e) {
if (open) open(id);
}).catch(() => {});
} else if (kind === 'email') {
import('./emailLibrary.js?v=20260815approvalsave1').then(mod => {
import('./emailLibrary.js?v=20260722emailfastindex1').then(mod => {
const open = mod.openEmailLibrary || (mod.default && mod.default.openEmailLibrary);
if (open) open({ uid: id });
}).catch(() => {});
@@ -1554,7 +1382,7 @@ export function buildImageBubble(imageUrl, prompt, model, size, quality, imageId
try {
const [galleryMod, editorMod] = await Promise.all([
import('./gallery.js'),
loadPanel('editor'),
import('./galleryEditor.js'),
]);
// Ensure the Gallery modal is open so the editor has a container
// to render into; switch its tabs to the Edit tab.
@@ -2046,19 +1874,23 @@ export function displayMetrics(messageElement, metrics) {
const isReal = metrics.usage_source === 'real';
const ctxPct = metrics.context_percent;
const model = metrics.model || 'Unknown';
const cost = _metricsBillableCost(
metrics,
model,
inputTokens,
outputTokens,
);
const cost = _billableCost(model, inputTokens, outputTokens);
// Nothing useful to show — bail out (only if ALL metrics are missing)
if (!responseTime && !inputTokens && !outputTokens && tps == null && !ctxPct) return;
// Rendering can occur when metrics arrive and again after [DONE]. The
// ledger mutation is idempotent for that shared payload.
recordSessionMetricsCost(metrics);
// Accumulate session cost (only on fresh metrics, not history reload)
if (!metrics._fromHistory) {
const _sid = window.sessionModule && window.sessionModule.getCurrentSessionId();
if (_sid && cost !== null) {
try {
const _costs = JSON.parse(localStorage.getItem(_COST_KEY) || '{}');
_costs[_sid] = (_costs[_sid] || 0) + cost;
localStorage.setItem(_COST_KEY, JSON.stringify(_costs));
} catch (_e) { /* ignore */ }
updateSessionCostUI();
}
}
// Keep token counts in the Message Stats popup; the footer should stay slim.
const costStr0 = cost !== null ? `$${cost < 0.01 ? cost.toFixed(4) : cost.toFixed(3)}` : null;
@@ -2348,7 +2180,6 @@ export function renderAskUserCard(payload, options) {
card.setAttribute('role', 'group');
card.tabIndex = -1;
const multi = !!aq.multi;
const isToolApproval = aq.kind === 'tool_approval' && !!aq.approval_id;
const emojiText = (value) => svgifyEmoji(uiModule.esc(String(value)));
const head = document.createElement('div');
@@ -2373,27 +2204,6 @@ export function renderAskUserCard(payload, options) {
card.appendChild(question);
card.setAttribute('aria-labelledby', question.id);
if (isToolApproval && aq.action) {
const action = document.createElement('div');
action.className = 'ask-user-option-desc';
const effects = Array.isArray(aq.action.effects)
? aq.action.effects.join(', ')
: '';
action.textContent = [
aq.action.tool || 'tool',
aq.action.content || '',
effects ? `Effects: ${effects}` : '',
aq.action.workspace ? `Workspace: ${aq.action.workspace}` : '',
aq.action.document_id ? `Document: ${aq.action.document_id}` : '',
aq.action.document_version != null
? `Document version: ${aq.action.document_version}`
: '',
aq.action.digest ? `Approval fingerprint: ${aq.action.digest}` : '',
].filter(Boolean).join('\n');
action.style.whiteSpace = 'pre-wrap';
card.appendChild(action);
}
const list = document.createElement('div');
list.className = 'ask-user-options';
card.appendChild(list);
@@ -2431,23 +2241,7 @@ export function renderAskUserCard(payload, options) {
}
if (!multi) {
row.type = 'button';
row.addEventListener('click', () => {
if (isToolApproval) {
card.remove();
document.dispatchEvent(new CustomEvent('odysseus:tool-approval', {
detail: {
approval_id: aq.approval_id,
decision: String((opt && opt.value) || '').toLowerCase(),
label,
document_id: aq.action && aq.action.document_id
? String(aq.action.document_id)
: '',
},
}));
} else {
send(label);
}
});
row.addEventListener('click', () => send(label));
}
list.appendChild(row);
});
@@ -2483,7 +2277,7 @@ export function renderAskUserCard(payload, options) {
});
other.appendChild(otherInput);
other.appendChild(otherSend);
if (!isToolApproval) card.appendChild(other);
card.appendChild(other);
chatBox.appendChild(card);
if (renderOptions.scroll !== false) {
@@ -2513,19 +2307,9 @@ export function addMessage(role, content, modelName, metadata) {
const textRaw = Array.isArray(content) ? markdownModule.renderContent(content) : content;
// --- Agent multi-bubble reconstruction from saved metadata ---
if (
role === 'assistant'
&& metadata
&& (
(Array.isArray(metadata.tool_events) && metadata.tool_events.length > 0)
|| (Array.isArray(metadata.round_texts) && metadata.round_texts.length > 1)
)
) {
if (role === 'assistant' && metadata && metadata.tool_events && metadata.tool_events.length > 0) {
const roundTexts = metadata.round_texts || [];
const roundModels = metadata.round_models || [];
const roundEndpointIds = metadata.round_endpoint_ids || [];
const roundEndpointLabels = metadata.round_endpoint_labels || [];
const toolEvents = metadata.tool_events || [];
const toolEvents = metadata.tool_events;
let pendingAskUser = null;
let lastWrap = null;
let firstMsgAi = null;
@@ -2533,20 +2317,16 @@ export function addMessage(role, content, modelName, metadata) {
const toolsByRound = {};
for (const ev of toolEvents) {
const r = ev.round ?? 1;
const r = ev.round || 1;
if (!toolsByRound[r]) toolsByRound[r] = [];
toolsByRound[r].push(ev);
}
const toolRounds = Object.keys(toolsByRound).map(Number);
const maxRound = Math.max(toolRounds.length ? Math.max(...toolRounds) : 0, roundTexts.length);
const maxRound = Math.max(...Object.keys(toolsByRound).map(Number), roundTexts.length);
const firstRound = (toolsByRound[0] || []).length ? 0 : 1;
for (let roundNum = firstRound; roundNum <= maxRound; roundNum++) {
const r = roundNum - 1;
const txt = r >= 0
? resolveDocumentPlaceholderLinks((roundTexts[r] || '').trim(), metadata)
: '';
for (let r = 0; r < maxRound; r++) {
const roundNum = r + 1;
const txt = resolveDocumentPlaceholderLinks((roundTexts[r] || '').trim(), metadata);
if (txt) {
const wrap = document.createElement('div');
@@ -2554,31 +2334,10 @@ export function addMessage(role, content, modelName, metadata) {
const roleEl = document.createElement('div');
roleEl.className = 'role';
const pair = replyModelPair(modelName, metadata);
const contModel = roundModels[r] || pair.actualModel || pair.requestedModel;
const contEndpointId = r < roundEndpointIds.length
? roundEndpointIds[r]
: pair.actualEndpointId;
const contEndpointLabel = r < roundEndpointLabels.length
? roundEndpointLabels[r]
: pair.actualEndpointLabel;
roleEl.textContent = modelRouteLabel(
pair.requestedModel,
contModel,
pair.requestedEndpointLabel,
contEndpointLabel,
pair.requestedEndpointId,
contEndpointId,
);
if (
pair.requestedModel
&& contModel
&& (
!sameModelName(pair.requestedModel, contModel)
|| (pair.requestedEndpointId && contEndpointId && pair.requestedEndpointId !== contEndpointId)
)
) {
roleEl.title = pair.requestedModel + ' -> ' + contModel
+ ' (' + pair.requestedEndpointLabel + ' -> ' + contEndpointLabel + ')';
const contModel = pair.actualModel || pair.requestedModel;
roleEl.textContent = modelRouteLabel(pair.requestedModel, contModel);
if (pair.requestedModel && contModel && !sameModelName(pair.requestedModel, contModel)) {
roleEl.title = pair.requestedModel + ' -> ' + contModel;
}
applyModelColor(roleEl, contModel);
if (r === 0) roleEl.appendChild(roleTimestamp(metadata?.timestamp));
@@ -2733,14 +2492,7 @@ export function addMessage(role, content, modelName, metadata) {
const isCompacted = metadata?.compacted;
const replyModels = replyModelPair(modelName, metadata);
const resolvedModel = replyModels.actualModel || replyModels.requestedModel;
var _roleText = role === 'user' ? 'You' : (isSlash || isCompacted) ? 'Odysseus' : modelRouteLabel(
replyModels.requestedModel,
resolvedModel,
replyModels.requestedEndpointLabel,
replyModels.actualEndpointLabel,
replyModels.requestedEndpointId,
replyModels.actualEndpointId,
);
var _roleText = role === 'user' ? 'You' : (isSlash || isCompacted) ? 'Odysseus' : modelRouteLabel(replyModels.requestedModel, resolvedModel);
if (role === 'assistant' && (metadata?.research || metadata?.research_clarification)) {
_roleText += ' (Research)';
}
@@ -2751,14 +2503,8 @@ export function addMessage(role, content, modelName, metadata) {
}
r.textContent = _roleText;
if (role !== 'user') {
const endpointChanged = Boolean(
replyModels.requestedEndpointId
&& replyModels.actualEndpointId
&& replyModels.requestedEndpointId !== replyModels.actualEndpointId
);
if (!isSlash && !isCompacted && replyModels.requestedModel && resolvedModel && (!sameModelName(replyModels.requestedModel, resolvedModel) || endpointChanged)) {
r.title = replyModels.requestedModel + ' -> ' + resolvedModel
+ ' (' + replyModels.requestedEndpointLabel + ' -> ' + replyModels.actualEndpointLabel + ')';
if (!isSlash && !isCompacted && replyModels.requestedModel && resolvedModel && !sameModelName(replyModels.requestedModel, resolvedModel)) {
r.title = replyModels.requestedModel + ' -> ' + resolvedModel;
}
if (!isSlash && !isCompacted) applyModelColor(r, resolvedModel);
r.appendChild(roleTimestamp(metadata?.timestamp));
@@ -3042,7 +2788,6 @@ const chatRenderer = {
getSessionCost,
resetSessionCost,
updateSessionCostUI,
recordSessionMetricsCost,
roleTimestamp,
stripToolBlocks,
copyMessageText,
+3 -3
View File
@@ -7,7 +7,7 @@ import Storage from './storage.js';
import themeModule from './theme.js';
import markdownModule from './markdown.js';
import sessionModule from './sessions.js';
import documentModule from './document.js?v=20260815approvalsave1';
import documentModule from './document.js?v=20260722emailfastindex1';
/**
* Handle a ui_control SSE event AI-driven UI manipulation.
@@ -156,7 +156,7 @@ export function handleUIControl(uiData) {
if (fn) fn();
}).catch(function(){});
} else if (panel === 'email') {
import('./emailLibrary.js?v=20260815approvalsave1').then(function(mod) {
import('./emailLibrary.js?v=20260722emailfastindex1').then(function(mod) {
var fn = mod.openEmailLibrary || (mod.default && mod.default.openEmailLibrary);
if (fn) fn();
}).catch(function(){});
@@ -205,7 +205,7 @@ export function handleUIControl(uiData) {
} catch (e) {
console.warn('open_email_reply existing draft update failed:', e);
}
import('./emailInbox.js?v=20260815approvalsave1').then(function(mod) {
import('./emailInbox.js?v=20260722emailfastindex1').then(function(mod) {
var fn = mod.openReplyDraft || (mod.default && mod.default.openReplyDraft);
if (fn) fn(uiData.uid, uiData.folder || 'INBOX', uiData.mode || 'reply', uiData.body || '');
}).catch(function(e) {
-23
View File
@@ -1,23 +0,0 @@
/** Build a terminal stream error while preserving provider-supplied text. */
export function createTerminalStreamError(payload = {}) {
const rawError = payload.error;
const message = (
payload.text
|| (typeof rawError === 'string' ? rawError : rawError?.message)
|| `Error ${payload.status || 'unknown'}`
);
const error = new Error(message);
error.name = 'TerminalStreamError';
error.terminalStreamError = true;
error.status = payload.status;
return error;
}
/** Only connection-class stream failures are safe to resubmit automatically. */
export function isRecoverableStreamError(error) {
if (!error || error.terminalStreamError || error.name === 'TerminalStreamError') return false;
if (error.name === 'TypeError') return true;
const message = (error.message || '').toLowerCase();
if (/\btool\b|unsupported|json|parse|\b4\d\d\b|\b5\d\d\b/.test(message)) return false;
return /network|fetch|connection|reset|closed|aborted|stream|tim(?:e|ed)\s?out|econn|eof/.test(message);
}
+1 -3
View File
@@ -15,7 +15,6 @@ let _getPlatform;
let _serverByVal;
let _isWindows;
let _buildEnvPrefix;
let _psQuote;
let _buildServeCmd;
let _detectBackend;
let _detectToolParser;
@@ -539,7 +538,7 @@ export async function _runModelDownload(panel, model, backend, hostOverride) {
if (srv.downloadDir) payload.local_dir = srv.downloadDir;
if (isWin) {
if (env === 'venv' && envPath) {
payload.env_prefix = '& ' + _psQuote(envPath.endsWith('\\Scripts\\Activate.ps1') ? envPath : envPath + '\\Scripts\\Activate.ps1');
payload.env_prefix = '& ' + (envPath.endsWith('\\Scripts\\Activate.ps1') ? envPath : envPath + '\\Scripts\\Activate.ps1');
} else if (env === 'conda' && envPath) {
payload.env_prefix = 'conda activate ' + envPath;
}
@@ -653,7 +652,6 @@ export function initDownload(shared) {
_serverByVal = shared._serverByVal;
_isWindows = shared._isWindows;
_buildEnvPrefix = shared._buildEnvPrefix;
_psQuote = shared._psQuote;
_buildServeCmd = shared._buildServeCmd;
_detectBackend = shared._detectBackend;
_detectToolParser = shared._detectToolParser;
+1 -3
View File
@@ -338,7 +338,6 @@ let _sshPrefix;
let _getPlatform;
let _isWindows;
let _buildEnvPrefix;
let _psQuote;
let _loadPresets;
let _savePresets;
let _copyText;
@@ -1972,7 +1971,7 @@ export async function _launchServeTask(shortName, repo, cmd, fields, hostOverrid
let envPrefix = '';
if (_isWindows()) {
if (_envState.env === 'venv' && _envState.envPath) {
envPrefix = '& ' + _psQuote(_envState.envPath.endsWith('\\Scripts\\Activate.ps1') ? _envState.envPath : _envState.envPath + '\\Scripts\\Activate.ps1');
envPrefix = '& ' + (_envState.envPath.endsWith('\\Scripts\\Activate.ps1') ? _envState.envPath : _envState.envPath + '\\Scripts\\Activate.ps1');
} else if (_envState.env === 'conda' && _envState.envPath) {
envPrefix = 'conda activate ' + _envState.envPath;
}
@@ -4403,7 +4402,6 @@ export function initRunning(shared) {
_getPlatform = shared._getPlatform;
_isWindows = shared._isWindows;
_buildEnvPrefix = shared._buildEnvPrefix;
_psQuote = shared._psQuote;
_loadPresets = shared._loadPresets;
_savePresets = shared._savePresets;
_copyText = shared._copyText;
+5 -12
View File
@@ -3934,7 +3934,7 @@ import { bindMenuDismiss, dismissOrRemove } from './escMenuStack.js';
leadingIcon: 'check',
action: 'View Message',
onAction: () => {
import('./emailLibrary.js?v=20260815approvalsave1').then(mod => {
import('./emailLibrary.js?v=20260722emailfastindex1').then(mod => {
const open = mod.openEmailLibrary || (mod.default && mod.default.openEmailLibrary);
if (open) open({
account_id: data.account_id || activeAccountId || null,
@@ -9401,9 +9401,9 @@ import { bindMenuDismiss, dismissOrRemove } from './escMenuStack.js';
/** Save manual edits */
export async function saveDocument({ silent = false, forceVersion = false } = {}) {
if (!activeDocId) return false;
if (!activeDocId) return;
const textarea = document.getElementById('doc-editor-textarea');
if (!textarea) return false;
if (!textarea) return;
const savingDocId = activeDocId;
saveCurrentToMap();
const localDoc = docs.get(savingDocId);
@@ -9422,7 +9422,7 @@ import { bindMenuDismiss, dismissOrRemove } from './escMenuStack.js';
});
if (res.status === 404) {
if (silent && localDoc?.language === 'email') {
return false;
return;
}
// Streaming/empty email drafts can leave a local tab pointing at a temp
// or already-deleted document. Do not keep surfacing autosave errors for
@@ -9434,7 +9434,7 @@ import { bindMenuDismiss, dismissOrRemove } from './escMenuStack.js';
}
_syncDocIndicator();
if (!silent && uiModule) uiModule.showError('Document no longer exists');
return false;
return;
}
if (!res.ok) throw new Error(`Document save failed: HTTP ${res.status}`);
const doc = await res.json();
@@ -9447,7 +9447,6 @@ import { bindMenuDismiss, dismissOrRemove } from './escMenuStack.js';
}
_syncDocIndicator();
if (!silent && uiModule) uiModule.showToast(forceVersion ? 'New version saved' : 'Document saved');
return true;
} catch (e) {
console.error('Failed to save document:', e);
const now = Date.now();
@@ -9455,7 +9454,6 @@ import { bindMenuDismiss, dismissOrRemove } from './escMenuStack.js';
uiModule.showError(silent ? 'Autosave failed' : 'Failed to save document');
_lastAutoSaveErrorAt = now;
}
return false;
}
}
@@ -9738,11 +9736,6 @@ import { bindMenuDismiss, dismissOrRemove } from './escMenuStack.js';
const container = document.createElement('div');
container.style.cssText = 'padding:20px;font-family:sans-serif;font-size:12px;color:#000;background:#fff;line-height:1.6;';
container.innerHTML = html;
// This container is detached, so the document-scoped flush mdToHtml
// schedules never sees it. Typeset the deferred math before html2pdf
// rasterises, or the PDF gets raw formula source. renderMath() returns
// immediately, without loading KaTeX, when there is nothing pending.
await markdownModule.renderMath(container);
const baseName = _getExportBaseName();
window.html2pdf().set({
margin: 10,
+1 -1
View File
@@ -5,7 +5,7 @@
import spinnerModule from './spinner.js';
import sessionModule from './sessions.js';
import { initEmailLibrary, openEmailLibrary, closeEmailLibrary, isOpen as isLibOpen, prewarmEmailLibrary, prewarmUnreadEmails } from './emailLibrary.js?v=20260815approvalsave1';
import { initEmailLibrary, openEmailLibrary, closeEmailLibrary, isOpen as isLibOpen, prewarmEmailLibrary, prewarmUnreadEmails } from './emailLibrary.js?v=20260722emailfastindex1';
import * as Modals from './modalManager.js';
import { applyEdgeDock } from './modalSnap.js';
import { buildReplyAllCc, extractEmail } from './emailLibrary/replyRecipients.js';
+4 -4
View File
@@ -5,7 +5,7 @@
import spinnerModule from './spinner.js';
import { styledConfirm, showToast, emptyStateIcon } from './ui.js';
import { folderDisplayName, sortedFolders } from './emailInbox.js?v=20260815approvalsave1';
import { folderDisplayName, sortedFolders } from './emailInbox.js?v=20260722emailfastindex1';
import settingsModule from './settings.js';
import * as Modals from './modalManager.js';
import { topPortalZ } from './toolWindowZOrder.js';
@@ -23,7 +23,6 @@ import {
_tryFoldHintSig, _foldSignature, _SIG_ICON, _QUOTE_ICON,
} from './emailLibrary/signatureFold.js';
import { state } from './emailLibrary/state.js';
import { getSettings } from './appConfig.js';
import { collapseSidebarToRail } from './modalSnap.js';
import { emailApiUrl } from './emailShared.js';
import { bindMenuDismiss, dismissOrRemove } from './escMenuStack.js';
@@ -994,7 +993,8 @@ function _syncEmailReminderBellVisibility(enabled) {
async function _loadEmailReminderBellVisibility() {
try {
const settings = await getSettings();
const res = await fetch('/api/auth/settings', { credentials: 'same-origin' });
const settings = await res.json();
_syncEmailReminderBellVisibility(settings.reminder_channel === 'email');
} catch (_) {
_syncEmailReminderBellVisibility(false);
@@ -6680,7 +6680,7 @@ function _wireAttachmentHandlers(reader, folder) {
ownerModal.classList.add('hidden');
}
}
const docMod = await import('./document.js?v=20260815approvalsave1');
const docMod = await import('./document.js?v=20260722emailfastindex1');
const load = (docMod && docMod.loadDocument) || (docMod && docMod.default && docMod.default.loadDocument);
if (typeof load === 'function') {
await load(json.doc_id);
+1 -49
View File
@@ -3,7 +3,7 @@
*/
import uiModule from './ui.js';
import { loadPanel } from './panels.js';
import { openEditor, closeEditor, isEditorOpen } from './galleryEditor.js?v=20260708match1';
import spinnerModule from './spinner.js';
import { makeWindowDraggable } from './windowDrag.js';
import { bindMenuDismiss, dismissOrRemove } from './escMenuStack.js';
@@ -15,54 +15,6 @@ const API_BASE = window.location.origin;
let _open = false;
let _galleryResizeHandler = null;
// ── Image editor, loaded on first use ──
// galleryEditor.js plus everything under js/editor/ is 54 modules / 576 KB.
// It used to be a static import here, so every page load paid for it even
// though most sessions never touch the Edit tab. The wrappers below keep the
// three call shapes the rest of this file already uses.
//
// closeEditor() and isEditorOpen() stay synchronous on purpose: if the module
// was never loaded there is no edit session to close, and none can be open.
let _editorMod = null;
let _editorLoading = false;
async function _loadEditor() {
_editorLoading = true;
try {
_editorMod = await loadPanel('editor');
return _editorMod;
} finally {
_editorLoading = false;
}
}
async function openEditor(...args) {
let mod = _editorMod;
if (!mod) {
try {
mod = await _loadEditor();
} catch (e) {
// Previously unreachable — a static import either loaded or the whole
// page failed. Now it can fail on its own (offline before the panel was
// ever cached), so say so instead of doing nothing.
console.error('[gallery] image editor failed to load', e);
uiModule?.showError?.('Failed to load the image editor');
return;
}
}
return mod.openEditor(...args);
}
function closeEditor(...args) {
return _editorMod ? _editorMod.closeEditor(...args) : undefined;
}
// True while the module is still in flight as well — the gallery-close paths
// use this to refuse to tear the container down under an edit that is opening.
function isEditorOpen() {
return _editorLoading || (_editorMod ? _editorMod.isEditorOpen() : false);
}
// Auto-refresh gallery when new image is generated
window.addEventListener('gallery-refresh', (e) => {
if (e?.detail?.source === 'chat-upload' && _sort !== 'recent') {
+2 -2
View File
@@ -3,7 +3,6 @@
// ============================================
import { IS_MAC, isAltGrEvent } from './platform.js';
import { getSettings } from './appConfig.js';
const _defaultKeybinds = {
search: 'ctrl+k', toggle_sidebar: 'ctrl+alt+b', new_session: 'ctrl+alt+n',
@@ -57,7 +56,8 @@ export function initKeyboardShortcuts(modules) {
window._odysseusKeybinds = { ..._defaultKeybinds };
// Load saved keybinds
getSettings()
fetch('/api/auth/settings', { credentials: 'same-origin' })
.then(r => r.json())
.then(s => { if (s.keybinds) window._odysseusKeybinds = { ..._defaultKeybinds, ...s.keybinds }; })
.catch(() => {});

Some files were not shown because too many files have changed in this diff Show More